The packed train was originally thought to be to due to split, with three carriages travelling to Szonbathely and the rest to Sopron, both near Hungary's western border with Austria.
However, the train was stopped at Bicske, about an hour from the Hungarian capital with scenes of chaos ensuing.
Riot police tried to clear the train and bus the refugees to a local reception centre, but many people refused to disembark the train. Others lay on the tracks to protest the move.
On a hot afternoon, many angrily protested, shouting “Germany! Germany!” and holding placards saying “Help” and “SOS”.
Journalists from media international outlets were then ushered away from the train. One of those journalists, James Mates, the Europe editor for Britain's ITV station tweeted late on Thursday afternoon that the first passengers had been taken away in a bus, supposedly to a nearby camp.
First 4 refugees taken from the train, it's assumed to a nearby camp. Going to be a slow process at this rate pic.twitter.com/VriC3DcRxS— James Mates (@jamesmatesitv) September 3, 2015
However, many others were refusing to go.
Hungary had allowed several thousand to board trains bound for Austria and Germany on Monday, but the following day the station was closed to anyone without an EU passport or a valid visa.
The move left around 2,000 men, women and children stranded around the station or in a makeshift refugee camp and scuffles broke out between police and migrants on Tuesday.
Earlier, a Hungarian police officer on the platform at Keleti train station told a reporter from the Austria Press Agency that the train would travel from Sopron to Munich, via Austria.
However a spokesman for Austrian Railways (ÖBB), Michael Braun, said the special train would not be travelling on from Sopron, as it had not yet been agreed that it could cross the border into Austria and Hungary had failed to inform the ÖBB about the train's departure. "We are trying to find a solution", he said.
He added that it would have been better for the train to take the normal route to Hegyeshalom, where the migrants could disembark and get another train which would travel on to Austria.
On Tuesday Hungarian authorities stopped migrants taking trains to Austria and Germany and some were involved in scuffles with police.
The next day, it closed Budapest's Keleti station to migrants, leaving some 2,000 people stranded and leading to a tense stand-off with demonstrations.
Early on Thursday the station was fully re-opened and hundreds of people stormed inside, cramming into trains. Hungarian Railways said however that there would be no trains going to western Europe.