After arriving at Vienna's Westbahnhof, many of the migrants then boarded a train to Salzburg, while others climbed on to another one headed for Munich, with police looking on, an AFP correspondent at the scene said.
Austrian security forces stopped two trains with several hundred migrants near the border with Hungary on Monday, a police spokesman said, hours after authorities in Budapest let them leave despite many not having EU visas.
The trains were halted near the Hungarian border town of Hegeshalom, where Austrian police proceeded to check their papers.
One of the trains was bound for Munich and carried 300 to 400 migrants, police spokesman Roman Hahslinger told AFP.
"It could well be that others will be stopped. It remains to be seen how many (migrants) will actually arrive," he said, adding that Austria was in close contact with Hungarian and German authorities.
Migrants who had already been registered at refugee processing centres in Hungary would be returned to Budapest, while the others would be allowed to continue their journey, Hahslinger said.
The migrants, many from Syria, were among some 2,000 people who were stuck for several days in makeshift refugee camps at train stations in Budapest.
Police had previously prevented them from leaving, even if they had valid train tickets and papers, because they did not have the required visa to move around the European Union's passport-free Schengen zone.
But on Monday there were no security forces present as the migrants rushed to get on trains leaving for Vienna, Munich and Berlin from Budapest's Keleti station.
People were running along the platform to catch an Austria-bound train scheduled to leave at 1110 GMT, with some helping to lift a woman in a wheelchair into a carriage.
There were confusing scenes as a Hungarian railway employee initially refused to allow the train to leave, saying it was packed beyond capacity and some people did not have the right papers to travel.
But the train eventually departed with a 20-minute delay.
Under current EU regulations, known as the Dublin provision, asylum-seekers must remain in the first European country they enter while their application is being processed.
Those who subsequently travel to other member states face deportation back to the EU country they originally entered.
Hungary has become a frontline country for migrants arriving via the western Balkans route as they flee war and unrest in the Middle East, Africa and Asia.
Many are trying to reach Germany, where the government last week eased asylum rules for Syrians to relieve pressure on southern European nations.
"I want to go to Hamburg. My brother arrived there 12 days (ago)," a 34-year-old Syrian told AFP at Vienna's central station, where he and his family had arrived from Budapest on Monday.
Mohammed, who fled the war-torn city of Aleppo with his wife and children a fortnight ago, said they had been stopped at the Austrian border for 25 minutes before being told to move on.
He added that his family had been forced to spend four days sleeping on the streets in Hungary.
Budapest has criticised Germany's decision to ease asylum restrictions, warning that it "built up the hopes of illegal immigrants".
The Hungarian government on Monday demanded that Berlin clarify "the legal situation, in order to eliminate this ambiguity and controversy".
"It is in our common interest that all member states abide by EU legislation. Order and legality must be restored at the borders of the European Union", said government spokesman Zoltan Kovacs.