Many carried placards condemning the government, which since December has been made up of Chancellor Sebastian Kurz's conservative People's Party (OeVP) and Vice-Chancellor Heinz-Christian Strache's far right Freedom Party (FPOe).
The government has made a crackdown on immigration one of its main priorities, pledging to speed up and toughen the asylum process and increase deportations. In last year's elections Kurz stood on a platform of regaining control of Austria's borders after the migrant crisis of 2015-16 saw more than 150,000 people seek asylum in the country of 8.7 million.
However, speakers at the rally, organised by left-wing and anti-racist groups, said the government's policies risked stigmatising foreigners, making them scapegoats for social problems.
"I'm here with you today because, 80 years after the Nazis' takeover of Austria, I don't want people to be discriminated against once again because of their origins," Daniela Gruber-Pruner, MP for the opposition Social Democrats (SPOe), told the crowd.
Many also pointed to recent revelations over persistent anti-Semitism in right-wing student fraternities which count prominent FPOe politicians as members.
"There's the spectre of right-wing extremism today in Europe," 70-year-old Monika Salzer, founder of the "Omas gegen Rechts" ("Grannies against the Right") activist group, told AFP.
She added that she was worried Austria could develop into a "guided democracy, in the style of [Hungarian Prime Minister] Viktor Orban".
She pointed to a recent controversy over raids at Austria's domestic intelligence agency by a police unit headed by an FPOe official as proof that the FPOe wanted to control state institutions. The FPOe denies this and says it rejects anti-Semitism and racism.
Organisers put turnout at the rally at 8,000, with police counting 3,400.
