The Social Democrats (SPÖ), which have ruled Vienna uninterrupted since 1945, scored 39.4 percent, down 4.9 points, while the Freedom Party (FPÖ) rose 5.3 points to 31 percent.
The Green party is projected to have won 11.6 percent (down 1 point) and the centre-right People's Party (ÖVP) 8.7 percent (down 5.3) - its worst ever result. The ÖVP's top candidate Manfred Juraczka has said he will step down.
The moderate Neos party made the most gains and looks set to win 6.2 percent and its first seat in the city council.
Opinion polls before the election had forecast a much closer result, with even a victory for the FPÖ on the back of unease among voters about Europe's refugee crisis.
The result, if confirmed, would in theory make possible a continuation of the SPÖ's coalition in Vienna with the ecologist Greens - together they would have 54 seats in the city council.
"Of course we would have preferred a neck-and-neck result, but this didn't happen. At the same time it has to be said that this is the SPÖ's worst result in its history," said FPÖ leader Heinz-Christian Strache, 46.
"I said before (the election) that I was ready to assume the responsibility of becoming mayor. Today it wasn't enough yet, but we took a step closer," he said.
Opinion polls before the election had forecast a much closer result, with even a victory for the FPÖ a possibility.
Voter turnout was high, at around 74 percent. A major issue for voters - mirrored across Europe - has been the arrival of hundreds of thousands of refugees into the continent this year.
A survey released on Sunday, which was commissioned by Austria's national public broadcaster, showed that for 65 percent of voters in Vienna, the migrant crisis was a major issue in deciding whom to vote for.
Help 'our poor'
Austria has seen more than 200,000 migrants enter the country since the beginning of September, most of whom travel onwards to Germany or Scandinavia.
But the government still expects some 85,000 asylum claims this year, making the Alpine country of 8.5 million people one of the highest recipients in Europe on a per-capita basis.
Strache, a former dental technician, has called for Austria to copy Hungarian Prime Minister Viktor Orban and put fences up on the borders - something rejected by Chancellor Werner Faymann of the SPÖ.
"Every property owner does this with his house, putting a fence around his garden and not leaving the doors and windows open, so unwelcome guests don't come," Strache said earlier this year.
But emulating Marine Le Pen, the head of France's far-right National Front, Strache has also toned down the anti-immigration rhetoric seen in earlier campaigns in his ten years as FPÖ head.
Gone are the slogans vilifying Islam. The posters all over Vienna were more subtle: "Help for our poor", they declare, adding in smaller letters "Instead of open doors for economic migrants".
"Security for our people," reads another, below a grinning Strache. "Instead of open borders for criminals".
At the same time, Strache campaigned on traditional SPÖ issues like alleviating poverty and more affordable rents, while even encroaching on liberal topics like slashing bureaucracy.
"He has shifted his political discourse," political analyst Anton Pelinka told AFP. "The xenophobic root was still there, but in a more latent form."