The jeering and whistling at times threatened to derail Werner Faymann's speech as he sought to defend the ruling coalition's handling of the migrant crisis and rising unemployment.
Many in the crowd gathered outside the city hall held up signs reading: "Retirement for Faymann! Step down now!"
Raising his voice, Faymann urged party supporters and trade unions to choose a "united path... for a fair, socially equal Austria".
The coalition, made up of Faymann's Social Democrats (SPÖ) and the conservative People's Party (ÖVP), has been haemorrhaging voters to the far-right Freedom Party whose presidential candidate Norbert Hofer won the first round of a presidential election last weekend.
Meanwhile candidates from the two main parties, which have effectively run Austria since the end of World War II, failed to even make it into the runoff on May 22.
The FPÖ also looks set to do well in the scheduled general election in 2018, consistently scoring more than 30 percent in voter opinion polls.
In light of this, there has been a growing rift in the SPÖ between those pushing for closer cooperation with the FPÖ and those strongly opposed to any rapprochement.
Several regional party heads have also demanded Faymann's resignation in recent days - a call he has so far rejected.
Austria received 90,000 asylum demands last year, the second-highest number in the bloc on a per-capita basis.
In response, the government introduced border fences and imposed tough restrictions on asylum-seekers.
But this has not stemmed growing dissatisfaction among Austria's regional leaders who say their states are unable to shoulder the burden of the many new arrivals.
In addition to the migrant crisis, Austria also no longer has the lowest unemployment rate in the European Union and Faymann's coalition, in power since 2008, has bickered over structural reforms.