He told a press conference on Monday that Austria needed a chancellor whose party stood behind him and that the government needed a new start.
Faymann's future looked highly uncertain after his party's candidate was knocked out of the presidential race and there were disagreements within the party over the migrant crisis, and whether to cooperate with the right-wing Freedom Party.
On April 24 the Freedom Party's candidate Norbert Hofer came a clear first in the first round of elections for the largely ceremonial post of president with 35 percent of the vote.
Signing off at the end of the press conference, Faymann said he was stepping down "for Austria" and that he was "extremely thankful to have been able to serve my country". The SPÖ will announce a new leader later on Monday afternoon.
The centre-left Faymann, 56, chancellor since 2008, said in a statement that he no longer had "strong backing" in his party, the Social Democrats (SPOe).
"As a result of this insufficient support I am drawing the consequences and resign my functions as party leader and chancellor, effective today," he said.
The SPOe and its coalition partner since 2008, the centre-right People's Party (OeVP), have dominated Austrian politics since World War II but their support has been sliding in recent years.
At the last general election, in 2013, they only just scratched together a majority, and polls suggest doing so again at the next scheduled vote in 2018 will be difficult.
Mirroring similar trends across Europe, the two main parties have been bleeding support to fringe groups, in Austria's case in particular to the far-right Freedom Party (FPOe), the former party of the controversial, late Joerg Haider.
The far-right has tapped into growing unease about immigration after Austria last year saw 90,000 people claim asylum, and around 10 times that number pass through at the high point of Europe's migrant crisis.
But the two parties have also presided over a rise in unemployment, with Austria losing its crown as the EU member with the lowest unemployment. The coalition has also squabbled over structural reforms.
Far-right on the rise
The FPOe is leading national opinion polls and on April 24, in the first round of elections to the largely ceremonial post of president, the FPOe's Norbert Hofer came a clear first with 35 percent.
Hofer, 45, who presents himself as the friendly and reasonable face of the FPOe, will now face Alexander van der Bellen, a former head of the Greens who came second, in a runoff on May 22.
The two hapless candidates from the ruling coalition parties were relegated into distant fourth and fifth places, failing to make it through to the runoff with just 11 percent of the vote each.
This historic failure means that for the first time since 1945, there will not be a president from within these two parties in Vienna's Hofburg palace.
This in turn could mean that the new president might make use of some of the considerable powers afforded to the head of state under Austria's constitution that until now have been not been used.
In theory the Austrian president can fire the government -- as Hofer has threatened to do if elected -- or dissolve parliament.
It was unclear on Monday who would succeed Faymann, with the government in theory having two more years to govern.
The popular mayor of Vienna, Michael Haeupl, will take over from Faymann on an interim basis as party chief, saying the SPOe needed a "phase of reflection".
Christian Kern, currently the head of the national railways company, and Gerhard Zeiler, former chief of national broadcaster ORF, have been touted as possible replacements.
(Updated with AFP reporting.)