The poem by Christian Schilcher, deputy mayor of the town of Braunau am Inn in Upper Austria, used the image of foreign rats integrating with Austrian ones to illustrate the dangers of "mixing" cultures and languages.
FPOe leader and Vice-Chancellor Heinz-Christian Strache said that Schilcher would quit his job and leave the FPOe "in order to avert any damage to the party".
The row overshadowed the FPOe's campaign launch ahead of European parliament elections next month.
Chancellor Sebastian Kurz, who has led a coalition between the FPOe and his own centre-right People's Party (OeVP) since late 2017, praised Strache's "clear actions".
"The resignation of the deputy mayor of Braunau was the only logical outcome of this abominable and racist poem," Kurz said.
Both Kurz's and Strache's parties have run on anti-immigration platforms but with May's elections looming Kurz has come under pressure to condemn outbursts from FPOe members.
The opposition Social Democrats (SPOe) said the coalition should be dissolved to protect "the country's image".
"There has to be a democratic consensus that human beings cannot be denigrated, insulted or humiliated," SPOe leader Pamela Rendi-Wagner said on Tuesday.
Strache himself was criticised last week for a Facebook post linking to a site which has published Holocaust denial and various anti-Semitic conspiracy theories.
The post was subsequently deleted.
Last week Kurz's predecessor as OeVP leader, Reinhold Mitterlehner, sharply criticised Kurz for the government's current direction, accusing it of scapegoating migrants and refugees.
In March the FPOe came under scrutiny for its ties to the nationalist Identitarian group, which received a donation from suspected New Zealand mosque attacker Brenton Tarrant.
After that episode Kurz demanded that the party break all ties with the Identitarians.