The film, which premiered in Vienna on Thursday night, depicted five musician friends – including the two councillors - singing in a Burgenland basement under a portrait of Adolf Hitler, surrounded by swastikas and SS flags.
Following widespread outrage from ÖVP leaders among others, the two councillors handed in their resignations on Thursday.
Police too have filed a complaint against the men and the prosecutor is currently deciding whether to initiate an investigation into their Nazi activities.
The councillors have issued a statement distancing themselves from their actions.
"With the deepest conviction we dissociate ourselves from any Nazi ideology and atrocities," they said.
"To prevent further damage to the community and the party, we have voluntarily decided to resign, with immediate effect. It was a mistake to participate in this film."
The men also added that, "at the time of filming, in 2009, we were not members of the council."
Gerald Hüller, ÖVP mayor of Marz, in the district of Mattersburg, issued a press release stating: "We do not need outcry from others. We have realized ourselves that our portrayal in the film leads to devastating conclusions and that we must take responsibility for our actions."
"Anyone who knows the two municipal councilors and other parties involved are aware of the distorted representation in this film. Regrettably, the musicians have been part of a movie scene set against the background of Nazi memorabilia."
Extract from the film by Ulrich Seidl called "In the Basement."
According to TV channel Puls 4, the Documentation Centre for Austrian Resistance responded to the case as follows: "It is expected, especially for political office-holders sworn to the Republic and laws of Austria, that as elected officials they should know about the issue of symbols in relation to the law. One has the impression here that Nazism and its symbols have seeped into popular culture."
ÖVP General Secretary Gernot Blümel urged the two ÖVP politicians to resign on Thursday evening.
"Extreme right ideology has no place in the ÖVP," said Blümel. "National Socialism is one of the darkest chapters in our history and should not be trivialized."
There was also criticism from other parties, including the Burgenland Socialist Youth and the Greens.
The incident was the second one this week involving apparent Nazi sympathizers, when a mayor in Carinthia was expelled from the far-right Freedom Party (FPÖ) for expressing support for Nazi views.