The Nazi symbol was sprayed in black paint overnight on Tuesday, around the entrance to the mosque and on some mosaic tiles.
It is not known who defaced the mosque but the State Office for the Protection of the Constitution has launched an investigation.
The mosque was opened in 2001 by the local Turkish cultural association. However, many locals protested in 2005 when the cultural association announced that it wanted to build a minaret to accompany the mosque.
Among the roughly 15,000 residents of Telfs, about 2,400 people signed a petition against the building of the mosque, and nearby homeowners threatened lawsuits.
Locals objected that the minaret would attract crowds and cause traffic congestion. Others complained that the minaret would represent a victory of Islam over Christianity.
A compromise was reached wherein the mosque's minaret would be only 15 meters tall, rather than the 20 meters originally planned by the cultural centre. It was also agreed that there would be no muezzin call for prayers from the minaret.