A group of protesters clashed with Erdogan supporters on the Reichsbrücke, according to the Heute paper.
According to eyewitness reports a group of counter-demonstrators threw stones and glass bottles at the anti-Erdogan marchers. Police intervened and used pepper spray to disperse the protesters.
Erdogan was in Vienna for a private meeting with 7,000 of his closest friends and supporters in Austria.
The visit has stirred controversy, with Freedom Party (FPÖ) leader Heinz-Christian Strache warning him to stay home, and Austrian foreign minister Sebastian Kurz telling him to avoid making damaging remarks in his speech.
The main demonstration route departed from Praterstern (near the iconic Prater ferris wheel that featured in the Orson Welles' movie, The Third Man), and made its way along Lasallestrasse, over the Reichsbrücke bridge, past the United Nations along Wagramer Strasse, and will end at Siebeckstraße, near the location where Erdogan is speaking.
At one point, the demonstration march stretched for more than 2km along the route, with police estimating numbers at 6000, while organizers claim 10,000.
According to AFP, a speech by Erdogan in a sports arena to members of Austria's 250,000-strong Turkish minority ahead of an expected presidential run in August drew some 6000 to 7000 supporters, police said.
A further 10,000 people watched his speech on a big screen outside the venue.
Austria's government had warned Erdogan against making provocative comments, and he appeared to heed the advice telling the crowd, "no one has anything to fear from us".
Trumpeting Turkey's economic growth under his stewardship, Erdogan said that Europe needed his country.
"Europe does not end where the river Danube flows into the Black Sea but begins where the Euphrates and Tigris begin," he said.