The 49-year-old suspect is accused of shouting “Heil Hitler” and making a Nazi salute at the rabbi in the town of Ebensee last week.
The rabbi was there with his grandson to visit a memorial commemorating the camp where his own father had been deported.
The police said the man had admitted the charges, which also included shouting “Mein Volk, Mein Reich, mein Fuhrer”, the motto of the Nazi regime.
Offences involving expressions of pro-Nazi sentiment are not uncommon in Austria, despite it having some of the world's strictest laws against such acts.
Concerns over extremism have been brought to the fore by the entry of the far-right Freedom Party (FPÖ) into the national government in December as a junior coalition partner.
Protests are expected outside a ball for student fraternities organised by the FPÖ in Vienna's former imperial Hofburg palace.
Vice-Chancellor and FPÖ leader Heinz-Christian Strache said he would renounce “all forms of anti-Semitism” in his speech opening the ball.
Earlier this week prosecutors opened an inquiry after the weekly Falter magazine revealed a student fraternity had published a songbook containing virulently anti-Semitic and pro-Nazi lyrics.
The FPÖ's Udo Landbauer came under political pressure over the revelations as he had been deputy chair of the fraternity.
Landbauer, who is the chief candidate in a regional election on Sunday, denied all knowledge of the texts and said he had been a child when they were published.