The Art Restitution Advisory Board "recommended unanimously... not to return the Beethoven Frieze by Gustav Klimt to the heirs of Erich Lederer," panel chair Clemens Jabloner told journalists.
The fresco, 34 metres (112-feet) long, two metres high and weighing several tons, is widely regarded as a central masterpiece of Viennese Jugendstil art nouveau from the early 20th century.
Klimt painted it directly onto the walls of Vienna's Succession gallery in 1902, and in 1915 it was sold to a Jewish industrialist, August Lederer.
However, the Nazis confiscated the frieze from the Lederer family in 1938.
After the end of World War II in 1945 Austria returned it to the family heir Erich Lederer, living in Switzerland, who then sold it to the Austrian Republic in 1972.
His descendants say however that Lederer sold it under pressure because Austria refused to allow him to take the frieze out of the country, and that the reported sale price of $750,000 (€680,000) was too low.
They launched a claim for its return in 2013.
On Friday Clemens Jabloner said that it was "not the case that the export procedure was used as a tool to force Lederer into an agreement".
But Marc Weber, a lawyer from Swiss law firm Lanter Rechtsanwälte representing some of the heirs, told AFP that the panel had "muddled up the facts."
"We are now considering taking their case to the European Court of Human Rights and/or to the United States," Weber said.
Since Austria passed a law in the 1990s covering the restitution of vast numbers of artworks stolen by the Nazis, thousands have been returned - including major works worth millions of euros.
Since 1986 Klimt's Beethoven Frieze has been on display in a specially built basement in Vienna's Secession Building, the work's original home, where it attracts hoards of visitors.
The gallery, itself a Jugendstil masterpiece topped by a golden dome, says that returning it would be "unjustified both legally and morally" and that leaving the frieze in its current place "maintains the art-historical connection between building and work".
The final decision on whether to return the work falls to Culture Minister Josef Ostermayer, who said on Friday that he would stick to the panel's recommendation, as Vienna has done in the past.
"Losing (the frieze) ... would of course have been a loss for Austria as a place for art and culture," Ostermayer told the Austrian Press Agency.