The Art Nouveau building, completed in 1898, is undergoing restoration and the culprits made use of the scaffolding to clamber up and prise off six to eight of the leaves, police spokeswoman Irina Steirer said.
The theft from the roof, visible from afar and a well-known landmark in the Austrian capital, took place on Monday night, Steirer said.
The building by architect Joseph Maria Olbrich was constructed to house works by Gustav Klimt and other artists after they quit the stuffy Künstlerhaus to found a new art association called the Secession.
Not everyone in Vienna, a vibrant hub of art, science and philosophy at the time, was impressed however, and the original site on the famous Ringstrasse boulevard was abandoned after a public furore.
For the renovation work, due to be completed at the end of May, around 2,500 leaves and 311 "berries" were restored and repaired over recent months and were in the process of being put back.
The leaves, around 60 centimetres long made of steel and coated with gold leaf, are worth less than 1,000 euros ($1,200) each, Secession spokeswoman Karin Jaschke told AFP.
"Security measures are being increased," Jaschke said.
READ ALSO: 'Lost' Klimt drawing discovered in Austria