Minister Wolfgang Sobotka made the remarks in a television interview on channel ORF.
The house has been regularly in the news, with the latest decision apparently resulting in a parliamentary bill to expropriate the house in the small town of Braunau am Inn, near the German border.
Controversy has raged since its current owner refuses to allow renovation or sale of the house, with fears that it could be turned into a shrine for neo-Nazis.
Sobotka told the TV audience that the expropriation was necessary, adding that "we have tried to clear up all possibilities for using it and buying it from the owner, but with no results."
"For me, a demolition ... would be the cleanest solution," Sobotka said.
A spokesman for the Interior ministry clarified on Sunday that the remarks were the personal opinion of the minister, and did not necessarily reflect official policy, in that the legality of a demolition order remains to be determined.
The minister has invited a commission of historians to make a recommendation on the further disposition of the house.
According to Interior Ministry spokesman Karl-Heinz Grundboeck, “We are currently examining the creation of a law, which would force a change of ownership and pass the property to the Republic of Austria.”
“We have come to the conclusion over the past few years that expropriation is the only way to avoid the building being used for the purposes of Nazi sympathizers," he said.
The approximately 800 square metre property currently costs the state €4,800 a month. The tenancy agreement states that it may not be used as a museum, or in any historical context, and that the owner must consent to any new tenant.