"(It) is clear that Hungary cannot take back these migrants," Defence Minister Istvan Simicsko told a joint news conference in Budapest with visiting Austrian counterpart Hans Peter Doskozil.
"In order to take them back they would have had to begin their journey here. But... they crossed several countries before arriving in Hungary. They didn't suddenly get here by magic, they crossed several safe countries", including Greece, Simicsko said.
Under the European Union's much-criticised Dublin Treaty, asylum claims must be processed by the first EU member state in which refugees arrive. Several countries have suspended sending them back to Greece however.
Austria, which saw 90,000 people claim asylum last year, the second highest per capita in the EU, says that there are several thousand migrants that Hungary should take back.
The Austrian defence ministry says around 150 migrants arrive in Austria every day from Hungary -- where most of them are registered -- having travelled from Greece through Macedonia, Bulgaria and Serbia.
Currently Austria is not sending any back to Hungary following a court decision last September that prevented an Afghan family being returned because of "inhumane conditions" in Hungary.
An Austrian interior ministry spokesman, Karl-Heinz Grundboeck, said however that this concerned an individual case and did not prevent people being sent back in the future.
"We have very clear expectations of Hungary. Hungary is after all a member of the EU, the Schengen zone and is party to Dublin. Of course our expectation is that in a Europe with this legal framework, returns to Hungary are possible," Grundboeck told AFP.
He said that ministry officials would travel to Hungary later this month. A meeting of the interior and defence ministers of Austria, Hungary and Slovenia is planned soon, the Austria Press Agency reported.
Hungary has long been criticised for its treatment of migrants, having sealed its border with Serbia last year with razor wire and making illegal border crossing a criminal offence punishable by jail.
This week the UN refugee agency (UNHCR) sharply criticised Hungary for conditions at a makeshift transit camp on the border with Serbia for people waiting to be admitted into Hungarian "transit zones".
"We remain concerned about Hungary's restrictive approaches and the dire situation asylum-seekers face outside the transit zones. Currently, only 15-17 people are admitted daily at each zone, leaving hundreds to suffer day and night without any proper support at the EU border," said Samar Mazloum, head of UNHCR's local field office.
"The current approach makes it very easy for human traffickers to further exploit desperate refugees and pushes them to take more dangerous routes when legal ways are shut."