"With what happened yesterday, (Turkey's) membership prospects are buried, in practical terms," said Chancellor Christian Kern.
"We are entering a new era," the Social Democrat told reporters in Vienna.
EU aid to Turkey to help it advance towards membership was now "obsolete," he added.
Like other European leaders, Kern also spoke out against any move to restore the death penalty in Turkey.
Separately, Foreign Minister Sebastian Kurz said that after the Turkish referendum, "we can no longer simply return to business as usual."
"We must be honest about the relationship between the EU and Turkey," Kurz said in a tweet.
"We need to end the EU entry negotiations and instead work to establish a neighbourhood agreement" with Turkey, he said, referring to relations that are close but lie below full membership.
Sunday's referendum focussed on a proposal to reinforce the powers of the Turkish president -- a move that critics say may worsen the country's rights record and steer it towards dictatorship.
The "Yes" campaign won by 51.41 percent, according to near-complete results released by the election authorities.
Angry opposition groups have cried foul and demanded a recount.
International observers on Monday said the referendum campaign was conducted on an "unlevel playing field".
On Sunday evening, Erdogan suggested he would back moves to bring back capital punishment which had been abolished in 2004 as part of Turkey's bid to join the EU.
Brussels has made it clear that any move to restore it would scupper Ankara's efforts to join the bloc.
Ankara's relations with Europe had been tense in the months leading up to Sunday's vote, driven by the government's crackdown after a failed coup last July.
Turkey has had an association agreement with the EU since 1963 and formally applied to join the bloc on April 14th 1987.
The talks have made only slow progress, hamstrung by questions in Brussels over human rights and democracy in Turkey.