Judges found that the earlier verdict had violated the constitution, noting that the environment did not hold "absolute priority" over other factors.
In February, the Federal Administrative Court (FAC) had argued that the project would result in a "significant" rise in greenhouse gas output, contravening domestic and international undertakings to reduce pollution.
Vienna airport appealed against the decision, which was also heavily criticised by Austrian businesses and a number of senior politicians.
The constitutional court said on Thursday that the FAC had committed several gross errors, including miscalculating the carbon dioxide emissions connected with the new runway.
The FAC must now issue a new ruling that takes into account these findings.
While Vienna's chamber of commerce and senior political figures welcomed the decision, environmental group Global 2000 warned that "the unrestrained growth of air traffic is not compatible with the goal of averting a climate catastrophe".
A key travel hub between western and eastern Europe, Vienna airport handled 23 million passengers last year and has been wanting to construct the runway for a decade.