"A consortium formed of (Houston-based) Marathon Oil and OMV decided to return its licences since general economic conditions for a long-term investment have not been met," a spokesman for the Austrian group said in Vienna.
The announcement came less than a week after OMV's new chief executive took charge. German Rainer Seele has said he wants to make Russia a key partner in the group's development.
In January Croatia granted OMV and Marathon Oil seven out of the ten licences it has awarded to explore for oil and gas in the Adriatic. Two were given to INA -- jointly owned by the Croatian state and Hungary's MOL -- and one to Italy's ENI and MEDOILGAS.
The contracts were expected to be signed in the coming weeks.
Officials in Zagreb confirmed the OMV-Marathon consortium's withdrawal, citing the unresolved border dispute with Croatia's neighbour Montenegro in the southern Adriatic -- where four of the seven exploration areas are located -- as the primary motive.
"The main reason for quitting is the issue of the border with Montenegro," Economy Minister Ivan Vrdoljak said in a statement by the national hydrocarbon agency.
Croatia has yet to resolve border disputes with its neighbours -- Bosnia-Hercegovina, Montenegro, Serbia and Slovenia -- all of whom were fellow members of the former Yugoslav federation that collapsed in a series of wars in the 1990s.
"The security we have offered was not a sufficient guarantee for the other side at the moment when a problem with the arbitration process with Slovenia has emerged," Vrdoljak said.
Croatia decided on Wednesday to withdraw from arbitration on a border row with Slovenia after revelations that a Slovenian member of a tribunal tasked with solving it had breached its impartiality.
The contracts for the remaining three licences for the Adriatic, as well as six licences awarded for onshore exploration for oil and gas, will be signed in September the hydrocarbon agency said.
"Due to a large interest for the remaining exploratory areas both onshore and offshore we are planning to publish a second tender," agency head Barbara Doric said in the statement.
Croatia, struggling with recession since 2008, was hoping that the Adriatic project would help revive its ailing economy and make the country energy independent.
But it has sparked concerns of environmental groups in the country, which depends heavily on tourism revenue.
Croatia produces about 60 percent of its own natural gas, importing the remaining 40 percent from Russia. The former Yugoslav republic imports 80 percent of its annual oil needs, while average annual domestic oil production is 500,000 tonnes.