The passengers were forced to undergo the extra checks before boarding the flight to the Austrian capital from Ataturk airport at 1440 GMT on Friday, Dogan news agency said.
In video footage from the agency, certain travellers were checked by a sniffer dog while others walked past security officers and those being searched.
The measure was taken in response to Austrian airport authorities' use of a sniffer dog to check passengers on a Turkish Airlines flight at Vienna International Airport on Thursday, the agency said.
The Turkish foreign ministry hit back at the "inappropriate" checks by Austrian officers, urging Vienna to "warn" the airport authorities in a statement late Thursday.
The Turkish consul general in Vienna went to the airport to conduct an "inspection", the ministry added.
But the Austrian finance ministry, in charge of customs, said it was a routine check that was not specifically aimed at Turkish Airlines.
The dogs are responsible for checking whether passengers leaving Austria are carrying large sums of cash over the authorised limit of 10,000 euros.
A finance ministry spokesman said such checks were carried out in a "discreet" way, regardless of the passengers' nationality, and until now had "never given rise to recriminations".
Relations between Turkey and European Union states including Austria have been strained since last year's attempted overthrow of President Recep Tayyip Erdogan and Europe's growing concerns over the subsequent crackdown.
Since the coup bid over 50,000 people have been arrested, including several European nationals, while more than 140,000 public sector workers have been sacked or suspended.