"The structure will be 250 metres long and will cover both the motorway and the main road," Helmut Tomac, the head of the police in the southern Austrian state of Tyrol was reported by Italian newspaper La Stampa as saying.
The guardrails of the motorway and main road are already being taken apart to make way for the barrier and road signs are set to be modified.
In addition to the barrier, a new registration centre will also be built on the Austrian side of the border, while traffic checks could be carried out as early as May from a car park just inside Austria.
The new barrier comes amid Austrian fears that the country is set to see a wave of asylum seekers from Italy following the closure of the so-called 'Balkan route' earlier this year. So far at least, these fears have been unfounded.
The construction of the barrier follows a meeting between Italy's Interior Minister, Angelino Alfano and the then Austrian Interior Minister, Johanna Mikl-Leitner in Rome on Friday, at which the pair outlined their desire to work together at the border to "safeguard the Schengen principles."
"This wall violates Schengen rules and is further proof that we are working in small national groups instead of trying to find a common solution," said Gianni Pitella, president of the Progressive Alliance of Socialists and Democrats, the second largest political group in the European Parliament.
Tomac, who is overseeing the building work, said however that the fence is not to stop people from getting in.
"The fence is not there to close the doors to Austria, but to make it possible to securely control entry," he was quoted as saying in the Kurier.
Mikl-Leitner, who had spearheaded Austria's tough stance against migrants in recent months and attempt to build 'fortress Europe', has stepped down from her post as Interior Minister over the weekend.
Since replacing her, the new minister Wolfgang Sobotka has said he will continue with the same hardline course of action.
Austria has set an upper limit of 37,500 for the number of asylum applications it is prepared to accept this year, which it is expected to reach by the summer. Last year the country accepted 90,000 asylum applications.