The 58-year-old man, identified by media as Josef B., was arrested after police discovered the family of six in a tiny secret room in the building in the village of Ruinerwold in the northern province of Drenthe.
The mystery surrounding the case deepened with reports that the son who had reportedly walked into a bar and raised the alarm about their captivity had nevertheless been active on social media this year.
"In the investigation, the man is currently suspected of being involved in illegal deprivation of liberty and harming the health of others," Dutch prosecutors said in a statement.
The man will appear before an examining magistrate on Thursday who will decide whether he should be kept in custody for longer.
Police also raided two properties as part of the investigation, including a former toy store owned by the suspect in the town of Zwartsluis, 16 kilometres (10 miles) from Ruinerwold.
The Austrian foreign ministry earlier said an Austrian national from Vienna was being held in relation to the case, but said the man did not want to have contact with Austrian officials.
Ministry spokesman Peter Guschelbauer told AFP that Dutch authorities had informed their Austrian counterparts that all others in the cellar case were Dutch.
'We never saw anything'
The mayor of Perg district in Upper Austria state said the arrested man had worked as a carpenter and lived alone for 10 years before leaving for the Netherlands in 2010, describing him as "reserved" and "discreet."
The man was reportedly known in Ruinerwold as "Josef, the Austrian" and Dutch media said he was the one who held the group, renting the house and planting vegetables for them in the garden.
"We used to see him, at first we thought he was our neighbour. And we always used to think that he lived alone there, that no one else lived there," Vincent de Gooijer, 46, who lived in a house overlooking the farm, told AFP.
"He was always busy working, we assumed he was doing up the farm. But he arrived early in the morning and left at the end of the afternoon, so he didn't sleep there or live there."
De Gooijer added: "I can't believe that such a thing is possible. And you know it's weird to say that we never saw anything all these years. But if these people never go out and they shut themselves away, there's nothing you can do."
According to media reports, the family had spent years "waiting for the end of time", although officials would not confirm that.
Police had earlier said they were held in a cellar but later said the family were kept "in a small room in the house that was closed off".
Ruinerwold's mayor told journalists: "I have never come across anything like this before."