The family were visiting the country over the Easter holiday. The names of the children have not been made public.
Mr Holch Povlsen owns the international clothing chain Bestseller.
He is also the biggest single shareholder in clothing giant Asos and is the UK's largest private landowner, according to the Times newspaper.
"Unfortunately, we can confirm the reports," a Bestseller spokesman said in an email. "We ask you to respect the privacy of the family and we therefore have no further comments."
Mr Holch Povlsen has a large property portfolio in Scotland, where he owns about a dozen estates including Aldourie Castle. He bought them through his company Wildland, which describes itself as a "landscape-scale" conservation project.
"It is a project that we know cannot be realised in our lifetime, which will bear fruit not just for our own children, but also for the generations of visitors who, like us, hold a deep affection the Scottish Highlands," Mr Holch Povlsen and his wife Anne say on the website.
"We wish to restore our parts of the Highlands to their former magnificent natural state and repair the harm that man has inflicted on them."
The death toll in the Sri Lanka attacks is now at 290, following a series of blasts at churches and luxury hotels on Sunday. Police have arrested 24 people, but no-one has claimed responsibility.
The vast majority of those killed are thought to be Sri Lankan nationals, including many Christians who died at Easter services.
Authorities say they believe 36 foreign nationals are among the dead, with most still unidentified at a Colombo mortuary.
The international victims include:
- At least eight British citizens - including two with joint US citizenship
- Three Danish citizens
- One Portuguese citizen and six Indian nationals
- Two engineers from Turkey, according to Turkish news agency Anadolu
- Two Chinese nationals, according to the China Daily
- Two Australians, Prime Minister Scott Morrison said
- One person from the Netherlands
- One person from Japan, according to Japanese media citing government sources