The MBCA Kim-Nirvana, a motorized outrigger with 173 passengers and 16 crew on board, capsized minutes after leaving the port of Ormoc.
Coast guard spokesman Armand Balilo said 127 people survived, while 26 were still listed as missing.
"Search and rescue operations are ongoing. Initially we learned that it was due to big waves," said Rey Gozon, director of the office of civil defense for the region.
Scores, sometimes hundreds, of people die each year in ferry accidents in the Philippines, an archipelago of 7,100 islands with a notoriously poor record for maritime safety. Overcrowding is common, and many of the vessels are in bad condition.
May Sopa told CNN Philippines that when she hit the water, other panicking passengers pushed her down, before a man holding on to a water container rescued her.
"I told him 'Please save me'," Sopa said, adding they bobbed on the water and tried to paddle ashore. Two other women and a boy around 10 years old also latched on to the same water container.
Television pictures showed orange rubber boats and white coast guard vessels bringing survivors ashore, including at least one toddler and some on stretchers, with the half-submerged ferry visible offshore.
Balilo said authorities were looking at various possible causes, including human error and bad weather.
"There was an occasional swell but the sea condition was manageable. Some motorized outriggers were able to sail," he told a local TV news channel.
"There was no gale warning and while there was a tropical depression, it was far from the area of the accident," he said.
Authorities took the captain and some crew members of the 33-tonne boat into custody, Balilo said, adding that a formal investigation would be conducted as soon as search and rescue operations were concluded.
Eli Borinaga, the vice mayor of Pilar town on an island to the south who had hoped to join the ferry but didn't make it on time, told local radio that there was only light rain at the time of the accident.
He cited a witness at Ormoc port who saw the boat make a sharp turn just before it capsized.