More than 100 are believed to be still buried in the collapsed building from a disaster that struck during the most important family holiday in the Chinese calendar - the Lunar New Year holiday.
The government in Tainan, the worst-hit city, said that more than 170 people had been rescued alive from the 17-story building, which folded like an accordion after the quake struck.
Mao Yi-chen, 20, was rescued soon after the magnitude-6.4 quake hit before dawn Saturday, and her older sister Mao Yi-hsuan was pulled out Sunday in serious condition. A rescue worker had handed over a photo album and homemade cards found next to her for her family to collect, said local official Wang Ding-yu.
"He said that 'maybe your home is damaged, but memories of the family can last,'" Wang said.
Wendy Chuang, a reporter in Taiwan, told CBS Radio News that the building was unrecognizable.
"Actually if no one told me, it's hard to tell that's a building because it just fell down and you can't tell which way it fell down actually," Chuang said. "You can't find where's the doors, where's the front, where's the back."
The extended Lunar New Year holiday officially started Monday, but celebrations were subdued and both President Ma Ying-jeou and President-elect Tsai Ing-wen canceled the handing out of envelopes of cash in their hometowns, a holiday tradition for Taiwan's leader.
The Tainan Disaster Emergency Center estimated that 118 people were still trapped at the site of the collapsed building, many at the bottom of the debris. Tainan Mayor Lai Ching-te said rescuers were able to reach many people by using information from residents who got out about the possible locations of those still inside.
Earthquakes frequently rattle Taiwan, but most are minor and cause little or no damage, though a magnitude-7.6 quake in central Taiwan in 1999 killed more than 2,300 people.
The spectacular fall of the high-rise, built in 1989, raised questions about whether its construction had been shoddy. Tainan's government said the building had not been listed as a dangerous structure, and Taiwan's interior minister, Chen Wei-zen, said an investigation would examine whether the developer had cut corners.
Eighth-floor resident Huang Guang-wei was pulled out Sunday morning from a different section from where he lived, showing how much the building twisted when it fell, Lai said. It took rescuers eight hours to get Huang out, hei said.
Among the fatalities was a 6-month-old baby girl who was pulled from the rubble and rushed to a hospital, where she was pronounced dead. A deceased man believed to be her father was pulled out 40 minutes later, Wang said.
A man in his 60s, whose son escaped and whose daughter-in-law was in serious condition, was trying to help rescuers pinpoint his grandsons. "My 11- and 12-year-old grandsons are still inside on the ninth floor," said the man, who gave only his surname, Huang. "I told my son not to buy an apartment here; it was suspiciously cheap."
Beside him, another man nodded in agreement as he waited for news of his relatives on the seventh floor.
The emergency center said 171 people had been rescued from the building, 92 of whom were sent to hospitals. More than 100 people were rescued from other parts of Tainan. Nine other buildings in the city collapsed and five careened.
Taiwanese broadcaster EBC showed video of volunteers trying to comfort the mother of a missing 20-year-old man, Chen Guan-yu. "He always thinks of me," said the woman, whose name was not given. "He worries about me being single and lonely and that no one is taking care of me."