Nearly 100 survivors have been pulled from the wreckage of the factory, which made shopping bags 20 km (12 miles) south of the city of Lahore, but rescuers say scores of workers had been crowded into the building's basement.
"We were working on the first floor when the roof collapsed," said one of the trapped workers, Liaqat Ali, who used his mobile to talk to a television station.
"Now, I can hear the rumble of heavy machines which gives me hope that I will come out alive."
Rescue officials on Wednesday said 150 people were believed to have been in the building when it collapsed. But rescuers had to move slowly, government officials said, to avoid further injuries to those still trapped.
Injured survivors said the factory's owner, who was adding a third floor, had ignored advice from his contractor and pleas from his workers to stop construction after cracks in the walls following a powerful earthquake last week.
"The factory owner had an exchange of harsh words with the contractor who had advised him to stop the work due to cracks appearing after the earthquake," Muhammad Ramzan, a worker trapped under the rubble, told rescue officials by telephone.
The quake of magnitude 7.5 killed more than 300 people in Pakistan and the northern parts of neighbouring Afghanistan and damaged thousands of buildings.
The owner of the factory was among the dead, said a government official, Muhammed Usman. Representatives of the factory management could not immediately be reached for comment.
Muhammed Younis Bhatti, an official of emergency responder Edhi Rescue Services, said 97 survivors had been pulled from the rubble.
Pakistan's construction sector is plagued by poor oversight and developers frequently flout building codes.
In September 2012, 289 people burned to death in a fire at a garment factory in the southern city of Karachi. On the same day, a fire at a shoe factory in Lahore killed 25.