At least 238 others suffered injuries when a powerful storm toppled the crane, according to the nation's civil defense authorities.
The crane fell 10 days before the start of the Hajj, an annual pilgrimage expected to bring 2 million people to Mecca.
The mosque is the largest in the world and surrounds Islam's holiest site known as the Kaaba.
'Everybody was pushing'
Witnesses posted photos and video on social media showing the crane crashing through the mosque roof. Scores of bodies, blood and debris lay scattered across the courtyard.
"We just washed and were getting ready to head to the Masjid al-Haram for the Maghrib prayer (sunset prayer)," said Yahya Al Hashemi, 30, a CNN iReporter who shot iPhone video of the crane striking the roof of the mosque.
"It was a sandstorm which turned to rainstorm and lot of the construction covering boards were flying around, and lots of cracking noises which unfortunately ended with this tragedy. ... Everybody was pushing trying to escape from inside towards the exits," Al Hashemi said.
The crane collapsed after a strong thunderstorm hit Mecca, bringing gusty winds that shifted direction and caused the local temperatures to drop, CNN meteorologists reported.
Trees uprooted, glass broken
The storm was so strong, it uprooted trees and broke windows throughout Mecca, said Khaled Al-Maeena, editor at the Saudi Gazette in Jeddah.
The crane fell during a lull in visitors at the mosque, he said.
"Had it happened an hour later it would have been much worse," he said. "Had it happened five hours earlier or four hours earlier, I think the death toll would have been more than a thousand.
Construction cranes surround the Grand Mosque, which is undergoing expansions to make the pilgrimage more manageable, he said.
"The irony is that all this expansion was being done to see to the welfare of the pilgrims," he said.
Victims from various nations
Nations scrambled to account for their pilgrims after the crash.
The Egyptian health ministry said 12 of its nationals were injured while India said it received reports that nine of its pilgrims were wounded.
Pakistan's foreign ministry said that 22 of its nationals were hospitalized in Saudi Arabia with severe injuries. Pakistani embassy officials are headed to Mecca from Riyadh to find out if there were any fatalities.
Saudi officials have not released a breakdown of the nationalities of the casualties.
No stranger to tragedies
Tragedy has hit Mecca before, often because of the throngs of people there for the Hajj.
In 2006, a stampede killed at least 363 people. As with other incidents, it happened during a religious ritual in which the pilgrims stone a symbolic devil.
Hundreds were killed in other stampedes in 2004 and 1998, and 1,426 died in 1990.
Islam requires every Muslim who is physically and financially able to make the journey to Mecca at least once in a lifetime.
Hajj occurs two months and 10 days after Ramadan ends, during the Islamic month of Dhul-Hijjah.
Nearly 800,000 pilgrims had arrived in Saudi Arabia by this week for Hajj, authorities said.