In a 3-2 majority verdict, the court called the practice "un-Islamic".
India is one of a handful of countries where a Muslim man can divorce his wife in minutes by saying the word talaq (divorce) three times.
The landmark court decision came in response to petitions challenging the so-called "triple talaq" custom.
The cases were filed by five Muslim women who had been divorced in this way and two rights groups.
Women's rights campaigners have hailed the court's decision as a historic win.
What is instant divorce?
There have been cases in which Muslim men in India have divorced their wives by issuing the so-called triple talaq by letter, telephone and, increasingly, by text message, WhatsApp and Skype. A number of these cases made their way to the courts as women contested the custom.
Triple talaq divorce has no mention in Sharia Islamic law or the Koran, even though the practice has existed for decades.
Islamic scholars say the Koran clearly spells out how to issue a divorce - it has to be spread over three months, allowing a couple time for reflection and reconciliation.
Most Islamic countries, including Pakistan and Bangladesh, have banned triple talaq, but the custom has continued in India, which does not have a uniform set of laws on marriage and divorce that apply to every citizen.
What did the court say?
Three of the five Supreme Court judges called the controversial practice "un-Islamic, arbitrary and unconstitutional". One of the judges, Justice Kurien Joseph, said the practice was not an essential part of Islam and enjoyed no protection.
The judges also said it was "manifestly arbitrary" to allow a man to "break down (a) marriage whimsically and capriciously".
Chief Justice JS Khehar, in a differing opinion, said that personal law could not be touched by a constitutional court of law. The opposing judgements also recommended that parliament legislate on the issue.
However this is not binding and is up to parliament to take up.
The Indian government, led by the Hindu nationalist Bharatiya Janata Party (BJP), has supported ending the practice. Prime Minister Narendra Modi has brought up the issue several times including in his Independence Day address on 15 August.
How are people reacting?
The judgement is being widely hailed as a major win for Muslim women and women's rights. The prime minister praised the "historic" ruling.
Shayara Bano, one of the main petitioners, said she appealed to people to accept the ruling and not politicise the issue. "I have felt the pain when family breaks. I hope no one has to go through this situation in future," she told reporters.
Hasina Khan, founder of the Beebak Collective which fought against triple talaq, called the verdict "historic". "We are extremely happy. Muslim women have struggled for years," she told the BBC.
Zakia Soman, an activist from Bharatiya Muslim Mahila Andolan, another of the groups which contested the practice, said Indian women of many religions had supported them. "It's a historic day for us, but it doesn't end here," she said.
The All India Muslim Personal Law Board (AIMPLB), which had maintained that the court had no jurisdiction over the matter, has yet to make an official statement on the ruling, but has convened a meeting to decide what its response should be, a report in Indian newspaper The Hindu said.
However, it quoted an executive member as saying that the judgement would have "wide ramifications" as it affected the religious rights of minority groups.
But the All India Muslim Women Personal Law Board (AIWPB), which had opposed the stand of the AIMPLB, said the judgment "could not have been better".
"It will change the entire landscape of Muslim families. It's now in the mainstream and will protect not only women, but children. Families will be more stable because children will also be protected," Chandra Rajan, an advocate for the group, told the BBC.
On social media the hashtags #TripleTalaq and #SupremeCourt began trending on Twitter India even as the verdict was being announced. The hashtag #Tripletalaq is also trending globally on Twitter.
Is talaq talaq talaq allowed around the world?
Dating from the 8th Century AD and not mentioned in the Koran, triple talaq divorces often conflict with countries' legal systems.
India has become the 23rd country to outlaw them, joining places as far apart as Egypt, Sri Lanka, the United Arab Emirates and Malaysia.
In countries including China and the UK a couple must go through the official channels to be legally divorced but there have been cases of individuals considering themselves divorced after the triple talaq has been said.
In Saudi Arabia the law leaves room for various interpretations of religious custom, and triple talaq is practiced.
This type of verbal divorce is practiced around the world but as it is illegal in so many countries, it is hard to say exactly how common it is.
Why must "talaq" be said three times? Under some interpretations of Islamic law, a man can divorce his wife and get back together with her - but only twice. After the third divorce, the marriage is completely over and cannot be started again without an intervening marriage to someone else.
Scholars are divided on whether it counts as a full and final divorce to say the word three times, or whether it needs to be said on three separate occasions.