The latest protests are a reaction to him sacking two finance ministers last week, further damaging confidence in the economy.
This comes on top of claims of widespread corruption, recent student protests and a succession battle in the governing African National Congress.
The marchers are using the hasthag #ZumaMustFall on social media.
The #ZumaMustFall hashtag is a modification of #FeesMustFall which was used to organise protests against a rise in university fees.
Before that, #RhodesMustFall was used to demand a statue of British colonialist Cecil Rhodes was removed from the University of Cape Town's campus.
At the scene: Karen Allen, BBC News, Johannesburg
"It feels like the start of something big," a woman in her 50s called Lianda told me.
It's the first protest she has been on since she saw friends and family killed during the 1976 Soweto uprising.
For her it was important to join the demonstrations with her teenage daughters "for the sake of their future". So along with thousands of others she came to register her displeasure at the ANC leader, who some increasingly see as a political liability.
But unlike the groundswell of protest that led to the recall of President Thabo Mbeke in 2008, Jacob Zuma still appears to have support within the National Executive Committee of the ANC.
Earlier this week, the party's top brass went on TV to deliver a message to a South African public still reeling from the events of the past week, that they still support their president.
Yet privately there is said to be considerable disquiet within the ANC.
This may not be an Arab Spring but watch this space. The discontent is growing louder.
The ANC has been in power since the end of apartheid in 1994 and won a landslide in general elections last year.
However, the economy is struggling, unemployment is around 25% and many accuse ANC officials of corruption.
Mr Zuma's decision to appoint the little known Des van Rooyen as finance minister, before replacing him four days later, was widely condemned.
The appointment led the rand to tumble to record lows, although it rose after he was replaced by the respected Pravin Gordhan on Sunday night.