He addressed sailors in Sevastopol harbour as part of celebrations marking the 1945 Soviet victory over the Nazis.
The Kiev government protested at the visit, calling it a "gross violation of Ukraine's sovereignty".
Kiev also reported that more than 20 people had died in a security operation against separatists in Mariupol.
Ukrainian Interior Minister Arsen Avakov said that about 20 pro-Russian protesters and one Ukrainian security officer had been killed in the southern port.
Ukrainian forces and pro-Russian separatists had clashed at the police HQ, which was set on fire.
Soviet 'iron will'
In Sevastopol, Mr Putin thanked the armed forces for their role in World War Two and hailed the incorporation of the peninsula into the Russian Federation.
He watched a fly-by of Russian aircraft and addressed seamen on naval vessels, as crowds gathered on cliffs overlooking the harbour.
He said: "I am sure that 2014 will go into the annals of our whole country as the year when the nations living here firmly decided to be together with Russia, affirming fidelity to the historical truth and the memory of our ancestors."
The BBC's Daniel Sandford in Sevastopol says Mr Putin was treated as a conquering hero as he walked through the main square and shook hands with Crimeans.
Mr Putin earlier addressed thousands during a huge, hour-long military parade in Moscow's Red Square.
He vowed to defend the "motherland" but did not mention Ukraine in his speech.
He told the crowd that 9 May, known as Victory Day in Russia, was a "day of grief and eternal memory" and stressed how the "iron will of the Soviet people" had saved Europe from slavery.
"It is a holiday when an overwhelming force of patriotism triumphs, when all of us feel particularly acutely what it means to be loyal to the motherland and how important it is to defend its interests," he said.
Russian state TV reported the Red Square parade was the largest such event in 20 years.
Nato's Secretary General Anders Fogh Rasmussen said Mr Putin's visit to Crimea was "inappropriate", adding: "We consider the Russian annexation of Crimea to be illegal, illegitimate and we don't recognise it."
US National Security Council spokesperson Laura Magnuson said: "We do not accept Russia's illegal annexation of Crimea. Such a visit will only serve to fuel tensions."
Ukraine's interim authorities held subdued memorials to mark the Soviet victory.
A brief veterans ceremony was held in Kiev's main park, in front of PM Arseniy Yatsenyuk and several former presidents.
The authorities feared pro-Russian activists would try to stoke violence if there were any higher profile celebrations.
Nazi Germany invaded the USSR - which included Ukraine - in June 1941 and advanced almost as far as Moscow before being driven back to Berlin in some of the fiercest fighting of the war.
Crimea was put under Ukrainian administration in 1954.
After the collapse of the USSR, Russia maintained a large military presence on the peninsula, and more than half of the region's population identified as ethnic Russian.
In the chaos that followed the ousting of pro-Russian President Viktor Yanukovych in February this year, Russian forces took over most of the peninsula.
Crimeans then held a referendum and voted to join Russia, though the vote was widely criticised as offering no real choice.
Pro-Russian separatists in south and eastern Ukrainian cities have now said they will hold secession referendums on Sunday.
Activists remain in control of many official buildings across the south and east despite a military operation by Kiev to remove them. Dozens have been killed in the unrest.