"There was no common view," Latvian Interior Minister Rihards Kozlovskis said, referring to a plan to transfer 60,000 migrants on a quota system.
Efforts to reach agreement over the Mediterranean migrant crisis will continue at an EU summit next week.
Separately on Tuesday, Italian police removed dozens of migrants camping at the French border.
Police in riot gear forcibly shoved some of them onto buses and took them away from Ventimiglia, not far from resorts in the French Riviera.
The migrants, mostly from Eritrea and Sudan, had been prevented from entering France.
The European Commissioner for Migration, Dimitris Avramopoulos, insisted that only a compulsory mechanism - not voluntary - would work to redistribute 40,000 migrants among the EU states.
The other 20,000 - many of them Syrian refugees - would be moved to Europe from UN refugee camps under a voluntary scheme, once their claims to asylum were proven to be genuine.
"Solidarity does not come in pieces," Mr Avramopoulos said after the interior ministers' talks in Luxembourg. "We made progress today, but we are not there yet.
"It's time we looked beyond national interests and avoided the lowest common denominator, the bargaining and finger-pointing."
He said the EU "tried a voluntary approach in the past and it didn't work" - referring to an attempt to ease Malta's migrant burden.
Earlier Italian Prime Minister Matteo Renzi warned that without agreement Italy would adopt a "Plan B" that would "hurt" Europe.
While some EU states including Germany and Austria back the quota idea, others argue that migrants should not be forced to move to countries where they do not want to settle. The UK, Denmark and Ireland have exemptions from the quota plan.
More than 1,800 migrants have died in the Mediterranean so far this year - a 20-fold increase on the same period in 2014. The majority had set sail aboard rickety, overcrowded boats from strife-torn Libya.
In recent weeks naval vessels, including the UK's HMS Bulwark, have rescued thousands of people at risk of drowning.
For the past few days, Italy has been involved in a row with France over the presence of more than 200 African migrants in Ventimiglia.
France has accused Italy of failing to respect EU asylum rules, but the Rome government has argued the migrants see Italy only as a transit country.
The Dublin Regulation says the EU country where migrants first arrive should deal with their asylum claims.
Going into the Luxembourg talks, French Interior Minister Bernard Cazeneuve said "we have to build solidarity for our Italian friends, and they have to build responsibility". Responsibility, he explained, must include "organising the return of irregular migrants".
Another priority for the EU is to send home many more migrants who have managed to stay in the EU illegally. Mr Avramopoulos complained that in 2013 the migrant return rate was just 33%.
The European Commission has proposed a "distribution key" for relocating migrants across the EU, taking account of factors such as population, gross domestic product and unemployment.
But the crisis has opened up fissures in the EU, the BBC's Europe correspondent Damian Grammaticas reports. Eastern nations like the Czech Republic, Poland and Hungary say they are adamantly opposed to quotas requiring them to take in thousands.
The UK says it is opting out of the plan, as it can under EU rules. Ireland also has an opt-out, but has pledged to "do its fair share" and take part in the EU's migrant distribution scheme.
Italy's 'Plan B'
Italy and Greece - which have already seen more than 100,000 boat people arrive on their shores this year - insist other EU nations must share some of the burden.
Mr Renzi is reported to have warned that, without a fair deal, Italy would start issuing temporary visas to enable migrants to travel beyond Italy under Schengen rules.
Although details of Italy's Plan B have not been given, reports suggest that Italy would refuse to allow migrants rescued by foreign navies to disembark on its shores.
The Schengen system removed compulsory passport controls at most of the EU's internal borders, to encourage free movement.
German Interior Minister Thomas de Maiziere said that "we're not looking to change Schengen, we're not for systematic border controls".
EU quota proposals
- 40,000 migrants who arrive from Syria and Eritrea over the next two years to be relocated across the EU
- Of that total, 24,000 to be taken from Italy, 16,000 from Greece
- "Distribution key" criteria to include a country's population, GDP and unemployment, as well as the number of refugees already taken in
- France and Germany are expected to receive more than 30% of the relocated refugees
- Separate proposal to resettle some 20,000 mainly Syrian refugees currently living in Jordan, Lebanon and Turkey