This rare moment of candor from the usually solidly diplomatic monarch was caught on camera by Buckingham Palace's own staff, as the Queen greeted a senior Metropolitan police officer.
"Oh, bad luck," she says, after hearing commander Lucy D'Orsi had been in charge of security for Xi's visit.
"I'm not sure whether you knew, but it was quite a testing time for me," D'Orsi tells the Queen.
"Yes, I did," she responds.
"They were very rude to the ambassador," the Queen adds, referring to Barbara Woodward, Britain's first female ambassador to China.
The Metropolitan Police said it does not comment on private conversations of officers.
A spokesman for Buckingham Palace said in a statement: "We do not comment on the Queen's private conversations. However the Chinese State Visit was extremely successful and all parties worked closely to ensure it proceeded smoothly."
Asked about the Queen's comments at a regular press conference, Chinese Foreign Ministry spokesman Lu Kang said "the visit launched a golden era in China-UK bilateral ties."
He added that he had "no knowledge" of any threats to call off the visit, referenced by D'Orsi in her comments to the Queen.
Two gaffes in one day
The Queen's remarks were made on the same day as UK Prime Minister David Cameron was caught on camera telling the monarch that Nigeria and Afghanistan were "fantastically corrupt."
Cameron, speaking about an upcoming anti-corruption summit, said they were "possibly the two most corrupt countries in the world."
His remarks caused consternation online, and sparked an angry rebuke from Nigeria's presidential spokesman, who said they were "embarrassing."
Out of character
While Cameron has had his fair share of gaffes in the past, the Queen -- who recently celebrated her 90th birthday -- is usually careful to stay out of politics.
At the time, she hailed Xi's trip -- during which she hosted the Chinese leader and his wife Peng Liyuan at Buckingham Palace -- as a "milestone" visit.
Other members of the royal family have been less circumspect.
In the 1980s, Prince Philip told British students in China that they'd get "slitty eyes" if they stayed in the country too long.
Prince Charles is a personal supporter of the Dalai Lama, who has described the two of them as the "best of friends," and skipped a state dinner with Xi, though he met the Chinese leader at another engagement during his trip. China regards the Dalai Lama as a "separatist" and regularly pressures foreign governments not to have dealings with him.
In a leaked journal, Charles also described Xi's predecessors as "appalling old waxworks."
The Queen's remarks come during a supposed "golden era" of UK-China relations.
During a trip to a pub with Cameron, Xi toasted the country's stronger bond and said he was "deeply impressed by the vitality of China-UK relations."
Since then however, cracks have started to emerge, over China's alleged abduction of a British bookseller from Hong Kong, and Beijing's aggressive moves in the South China Sea.
In an opinion piece published in British media last week, China's ambassador to the UK, Liu Xiaoming, accused London of "ramping up" the issue and "meddling" in the region.