Prince Andrew refused to support Prince Charles and Camilla during a difficult time in their relationship. And until today, the Duchess of Cornwall has not forgotten about it.

To make things worse, the Duke of York reportedly “spread poison” about the royal couple instead of building them up with Queen Elizabeth II. In his biography, royal author Tom Bower said that this just made Prince Andrew and Prince Charles more distant.

“Mindful of Diana’s prediction on Panorama that he would never be King, Charles convinced himself that even Princess Diana and Sarah Ferguson, were hatching plans to replace him as heir,” he said. During Bower and Prince Charles’ conversation, the heir to the throne accused his younger brother of wanting to be him.

The dad of two also said that he should have just allowed Prince Andrew to work for him. This way he would become more helpful. Camilla also had the same thoughts about her brother-in-law. A source said that the Duchess of Cornwall believes that Prince Andrew could have done more for her and Prince Charles.

“The Queen listens to Andrew and he could have helped his brother at a time when he had few allies within the family,” the source said. “In fact, I would go so far as to say he was deeply unhelpful when support would have meant a huge amount to her and the Prince. Looking back to when there was little public sympathy for Charles and Camilla, if not open hostility, it was a case of Andrew playing politics. Camilla hasn’t forgotten,” the source added.

Previous reports also claimed that Sarah Ferguson’s ex-husband has always been jealous of his older brother. In the past, Prince Charles expressed his desire to slim down the monarchy. This decision would mean that Prince Andrew’s daughters, Princess Beatrice and Princess Eugenie, would have lesser involvement in the royal family.

And royal experts also claimed that Prince Charles has never been close to Fergie. In fact, Princess Diana’s ex-husband doesn’t understand why Sarah is still very much a part of Prince Andrew’s life when they have already been divorced for decades.

Royal Family Queen Elizabeth the Queen Mother poses 04 August 1989 outside her London Clarence House residence for photographers with Queen Elizabeth (R), Sarah Ferguson, Prince Charles, Princess Diana and other members of the Royal Family on her 89th birthday. JOHNNY EGGITT/AFP/Getty Images