The British line of succession dictates who ascends the throne when Queen Elizabeth dies. Although the monarchy will need to pass the crown directly to Prince Charles when she dies, a formal royal butler working for Princess Diana believes the Princes of Wales will never be king.

According to Paul Burrell, the monarchy might get a major shakeup when Queen Elizabeth passes away. Burrell, who worked in the royal household for more than ten years, said in an interview that when his time to reign comes, Prince Charles might “do the right thing” by abdicating. “When the Queen dies and she’ll be a hundred and something, she will never abdicate. When she dies, I think Charles will do the right thing and say, ‘I’m far too old for this responsibility,’” Burrell explained.

According to Burrell, while Queen Elizabeth is already starting to hand over some of her important royal roles to Prince Charles, the truth is that the world and the Commonwealth prefer a much younger king than Prince Charles. The way forward for the royal family, he said, is to hand over the crown to Prince William.

Prince Charles’ age is not the only factor seen to hinder his ascension to the throne. For years, surveys have consistently suggested that the public is not keen on having him as king or his wife, the Duchess of Cornwall, as queen. The public’s enthusiasm for Prince Charles took a nosedive soon after the public breakdown of his marriage with Princess Diana in 1992, allegedly due to his infidelity.

Although the Prince of Wales’ reputation managed to slightly pull out of its nosedive in 1997, a majority of the UK would still rather have Queen Elizabeth retain her position than pass on the crown to her firstborn. A separate survey conducted a couple of years ago also showed that nearly half of the people questioned Prince Charles’ capacity to become king and wanted him removed from the line of succession in favor of his son, Prince William, who is the second in line to the throne.

As of now, Burrell’s prediction about Prince Charles’ fate in the monarchy is consistent with the voice of the Brits. But since who reigns as the next king or queen is not a popularity contest, Prince Charles will be England’s next king by default unless he chooses to abdicate.

Prince Charles Prince Charles, Prince of Wales, speaks during the Governor-General’s Reception at Government House. Prince Charles might not be the next king, according to former royal butler Paul Burrell. Chris Jackson/Getty Images