For fans, working for the royal family might be a dream job. After all, one will be surrounded with luxurious palaces, and there’s always that chance that one will be able to get a glimpse of Queen Elizabeth II, Prince Charles, Prince William or Prince Harry. But Princess Diana’s former bodyguard revealed that while it has its perks, working for a royal might not be for everyone.

Ken Wharfe worked for the royal family as William and Harry’s bodyguard in 1986. He later became Diana’s protection officer in 1988, a post that he held until 1993.

But Wharfe revealed to columnist Angela Mollard that while working as the bodyguard of a royal is indeed a prestigious post, the job might not suit everyone. “He said it can be a really hard job,” Mollard said in the podcast “New Idea Royals.”

Wharfe said that some of the job’s perks are that one gets to tour around the world a lot and be housed in luxurious accommodations. “He said, the rewards are terrific, you travel well you get free accommodation — there’s lots of staff who live at the palace — you get to go on tours, particularly in his job,” the writer narrated.

Of course, those amazing perks come hand in hand with a few challenges that the post might only appeal to a few persons in the end. “But he also says that the jobs are very demanding and there are lots of perks, but you have to be a person that fits right,” Mollard continued.

Those who work as protection officers for Diana and the rest of the royal family back then might have to sacrifice vacation time with their families. “If you want to travel with your family, [working for the royals] makes it difficult because you have to be round a lot,” the columnist added.

It’s a good position for those who don’t mind working on holidays. “Of course, they get paid holidays but it’s a very unique position, you have to be on [c]all a lot. Some people would love it some people would hate it and that’s the point that [Ken] makes,” the writer said.

Prince William, Prince Harry and Princess Diana Diana, Princess of Wales (R); Princes William (L); and Harry applaud during the Wales vs France Five Nations Cup match at Cardiff Arms Park on February 1, 1992. JEAN-PIERRE MULLER/AFP/Getty Images