A source close to the Duke of Sussex tells PEOPLE that Prince Harry will always protect his wife and family from what he sees as negative and agenda-driven media narratives.

Prince Harry is reportedly “hurt and upset” by the criticism his wife, Meghan Markle, received following her recent trip to Paris.

While attending Paris Fashion Week earlier this month, the Duchess of Sussex shared a video on Instagram showing her riding in a car through the French city with her feet up. However, critics called the move insensitive, saying she was close to the Pont d’Alma tunnel, where Harry’s mother, Princess Diana, tragically died at age 36 in 1997.

However, friends told The Daily Mail that it was a “hell of a stretch” to say she purposefully drove past the location — and that Harry was frustrated by the backlash.

“Harry was left hurt and upset. Diana’s death was used as a stick to beat his wife with. [Meghan] did not even pass close to the tunnel,” a source told the outlet. “The whole thing is a joke, but not a very funny one for Harry.”

They added, “He was left feeling hurt, but the depressing thing is he wasn’t even that surprised. He was more despondent than shocked.”

A source close to the Duke of Sussex tells PEOPLE that Prince Harry will always protect his wife and family from what he sees as unkind, negative and agenda-driven media narratives.

Meghan, 44, made her debut at Paris Fashion Week on Oct. 4, attending the Balenciaga show to support Pierpaolo Piccioli, the label’s newly appointed creative director.

In her now-defunct lifestyle blog, The Tig, Meghan mused in 2014, “Oh, how I love the city of lights. Yes, it has been romanticized in our minds from cinematic shots of the Eiffel Tower to long walks on the River Seine — from ‘Before Sunrise’ to ‘Amélie’ to Jean-Luc Godard’s ‘Breathless.’ The food, the accent, the je ne sais quoi of the women with their inherent chicness – I sigh un petit peu just typing this.”

Prince Harry, 41, recounted in his memoir, Spare, that he drove through the same tunnel where his mother died while attending the 2007 Rugby World Cup semifinal in Paris.

“It had been a very bad idea,” he wrote. “I’d had plenty of bad ideas in my twenty-three years, but this one was uniquely ill-conceived. I’d told myself that I wanted closure, but I didn’t really. Deep down, I’d hoped to feel in that tunnel what I’d felt when JLP gave me the police files—disbelief. Doubt. Instead, that was the night all doubt fell away. She’s dead, I thought. My God, she’s really gone for good.”

Although Meghan never got to meet her mother-in-law, Harry has spoken about the similarities between his late mom and his wife.

When the Duchess of Sussex appeared as a guest on The Jamie Kern Lima Show in April, the host recalled: “So Harry said publicly, ‘So much of what Megan is and how she is so similar to my mum. She has the same compassion. She has the same empathy. She has the same confidence. She has this warmth about her.’ How do you feel about him sharing that?”

“It’s beautiful,” Meghan replied. “I wish I could have met her.”