Tina Knowles does not play when it comes to her ultra-famous family, and in this week's World's Most Beautiful issue, the author of the sweeping memoir Matriarch, 71, is getting candid about how one rumor in particular deeply affected her.

Back in 2011 Beyoncé and her husband, rapper Jay-Z, were excitedly preparing to welcome their first baby, Knowles’ second grandchild Blue Ivy Carter. But joyous as the occasion was for all who were actually involved, Knowles reveals it was also “one of the worst times of my life."

Amid Beyoncé's first pregnancy the star was bombarded with widespread, unfounded rumors that she wasn't indeed pregnant, kicked off by a video that circulated, appearing to show fabric over her protruding belly flattening out. Moreover, the rumors persisted as many online declared she hadn’t actually given birth to Blue, who was born on Jan. 7, 2012.

It all infuriated Knowles, who tells PEOPLE: "My family is so precious to me and babies are so sacred. So to hear people say the most horrendous things and call my whole family a lie, because we would all have to be part of that conspiracy and lie about something so sacred."

She adds. "It was the worst because I couldn’t say anything. My daughter was saying, ‘No, you’re going to make the story bigger.' But it didn’t go away for a long time. It’s one of the most painful things for me, that people could be that disgusting."

For Knowles, the most difficult part of fame "is that people feel they can dog you out any kind of way and you’re not supposed to get upset," she says, "but we’re still humans, and it hurts."

That said, Knowles has had to get used to the spotlight and speculation, even if it still angers her. "I deal with lies all the time. I'm dealing with lies now, that I'm having, like, an invisible zipper at my mouth to talk about things now because there's always these crazy rumors that are out there and, people just get to lie and they don't have to suffer any consequences, you know, it is very sad."

When it came to crafting her revealing new book, "It was very carefully written at first with," she says. "But then it was like, this doesn't even make sense because there's only, you know, all the positive stuff. And there was a lot of positivity. But it's my story. It's just the truth."

Matriarch is available now.

For more on Tina Knowles's incredible life story and how she raised superstars, pick up this week's issue of PEOPLE, available now.