Kalvin Phillips says that "narratives" which followed former manager Pep Guardiola's comments about his weight left him feeling "frustrated".
Manchester City boss Guardiola has since apologised for saying in December 2022 that Phillips had come back from the Qatar World Cup "overweight".
Midfielder Phillips, who is on a season-long loan at Ipswich, told the "My Mate’s a Footballer" podcast that speculation on social media, "just kind of grew and grew," following those comments.
Speaking as a guest alongside his former Leeds team-mate Patrick Bamford and comedian Joe Wilkinson in the latest episode of the BBC Sounds series, the 28-year-old said: "Every club that I'd go to, where I spoke to the manager and the nutritionist and stuff like that, they'd always speak about weight before they'd say anything else".
He also spoke about Guardiola's footballing vision, negativity from ex-players around England at Euro 2024 and loving life under Ipswich manager Kieran McKenna.
Phillips joined Manchester City in 2022, in a deal worth £45m. He was loaned to West Ham 18 months later, having made just two Premier League starts under Guardiola out of a potential 58.
He told the podcast about being "gobsmacked" by the Spanish manager’s footballing brain on his arrival at Etihad Stadium.
City's first Premier League match following the midfielder's signing was a 2-0 win away against West Ham. Phillips claims that Guardiola told the team in a meeting two days prior exactly how the game was going to play out.
"He said 'West Ham are going to start well, they're going to put us under pressure and then we're going to have a little bit of time, we're going to get opportunities to get the ball to Kevin [de Bruyne] and then if Kevin plays it behind for Erling [Haaland], we're more than likely going to score'," Phillips remembered.
In the event, Haaland scored twice, with an assist from De Bruyne for his second.
"He mentioned different ways that we could attack and then how they're going to press us and then you'd seen it happen in real life," says Phillips.
At Euro 2020, Phillips started every game as England reached the final and was named England's player of the year that season.
However, following an ultimately unsuccessful loan at West Ham, the midfielder was left out of Gareth Southgate's squad for Euro 2024.
Phillips says he "couldn't believe" how much negativity there was towards the team during the tournament from former players.
Following a disappointing 1-1 draw with Denmark during the group stages, pundits including ex-England internationals Alan Shearer and Gary Lineker criticised the team.
Phillips said he found it "absolutely crazy" to see how well the squad "did to keep that [negativity] out" as they made it to the final of the competition, finishing runners-up to Spain.
Phillips joined Ipswich over the summer.
The Leeds-born midfielder said that, despite the fact “I'd never expect myself to be down here" he is "loving" life in Suffolk so far.
He said he spoke to Ipswich defender Leif Davies, who he knows from their time together at Leeds, before making the move
"Leif was Facetiming me every day for the three weeks before I signed," Phillips said.
He also added that he knew "straight away" that manager McKenna would "put trust in me and make sure that everything that he did was for my benefit".
Phillips has played three league games so far this season for Ipswich, who have yet to win a game.