In a new interview with Australia's "60 Minutes," Smith gets candid on just how much jabs at his weight have hurt him.
"If someone called me fat, that affects me way more than someone calling me a f----t," he admits, referring to his sexuality. "I think just because I've accepted that, if someone calls me a f----t, it's like, I am gay and I'm proud to be gay so there's no issues there. If something calls you fat, that's something I want to change."
Smith says a big part of being overweight was his loneliness.
"I genuinely think I gained about two stones [28 pounds] when I moved [to London]," he says about relocating from a small village to London when he was 18 to pursue a music career. "Just get loads of chicken and watched 'Lost' every night after work."
"The Lonely Hour, that's where it all came from," he jokes about the title of his critically-acclaimed album.
While hours in front of the TV didn't help, emotional eating was a big part of his weight gain.
"When I was at school and wasn't having a great time or when music wasn't going very well. I would eat, eat," the 6'4" star explains. "Eating would make me feel better, when I felt lonely I would eat."
These days, the "Stay With Me" singer is looking better than ever. Last week, he shared that he's only four pounds away from his weight goal.
"Beautiful catch up with the incredible @ameliafreer xx 4 pounds away from my goal weight and feel happier and healthier than ever #eatnourishglow," he Instagrammed. Smith has credited Amelia Freer, a nutritional therapist and author of "Eat. Nourish. Glow: 10 Easy Steps for Losing Weight, Looking Younger & Feeling Healthier," for his incredible results.
"[Amelia Freer] has completely transformed my relationship with food," Smith wrote a month ago on Instagram. "It's not even about weight loss it's about feeling happy in yourself. Love you Amelia & thank you for making me feel so happy inside and out."