Simon Cowell remains very close with the guys of One Direction, but claims that upon discovering them on the U.K.'s "X Factor" in 2010, it wasn't easy to work with them.

The 56-year-old music mogul "half-jokingly" tells Billboard that Harry Styles, Louis Tomlinson, Zayn Malik, Liam Payne and Niall Horan were "a nightmare" when they were coming up in the music industry.

WATCH: One Direction Will Give You All the Feels With Their 'History' Music Video

"They were like five puppies," he quips. "Really excited, loads of opinions, always wanted to hang out."

"I'm sure we were a nightmare," Tomlinson agrees with a laugh. "We were at the height of our immaturity, always running around and being mischievous. There were so many meetings where Simon had to snap us to attention, but he was very patient."

MORE: Zayn Malik Says He Wasn't Allowed to Grow a Beard or Dye Hair While in One Direction

Six years later, the 24-year-old singer insists that Cowell is "lovely and kind," a far cry from the tough-love judge he portrays on "American Idol," "X Factor," and soon, "America's Got Talent." "Simon is one of the first people I told about my son, Freddie," he gushes. "You can trust him."

Tomlinson adds that Cowell was very understanding when the boy band decided to take a hiatus. "Not once did he put any pressure on us," he says. "Most people would've tried to give us at least a nudge. But he didn't. That's a real reflection of his character."

EXCLUSIVE: One Direction's Harry Styles Reveals Pre-Fame Aspirations

That being said, Cowell says he's not convinced that One Direction will stay together. "I don't know if it's a hiatus or a breakup, to be honest," he shares with Billboard. "In a weird way, I don't want to know. I don't think they've had enough time to experience what it's like not being in the group to really answer that."

In March 2015, Malik announced he was leaving the group to pursue a solo career, and Cowell signed the 23-year-old singer to RCA Records, which, like 1D's label Syco, is under Sony. "The boys hadn't bought into the fact that he was leaving, so out of respect for them, it wasn't the right thing to put him on Syco," he explains. "What's good for Zayn is good for us, good for the show and good for the brand."

WATCH: Zayn Malik Admits He's Only Been in Touch With One Member of One Direction

While Malik prepares for the release of his first solo album this month, rumors are swirling that Harry Styles is also considering going out on his own. "He'll work out what kind of record he wants to make because he's got great taste -- which is always a help," Cowell says. "All the writers and producers want to work with him, understandably, but he probably won't rush into it."

Cowell may have his suspicions as to the future of One Direction, but Styles assured ET in November that the band is not breaking up. "I think it's important for us to take the break and have it as a break," he said. "Obviously it would be naive of us to think that people aren't going to read into that in a ton of different ways, but we're just gonna take a little break next year."