It's part rugby, part dodgeball and part fantasy. But as Harry Potter's magical, fictional game of Quidditch takes off as a real sport, it's only part fun and games.

Quidditch, the brainchild of Harry Potter author J.K. Rowling, has taken flight in hundreds of colleges and high schools in America. It's given birth to an international organization and it's even inspired a competitive World Cup, which is being played in the heart of Manhattan this week.

But the Quidditch craze isn't all smooth soaring.

Critics say the game is more joke than sport and it's hard to tell which is more costly: the money spent playing it or the embarrassment it brings their institutions. And others say the efforts by some to bring legitimacy to the game -- including holding tryouts, having more structured competitions and even seeking NCAA recognition -- are threatening to take the fun out of something that was supposed to be . . . fun.

Alicia Radford, spokeswoman for the International Quidditch Association -- yes, there's an International Quidditch Association -- says she never anticipated Quidditch would take off the way it has.

"Quidditch got its start in 2005 at Middlebury College. It was just kind of a Sunday afternoon dorm sport," she told FoxNews.com.

But what happens in Vermont doesn't stay in Vermont, and the game spread to more than 400 colleges, 300 high schools, and 12 other countries and spawned the association -- a registered, nonprofit organization that codified the game's official IQA rules.

"Three chasers on each team use the quaffle, which is a slightly deflated volleyball, to score; they can score in any of the three hoops and goals are worth 10 points each," Radford explained.

Standing in the chaser's way are a goalkeeper and two beaters from the opposing team who aim to temporarily knock them out of the game with dodgeballs, er, ‘bludgers.'

And all of this is done with a broom between their legs … and most of them wear capes … and they can tackle … and it's coed. 

But wait, there's more!

No game of Quidditch is complete without a snitch, which in the books is a small, fast-flying gold ball that a team's seeker must catch to gain extra points and end the game. 

But how do you replicate a flying, gold orb that's hard to see and even harder to catch?

"It's a person dressed in as much gold as possible, and they have a tennis ball stuffed in a sock hanging out of the back of their shorts," Radford said.

The snitch can go almost anywhere it wants and use almost any means necessary to avoid being caught.

"Snitches have been known to ride on bicycles across the field; snitches have hid in families in the stands; in 2008 a snitch appeared at the top of one of the buildings," Radford said. "… I even saw a snitch once take off the seeker's cape and wave it at him like a matador."

But one thing that neither the snitch, nor any other player, can do is fly, leading some to say that calling it a sport outside of Hogwarts is hogwash.

"Maybe the witches and wizards of the Harvard Quidditch team know a spell to ward off budget cuts," the Harvard Crimson reported in an article about the Harvard team's $600 club sport grant.

That's right . . . $600.

"The startup expenses can be a little more intense than if you were to start up, say, an ultimate Frisbee league, and all you needed to buy was a Frisbee," Radford said. "You need a volleyball and two dodgeballs; the hoops can be a little more expensive…"

And then there are the brooms.

Players are allowed to play with any "wooden or plastic pole at least 40 (inches)," according to the rules, but the IQA Guidebook to Getting Started recommends teams buy brooms like the Scarlet Falcon – a hand-crafted doozie made by a company in Florida for $60 a pop. (Note: It still doesn't fly.)

And though most teams fund themselves, critics say Quidditch is drawing attention and news coverage away from other, more serious school accomplishments.

"Quidditch is sapping our school's funds, is causing me to be ridiculed by all my friends, and might cost me a job," Middlebury senior Zachary Harris wrote to the "Stop Quidditch!" Facebook group.

But if some Quidditch players have their say, the game will fly even higher.

Valerie Fischman, a Quidditch player at the University of Maryland, says she's working to have the NCAA officially recognize the sport.

"Funding would be more easily acquired, Quidditch pitches would be a common sight on campus, playing against other teams would be a breeze," Fischman wrote in an Op-ed posted on the IQA website. "And how cool would it be to have your athletic department backing everything surrounding the sport!"

Others players say that may be putting the broom before the wizard, but that the sport definitely needs to continue to develop. 

"I don't think we want to alienate the people who play the sport just because it's fun," Radford said. "… But at the same time, over the last five years, Quidditch has acquired and developed some really good athletes who want to take the sport more seriously, and having good opportunities for them to play competitively is important to keeping the sport moving forward."

"There's the NBA and there's your local rec league, and I think we can have the same kind of duality," she said.

But all Quidditch fans agree the most important aspect of the game is fun, which Radford says, isn't going anywhere.

"At Quidditch tournaments there are vendors, there are a capella groups, we have trained improv comedians as announcers … it's not just about the sport, it's about the atmosphere around it," she said. "It's a competitive game, but it'll always be a very fun atmosphere."