Hazardous contact sports, like Quidditch, may be off limits for observant Jews. Even skiing may be forbidden (per DB and (?) Rabbi Willig). Nevertheless, sports and exercise are supported by Jewish law (halakhah). So this is out of bounds:
"The town of Kiryat Sefer has between 20 and 30 thousand residents (at last count). They are now constructing their first swimming pool (see Kiddushin 29a). They have yet to construct a single basketball court." Recent post at On the Contrary!
Sample halakhic arguments in favor of sports as exercise:
"Rav Moshe [Feinstein] writes that providing a swimming pool for students in the summer constitutes an act of kindness, since they need a place to cool off in the heat and sometimes this can also bring them to exert themselves more in their Torah study." (Rabbi Dr. Asher Meir citing Igrot Moshe Even Ha-ezer IV:61)
"I would think that baseball, being a means of physical exercise that isn't too boring to expect a kid to pursue regularly, is a chiyuv di'Oraisa midin [duty directly from the Biblical law] 'vinishmartem mi'od linafshoseichem' [safeguard yourselves greatly]. Anyone who has a stationary bike collecting dust in the basement knows that the hardest part of exercise is making it interesting enough to make you keep at it. Baseball qualifies." R. Micha Berger, Jan 2000 Avodah list. [Trans mine.]
See also, Gedolim and ballplaying stories from Avodah.