The Crusaders (8-20, 7-7), who led 47-46 at halftime, scored the first eight points of the second half and pulled away from the Mountain Hawks (15-14, 6-8).
Holy Cross shot 53.8 percent (14 of 26) in the second half to Lehigh's 34.5 percent (10 of 29). Brown had 18 points after halftime, making 6 of 7 shots.
C.J. McCollum, the nation's eighth-leading scorer at 21.9 points per game, led Lehigh with 19 points but made just 5 of 17 shots. Gabe Knutson added 15 points and Mackey McKnight 14 for the Mountain Hawks.
Andrew Beinert scored 15 points for Holy Cross.
The Crusaders will host Lafayette in a first-round tournament game on Wednesday, while Lehigh will be the No. 4 seed and host Navy.