A congressman from Southern California is facing backlash over recent remarks that, in his view, homeowners have a right to refuse to sell their property to gays and lesbians, reports said.

U.S. Rep. Dana Rohrabacher, R-Calif., a 15-term member of Congress, reportedly made the comments May 16 at a meeting in Washington, D.C., of a delegation from the Orange County Association of Realtors.

“Every homeowner should be able to make a decision not to sell their home to someone [if] they don’t agree with their lifestyle,” Rohrabacher told the gathering, according to Wayne Woodyard, a former president of the Realtor organization, the Orange County Register reported.

Rohrabacher, who represents California’s 48th Congressional District in coastal Orange County, confirmed the “accuracy of the sentiment” Thursday, the Register reported.

He said homeowners should have the right to “choose who they do business with.” He distinguished race-based discrimination from refusing to do business with a person for their “lifestyle or political beliefs.”

“We’ve drawn a line on racism, but I don’t think we should extend that line,” Rohrabacher said.

A Realtor gay-rights group reportedly protested Rohrabacher’s statement, prompting the National Association of Realtors (NAR) to withdraw its support for Rohrabacher, the newspaper reported.

“We certainly hope that Congress will … support the elimination of housing discrimination based on sexual orientation or gender identity,” NAR, a trade group of 1.3 million members. said.

Rohrabacher faces 15 challengers in his “toughest re-election campaign,” including eight Democrats and former Orange County GOP chairman Scott Baugh, the report said.

One of the Democrats running against Rohrabacher is Harley Rouda, son of a former Realtor association president, the Register reported.

Click here for more from the Orange County Register.