GOP Rep. Allen West said Thursday there's "not a chance" he'll back down from his criticism of the Democratic Party chairwoman, rebuffing calls to apologize for calling her "vile" and "not a lady."

West leveled the insults in a heated email to fellow Florida Rep. Debbie Wasserman Schultz in response to her criticism of him on the House floor. Several House Democrats have since demanded West retract his statements, but West told Fox News he's been "patient" with Wasserman Schultz long enough and had to "make a stand."

"It comes to a point when I have to stand and I have to defend myself, and I can be very direct in my language but so be it," he told Fox News on Thursday. "I stand by what I said."

West noted that the two Florida representatives have a "history." Specifically, Wasserman Schultz helped lead a protest outside his campaign headquarters last October at which she accused him of disrespecting women.

"It's just been a pattern of this and it's continued on," West said.

His email, though, has revived complaints from female lawmakers. Several female House Democrats penned a letter Wednesday to House Republican leaders describing his comments as "sexist" and urging the leaders to "disavow" them. One lawmaker suggested the leadership take West out to the "woodshed" to set him straight.

"By any objective viewing of her floor speech, (Wasserman Schultz) was well within the rules of decorum for our chamber," they wrote. "Yet, Mr. West's response was completely out of order and an offensive personal attack with a clearly sexist tone."

Meanwhile, both sides denied reports and audio tape from Wednesday that seemingly showed West backing off his original comments. In the tape, West could be heard telling a Huffington Post reporter: "I just apologized."

It's unclear what he was referring to -- a West aide immediately shot down claims that he had apologized, calling them "absolutely false." Wasserman Schultz likewise said she had not heard from West.

Both lawmakers claim they are now waiting for an apology from the other. West also tweeted Thursday: "U.S is in crisis & everyone wants to talk about congressional members not getting along,it's insulting to Americans to waste airtime on this."

West and Democratic groups are, meanwhile, trying to raise money off the incident.

Reacting to the outrage among Democrats, the chairman of the Congressional Black Caucus initially said he would have a talk with West -- the caucus' only Republican. However, Rep. Emanuel Cleaver, D-Mo., told Fox News on Thursday that this is a matter for the Republicans, not the CBC, to handle.

So far, Republican leaders have said little about the incident.

The dispute started Tuesday on the House floor, where Wasserman Schultz had said: "The gentleman from Florida, who represents thousands of Medicare beneficiaries, as do I, is supportive of this plan that would increase costs for Medicare beneficiaries. Unbelievable from a member from south Florida."

She was referring to a House bill to cut and cap spending in exchange for a debt-ceiling increase.

Taking offense, West afterward sent out his email, which said: "Look, Debbie, I understand that after I departed the House floor you directed your floor speech comments directly towards me. Let me make myself perfectly clear, you want a personal fight, I am happy to oblige. You are the most vile, unprofessional and despicable member of the US House of Representatives. If you have something to say to me, stop being a coward and say it to my face, otherwise, shut the heck up."

West also said her actions proved she is "not a Lady."