Story highlights

He "of course denied it," Mike McCaul said. "I have no reason to not believe what McCarthy said."

McCarthy ended his bid to become House speaker Thursday

Washington CNN  — 

House Majority Leader Kevin McCarthy has repeatedly said his decision not to run for House speaker came down to math – he couldn’t get 218 votes after a bloc of conservative members said they would not support him. But he has also been forced to address tough questions about his personal life.

During a closed-door meeting with Texas Republicans last week when he was still a candidate for speaker, McCarthy was confronted “point blank” with a question about whether he carried on an affair with a fellow member of Congress, Rep. Mike McCaul told CNN. McCaul’s account was confirmed by another GOP lawmaker who was in the room but asked not to be named because of the sensitivity of the conversations.

There is no evidence that the rumor is true.

He “of course denied it,” McCaul said. “I have no reason to not believe what McCarthy said.”

McCarthy on Thursday afternoon stunned Washington when he pulled out of the race for House speaker moments before a vote he was expected to win. Some conservative media outlets suggested the decision was connected to emailed threats from a conservative activist to expose a rumored affair between McCarthy and Rep. Renee Ellmers, R- North Carolina. CNN has not independently verified the email or its contents.

Both Ellmers and McCarthy have strenuously denied it. But the rumor has spread around Capitol Hill and appears to have at least indirectly affected the chaotic events of the week.

A few days before McCarthy pulled the plug on his bid for speaker, Rep. Walter Jones, R-North Carolina, had circulated a cryptic letter suggesting anyone running for leadership with skeletons in their closet should drop out – but he has several times declined to say the letter was targeting McCarthy and has not publicly accused any specific person of wrongdoing.

“It’s very ugly. This is where politics gets very ugly and slanderous,” McCaul said. “It’s very very hurtful. … It was out there. Then he addressed it and everyone moved on.”

Ellmers made a statement about the rumors to CNN.

“As someone who has been targeted by completely false accusations and innuendo, I have been moved by the outpouring of support and prayers from my colleagues, constituents and friends. Now I will be praying for those who find it acceptable to bear false witness,” she said.

Ellmers also addressed her colleagues Friday, a GOP lawmaker who asked not to be named told CNN, without explicitly mentioning the rumors.

The North Carolina Republican stood up during a Republican conference meeting and thanked party colleagues for their thoughts, prayers and support the last several days, the lawmaker said, adding that Ellmers didn’t say what she was talking about specifically.

Ellmers noted there are only 23 women in the Republican conference – and how tough is being a woman on the Hill – but said “she’s a tough cookie.”

McCarthy said he pulled out of the race after realizing his bid to become speaker was having trouble getting the votes he needed to win on the floor amid opposition from the most conservative flank of the conference, and that stepping aside was the best thing for party unity.