"I know this woman hasn't met me in a nightclub with Tiger because I've never been to a nightclub with Tiger," Williams told FOXSports.com Sunday from his home in New Zealand.
Williams has steadfastly -- and publicly -- maintained that he knew nothing of Woods' secret life of porn stars and cocktail waitresses, prompting rumors that he'd be fired for talking too much.
A high-profile golfer has been laying bets that Williams would be axed and replaced with Billy Foster, a well-respected Englishman who caddies for Lee Westwood.
Foster, who said last week that he knew nothing of any such plan, filled in for Williams at the 2005 Presidents Cup so Williams could be with his wife, Kirsty, as she gave birth to their first son.
But Williams confirmed Sunday that he would be on Woods' bag for his comeback tournament next month at Augusta National.
"I told Tiger I'd be there," he said.
The two have been together since 1999. Williams has been beside Woods for 13 of his 14 majors, only missing the 1997 Masters where Mike "Fluff" Cowan was on the bag.
James, who claims a three-year affair with Woods, told a New Zealand radio station over the weekend that she had met Williams while he'd been partying with Woods.
"If she's met me maybe it was in passing at a golf course, that's all I could think of," Williams said, "But as far as I'm concerned, I've never met this woman.
"I'm tired of having my name dragged through the mud over this. I've done nothing wrong. As I've repeatedly said, I knew nothing about what was going on."
Although many have doubted that Williams' claims that he was kept in the dark, Woods' former swing coach, Butch Harmon, isn't among them.
"I have to agree with Steve," he said in a recent radio interview. "I know Steve very well. Actually I'm the one that hired Steve when Tiger wanted to get rid of Fluff and go in a different direction.
"I would agree with Steve and I'll tell you why. Number one, Steve's wife and (Tiger's wife) Elin are very good friends and Steve's a good friend of Elin. I think had he known something, I think he would've talked to Tiger about it.
"He's a very straightforward person and very hard-nosed kind of guy. I think they have the kind of friendship and relationship that he would've said something to him about it because he was really friendly with Elin.
"It just lets you know how secretive this all really was."