The temporary restraining order says Sheen must also avoid the actress' home and car. The judge granted Sheen once-weekly visits with 2-year-old Sam and 10-month-old Lola, and scheduled another hearing for May 12.
Richards, 35, alleged in court papers that Sheen pushed her, shoved her, and threatened her and her parents. The papers also include transcriptions of telephone messages Richards said Sheen left in April 2005.
Sheen, 40, denied the allegations, and the judge granted his request that he and Richards be allowed to record telephone conversations with each other.
Sheen said in papers signed Thursday that he and Richards had agreed to counseling to "address parenting issues and to establish a mutually acceptable custody schedule," but that Richards stopped participating after two sessions.
He also alleged that he had been seeking overnight visits with the children but that last week, Richards refused to allow him any contact with them.
"I deeply regret her response to my request for the court to decide what's best for our children has taken the form of baseless allegations that I deny," Sheen said in a statement.
A call to Richards' spokeswoman was not immediately returned Friday.
Richards filed for divorce in March after three years of marriage. The couple met while shooting "Good Advice" in 2000.
Sheen stars in the CBS sitcom "Two and a Half Men." His screen credits include "Platoon," "Wall Street" and the "Hot Shots!" movies.
Richards has appeared in such films as "Wild Things" and "Starship Troopers."