Williams made 41 errors, while the 16th-seeded Petrova had 10 aces on Wednesday.
Williams had looked out of form in a marathon match against Vera Dushevina on Tuesday, which lasted 3 hours, 26 minutes. It was the longest match of her career.
Last year, Williams was beaten in the opening round in her first appearance at the Madrid Masters.
Williams' sister, Venus, moved into the fourth round by beating Francesca Schiavone 3-6, 6-1, 6-2.