Oksana Grigorieva could be checkmated into silence .... silence that could make it impossible to prosecute Mel Gibson for domestic violence -- sources tell TMZ.

As we first reported ... the L.A. County Sheriff's Department is conducting what we're told is a "serious" extortion investigation.  Our law enforcement sources tell us investigators have "numerous" emails, texts, documents and witness statements that could lead to criminal charges.

If the D.A. files extortion charges against Oksana ... sources familiar with the case tell us it virtually forces her to take the Fifth -- in other words, she'd refuse to testify against Mel Gibson because what she says could incriminate her.

Although the D.A. often prosecutes domestic violence cases with uncooperative alleged victims ... sources involved in this investigation tell us Oksana's testimony is critical, particularly because the physical evidence is shaky.  For starters, we've already told you about her dentist changing his story, not reporting the alleged violence to authorities and failing to get pictures that show injury.

We're told the D.A. considers the two investigations "connected."  If the D.A. files extortion charges, our sources say there's a good chance it could flat-out torpedo a domestic violence prosecution.

And ... if the D.A. decided not to file an extortion case against Oksana to preserve a domestic violence case against Mel, his lawyers would have a field day in court.

The plot thickens.