Updated

The search for a "gay gene" may be off-target, new research finds. Another process called epigenetics that switches genes on and off may explain why homosexuality runs in families.

Epigenetics are heritable changes caused by factors other than DNA. Instead of traits getting passed down through the genes, epigenetic change happens because of the way genes are regulated, or turned on and off.

These genetic regulators may be the reason homosexuality persists in nature despite the fact that gay people are less likely to reproduce, suggests the new study published in the journal The Quarterly Review of Biology.

"These things have evolved because they're good for the parents, but they sometimes, not [with] high frequency, but sometimes carry over" into offspring, study researcher William Rice, an evolutionary geneticist at the University of California, Santa Barbara, told LiveScience. In a male fetus, Rice and his colleagues write, an epigenetic change that benefited the mother may lead to "feminization" of sexual preference — homo- or bisexuality. The same may be true for epigenetic changes passed down by dad to a female fetus. (The terms feminization and masculinization of sexual preference refer to sexual orientation only — not to physical or personality traits of the offspring.)

The findings add to past research suggesting gay men haven't died out, because female relatives of gay men tend to have more children on average than other females. The study researchers specifically found that two genes passed on through the maternal line could produce this effect.

Hormones, epigenetics and orientation

Rice and his colleagues focused on epi-marks, which are molecular changes that act like temporary "switches" to turn genes on and off. If a gene is a blueprint, the epi-mark is the construction foreman who makes sure the product gets built. An epi-mark also determines when, where and how much a gene is expressed, according to the National Institute for Mathematical and Biological Synthesis.

These molecular switches are usually erased very early in the developmental process, but they can be passed down from generation to generation, too, Rice said.

Some epi-marks are particularly important during fetal development, when they promote normal physical development in the sexes despite natural variations in testosterone during pregnancy. Researchers know that fetal exposure to too much testosterone can masculinize the genitals, brain or behavior of a genetically female fetus. Likewise, too little testosterone can make a genetically male fetus more feminized. [Image Gallery: Fetal Development in Womb]

But here's the catch: There's lots of overlap between the levels of testosterone male and female fetuses get exposed to. That means there must be another side to the story, Rice and his colleagues wrote.

That side appears to be epigenetics, Rice said.

"Early in development, we think these epi-marks are laid down so that girl fetuses will be relatively insensitive to testosterone and male fetuses will be relatively sensitive to testosterone," Rice said.

Biological behavior

Thus, if an epi-mark that kept a mother from getting exposed to high testosterone in development gets passed on to her son — the opposite sex — it could desensitize him to testosterone, contributing to his sexual preference for men. Similarly, if a male-specific epi-mark from dad gets passed to a daughter, it could "masculinize" her sexual preference, making her more interested in women. [5 Myths About Gay People]

These findings could explain why twin studies show that homosexuality runs in families, but no "gay gene" can be found, Rice said. In identical twins, there's about a 20 percent chance that if one twin is gay, the other will be too. If genetic change were responsible for homosexuality, you'd expect a much higher match, Rice said. Epigenetics, however, can explain the heritability without the need for a specific genetic change.

The hypothesis could be tested by examining epigenetic marks in parents of kids with gay versus straight offspring, Rice said. There are, of course, concerns that this knowledge could be used by parents who want to avoid gay offspring, Rice said, but that concern already exists around certain hormonal conditions in utero, which are known to contribute to an increased chance of offspring being lesbians.

"That cat's already out of the bag," Rice said. He added that an understanding of the biological underpinnings of homosexuality could help emphasize that same-sex behavior is not "unnatural."

"In fact, it's a major part of the natural world," Rice said. Fourteen percent of Western gulls raise chicks in female-female pairs, he pointed out. And 8 percent of male sheep show zero interest in fertile ewes, but get sexually excited by other rams.

"We think that people who are gay or lesbian have a right to know, what is the biological foundation for this condition?" Rice said.

Follow Stephanie Pappas on Twitter @sipappas or LiveScience @livescience. We're also on Facebook & Google+.

Copyright 2012 LiveScience, a TechMediaNetwork company. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.