You know what they say — breaking up is hard to do — and the same holds true when it comes to your doctor.

That’s why patients, just like people in personal relationships, often make up excuses to put off the inevitable. But Dr. Keith Ablow, a psychiatrist in a private practice and a FOX News contributor, says that’s not a good idea, especially when it comes to your well-being.

“I think there are all kinds of logistical excuses people make to stay in a situation that is suboptimal,” Ablow said. “But this is your health. Like any relationship, it’s worth discussing and presenting your concerns.”

So, how do you know when it’s time to have the dreaded “talk"?

“If you have an internal sense that you either don’t understand your treatment plan or the sense that you don’t trust your doctor — either situation alone is probably grounds for leaving,” Ablow said.

Other reasons to leave include:

— If you feel ignored by your doctor;

— If you don’t understand your condition or the course of treatment;

— If people around you keep telling you that they don’t understand where your care is heading.

And don’t be fooled. It’s not just patients who want to end the relationship.

Doctors sometimes feel the same exact way, Dr. Suzanne Gilberg-Lenz, an obstetrician and gynecologist, and a regular contributor for Momlogic.com, told FOXNews.com.

“Don’t think we don’t want to break up with you guys, too,” she said.“It can be an intense relationship and sometimes it’s not a good fit and we feel that, too.

"We’ll stick it out, but when the patient feels it's not a good fit — the doctor often feels the same way. If that’s the case, the relationship needs to end just like any other relationship.”

Gilberg-Lenz, who practices in Los Angeles, said it’s the patient’s prerogative to leave a doctor at any time, but there is a right and wrong way of going about it.

“In general, the least mature way to leave your doctor is the sneak attack, when the doctor gets a medical records release form and the patient is gone,” she said.“The most mature way is to discuss with your doctor about why you’re not clicking. Don’t just run away and leave, because maybe something can be salvaged.”

This is particularly true in a group practice setting. Gilberg-Lenz said she encourages people who may not be happy with one doctor in a practice to stay in the office and see another doctor within the group.

In the end, if you do decide to leave your doctor, there are a few things you need to handle to make it a smooth transition, and that includes getting your medical records transferred.

“Sometimes people feel mortified or worried to ask for their medical records because their doctor is going to know they left,” Gilberg-Lenz said. “But I’ve got to tell you — I have a full life and if you leave my practice, I’m still going to be able sleep at night.”

Getting your medical records may sound like a daunting task, but it’s actually pretty easy. Most doctors’ offices have a release form that you can use to request your records. Once you fill out all the proper information, you can usually have the records sent directly to your new physician. But remember, there may be a fee. (Some states do not allow doctors to charge fees for the transfer of medical records, so it’s best to check before handing over the cash.)

The bottom line: Do what is best for you, and in the end everyone will be better for it.

“There can be tremendous power in terms of one's health when you believe in your physician,” Ablow said. “It can be comforting and life sustaining. But you wouldn’t stay in a friendship or a marriage in which your needs are consistently disregarded, and you shouldn’t stay in a doctor/patient relationship if your needs are disregarded either.”

Click here for more information on medical records from the NIH.