I have turned down two callings in my life. The first was about two months after I got off my mission. The bishop called me into his office and asked me to teach a primary class. I turned him down because I had just gotten a job in a town about 40 miles away and I was moving. This taught me an important lesson, not all callings are inspired. Sometimes callings are simply to fill a position.
The other time I turned down a calling was about 20 months ago. I had been the substitute gospel doctrine teacher, meaning when the regular GD teacher was out of town, they asked me to teach. One Sunday the bishop called me into his office and asked me to be the full time GD teacher and I reluctantly turned him down. The reason was my then 85 year old mother was in a convalescent hospital. She had several close calls when she was in the hospital, sometimes those close calls were because of inattention by the staff. Because of this, my sister and I decided that we needed to be there at the hospital as much as humanly possible. This would serve two functions, one, so we could hopefully catch any mistakes the hospital made before they became major mistakes, and two so the hospital knew we cared about our mother. This is critically important because it seems that many families only visit their loved ones once a week for 20 minutes on a Saturday and then they wonder why Dad is dead after a couple of months. I can't tell you how many times we visited the hospital and somebody who was never visited was convinced I or my sister was their son or daughter, it was heart-breaking. We were convinced that people who were never visited lived shorter, more unhappy lives, and got substandard care. So during the day when I was at work my sister visited my Mom for about 6 hours, then I would take over when I got off work and would stay with Mom 6-8 hours until visiting hours were over. On my days off, I would spend much more time than that with Mom.
Because of this, I turned down the bishop's request to be GD teacher. As a substitute GD teacher, I would spend between about 10-12 hours a week preparing for my class. I was not someone who just went by the manual and that was it. I used the manual as a guideline and really put a lot of time and effort into my preparation for the class. So i told the bishop that because of Mom's situation, I didn't feel that I could give the class the attention that it needed. As I was walking out of the office, the bishop stopped me and thanked me for being honest. He said, he wished people would be more honest about callings and accept them only to then not fulfill them. I told him if things changed with Mom I would be more than willing to accept the calling and then left the office.
The doctors told us Mom would never leave the hospital, but the Lord had other plans. Seven months later, Mom went home. She has now been home for over a year, living with me. It's taken a lot of adjustment on my part, particularly with my work situation, but it has been so worth it! The doctors look at her with amazement as they see this now 87 year old woman, someone who was completely bed bound in the hospital, now scoot around with her walker. I wouldn't be surprised if she's walking without the walker in 5-6 months. My sister and I are convinced that part of the reason she's home is because of the time we dedicated to being with her when she was in the hospital. Two weeks after Mom got out of the hospital, the bishop again called me into his office and asked me to be the full-time GD teacher. This time I accepted and held the calling until 3 weeks ago when our ward boundaries were changed.
The moral to this story? Do what is best for you. Ask the Lord in prayer what to do. Don't be afraid to speak up and talk to the person asking you to fulfill the call if you have problems or reservations about the call. This doesn't mean to look for ways to get out of the call, but if you seriously believe that you can't fulfill the obligations of the call, say so and then tell the person you'd be willing to serve in another less demanding capacity.