Ask to be released. If you physically are not up to it then you should be released and maybe put into a different calling that will still keep you active in church but not demand so much of your time. Just be honest with your bishop. As for facing your ward, I can understand that it would be hard for people to think horrible things because you asked to be released. But that is between you, the bishop, the relief society president and God. You need to decide what is best for you and your family.