I remember having a problem with this topic while teaching someone on my mission. He had read an article from a Liahona or somewhere that had a statement about how family shouldn't be overshadowed by Church callings, and then he pointed out a verse about only serving God and having nothing else before Him.
Like LM said, the ideal would be to worship God together with your family. But you should try to have a good balance in life. Don't spend all day at work or working on some Church calling and neglect your family, and don't do the opposite and neglect your job.
Elder Oaks said this in General Conference (Oct. 07) "The First Presidency has called on parents 'to devote their best efforts to the teaching and rearing of their children in gospel principles. … The home is the basis of a righteous life, and no other instrumentality can take its place … in … this God-given responsibility.' The First Presidency has declared that 'however worthy and appropriate other demands or activities may be, they must not be permitted to displace the divinely-appointed duties that only parents and families can adequately perform.'
(“Good, Better, Best,” Ensign, Nov 2007, 104–8)
In a way, you can be serving God and honoring Him by doing your best with the responsibilities He has given you. But anytime family can lead you away from the Gospel, you should stand firm and be the anchor and the example for faithfulness.
