First off, we should all understand that everyone has different opinions on how the Garment affects their life. For some, it's no big deal but for others, it's quite unpleasant. That's why it often comes up so often in these forums. Unfortunately, the Church rarely addresses these concerns directly so it's mainly up to the members/community to reconcile these kinds of questions.
But pertaining to the original question, sometimes I find the language of the TR question to be a bit strange "Do you wear the garment both night and day as instructed in the endowment and in accordance with the covenant you made in the temple?" Maybe it's semantics but that question makes it sound like we're told to wear them night and day during the endowment but we're not. We're just told to wear them throughout our lives. And sure we make lots of covenants at the temple, but in the TR question, "covenant" is singular which implies that we made a specific one regarding the Garment in the temple so I can see why people struggle to remember what or when that one is/was.
And then on top of that, it feels like we get most of our instruction outside of the temple. We get told how to dispose of them, specific counsel regarding yard work, the 4 S's, etc. So yes, sometimes the whole reasoning behind the Garment and its host of rules can be an obstacle. And for some it's a difficult obstacle because it's a daily/hourly/by-the-minute commitment. For me, I take it to be a challenge to test my willingness to be obedient. I do hope that someday the challenge will be over, whether the Church changes its policy or my mind is eased with understanding.