I am wondering where in Australia it is cold this time of year
I am sorry you had a negative experience because of this man. As you continue attending you will find it isn't uncommon that people trying to do what they think is the right thing end up causing offense. It has happened to me many times, and I know that I have been on the other side as well.
In my opinion wearing pants to church (without being immodest or rebellious, as is the case you described) isn't going to interfere with your relationship with God. The Church standards can be conservative when compared with Australian culture but the intention is reverence and respect, not control as emphasized in this article.
https://www.lds.org/new-era/2008/11/to-the-point/why-do-women-still-have-to-wear-dresses-or-skirts-in-all-our-meetings?lang=eng