It's not the heat, it's the humidity.
The gardens are beautiful, but we went to HK in September and it was as hot as Hades. We were dying in the heat at the gardens, there are SOOOOO many stairs EVERYWHERE! Great atmosphere, wrong time to go. Like walking up 7,000 stairs in a steam bath.