To make it even worse most pools are either (1) in direct sunlight without shade or (2) inside in a poorly vented, non-air conditioned building that echoes and feels like a giant sauna. I wouldn't normally want to spend time in either environment and the benefits of swimming don't offset these environments.
I did enjoy swimming with the grandkids yesterday evening and especially that feeling of buoyancy which makes me feel 60 pounds lighter. On the other hand, the 60 pounds felt like a ton of bricks when I started climbing out of the pool. Do I really need that sort of hassle?