Speaking as someone who had the same kind of reservations, I think you sound exactly like me, and if I had any advice to my past self, it would be "get on it as early as it's available, start much lower than anyone thinks is sane, and never ever stop".
I got a concierge doctor who absolutely evaluates what I ask for skeptically, and pay for my GLP medications out of pocket, so I feel ethically totally comfortable with this position. The downsides are real and significant, it's not a "fun" medication, but just the change in how I feel is worth every penny, let alone the massive reduction in heart attack risk, liver enzyme numbers, and so on.
What downsides have you experienced?