I believe epoxy-filling them a-la IronKey is a good idea, but for longevity reasons, I'd prefer the rechargeable battery to be replaceable.
Maybe it increases the price quite a lot, IDK.
Access to battery terminals enables power side-channel attacks such as DPA and CPA.
I had an old Ledger die recently and replaced it with a new one. Didn't even consider being potentially able to replace the battery, but it would have been nice.
On the plus side, I did get to do a "fire drill" of my recovery process, which did work, so that was nice.