I share this worry.

You can configure the storage template for the photos and include an "album" part, so if a photo is in some album it'll get sorted into that folder. Then the file tree on disk is as you wish.

I haven't tested what it does when a photo is in multiple albums, but it does handle the no album case fine as well.