I know that Alpine has a package manager but I haven't tried it because I worry that Musl might present problems compiling software (I sometimes like to compile things myself, like Vim or Emacs). Is this worry valid?
Also are there any issues compiling Go and Rust programs on Alpine?
If you're only building common packages I wouldn't expect trouble. For example, I don't see any patches in the official vim package: https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/c...