> if I run my "build" processes (npm, composer, etc) on the server at deploy time

It's perfectly fine, as long as you accept the risks and downsides. Your IP can get ratelimited for Docker Hub. The build process can exhaust resources on the host. Your server probably needs access to internal dev dependencies repository, thus, needs credentials it would not need otherwise. Many small things like that. The advantage is simplicity, and it's often worth the risk.