Yeah I have seen this issue a few times. If you use Docker build arguments that way add `--provenance=false` to get rid of all that build metadata. Build secrets are still better since they allow you to scope the secret inside of the Dockerfile. Also, the metadata can be useful to inspect images.
Yeah honestly I wasn't too familiar with this beforehand but now have a sense of the best practices going forward