alt Hacker NewsOccam's razor is perfectly empirical: "entities must not be multiplied beyond necessity". It's what people repeatedly accuse string theory of violating in low-rent popsci criticism.
The other things you refer to are still Occam's razor: symmetry is handy because it eliminates symmetry-breaking entities even though we know they can happen in the standard model (Higgs) and "beauty" is really just another way of saying Occam's razor - you'd prefer your theory to not be full of dozens of free parameters because it starts to fit any possible outputs and be less predictive.
At all points the issue is that unless you've fully explored a simpler space with less entities, don't start adding them because you can always keep adding them to solve any problem but predict nothing (ala epicycles keeping geocentric solar models alive. You could probably run a space program assuming the Earth is the center of the universe, but it would be fiendishly difficult to model).
You seem to be intuiting some kind of chi squared minimization. It is true that fewer free parameters constrain models, but there is nothing in nature that prefers simplicity. That is probably the most annoying thing to us physicists. Even thermodynamics is always shoving us toward disorder. Just look at plasma physics some time for deterministically intractable problems stemming from four little equations (one if you like tensors).
I think it's better to think of most real world models as being low dimensional-ish, where there is a decaying power law of eigenvalues, and most are quite small, though not zero. You can get quite far by looking at the largest modes and ignoring small ones, but you're not exact, so you're not seeing The Truth, or whatever. Forcing your self to use fewer parameters is a way of denoising, however, that is quite effective.