Agreed. One at a time testing (OAT) has been outdated for almost a century at this point. Factorial and fractional factorial experiments have been around for that long and give detailed insights into the effect of not just single changes but the interaction between changes, which means you can superpower your learnings as many variables in DL do in fact interact.

Or, more modern Bayesian methods if you're more interested in getting the best results for a given hyperparameter sweep.

However, that is not to detract from the excellent effort made here and the great science being investigated. Write ups like this offer so much gold to the community.