I started building https://github.com/kcc999/sidechain

It's a CLI Framework for running evaluations against LLMs.