Are the benchmarks comparing just the models while keeping the harness the same (the open-source Toast harness)?
yes for the retrieval benchmarks. For officeqa pro v2 we used Codex (as databricks did) and for Harvey LAB we used the vanilla harvey benchmark. For these benchmarks we added minimal tools to use mixedbread search and toast 1.
alt Hacker News
yes for the retrieval benchmarks. For officeqa pro v2 we used Codex (as databricks did) and for Harvey LAB we used the vanilla harvey benchmark. For these benchmarks we added minimal tools to use mixedbread search and toast 1.