logoalt Hacker News

gdcbe • yesterday at 9:19 AM • 0 replies • view on HN

Hi HN, I'm the author.

This project started about three years ago as a small piece of a proxy project (ramaproxy.org): on every request we pick a random upstream proxy that matches a handful of flags and a country or city. The data barely changes, the queries happen constantly. Scanning a Vec got slow, and SQLite felt like a lot of database for "give me a random row with these flags".

VennDB is a Rust library: you add a derive macro to a struct, and it generates a database type and a query builder for it. Bools become bitmasks, filter values become sets of rows, and a query is the intersection of those sets. It can pick a random match without finding all of them first.

It is mostly a bitmap index, generated from your struct so you don't have to write and maintain it yourself. It is also narrow on purpose: append-only and in-memory, with no updates or deletes. As such it is not really a database in the traditional sense of the word, but as I was not sure how to name it differently given it does fulfill a niche subset of usecases that I would otherwise use an (in-memory) database for it seemed fitting.

We just released 0.7, which rewrote the engine. In the README compares it with a hand-written roaring bitmap index, SQLite and a plain Vec. roaring gets close on some queries. We're not database experts, so if the benchmarks look unfair or there's prior art we missed, I'd like to hear it.