Full-Text Search with DuckDB

DuckDB is great, and DuckDB-WASM is magic.

I build a whole LLM benchmark system around it that lets you run the whole benchmark in your browser: https://sql-benchmark.nicklothian.com/#sample-queries-and-sq...

Click on a cell and you can run the SQL the LLM generated vs what the solution is: https://sql-benchmark.nicklothian.com/?highlight=ggml-org_ge...

The hardest part is getting people to understand that it is interactive! People expect a document-looking webpage to be static, but we can do so much better!

It seems that DuckDB by default downloads and runs extensions at runtime when you use certain features? This seems unnecessarily risky.

https://duckdb.org/docs/current/extensions/overview#autoload...

I would love to have more detail on this mechanism.

I'm relatively new to DuckDB (coming from SQLite) and I love it so far. Some parts are magical (described in the previous article by the same author: https://peterdohertys.website/blog-posts/dab-of-duck.html)

You can point DuckDB to almost any data source and boom, you get an SQL table that you can search, sum, or join to any other data. Or you can attach existing databases from completely independent db systems, and query and join them as one, without having to first importing anything.

It feels exhilarating (if you're into that sort of thing!)

Has anyone used DuckDB (or anything else) to create an open source way to publish a mailbox so that a regular person can browse it and search it?

I'm aware of jmail.world, but they haven't (yet?) published the source code.

I had Claude hack something together recently: https://healdsburg-youcubed-emails.vercel.app/

It works fine for this small set of emails, although the search isn't great, and there was more preprocessing that I would have liked. (I would prefer to be able to point a single binary at a pst or mbox file, and have it magically serve it like this, even if it means I need a VPS to serve it.)

Anyone tried both duckdb and clickhouse local?

Clickhouse seems less marketed, but seems quite similar.