Tin: full-text search for Postgres

I think what we're seeing with every database company providing new full-text search capabilities is an example of AI coding productivity showing up in the real world.

It started with paradeDB and pg_search https://www.paradedb.com/blog/introducing-search

Timescale has pg_textsearch https://github.com/timescale/pg_textsearch

Neon and Databricks have Lakebase Search https://docs.databricks.com/aws/en/oltp/projects/lakebase-se...

Now PlanetScale.

AFAIK all of these are implementations of the BM25 algorithm. You can just tell an agent to read about BM25 and implement it in your system of choice. Cool to see. Seems like there's still a lot of juice to be squeezed out of how it's architected and integrated into each system, but you can't help but wonder if this will lead to aggressive commodification

If anyone's curious - https://planetscale.com/docs/postgres/search/get-started#loc...:

They're not providing a local extension with the same performance at the time - it's only offered on their cloud services.

The local version https://github.com/planetscale/lead is mainly just for testing the syntax, it doesn't have the same perf characteristics.

Please read the Postgres manual. It has incredible built-in search capability.

Postgres does have pg_fts (tsvector/tsquery/tsrank) which is a quite sophisticated full text search package integrated with functional indexing and query optimization. Why would I use something vibecoded that isn't part of core Postgres instead?

Interestingly enough SQLites FTS supports Lucene queries out of the box with great performance characteristics. IIRC only writes become pretty slow after a while. I’ve always wondered what exactly would prevent PostgreSQL from strapping that implementation into its own database. My experience with ts_query hasn’t been particularly rosy. It can be better than LIKE but only marginally so and at the cost of insane index sizes… If this extension becomes open source and we can test it out in the real world I’m sure there’s a sweet spot

I struggle with FTS inside SQL (SQLite and MSSQL). There is often a fairly significant impedance mismatch between the relational concerns and how the documents need to be stored.

I've always preferred to use SQL as the system of record and then build/maintain an external Lucene index. Do we think these integral FTS capabilities are at the point where a hybrid architecture doesn't make sense anymore? How much customization exists in this provider?

Please note possible name collision with PostGIS Triangulated Irregular Network (TIN) data type.

I want to try Planetscale... but we're addicted to (and totally dependent on) Neon's branching model. They really got us hooked on that!

This was already posted and ignored at https://news.ycombinator.com/item?id=49751888 so i’ll ask the same question: again:

I don’t see any github link, is this 21st century embrace, extend, extinguish ?

I suppose it all depends on the scale of your project, but I've had pretty good luck using both MySQL's and SQLite's FTS capabilities. Surprised to hear that Open Source champion Postgres didn't have up-to-snuff FTS up to now...?

It’s just me or there are others who keep seeing these updates and think mongodb had all of this years ago?

Seriously so happy to be running our production stack on mongo.

it does what it says on the TIN

What’s funny is I worked with a company with planet in the name Who could really use a full text search that was great in the Postgres

What are the advantages of Tin over using ts_vector with gin and gist indexes?