logoalt Hacker News

sensodineyesterday at 5:59 PM0 repliesview on HN

> it leaves me completely baffled about what s2 does, what problem s2 is trying to solve, or who the intended audience of s2 is

Regarding S2 generally (not just s2-lite), the intent behind it is to turn the core data structure from streaming platforms (like Kafka) into a serverless primitive -- kinda similar to what object storage did for file storage.

So if you are already in the world of working with streaming platforms, S2 gives you a simpler API, bottomless storage (S2 itself uses object storage for durability), and no limits on the quantity of streams you can create and work with. Streams also all have URIs and are directly accessible over REST with granular access controls.

This enables new types of patterns for working with streams, other than just the traditional ones where people typically reach for streaming platforms (like CDC, ETL pipelines, etc). An agent can have its own stream to serialize state onto, for instance; you can use a stream as a durable transport layer -- e.g., you want to reliably provide a flow of data (tokens from a model, financial ticker data, etc) to a user and allow them to resume from exactly where they left off if they are disconnected, for instance; you could use streams as a durable ingest buffer, for collecting data that will eventually reside in an OLAP like Clickhouse.