logoalt Hacker News

Do you even need a database?

138 pointsby upmostlytoday at 12:26 PM205 commentsview on HN

Comments

pstuarttoday at 5:23 PM

In order to ask this question it's important to understand the lifecycle of the data in question. If it is constantly being updated and requires "liveness" (updates are reflected in queries immediately), the simple answer is: yes, you need a database.

But if you have data that is static or effectively static (data that is updated occasionally or batched), then serving via custom file handling can have its place.

If the records are fixed width and sorted on the key value, then it becomes trivial to do a binary search on the mmapped file. It's about as lightweight as could be asked for.

show 1 reply
rasengantoday at 5:18 PM

Sounds like a good way to waste the only scarce resource: time.

cratermoontoday at 4:51 PM

I worked one place that shoehorned SQL Server into a system to hold a small amount of static data that could easily have been a config file or even (eek) hard-coded.

ForHackernewstoday at 2:10 PM

Surprised to see this beating SQLite after previously reading https://sqlite.org/fasterthanfs.html

show 1 reply
amw-zerotoday at 4:49 PM

I think so, yea.

MattRogishtoday at 2:33 PM

"Do not cite the deep magic to me witch, I was there when it was written"

If you want to do this for fun or for learning? Absolutely! I did my CS Masters thesis on SQL JOINS and tried building my own new JOIN indexing system (tl;dr: mine wasn't better). Learning is fun! Just don't recommend people build production systems like this.

Is this article trolling? It feels like trolling. I struggle to take an article seriously that conflates databases with database management systems.

A JSON file is a database. A CSV is a database. XML (shudder) is a database. PostgreSQL data files, I guess, are a database (and indexes and transaction logs).

They never actually posit a scenario in which rolling your own DBMS makes sense (the only pro is "hand rolled binary search is faster than SQLite"), and their "When you might need" a DBMS misses all the scenarios, the addition of which would cause the conclusion to round to "just start with SQLite".

It should basically be "if you have an entirely read-only system on a single server/container/whatever" then use JSON files. I won't even argue with that.

Nobody - and I mean nobody - is running a production system processing hundreds of thousands of requests per second off of a single JSON file. I mean, if req/sec is the only consideration, at that point just cache everything to flat HTML files! Node and Typescript and code at all is unnecessary complexity.

PostgreSQL (MySQL, et al) is a DBMS (DataBase Management System). It might sound pedantic but the "MS" part is the thing you're building in code:

concurrency, access controls, backups, transactions: recovery, rollback, committing, etc., ability to do aggregations, joins, indexing, arbitrary queries, etc. etc.

These are not just "nice to have" in the vast, vast majority of projects.

"The cases where you'll outgrow flat files:"

Please add "you just want to get shit done and never have to build your own database management system". Which should be just about everybody.

If your app is meaningfully successful - and I mean more than just like a vibe-coded prototype - it will break. It will break in both spectacular ways that wake you up at 2AM and it will break in subtle ways that you won't know about until you realize something terrible has happened and you lost your data.

Didn't we just have this discussion like yesterday (https://ultrathink.art/blog/sqlite-in-production-lessons)?

It feels like we're throwing away 50 years of collective knowledge, skills, and experience because it "is faster" (and in the same breath note that nobody is gonna hit these req/sec.)

I know, it's really, really hard to type `yarn add sqlite3` and then `SELECT * FROM foo WHERE bar='baz'`. You're right, it's so much easier writing your own binary search and indexing logic and reordering files and query language.

Not to mention now you need a AGENTS.md that says "We use our own home-grown database nonsense if you want to query the JSON file in a different way just generate more code." - NOT using standard components that LLMs know backwards-and-forwards? Gonna have a bad time. Enjoy burning your token budget on useless, counter-productive code.

This is madness.

SyndicateLinkstoday at 4:36 PM

[dead]

fatih-erikli-cgtoday at 1:45 PM

I agree. Databases are useless. You don't even need to load it into the memory. Reading it from the disk when there is a need to read something must be ok. I don't believe the case that there are billions of records so the database must be something optimized for handling it. That amount of records most likely is something like access logs etc, I think they should not be stored at all, for such case.

Even it's postgres, it is still a file on disk. If there is need something like like partitioning the data, it is much more easier to write the code that partitions the data.

If there is a need to adding something with textinputs, checkboxes etc, database with their admin tools may be a good thing. If the data is something that imported exported etc, database may be a good thing too. But still I don't believe such cases, in my ten something years of software development career, something like that never happened.

show 3 replies
linuxhansltoday at 4:50 PM

Hmm... Sure, if you do not need a database then do not use a database.

Don't use a sports-car to haul furniture or a garbage truck as an ambulance. For the use case and scale mentioned in the article it's obvious not to use a database.

Am I missing something? I guess many people are the using the tools they are familiar with and rarely question whether they are really applicable. Is that the message?

I think a more interesting question is whether you will need a single source of truth. If you don't you can scale on many small data sets without a database.

I will say this before I shut up with my rant: If you start with a design that scales you will have an easier to scale when it is time without re-engineering your stack. Whether you think you will need to scale depends on your projected growth and the nature of your problem (do you need a single source of truth, etc.)

Edits: Spelling