alt Hacker NewsI've worked with large MySQL databases that used row-based replication and things like an UPDATE or DELETE that affected millions of rows had to be applied in batches there, because otherwise one SQL query might result in a million updated rows needing to be sent to all of the replicas at once.
Replies
Groxx • today at 9:09 PM
I've built custom batch-processors because percona-toolkit's automatic stuff was far too aggressive :|
Every DB needs it, eventually. Even NoSQL darlings like Cassandra - I've seen it go into a resource-constrained death-spiral on stuff that should be async / non-blocking and safe. If you need to stay up, it's always worth planning on, and making sure your logic works during long-running gradual migrations.
Yeah, I think anyone that's done significant database work has come to the understanding that large updates need to be done in batches, otherwise you nuke performance.
Once you get to about 1M rows of data, batching is essential.