Sqldef: Idempotent schema management tool for MySQL, PostgreSQL, SQLite

If someone is looking for a more comprehensive coverage for Postgres, please check out https://github.com/pgschema/pgschema.

I built this last summer and thought it was fairly complete. And it turns out I was naive. 6 months later, I have resolved ~100 issues reported by the users.

I use https://github.com/ariga/atlas for this, migration based and schema base both has there good and bad side, I prefer both even in one project, schema based can make dev faster, eaiser, migration based make feel reliable.

How does this compare to Entity Framework (for dotnet) migrations or sqitch/liquibase.

I get the declarative nature of this, but migration of schemas is not purely declarative, particular on large production databases. An ideal schema manager would understand the costs of particular migrations (perhaps by using the table stats and EXPLAIN) and apply that back to the migration strategies so that downtime is minimized/eliminated.

Adding or remove columns or indexes can trigger major database table scans and other problems, especially when partitioning conditions change.

We've rolled our own variant of this, which uses an XML file as input. Just easier to parse than SQL schema definitions. The tool compares the schema defined by the XML with the DB, and applies changes as needed.

We used Sybase SQLAnywhere, and a complication there was that if you had materialized views against a table, you had to drop and recreate that view when adding or removing columns in the table. And when you recreate it, you of course have to remember to recreate the indexes on that materialized view.

Tracking this in case you have multiple materialized views touching multiple tables became a bit tricky, and you don't want to do the "dumb" thing to just always drop and recreate, or per-column, since some of them might take an hour or so to recreate and reindex.

The tool has some built-in safeties like never dropping columns if it's missing (have to add explicit element for that in the XML), and only performs safe column definition changes, ie integer to varchar(50) is safe, integer to varchar(3) is not, etc.

It really made database migrations very painless for us, great since we've have hundreds of on-premise installations. Just modify the XML, and let the tool do its job.

And since its our tool, we can add functionality when we need to.

This is very cool!

Also means I can stop with my hobby project that was supposed to do the same. Wasn't far along and haven't worked on it in months anyway.

So I'll spend my time on another hobby project then that also solves something that is already solved 100 times over, but I don't like the other solutions (simple log monitoring including systemd and sending error emails if something is found).

This post from 2022 is the one I keep going back to

https://david.rothlis.net/declarative-schema-migration-for-s...

is there anything (open source) similar to microsoft database project but that would work for postgresql

i like the following about it 1. database schema is regular code 2. make schema change declaratively 3. packaging (.daspac) and deployement script

most open source tools , seem to be after the fact tools, that do diffs ms db project, handle the code from the start in a declarative, source code managed way

I'm always in the market for new sql tooling, but I'm wondering what's the use case there?

Isn't it much quicker to write a one line migration vs copying the DDL, then adapting it to the desired state and then getting getting the migrations from that tool? Or am I misunderstanding something?

Looks like this is only useful for empty databases. Which severely limits possible use cases.

Schema management is only a small part of the problem, and I don’t think this tool handles data migrations. E.g. if I reshape a JSONB column into something more structured, I don’t think it would be able to handle that. Or if I drop a column the backwards migration it generates ADD COLUMN … NOT NULL, which is obviously unusable if the table has any data in it already.

Wow. I saved the link for emergencies. And I sent it to all my team. It is like a diff tool but to advance the schema.

I renamed a column and it added a new one.

How is this better than Liquibase? Isn’t there a risk of missing schema evolution steps when you only diff current vs target? (Because current can be v1 and target v3 and you might have lost column migration logic from v2)

MS' Sql Server Data Tools is such an abominable garbage fire that I have no interest in these kind of tools. Besides being a buggy mess, it's very often insufficient - you end up having to maintain migrations anyways because often you have to inject data into new tables or columns, or rename columns, etc.

Anyone with real-world usage of this one vs atlas. I've used atlas but it seems to be moving away from FOSS (paywalled-fearures). Its also struggled with some migrations that were a bit more than not trivial (like trying to play changes in the wrong order)

Looks great! Could this work with duckdb?

Looks pretty cool. Lately I've been using ChatGPT to generate SQLite schema migrations and it works shockingly well. I give it the original schema and new schema and it generates the statements including caveats and gotchas to watch out for.