Connecting directly with the author of the project that inspired me is awesome.

Let's collaborate and see how we can make our two projects work together. DuckDB has a feature that can write to SQLite: https://duckdb.org/docs/stable/core_extensions/sqlite. Starting from Parquet files, we could use DuckDB to write into SQLite databases. This could reduce ingress time to around five minutes instead of a week.

If I have some free time this weekend, I would definitely like to contribute to your project. Would you be interested?

As for my background, I focus on data engineering and data architecture. I help clients build very large-scale data pipelines, ranging from near real-time systems (under 10 ms) to large batch processing systems (handling up to 1 billion business transactions per day across thousands of partners). Some of these systems use mathematical models I developed, particularly in graph theory.

Happy to chat.