We shared with the author how databricks multi-node and single-node graphframes were wildly inefficient for this kind of thing: we were measuring doing billion-edge graph traversals & scans in single node in-memory in seconds with regular dataframe (cudf) libraries, so the core of pagerank, which is magnitudes more efficient than their original spark approach

So then the question became pandas/polars/datafusion/duckdb/etc, must of which are rust/native. I'm curious myself why datafusion vs others, it's an interesting project :)