Cool thought! We have an internal version of map reduce implemented in waymark that worked pretty well (though uses Redis for its intermediate values so it's technically not an end-to-end integrated solution).

In principle I would love to have something like this working for arbitrarily large ETL flows; ie. write the logic you want and it tries to figure out a good distributed algorithm to get the outputs. At the moment flowing data through the cluster is the biggest bottleneck. We generally assume that action inputs & outputs themselves are pretty small at the boundary even if their containerized context is quite large.

Where did you hit the most snags with caching?