Working on mrjob was a big part of my first job out of college. Fun to see it get mentioned more than ten years later.

What some commenters don't realize about these bureaucratic IO-heavy expensive tools is that sometimes they are used in order to apply a familiar way of thinking, which has Business Benefits. Sometimes you don't know if your task will take seconds, minutes, hours, days, or weeks on one fast machine with a well-thought-out program, but you really need it to take at most hours, and writing well-thought-out-programs takes time you could spend on other stuff. If you know you can scale the program in advance, it's lower risk to just write it as a Hadoop job and be done with it. Also, it helps to have an "easy" pattern for processing Data That Feels Big Even If It Isn't That Big, Although Yelp's Data Actually Was Big. Such was the case with mrjob stuff at Yelp in 2012. They got a lot of mileage out of it!

The other funny thing about mrjob is that it's a layer on Hadoop Streaming, which is a term for when the Java process actually running the Hadoop worker opens a subprocess to your Python script which accepts input on stdin and writes output on stdout, rather than working on values in memory. A high I/O price to pay for the convenience of writing Python!