I've talked to some people who use Riak as an S3 and reported that their reasoning for sticking to it is 2 to 4 orders of magnitude faster than S3. This blew my mind, but then they showed me 20ms response times from their Riak cluster and 2-second response times from S3. Now, I think these aren't standard, and in my experience, I'd say Riak is more like a single order of magnitude faster.
I would imagine that most of that is probably due to latency. Which might already be worth the extra effort, depending on the usecase.