Why would you use S3 on top of NFS?

I mean you can, it would simplfy the locking somewhat.

But if you are doing file sharing for apps inside a network you manage, just use NFS, and maybe worry about the locking later.