if the input is larger than ~1.5GB.
...on a system where ints are 32 bits. The bottom line is that this doesn't appear to be easily exploitable and IMHO it should be the callers of EVP_EncodeBlock which need to be inspected to determine the actual risk.
using a nontrivial formula
It's base64. 4/3 expansion; round it up and add padding. What's "nontrivial" about that?
and the caller does not pass in the output buffer length, so the implementation cannot validate it
Not necessary, because the caller is responsible for allocation.
The question can be reduce to "who do the length validation".
In my case, if i write a public-facing API, i never trust the caller will do the correct thing, thus the caller must provide all the critical parameters, and i must validate all of them.