But if the pimpl size grows too big, you're forced to break ABI?

And before it grows too big, it wastes memory. For your use cases it may not matter, and the saved pointer indirection may be more important, but maybe the person who has a million item vector of objects doesn't appreciate a 300% "just in case" memory overhead. The overhead may also hurt cache hits.

If you're doing this to save the pointer indirection, you should benchmark it for every use case, since negative cache effects may dwarf that gain.

Then again, extra padding can also help performance, for some workloads (especially multi threaded read/write against a vector of objects).

So without further context, there's no way to say if your way hurts or helps. It's certainly not a general solution.