That's actually exactly where I started. The initial idea involved a YOLO nano model to classify images, deciding what to invert and what not to. It worked as a concept, but during the feasibility analysis I realized that for native PDFs it wasn't necessary: the format already tells you where the images are. I walk the page's operator list via getOperatorList() (PDF.js public API, no fork) and reconstruct the CTM stack, that is the save, restore and transform operations, until I hit a paintImageXObject. The current transformation matrix gives me the exact bounds. I copy those pixels from a clean render onto an overlay canvas with no filters, and the images stay intact. It's just arithmetic on transformation matrices, on a typical page it takes a few milliseconds.

Your approach with a classifier makes a lot more sense for the generic web, where you're dealing with arbitrary <img> tags with no structural metadata, and there you have no choice but to look at what's inside. PDFs are a more favorable problem.

A case where a classifier like yours would be an interesting complement is purely vector diagrams, drawn with PDF path operators, not raster images. Veil inverts those along with the text because from the format's perspective they're indistinguishable. In practice they're rare enough that the per-page toggle handles them, but it's the honest limitation of the approach.