logoalt Hacker News

jiggawattstoday at 12:43 AM2 repliesview on HN

I've been studying machine learning during the xmas break, and as an exercise I started tinkering around with the raw Bayer data from my Nikon camera, throwing it at various architectures to see what I can squeeze out of the sensor.

Something that surprised me is that very little of the computation photography magic that has been developed for mobile phones has been applied to larger DSLRs. Perhaps it's because it's not as desperately needed, or because prior to the current AI madness nobody had sufficient GPU power lying around for such a purpose.

For example, it's a relatively straightforward exercise to feed in "dark" and "flat" frames as extra per-pixel embeddings, which lets the model learn about the specifics of each individual sensor and its associated amplifier. In principle, this could allow not only better denoising, but also stretch the dynamic range a tiny bit by leveraging the less sensitive photosites in highlights and the more senstive ones in the dark areas.

Similarly, few if any photo editing products do simultaneous debayering and denoising, most do the latter as a step in normal RGB space.

Not to mention multi-frame stacking that compensates for camera motion, etc...

The whole area is "untapped" for full-frame cameras, someone just needs to throw a few server grade GPUs at the problem for a while!


Replies

pbalautoday at 10:21 AM

> Something that surprised me is that very little of the computation photography magic that has been developed for mobile phones has been applied to larger DSLRs. Perhaps it's because it's not as desperately needed, or because prior to the current AI madness nobody had sufficient GPU power lying around for such a purpose.

Sony Alpha 6000 had face detection in 2014.

show 1 reply
AlotOfReadingtoday at 2:39 AM

This stuff exists and it's fairly well-studied. It's surprisingly hard to find without coming across it in literature though, the universe of image processing is huge. Joint demosaicing, for example, is a decades-old technique [0] fairly common in astrophotography. Commercial photographers simply never cared or asked for it, and so the tools intended for them didn't bother either. You'd find more of it in things like scientific ISP and robotics.

[0] https://doi.org/10.1145/2980179.2982399

show 1 reply