Seeing beyond BMI: Estimating cardiometabolic risk with smartphone imagery

Beating Smartwatch based Body Composition Analysis (using Bio impedance snalysis) approach isn't a great feat as its single point crude sensor is widely inaccurate compared to production grade multi segment multi frequency inbody machines using same technique

Google PhotoScan (~2.1 pp MAE) may put it between DXA and well performed multi site Bodyfat Caliper by an skilled operator and WAY ABOVE US Navy circumference, consumer BIA ( smart scale with handlebars), poorly performed calipers.

But i remember microsoft research paper:

>In 2022, a Microsoft Research associated team published “Smartphone camera based assessment of adiposity”. Their Visual Body Composition (VBC) system used ordinary smartphone photographs plus a CNN to predict body fat percentage against DXA. In a 134 person, two site validation cohort, it achieved 2.16 ± 1.54 percentage point mean absolute error, with concordance to DXA of CCC = 0.96.

So from 2022 to 2026, we still don't have a model which can provide benefit to people.

This can help so many people using apps which currently simply wrap Gemini Flash to detect "body fat."

Unfortunately, none of these models, code, or training data are available freely.

These companies are seriously withholding public benefit. Currently, a guy who wants to get a DXA scan has to pay $50 in the US and 1,500-4,000 INR in India.

In poor countries this DXA may not be affordable to vast majority, if models are released we can perhaps setup even a "photo booth" with better cameras and widely available sensors

Companies spend billions of dollars on charity but not where it can produce benefit to the average person; so much technical talent, yet not a single public free model. This is actually a life changing opportunity if you can get it done!

Putting this out here in case anyone with metabolic disorders may find it useful. Labcorp has a recently added blood screening, the 'Metabolic Vulnerability Index' aka MVX. This looks outside the 'standard of care' cholesterol/glucose/bp triad which typically results in managed decline to identify underlying causes of said triad.

A metabolic specialist who understands how these work is essential to the recovery process.

I wonder if this could be used by insurance companies to determine premiums?

If this gets good enough, you could imagine your phone periodically tracking body composition and combining it with wearables, bloodwork, glucose, etc. Suddenly some measurements that currently require a clinic and specialized equipment become essentially free and continuous.

...at the same time, giving Google/Apple photos of your body is creepy

I expect Apple Health to integrate this kind of cool functionality in the future. Heck, I'm not sure why I can't have it pull my labs by connecting other applications.

I’m kinda surprised Apple doesn’t do something like this. I would imagine it’s a lot easier with phones with LiDAR

Been meaning to try this myself

Google already has a product called PhotoScan.

Come on guys!

waist-to-hip ratio already took us beyond BMI, a long time ago.