Weighing yourself only tells you your total weight; to know your body composition, such as body fat and visceral fat, you need professional medical equipment like dual-energy X-ray absorptiometry (DXA) scans at a hospital, or bioelectrical impedance analysis (BIA) sensors on wearable devices. Google Research recently published a deep learning framework called PhotoScan that claims to estimate body composition metrics such as body fat percentage and visceral fat ratio from just a few 2D photos taken with a phone, with accuracy approaching DXA scans and even outperforming BIA sensors on smartwatches.
Google PhotoScan AI estimates body fat from phone photos with accuracy approaching hospital DXA scans.
The invisible details of BMI
Looking at overall body fat percentage alone is not enough; additional body composition metrics can provide deeper clinical information. The A/G ratio (Android-to-Gynoid fat ratio) compares the proportion of fat accumulated in the trunk (apple-shaped) versus the hips and thighs (pear-shaped); the V/S ratio (Visceral-to-Subcutaneous fat ratio) distinguishes between metabolically active visceral fat surrounding the internal organs and subcutaneous fat beneath the skin. Studies indicate that an elevated A/G ratio and higher visceral fat mass are strongly associated with the prevalence of insulin resistance.
The current gold standard for measuring body composition is the DXA scan, which offers extremely high accuracy. However, the equipment is expensive, requires specialized clinical infrastructure, and exposes subjects to low-dose radiation, making it unsuitable for routine screening. The BIA technology commonly used in wearable devices and smart scales estimates body composition by measuring differences in how quickly a weak electrical current travels through body tissues. Convenient as it is, it can only provide a basic body fat percentage and cannot show where fat is distributed.
How does PhotoScan work?
PhotoScan’s approach first pretrains a deep neural network on over 35,000 participant records from the UK Biobank, then fine-tunes it on a new cohort of 677 adults. During training, 2D photos taken by smartphones are aligned with DXA scan results, allowing the model to learn to extract body geometry information directly from images, bypassing the clinical measurement process.
The team also incorporated body composition data from MRI scans into training, and finally fine-tuned with images captured by smartphone cameras to make the model more aligned with real-world usage scenarios.
Accuracy: outperforms wearables, claimed to be close to DXA
In 5-fold cross-validation on the PhotoBIA group, the fine-tuned PhotoScan achieved a mean absolute error (MAE) of 2.15 for body fat percentage prediction, while the BIA model estimated the same data with an error of 2.91. The mean errors for the A/G ratio and V/S ratio were 0.107 and 0.094, respectively. In the independent MetabolicMosaic validation group, the body fat percentage error was 2.13, and the errors for both A/G and V/S ratios decreased to 0.085, demonstrating consistent model performance across different populations.
To put it in everyday terms, in Google’s testing, the photo-based method’s average estimate differed from DXA readings by about two percentage points, while under the same testing protocol, the smartwatch-style impedance sensor had an error of close to three percentage points.
The team further tested PhotoScan’s ability to predict insulin resistance, comparing five feature combinations using a gradient boosting classifier. The baseline model using only basic data such as age, sex, and BMI achieved an AUROC of 0.692. Adding PhotoScan’s body composition metrics raised the AUROC to 0.760 with an NRI (net reclassification index) of 0.593, approaching the performance of directly using clinical DXA data (AUROC 0.773, NRI 0.748). In the control group, adding BIA data yielded almost no improvement, because BIA only provides body fat percentage, whose feature importance is far lower than that of the A/G ratio and V/S ratio.
is part of the same puzzle as the Pixel Watch
The timing of this research release is quite interesting. Google just unveiled the Pixel Watch 5 last week, and the new watch’s Health Guardian toolkit includes an Insulin Resistance Trends feature that uses heart rate and sleep data to estimate the body’s response to blood sugar fluctuations. PhotoScan looks like the next piece of the same puzzle, giving Google a second camera-based pathway to estimate the same risk.
Products such as the Samsung Galaxy Watch Ultra 2 also rely on BIA technology to estimate body composition, and Google’s research directly challenges this technical approach. The research team concluded that clinical DXA imaging is the most accurate but lacks scalability, wearable BIA sensors are convenient but limited to basic body fat percentage, and PhotoScan offers a middle-ground option that estimates detailed body composition using standard phone images, with accuracy approaching that of DXA.
Google has not yet revealed when or whether PhotoScan will become an actual product, and the research remains at the prototype stage. First, convincing users to hand over potentially unflattering photos for analysis, no matter how sound the privacy policy, is a significant ask. Second, even when the data is laid out in front of users, driving them to make real changes to their diet and lifestyle is an even harder challenge.
However, the potential value of this type of research for public health should not be underestimated. According to statistics from the National Institutes of Health (NIH), nearly three-quarters (73.1%) of American adults are overweight or obese, yet comprehensive body composition assessment has long relied on expensive X-ray analysis. If a smartphone camera could provide body composition data approaching hospital-grade accuracy, it would offer a viable vehicle for large-scale insulin resistance risk screening.
Source: KOCPC Chinese