Tesla’s purely visual autonomous driving solution has always been a focus of debate in the industry – without LiDAR, without radar, and only using cameras, how can it drive safely in harsh environments such as night, dense fog, and strong light? Elon Musk posted a comparison chart on the X platform on May 9, 2026, showing the core secrets of Tesla FSD to the public for the first time:Photon count reconstruction。
Musk shares Tesla FSD “photon counting reconstruction” technology
The left side of this comparison picture is an RGB image that is familiar to the average human eye. Under strong backlight, the image is almost completely white, and all details are lost in the glare. The right side is an image reconstructed by Tesla AI from the original data of the same set of photosensitive elements. The outlines of houses, trees, and vehicles are clearly discernible. “This is why the Tesla FSD can still see clearly at night or in extreme glare,” Musk wrote.
The human-perceived RGB is image 1 and the Tesla AI photon count reconstruction is image 2.
This is why Tesla FSD can see so well at night or through extreme glare. pic.twitter.com/ttBMzgpJtd
— Elon Musk (@elonmusk) May 9, 2026
Skip the ISP: Directly from photons to control
To understand the key to this technology, you must first understand the image processing process of a general digital camera. When light passes through the lens and hits the photosensitive element (CMOS sensor), the photosensitive element records the number of photons received by each pixel – that is, the original Bayer array data. But before this data enters our eyes, it will go through a series of image signal processor (ISP) processing: demosaicing, white balance, color correction, gamma correction, compression into JPEG… and finally output into an 8-bit RGB image that “looks beautiful”.
The problem is that information is being lost at every step of the ISP’s process. An 8-bit RGB image has only 256 levels of brightness per color channel (16.78 million colors in total), while the original output of the Sony IMX490 sensor used by Tesla is 12-bit data, with 4,096 levels of brightness per pixel. The information gap is as high as 16 times.
Tesla’s approach is:Just skip the ISP entirely. The raw Bayer data from the camera’s sensor is directly sent to the Tesla self-driving computer’s neural network for analysis without any traditional image processing. When Musk was asked by analysts at the first quarter earnings conference in 2025, “Will strong light blind the camera?” he replied: “In fact, it does not. We use a method called direct photon count. Generally, after ISP processing, if the image is shot directly against the sun, the screen will turn completely white. But what we read is the raw photon count data.”
Patent Revealed: Bit-Augmented Convolution Architecture
In January 2026, the U.S. Patent and Trademark Office published a Tesla patent titled “Direct Raw Bayer Image Input to Compute Hardware” (direct raw Bayer image input to computing hardware), patent number US 20260019717 A1. The patent details how Tesla overcomes the computational challenges of high-bit raw data on existing hardware.
Traditional deep learning accelerators (NPU, Tensor Core) are usually optimized for 8-bit integer operations and cannot directly process 12-bit or 16-bit raw photosensitive data. Tesla’s solution is a technology called bit-augmented convolution: split the high-bit data into multiple low-bit planes (for example, split 16-bit into two 8-bit planes, MSB and LSB), perform convolution operations separately, and then merge the results. This architecture not only retains the complete dynamic range of the original data, but also dynamically switches the computing precision under different lighting conditions: using full precision in extreme environments and reducing power consumption in normal environments.
This also explains why Tesla’s FSD can minimize the end-to-end delay from “photon-to-control”: there is no need to wait for ISP processing, no need to go through JPEG compression and decompression, and every photon on the photosensitive element can be turned into steering wheel and accelerator instructions at the fastest speed.
Not suddenly: The layout starts in 2021
Tesla’s “photon counting” strategy is not a recent idea. As early as November 2021, Musk said in response to FSD’s visual training method on
Tesla has been moving in this direction ever since. The FSD V12 launched in 2024 introduced end-to-end neural networks on a large scale for the first time. The FSD V13 in 2025 further enhanced the processing capabilities of raw photosensitive data. At the first quarter financial report meeting in 2025, Musk officially revealed the technical name of “direct photon counting” to Wall Street analysts.
This announcement in May 2026 was the first time that Tesla allowed the outside world to directly see the power of this technology in a visual way. Musk chose to disclose this technology at a critical moment when FSD enters the Unsupervised (fully unsupervised) stage, which obviously has strategic intentions: on the one hand, it responds to the outside world’s doubts about the safety of purely visual solutions, and on the other hand, it also paves the way for the Robotaxi service expected to be launched later this year.
Why this is important for the self-driving industry
Currently, most self-driving companies (such as Waymo, Cruise, and Baidu Apollo) adopt a multi-sensor fusion solution of LiDAR + radar + camera. Although LiDAR performs stably at night and in low-light environments, it is expensive (thousands to tens of thousands of dollars per set) and is also affected by weather such as heavy rain and dense fog.
Thanks Elon. Some LiDAR diehards refused to believe this when I told them, still insisting this was where LiDAR was required. I captured this last year. My eyes saw even less than this. Yet the Autopilot had no trouble seeing the road and the other vehicles. Today, Elon explained… pic.twitter.com/TtzXhXkmBk
— David Lee YL (@David_Lee) May 9, 2026
The biggest advantage of Tesla’s vision-only approach is cost: a set of cameras costs far less than any LiDAR system, allowing Tesla to deploy the same self-driving hardware in millions of production vehicles. The breakthrough in photon counting technology means that the performance of pure vision solutions in harsh environments may have surpassed multi-sensor fusion solutions, because LiDAR and radar are ultimately unable to identify the text on traffic signs or the color of traffic lights, and Tesla’s AI can do far more than traditional computer vision after obtaining raw optical data.
Today’s AV stacks infer traffic signal and brake light state from camera pixels, then try to align them to lidar geometry. Layers between sensing and decision.
Humans see color and depth together because they arrive together. Rev8 does the same: 48-bit RGB + 3D depth, same… pic.twitter.com/JgEALPo3IS
— Ouster (@ousterlidar) May 8, 2026
What’s more, the technology’s applications aren’t limited to cars. The Optimus humanoid robot being developed by Tesla will also use the same visual architecture. In environments with complex lighting conditions such as factories, warehouses, and homes, an AI vision system that can directly read raw photon data will have huge environmental adaptation advantages.
HW5 and AI5 chips: Hardware upgrade route
Tesla’s latest Hardware 4 (AI4) already has a built-in neural network accelerator that supports original Bayer data processing. The next-generation AI5 chip (which has just been announced to have been tape-out) will further enhance its capabilities in this area. AI5’s original computing performance is claimed to be 8 times that of AI4, and its memory capacity is 9 times higher. It is designed to process higher-resolution, higher-bit-depth raw photosensitive data.
This means that with hardware upgrades, Tesla FSD’s visual capabilities still have huge room for growth, moving from the current 12-bit to 16-bit or even higher precision, evolving from raw data processing of a single camera to real-time fusion of multi-camera data, and evolving from simple object recognition to complete scene understanding.
Criticism and Doubt: Still to be verified
Of course, the technology is not without controversy. Physicist Sean Kirkpatrick (former director of AARO) has pointed out that in the absence of complete independent verification, any claims about visual systems need to be viewed with caution. Some people in the industry also believe that even if photon counting technology can handle extreme lighting, there is still an insurmountable ceiling for purely visual solutions in the face of physical obstructions such as heavy rain, heavy snow, and dust.
It is also worth noting that although the comparison chart released by Musk this time is shocking, it does not provide any standardized test data or third-party verification results. Until FSD Unsupervised goes on the road on a large scale, these claims will remain at the level of technical demonstrations.
in conclusion
Tesla’s “photon counting reconstruction” technology represents a fundamental shift in visual thinking for self-driving cars: Rather than figuring out how to make cameras see what humans can see, AI can directly read everything the cameras can record. While traditional ISPs throw away a lot of critical data to make pictures “look good,” Tesla chooses to keep all the information and let the neural network decide which data is useful.
From the first hint in 2021 to the full reveal in 2026, Tesla has been on this road for nearly five years. With the imminent arrival of AI5 chips and the launch of Robotaxi services, photon counting technology will move from technical demonstration to actual testing, which will also determine the ultimate fate of purely visual self-driving solutions.
Source: KOCPC Chinese