Recently, Tesla’s Vice President of AI team, Ashok Elluswamy, delivered an online speech at the International Conference on Computer Vision (ICCV), focusing on Tesla’s latest progress and core philosophy in the field of autonomous driving. He also revealed how Tesla promotes the intelligence of real-world autonomous driving through end-to-end neural network architecture, with data scale and neural simulation at its core.
End-to-End Learning: Letting AI Handle Everything from Perception to Decision-Making in Cars
Tesla’s Full Self-Driving (FSD) system is built on an end-to-end neural network that directly processes image inputs from the vehicle’s multiple cameras, combined with vehicle speed, navigation maps, audio signals, and dynamic sensor data, to generate vehicle control commands such as steering and acceleration. This stands in stark contrast to the “modular autonomous driving architecture” used by most competitors. The latter typically relies on a large number of sensors and splits the system into three components: Perception, Prediction, and Planning. While this approach is easier to manage in the early stages of development, it lacks holistic optimization capability when faced with the complexities of real-world scenarios, making it difficult to achieve human-like intuitive judgment.
Tesla argues that the key advantages of the end-to-end architecture include:
-
Learning human values from data.Human drivers integrate ethics, risk, and intuition when making decisions. These values are difficult to define explicitly in code, but they can be learned and imitated through big data.
-
Integrated learning across module boundariesIn an end-to-end architecture, the error gradient can be fed back all the way from the control output to the perceptual input, enabling the entire system to achieve global optimization during training.
-
Scalable with consistent computational latency.The same architecture can be synchronized and optimized across millions of vehicles, ensuring predictable and stable system latency.
-
Adapting to “The Bitter Lesson”History shows that as computing power and data scale expand, general learning methods will ultimately surpass hand-designed rule systems.
He also gave some examples:
Case 1: The Choice Between a Puddle and the Oncoming Lane
In a video, the AI must decide whether to go around a large puddle ahead, even if it means temporarily veering into the oncoming lane. Traditional rules would prohibit such behavior, but in that scenario, the road is empty and visibility is clear. The AI correctly determines there are no oncoming vehicles and therefore chooses to safely go around.
FSD V12 avoids standing water
(I know hand held phone videos are a no no, and I’m sorry, but I had to share) pic.twitter.com/3vXmJBw0c1
— ΛI DRIVR (@AIDRIVR) February 23, 2024
Case 2: Understanding Animal Intentions
The other two videos demonstrate how AI distinguishes between two scenarios: a group of chickens crossing the road versus a group of geese merely lingering by the roadside. Traditional modular systems require defining complex behavioral classification and planning logic, whereas end-to-end models naturally capture this kind of “intention” through latent representations. As a result, without explicit rules, the AI learns to “understand” the difference in animal behavior.
Why did the chicken cross the road? To see if FSD v12.3 would stop for it.
I didn’t intervene at all.@AIDRIVR @DirtyTesLa @WholeMarsBlog @SawyerMerritt @chazman @Tesla_AI pic.twitter.com/hhgPlmP8gP
— Kyle Westwood (@SnowmanSmasher) March 24, 2024
AJ @OGMarsLife and I made a video today about Actually Smart Summon, and we had some amazing results. For those that need the TLDR, here is a highlight clip, but I recommend heading over to AJ’s post and watching the full video. https://t.co/zlEbJtSIsp pic.twitter.com/OfBAaiJeHT
— Chuck Cook (@chazman) September 28, 2024
Overcoming the challenges of end-to-end learning
The Curse of Dimensionality and the Data Deluge
Real-world driving environments are extremely complex. When a vehicle is on the road, Tesla’s autopilot system must simultaneously handle:
-
7 camera lenses, 36 fps, 5 megapixels per frame, 30-second history recording
-
High-frequency vehicle dynamic data (100Hz)
-
Navigation map and route information
-
High-sampling-rate audio data (48kHz)
These data are equivalent to approximately 2 billion input “tokens,” which the AI must ultimately compress into two outputs: steering angle and acceleration. Learning true causal relationships from this data while avoiding spurious correlations is an extremely challenging task. Fortunately, Tesla has the world’s largest database of real vehicle data:The daily data volume is roughly equivalent to 500 years of driving experience for an average person.Through an efficient data engine, Tesla can select the most diverse and valuable segments from massive amounts of data to train AI, enhancing its ability to generalize to rare situations.
If training with such corner case data, you obtain solutions for various extreme scenarios. Here is an example of how an AI model learns to proactively avoid potential collisions. Impressively, the AI reacts in about 5 seconds, when it is still far from obvious that the situation could escalate into a crash. The AI needs to understand that it is drizzling outside, the leading car might be skidding, it could hit an obstacle, and then bounce back into the ego vehicle’s path, so braking now is prudent. Only a highly capable AI system can predict these possible situations in advance.
Interpretability and safety verification
Although end-to-end models are powerful, their “black box” nature also raises concerns about interpretability and safety. Tesla therefore developed an architecture that outputs intermediate semantic markers, allowing the AI to generate meaningful “intermediate reasoning tokens” during operation for monitoring and debugging.
One of the representative technologies isGenerative Gaussian SplattingTraditional 3D Gaussian reconstruction relies on wide-baseline camera viewpoints and is computationally intensive; however, Tesla’s generative approach achieves high-quality 3D reconstruction in approximately 220 milliseconds using only onboard cameras, requires no initialization, handles dynamic objects simultaneously, and is jointly trained with the main AI model.
This technology can reconstruct environments from new perspectives and even generate semantically rich natural language reasoning. Lightweight versions of such models are already running onFSD v14.xAmong the versions.
Evaluation and Simulation: Building the “Neural World”
No matter how much data you train on, it’s still not enough, because a model’s low loss on open-ended prediction does not necessarily mean superior real-world performance. To solve this problem, Tesla developedNeural World Simulator:A virtual environment that can test AI decisions in a closed-loop manner.
This simulator was trained by Tesla, but its mission isBased on the current state and actions, predict the future world state.It can generate high-resolution, multi-camera image sequences and respond in real time to commands from the driving policy model. (The video below is not actual footage, but generated by Tesla’s neural world simulation model.)
The simulator can not only reproduce real driving history, but also generateAdversarial Scenarios，用以測試AI在極端條件下的表現。
By adjusting computing resources, the simulator can generate complete driving footage from eight cameras at 24 frames per second in real time, allowing human drivers to “actually test” AI behavior in a virtual world (the footage below is generated in real time by the simulator and is genuinely drivable).
Cross-Domain Applications: From Self-Driving Cars to Humanoid Robots
Tesla emphasizes that these technological breakthroughs not only serve autonomous driving, but alsoThe foundation of Artificial General Intelligence (AGI). The same neural world simulation and visual generation models have now been extended to the humanoid robot Optimus.。
Inside Tesla’s Gigafactory, Optimus uses this neural simulation environment to learn navigation and operational tasks. The video shows that different robot actions are accurately reflected in the simulation, demonstrating a high degree of consistency between the end-to-end model and the simulator. This means that Tesla is gradually building aAn intelligent agent platform shared by vehicles and robots。
Furthermore, such simulation systems can be used for large-scale Reinforcement Learning, allowing AI to iterate infinitely in a closed loop and surpass human performance, moving toward “superhuman-level driving and action intelligence.”
If you’re interested in the content of Tesla’s AI Vice President Ashok Elluswamy’s talk at the International Conference on Computer Vision (ICCV) and want to dive deeper, you can watch the video I’ve translated below: