Many people want to buy a graphics card to run local LLM, but new graphics cards are very expensive now. Not everyone has the budget, so they may instead look for second-hand entry-level mid-range graphics cards such as RTX 3060 and RX 7800 XT. In fact, except for consumer-grade graphics cards, server cards in the past may not have higher CP values.
Recently, a foreign YouTuber tested an 8-year-old server graphics card “NVIDIA Tesla V100”. The speed and efficiency of running models such as gpt-oss-20b and Google Gemma 4 surpassed the RTX 3060 12GB and RX 7800 XT. This graphics card cost $10,000 back then, but now the SXM2 version on eBay abroad only costs $100.
The second-hand price of NVIDIA V100 from 8 years ago is only US$100. After modification, it runs local LLM 42% faster than RTX 3060.
Foreign YouTube channel Hardware Haven recently uploaded a video titled “This $200 AI GPU shouldn’t be so easy to use.” The content shared that they actually measured the LLM performance of an NVIDIA Tesla V100 16GB, and the results were amazing.
But what you need to know first is that NVIDIA Tesla V100 16GB has two versions: PCIe and SXM2 interface. Their one has an SXM2 interface, so if you want to install it on a consumer motherboard, you need an “SXM2 to PCIe x16 adapter”. It does not come with a fan, and you have to purchase an 80mm Noctua fan to match their own 3D The fan shroud is printed, so the total cost is actually $234.64. The V100 also has no image output at all and is just a pure computing card. Therefore, this card cannot be used as a general graphics card and can only be used to run AI inference or HPC calculations.
Below is the actual measured data. Hardware Haven compared the V100 system with its daily use of RX 7800 XT and RTX 3060 12GB, and ran two open source models on Ollama.
For the gpt-oss-20b part, the V100 ran out of 130 tokens/s, while the RX 7800 XT only ran about 90 tokens/s, with a gap of about 44%:
In the Google Gemma 4 e4b (Ollama + OpenWebUI) part, the V100’s score is 108 tokens/s, and the RTX 3060 12GB is 76 tokens/s, with a gap of about 42%:
However, the power consumption of the V100 is relatively high, reaching 293W, while the RTX 3060 12GB is 235W:
In order to make the power consumption fair, Hardware Haven set the total system power consumption at 170W. The V100 ran out of 95 tokens/s, and the RTX 3060 was 68 tokens/s, which is still nearly 40% ahead:
The power consumption is reduced to 100W to compare the energy efficiency (how many tokens can be generated per 1W of electricity). The V100 is 0.55 tokens/s/w and the RTX 3060 is 0.39 tokens/s/w. Therefore, the V100 is obviously more efficient, about 41% ahead:
I really didn’t expect that a 2017 server card could significantly surpass the 2021 RTX 3060 and the 2023 RX 7800 XT in a specific LLM inference test.
V100 SXM2 16GB uses HBM2 high-bandwidth memory, with a memory bandwidth of up to 900 GB/s; RTX 3060 12GB is GDDR6, 192-bit channel, with a bandwidth of only 360 GB/s. LLM infers that this workload is very sensitive to memory bandwidth.
In addition, V100 is NVIDIA’s early data center GPU that introduced Tensor Core. It was originally designed for deep learning and HPC workloads, so it is not surprising to win in this type of LLM inference test.
Full video:
Source: KOCPC Chinese