sarvam-translate on three Windows laptops: 17.5 to 73.6 tok/s
The public sarvam-translate model ran fully on the NVIDIA GPU in all three Windows laptops I tested, including one with a GTX 970M from 2014. With llama.cpp, generation ranged from about 17.5 tok/s on the GTX 970M to 73.6 tok/s on an RTX 4060 Laptop GPU.
sarvam-translate is fine-tuned from Gemma-3-4B-IT. I used a Q5_K_M quantization that occupies 2.63 GiB, or about 2.83 GB. I tested it with llama.cpp using CUDA 12 and with Ollama; the harness also provided an optional LM Studio path.
The three laptops
All three machines ran Windows 11. The model was fully GPU-offloaded on each one, with all 35 layers on CUDA.
| GPU | CPU | VRAM | GPU generation |
|---|---|---|---|
| GTX 970M | i7-6700HQ | 6 GB | 2014 Maxwell |
| RTX 4050 Laptop | i7-13620H | 6 GB | 2023 Ada |
| RTX 4060 Laptop | i7-13650HX | 8 GB | 2023 Ada |
I do not own these laptops. I put together a self-contained PowerShell script, and friends ran it on their machines and sent the logs back. The script auto-detected the hardware, validated the pipeline with a tiny model first, then ran sarvam-translate through llama.cpp and Ollama while logging every stage. It installed nothing system-wide.
Issues in the test harness
The problems I encountered were in my harness, not in sarvam-translate or the inference runtimes.
- Path resolution failed in some launch modes.
$PSScriptRootwas empty, so the model path collapsed and both engines failed to find the file. - The llama.cpp log format had changed. One build used a different throughput line, so my old parser reported 0 tok/s even though generation was working.
- A 32-bit PowerShell host misreported the environment. It exposed the CPU as
x86and redirectedSystem32toSysWOW64, which hidnvidia-smiand could have silently forced a CPU-only run. - Ollama import appeared to hang because stdout was not being drained. The server filled its roughly 64 KB pipe buffer and blocked. Redirecting the output to a file fixed it.
- A successful Ollama import was reported as a failure.
Start-Processreturned a blank exit code, so I changed the check to verify whether the model had actually registered.
An independent code review found a few more harness problems, including a summary path that could print PASS with empty result tables. I fixed those before using the final logs.
Results
For all three laptops, the throughput measurements are medians of five prompts at temperature 0, with the warmup discarded.
| GPU | llama.cpp generation | Ollama generation | Prefill | Run scope |
|---|---|---|---|---|
| GTX 970M | ~17.5 tok/s | ~16.5 tok/s steady-state | ~58 tok/s | Five-prompt test |
| RTX 4050 Laptop | ~48.5 tok/s | ~48 tok/s | ~300 tok/s | Five-prompt test |
| RTX 4060 Laptop | ~73.6 tok/s | ~71.5 tok/s | ~485 tok/s | Five-prompt test |
On the GTX 970M, the per-prompt llama.cpp medians were 17.4–17.9 tok/s and prefill was 55–61 tok/s. The model also ran with an 8K context and about 2 GiB of VRAM headroom.
The five-item smoke test covered Hindi, Tamil, Bengali, Marathi, and English. I measured throughput; I did not formally score translation quality. One Tamil response was subtly wrong: it changed an explicit instruction to “transfer the money” into a passive statement. A plausible translation is not necessarily a verified one.
Comparison with nominal memory bandwidth
The generation results rose with nominal, spec-sheet memory bandwidth.
| GPU | Nominal memory bandwidth | llama.cpp generation | Approx. weight-streaming utilization* |
|---|---|---|---|
| GTX 970M | ~120 GB/s | ~17.5 tok/s | ~41% |
| RTX 4050 Laptop | ~190 GB/s | ~48.5 tok/s | ~72% |
| RTX 4060 Laptop | ~256 GB/s | ~73.6 tok/s | ~81% |
* Weight-streaming utilization is tok/s ÷ (bandwidth ÷ 2.83 GB). This is an approximate weight-streaming ceiling based on nominal bandwidth, not a measured hardware roofline.
Generation throughput increased with nominal memory bandwidth, while the two Ada GPUs also reached a higher fraction of the approximate weight-streaming ceiling than the Maxwell GPU.
From the GTX 970M to the RTX 4060, throughput increased by about 4.2× while nominal bandwidth increased by about 2.13×. Bandwidth alone does not explain the full difference: the two 2023 Ada GPUs also reached a higher fraction of the approximate ceiling than the older Maxwell GPU.
This is an observed correlation across three laptops and three different llama.cpp builds, not a controlled study. The practical result is that on-device decode speed depended much more on memory bandwidth than on the GPU’s age alone.
Scope: this test used the public GPL-3.0 sarvam-translate model as a representative 4B translator on real NVIDIA client hardware. It is not a benchmark of any proprietary production stack.
Thanks to Adesh and Aditya for running the tests on their laptops and sending me the logs.