Technical Evaluation Report
ThaiLLM-30B: NVFP4 Quantization on NVIDIA DGX Spark
A controlled BF16-versus-NVFP4 comparison — accuracy, language fidelity, inference performance, and cross-model context
1. Executive Summary
The NVFP4-quantized ThaiLLM-30B is recommended for deployment in Thai-language workloads on DGX Spark. Quantization reduces the model from 61.1 GB to 18.1 GB (3.4×), increases decode throughput 2.3–2.5× (27 → 63 tokens/s single-stream), and reduces time-to-first-token by 2.0–2.7× — while Thai-language capability remains statistically indistinguishable from the BF16 original (ThaiExam 0.619 → 0.614, p = 0.79; no significant change across 3,890 paired Thai test items). A small, statistically confirmed cost of approximately 0.8 accuracy points — pooled over 19,786 paired questions (ThaiExam counted once, via the letter-based template) — falls almost entirely on English reasoning benchmarks. Expert qualitative review found no systematic degradation in Thai generation; the single caution concerns verbatim quotation (e.g., legal statutes), for which the BF16 model or retrieval grounding is advised.
2. Accuracy Evaluation
Both checkpoints answered identical questions under identical serving conditions (seed 0, deterministic log-likelihood scoring, BF16 KV cache on both sides, prefix caching disabled). Figure 1 shows each benchmark as a paired comparison; whiskers indicate ±1 standard error. Significance is determined by exact McNemar tests on the paired outcomes. No Thai benchmark shows a statistically significant decline. Small but confirmed declines appear on MMLU and HellaSwag (English).
Figure 1 · Benchmark accuracy, BF16 → NVFP4
Zero-shot unless noted · ThaiExam scored by answer letter (model-card protocol)
Table 1 — full values, standard errors, paired flips, and p-values
Figure 2 · Accuracy change by category (NVFP4 − BF16), paired McNemar tests
Across 19,786 shared questions (ThaiExam counted once): BF16 uniquely correct on 727, NVFP4 uniquely correct on 567
3. Language Modeling and Token-Level Fidelity
Bits-per-byte measures raw-text compression and is tokenizer-independent. The absolute information loss is nearly language-neutral (+0.014 Thai vs +0.016 English bits/byte). Teacher-forced agreement — whether both models predict the same next token given identical context — is the most sensitive fidelity probe available; Thai agreement exceeds English, consistent with the accuracy findings.
Figure 3 · Perplexity — bits per byte (lower is better)
Thai: 1,000 docs · EN: full 62-doc test set · 8,192-token windows
Figure 4 · Teacher-forced top-1 agreement
40 held-out passages · 46,272 scored positions
4. Inference Performance
Measured with vllm bench serve (random dataset, fixed output lengths, seed 0, median of three runs after a discarded warm-up). The container selected native FlashInfer-CUTLASS NVFP4 kernels on SM121; no speculative decoding was used. Decoding on this platform is memory-bandwidth-bound, so the 2.3× throughput gain follows directly from the 3.4× weight compression.
Figure 5 · Output throughput — tokens/s (higher is better)
Figure 6 · Time to first token — ms, median (lower is better)
Figure 7 · Resource footprint
5. Cross-Model Context — Seven Models on One Machine
The same Thai tasks were run on five additional locally available models under the same protocol (each model uses its own tokenizer; the five references are instruction-tuned and scored without chat templates, which is mildly conservative for them). Two observations stand out: quantization differences (blue → orange) are far smaller than any between-model difference, and ThaiLLM remains the strongest raw-Thai language model on the machine, while the newer-generation Qwen3.6-27B leads exam and comprehension tasks.
Figure 8a · ThaiExam
Figure 8b · Belebele-TH
Figure 8c · Thai bits/byte (lower better)
instruction-tuned models pay a raw-text penalty
6. Qualitative Assessment
Twelve domain prompts (Thai news, law, medicine, education, business, travel, mathematics, factual QA, translation; English economics and science; Python) were generated greedily by both models and reviewed by a Thai-fluent evaluator. Result: 8 equivalent · 3 favoring BF16 · 1 favoring NVFP4. Thai orthography is fully intact in both; repetition artifacts occur equally in both (a property of greedy decoding on a base model, not of quantization); every factual claim tested was correct in both versions.
The one meaningful regression class is verbatim precision: the NVFP4 model paraphrased Civil & Commercial Code Section 420 (“บุคคลภายนอก” in place of “บุคคลอื่น”) where BF16 quoted the statute exactly. For legal-citation and other precision-quoting applications, retain BF16 or add retrieval grounding.
7. Method Summary and Limitations
Quantization used hf_ptq.py --qformat nvfp4 --kv_cache_qformat none with a calibration set of 256 Thai Wikipedia and 256 English news documents — the half-Thai calibration is the most likely reason Thai capability survived fully, in line with published findings on calibration-language effects. Attention projections and all 128 experts per layer are quantized (W4A4); router gates and the output head remain BF16; the KV cache was deliberately left unquantized on both sides so that only weight/activation quantization is measured.
Limitations. (1) No large-scale human evaluation — the literature indicates automatic metrics can understate generative quality loss; our 12-domain expert review mitigates but does not eliminate this risk. (2) ThaiLLM-30B is a base model; conclusions should be re-verified after instruction tuning by re-running this suite (≈ 25 minutes to re-quantize, scripts provided). (3) MMLU was sampled at 50 items per subject; a single seed was used throughout. (4) Reference-model figures are capability snapshots, not controlled comparisons.
8. Conclusions and Recommendations
- Deploy NVFP4 for Thai serving on DGX Spark. Speed and memory gains are large; Thai quality loss is not statistically detectable. In production, adding
--kv-cache-dtype fp8approximately doubles KV-cache capacity. - Retain BF16 for verbatim-quotation workloads (legal, regulatory), or pair NVFP4 with retrieval so citations come from documents rather than model weights.
- Fine-tune first, then re-quantize. The production sequence is instruction-tuning on the BF16 base, NVFP4 quantization of the tuned checkpoint with half-Thai calibration, then this evaluation suite as a release gate.
- Further speed is available post-tuning via speculative decoding (EAGLE/draft-model, ≈1.5–2× reported in comparable setups); SM121 kernels also continue to improve with each container release.
Planned next phases. (1) Draft-model speculative decoding — the Qwen3-30B-A3B base ships no MTP head (unlike the Qwen3.6-generation models behind community 97–120 tok/s figures), but a same-tokenizer drafter such as Qwen3-0.6B is expected to add ~1.3–1.6× decode speed losslessly (63 → ~80–100 tok/s). (2) Thai instruction SFT on the BF16 base, then re-quantization with this exact recipe and this suite as the release gate. (3) EAGLE-3 head training post-SFT (~1.5–2× additional). (4) FP8 KV cache in production serving — disabled here only to isolate weight quantization.