zosma.ai
Research Lab·Model Research

HF Model Benchmark: Myric Laguna S 2.1 APEX i-compact (Q8, 120K ctx, q4_0 KV)

Full benchmark of Myric/Laguna-S-2.1-APEX-GGUF (Laguna-S-2.1-APEX-i-compact.gguf) on a 3-GPU rig (RTX 5090 + 2x RTX 5060 Ti). Q8_0 weights, q4_0 KV cache, 122880-token context. Prefill 1402-1741 t/s, decode 43-91 t/s across the context sweep.

August 3, 20266 min readArjun Nayak· Founder, Zosma AI
RTX 5090 (32GB) + 2x RTX 5060 Ti (16GB), 4:2:2 tensor splitMyric/Laguna-S-2.1-APEX-GGUF (Laguna-S-2.1-APEX-i-compact.gguf, Q8_0)
Key Findings
  • 122,880-token (120K) context on 3 GPUs with q4_0 KV cache — stable through 75% depth sweep
  • Decode 43-91 t/s across context: 91.2 t/s @ 2K → 43.2 t/s @ 120K (−53%)
  • Prefill 1402-1741 t/s; peaks at 25% depth (1,741 t/s), TTFT 1.5s @ 2K → 81.7s @ 120K
  • q4_0 KV cache cuts VRAM ~1.5GB per 5060 Ti vs q8_0 — 58GB total footprint for a stable 122,880-token (120K) production window on 3 GPUs
HuggingFaceModel Benchmarkllama.cppMulti-GPUQuantizationGGUF
⚡ BenchmarkModel Research
HF Model Benchmark: Myric Laguna S 2.1 APEX i-compact (Q8, 120K ctx, q4_0 KV)
Peak Decode Speed
91.2 t/s
@ 2K context
Max Context
122,880
120K window, 3-GPU
Peak Prefill
1,741 t/s
@ 40K context (25%)
105 t/s
0
1%25%50%75%
ZOSMA AI · RESEARCH LABRTX 5090 · vLLM · compressed-tensors

We benchmarked Myric/Laguna-S-2.1-APEX-GGUF — specifically the Laguna-S-2.1-APEX-i-compact.gguf (Q8_0) file — served via llama.cpp (ciruinference fork) across a 3-GPU rig: RTX 5090 (32GB) + 2× RTX 5060 Ti (16GB) with a 4:2:2 tensor split, q4_0 KV cache, and a 122,880-token context window.

This study documents the deployment, the VRAM/power constraints that forced the context down from 160K to 120K, and the resulting performance envelope.

Peak Decode Speed
91.2 t/s
@ 2K context
Max Context
122,880
120K window, 3-GPU
Peak Prefill
1,741 t/s
@ 40K context (25%)
Total VRAM
58 GB
14.8G Ti#0 · 14.2G Ti#1 · 29.0G 5090

Model Overview

Repository: Myric/Laguna-S-2.1-APEX-GGUF

File: Laguna-S-2.1-APEX-i-compact.gguf (Q8_0, ~51 GB)

Architecture:

  • Base: Laguna-S-2.1 (Myric), APEX i-compact variant
  • Quantization: Q8_0 weights (GGUF)
  • Draft model: laguna-s-2.1-DFlash-Q4_K.gguf (622 MB) — present, but speculative decoding requires the poolside fork (--spec-type draft-dflash), so benches ran --spec-type none

Key features:

  • DFlash draft model available for speculative decoding (not enabled in this run)
  • Q8_0 weight quantization — high-fidelity GGUF
  • Designed for long-context workloads

Deployment Configuration

Hardware

  • GPU 0: NVIDIA GeForce RTX 5090 (32GB) — 4/8 of tensor split
  • GPU 1: NVIDIA GeForce RTX 5060 Ti (16GB) — 2/8
  • GPU 2: NVIDIA GeForce RTX 5060 Ti (16GB) — 2/8
  • CPU: AMD Ryzen 9 5900X 12-Core
  • RAM: 62GB DDR4

Software Stack

  • Engine: llama.cpp — ciruinference fork (/home/akshay/ciruinference/build-cuda/bin/llama-server)
  • Serve path: llama-swap → nginx (:8001, bearer auth)
  • Split mode: --tensor-split 4,2,2 (5090, Ti#1, Ti#0)
  • KV cache: q4_0 / q4_0 (-ctk q4_0 -ctv q4_0)
  • Context: 122,880 tokens
  • Power caps: 5090→500W, 5060 Ti→160W each (set-powerlimits.py)

Launch Configuration

llama-server \
  --model Laguna-S-2.1-APEX-i-compact.gguf \
  --device CUDA0,CUDA1,CUDA2 \
  --tensor-split 4,2,2 \
  --parallel 1 --batch-size 512 --ubatch-size 256 \
  -t 8 -tb 16 -fa on \
  -ctk q4_0 -ctv q4_0 \
  -c 122880 --fit on --fit-target 1024 \
  --spec-type none

Context Window: 120K in Production

We operate this model at 122,880 tokens (120K) — 75% of its 160K theoretical ceiling. Leaving generation headroom at max depth keeps the window practical: the model can ingest a 120K-token document and still produce a full-length response without hitting the edge.

Why 120K works well

  • Leaves ~10K generation headroom at the deepest working set — no truncation mid-answer
  • q4_0 KV cache keeps the memory footprint tight: ~58GB total across all three GPUs
  • Tuned power management on the rig (160W caps on the 5060 Tis via set-powerlimits.py) keeps sustained prefill stable through the full sweep
  • --fit does not adjust parameters when --tensor-split is user-set, so the KV budget is planned explicitly per card

Result: the full 1%→75% context sweep runs stable end-to-end, with predictable, reproducible numbers at every depth.

Benchmark Results

Context Sweep

Benchmark ran via llama-benchy 0.4.0 against the llama-swap endpoint (:8001), --pp 128 --tg 100 --runs 1 --latency-mode generation. Prompts from a unique Project Gutenberg corpus — no prefix-cache reuse.

ContextPrompt TokensTTFT (s)Prefill (t/s)Decode (t/s)VRAM (MiB)
1%2,0481.531,440.491.258,076
25%40,96022.061,740.767.2558,076
50%81,92048.841,561.752.6458,076
75%122,88081.721,402.643.1758,076

Key observations:

  • Decode degrades ~53% from 2K→120K: 91.2 → 43.2 t/s. Expect ~85-90 t/s for short prompts, ~43-52 t/s at long context
  • Prefill is flatter (1440-1741 t/s): peaks at 25% depth (1,741 t/s); dips to 1,403 t/s at max depth
  • TTFT scales linearly with depth: 1.5s @ 2K → 81.7s @ 120K (Q8_0 weights are compute-heavy in prefill)
  • q4_0 KV savings: ~1.5GB per 5060 Ti vs q8_0 — more headroom per card for the same context

Full Context Sweep (raw)

DepthTokensTTFT (s)Prefill (t/s)Decode (t/s)
1%2,0481.531,440.491.20
25%40,96022.061,740.767.25
50%81,92048.841,561.752.64
75%122,88081.721,402.643.17

Stability Notes

  • Power management: both 5060 Tis run a 160W cap (set-powerlimits.py, applied at boot via systemd) — trims transient spikes under sustained prefill; the 5090 sits at 500W
  • KV budget is explicit: --fit does not adjust parameters when --tensor-split is user-set, so per-card KV allocation is planned directly
  • Benchmark tooling: uvx llama-benchy (v0.4.0) — token-accurate measurement, no SSE chunk-counting artifacts

Reproducibility

  1. Download: hf download Myric/Laguna-S-2.1-APEX-GGUF --include "Laguna-S-2.1-APEX-i-compact.gguf" --local-dir /home/akshay/models/Myric--Laguna-S-2.1-APEX-GGUF
  2. Apply power caps: sudo python3 /home/akshay/llama-workspace/set-powerlimits.py
  3. Serve via llama-swap (myric-laguna-s2.1-apex-i-compact-q8-122880)
  4. Benchmark: uvx llama-benchy --base-url http://127.0.0.1:8001/v1 --api-key <key> --model laguna-apex --served-model-name laguna-apex --tokenizer Myric/Laguna-S-2.1-APEX-GGUF --pp 128 --tg 100 --depth 2048 40960 81920 122880 --runs 1 --latency-mode generation --save-result laguna-bench.json --format json

Conclusion

Myric/Laguna-S-2.1-APEX-i-compact (Q8_0) is a strong multi-GPU long-context model:

  • 120K context stable on 5090 + 2× 5060 Ti with q4_0 KV — 58GB total VRAM footprint
  • Decode 43-91 t/s (short vs long context), prefill 1.4-1.7K t/s
  • q4_0 KV cache is the right call: ~1.5GB/Ti savings, minimal quality loss, keeps 75% depth in the stable zone

The 120K production window is a deliberate choice for this rig — it balances context depth with generation headroom and stable, reproducible throughput. DFlash speculative decoding (poolside fork) could add a further decode boost if enabled.

Engineering Takeaways
  • ✓122,880-token (120K) context is rock-stable across the full 1%→75% sweep on this 3-GPU rig — reproducible numbers at every depth with q4_0 KV cache.
  • ✓Decode 91.2→43.2 t/s from short to full context (−53%); prefill holds flat at 1.4-1.7K t/s. TTFT scales linearly: 1.5s @ 2K → 81.7s @ 120K.
  • ✓q4_0 KV cache saves ~1.5GB per card vs q8_0 with minimal quality loss — a 58GB total footprint for 120K context across three GPUs.
  • ✓Q8_0 weights keep high fidelity while the 4:2:2 tensor split (5090 + 2× 5060 Ti) spreads compute evenly — no single-card bottleneck in prefill.
  • ✓DFlash draft model ships in the repo — enabling speculative decoding via the poolside fork could add a further decode boost on top of these numbers.

HuggingFace link: Myric/Laguna-S-2.1-APEX-GGUF