Portfolio / Vol. 012026

I make inference cheaper. And I show the receipts.

I build and measure the systems layer of ML: quantization, KV memory, batch scheduling, and speculative decoding. Ten case files from a twenty-repo public inventory. Real CPU receipts. GPU figures are labelled models, never borrowed numbers.

Selected systems GitHub

Every number on this page ran on a CPU. No GPU needed to check any of it.

5.69× tokens per target call sdecode

59.5% fewer KV blocks pagemem

94% lower TTFT cplan

3.98× weight compression bithash

SELECTED RECEIPTS / READ FROM COMMITTED JSONEVERY NUMBER LINKS TO ITS SYSTEM

01 / WORK

Selected systems

Ten case files from the public inventory. Every claim links back to the repo and the receipt that proves it.

01Speculative decoding2026

sdecode

5.69× tokens per target call.

Six words per slow step, instead of one.

A measured draft/verify harness for real models: a smaller drafter proposes, the target verifies a batch, and every lane is checked token-for-token against the naive target. The receipt keeps the wins and the losses.

TARGET / DRAFTER: gpt2 / distilgpt2
DRAFTER COST c: 0.63
BEST LANE: fixed K=8
TOKENS / CALL: 5.69×
IDENTITY: 100% token match
ARTIFACTS: app · film · ledger

Negative result included: adaptive K averaged 4.16× on this pair; both lanes stay in the receipt.

Open repo Launch app receipt

sdecode session app showing a recorded run, lane selector, and generated text — APP / target-call budgetmeasured on CPU

TOKENS / CALL5.69×

TARGET CALLS10.7 avg

MATCH100%

02KV memory2026

pagemem

PAGED KV / BLOCK MAP204 / 504

59.5% fewer blocks

The same card holds about 2.5 × more conversations.

The allocator shares a 512-token system prefix and pages the rest. The 504-block preallocated baseline becomes 204 blocks, with the block map committed next to the numbers.

Open repo receipt Launch app

03Serving / scheduling2026

cplan

CONTINUOUS VS STATIC94% lower

94% lower TTFT

The wait for the first word falls by 94%.

A continuous scheduler lets every slot refill the moment it frees. Static batching waits for the longest job. The receipt records TTFT and p99 at four request rates.

Open repo receipt Launch app

ALSO SHIPPING / 07

Seven more, still measured.

All repositories

04 softfp8 Quantization 2.0× weight bytes / 4.87% rel. error 05 latq Quantization 15.8× compression / 0.607 rel. error 06 fusednorm Kernels 33% less norm-site traffic 07 kvstamp KV cache 8.0× KV compression / 1.14 error 08 roofline-planner Planning 34 deployment plans / 5 models × 9 GPUs 09 gembench Kernels 1.8× GEMV throughput / 64.7 ms 10 bithash Compression 3.98× compression / 4.02 bpw

02 / INVENTORY

Twenty repos. Five layers.

The ten case files above are the deep cuts. This is the complete public inventory, grouped by the inference layer each repo works on. One receipt-backed metric per card.

20public repos

05themes

20receipt links

04live apps

New here? Start with the KV desk — one interactive tour of attention memory, built from three receipts.

Decode loop

Propose, verify, constrain, stream. Four ways to shorten the token loop without changing the output.

01 Speculative decoding

sdecode

5.69× tokens per target call

Six words per slow step, instead of one.

Measured draft/verify on real GPT-2 weights, token-identical to the naive lane.

repo receipt app

02 Speculative decoding

treetrace

5.28× packed node pass

A kernel 5.3 × faster that only moves the finish line 1.06 ×.

Random 2-layer CPU model: nearby drafts accept 3 of 4 tokens. The packed node pass is 5.28× faster, but end-to-end is only 1.06×; independent drafts regress.

repo receipt

03 Structured decoding

pushdown

25/103 legal tokens

One gate token in 103 passes; the JSON value state keeps only 25.

Stack-aware JSON mask keeps 25 of 103 tokens at the value state. A 1,024-deep synthetic stack parses in 51 ms; not a production DFA.

repo receipt

04 Token streaming

streamdetok

60.1× faster / 0 replacements

Text streams without ever splitting a character in half.

CPU byte-level test: 128-byte chunks over a 2,880-byte payload, 60.1× faster than the custom path with zero replacements, versus 18 naive splits. No tokenizer context.

repo receipt

Serving control

Batch, chunk, admit, route. Control the queue before it controls p99.

05 Serving / scheduling

cplan

94% lower TTFT

The wait for the first word falls by 94%.

Continuous batching against a static scheduler, with p99 in the same receipt.

repo receipt app

06 Serving / scheduling

chunkpipe

71.7% lower p95 TTFT / 1 burst/s

Long inputs are cut into chunks so nobody waits behind one giant job.

Synthetic CPU service model, Poisson bursts, six requests each: 128→8192-token chunks cut p95 TTFT from 21.7s to 6.1s at 1 burst/s; the 5/s gain is 46.7%.

repo receipt

07 Admission control

gatekeeper

71.8% lower p95 response

A bouncer that keeps the queue calm by turning away the flood.

Synthetic CPU open-loop burst, 300 requests: token bucket and GCRA hold p95 at 4.07s versus 14.46s for fixed window; admitted count 90 vs 88.

repo receipt

08 Prefix routing

radixlite

92.3% prefill saved

Similar questions route to the same worker and reuse the prefill.

CPU token-slot simulation: cache-aware routing saves 92.3% of prefill tokens at a 128-slot budget; round-robin saves 73.4%. Routing costs 58.1ms vs 23.9ms.

repo receipt

Memory & attention

KV pages, token budgets, expert residency, paged attention. The memory layer, measured.

09 KV memory

pagemem

59.5% fewer blocks

The same card holds about 2.5 × more conversations.

Prefix-shared paged KV; a 504-block baseline collapses to 204 blocks.

repo receipt app

10 KV cache

kvstamp

8.0× KV compression / 1.14 error

An eighth of the attention memory, if you keep the right tokens.

Token importance at a fixed budget, swept until the error knee is visible.

repo receipt

11 Expert cache

expertcache

14.2% fewer misses / hot scan

Cache the experts you actually use — while the pattern holds.

CPU expert-cache policy pack: cost2q cuts misses 14.2% and median CPU 10.7% on the hot scan; phase-shift regresses, and both cases stay in the receipt.

repo receipt

12 Attention kernel

pagedattn

9/9 parity cases / 3.0e-7 max err

Same answer, different memory layout — 9/9 parity cases.

Synthetic q/k/v on CPU torch: streaming paged attention vs dense gather, 9/9 parity cases, max abs error 3.0e-7. Parity, not a speed win; no CUDA claim.

repo receipt

Numerics & compression

Weights, lattices, codebooks, bit-level softmax. Fewer bits, visible error.

13 Quantization

softfp8

2.0× weight bytes / 4.87% rel. error

Half the memory, and the answers come out the same.

W8 weight path, measured output error and a labelled bandwidth model.

repo receipt

14 Quantization

latq

15.8× compression / 0.607 rel. error

Four weights in the space of one, near the theoretical floor.

D8/D4/Z2 lattice ladder from 4 to 1 bit per weight; the 1-bit point sits within 1% of the Gaussian R-D floor.

repo receipt

15 Compression

bithash

3.98× compression / 4.02 bpw

3.98 × smaller weights from a 4-bit codebook.

A four-bit codebook study: uniform16 beats odd16 on dense weight rows.

repo receipt

16 Numerics

bitforge

100% decidable argmax / 0.00239 max dev

A 4-bit softmax still wins 100% of the decidable argmax calls.

4-bit softmax path with 100% argmax agreement on decidable rows and 0.00239 max probability deviation; near ties are reported separately.

repo receipt

Kernels & topology

Norm traffic, GEMV, placement, collectives. The shapes around the model.

17 Kernels

fusednorm

33% less norm-site traffic

Three trips to memory become one — 1 MiB saved on every word.

RMSNorm, residual, and SwiGLU epilogue fused into one pass; 1 MiB saved per LLaMA-3 8B decode token.

repo receipt

18 Kernels

gembench

1.8× GEMV throughput / 64.7 ms

The slow way, the easy way, and the fast way, on a plain CPU.

A bandwidth-honest GEMV ladder: 119.2 ms naive rows vs 64.7 ms torch.mv, measured on CPU.

repo receipt

19 Planning

roofline-planner

34 deployment plans / 5 models × 9 GPUs

Will it fit, and how fast — 34 answers against one memory wall.

Fit-check, roofline regime, and decode throughput for every plan.

repo receipt app

20 Collectives

ring

2.01 GB/s busbw / 4 ranks

A ring of four ranks reaches 2.01 GB/s, then falls at eight.

Five rank counts on loopback TCP: N=4 busbw 2.01 GB/s; N=8 falls to 0.99 GB/s. Formula-normalized, not NCCL or GPU scaling.

repo receipt

— / THE LOSSES

What didn't work.

Six experiments that regressed, published on purpose. A receipt you can trust is one that also shows the misses.

01treetrace

Tried: Pack a draft tree into one fast node pass.
Expected: the fast kernel carries the whole step.
Got: End to end it is only 1.06 × ; independent drafts accept 0 tokens.

receipt

02expertcache

Tried: Rank expert cache entries by cost instead of recency.
Expected: fewer misses than plain LRU.
Got: On the phase-shift trace it regresses: 78 misses against 62 for LRU.

receipt

03roofline-planner

Tried: Buy throughput by batching decode to 64 sequences.
Expected: total tokens per second should climb.
Got: Total throughput falls in 34 of 34 plans; worst case 0.33 × the single-stream rate.

receipt

04sdecode

Tried: Let the drafter choose its own run length (adaptive K).
Expected: beat a fixed guess.
Got: Adaptive averaged 4.16 × against 5.69 × for fixed K=8 on this pair.

receipt

05ring

Tried: Add ranks to raise all-reduce bandwidth.
Expected: bandwidth should keep scaling.
Got: Bus bandwidth climbs to 2.01 GB/s at four ranks, then falls to 0.99 at eight.

receipt

06pushdown

Tried: Track a deep JSON stack with a pushdown automaton.
Expected: a production-ready structured-decoding mask.
Got: It keeps only 25 of 103 tokens at the value state — a teaching model, not a production DFA.

receipt

03 / RECEIPTS

The receipt is the résumé.

Not a screenshot of a benchmark. The committed JSON that generated every number on this page.

If it isn't in receipts/, it isn't on this page.

The page is a build artifact. It reads the receipts, then writes the cards, the proof strip, and the evidence plates. Change a number in a repo and the site changes with it; there is no second copy to drift.

A real measurement firstCPU timings, Triton interpreter parity, real HF weights. GPU figures are labeled as models.

A harness you can runCLI, test, or live app — then a receipt generated by executing it. sdecode's session app is on this page.

Tests that defend the claimAssertions live next to the number. CI fails when a receipt drifts.

Browse the source

sdecode / receipts/real_run.json open raw

Loading receipts…

source: receipts/real_run.json

04 / METHOD

Method, not magic.

The same contract in every repo, so the numbers can be compared instead of admired.

“A benchmark is an argument. A receipt is evidence.”

Measure on the machine you have.

CPU timings, interpreter parity, real model weights. No invented H100 numbers; GPU projections are labeled models with the assumptions attached.

Make the claim runnable.

A CLI, a test, or a live app: a path to the number that a stranger can execute. If a claim cannot be reproduced from the repo, it does not count.

Let CI defend the claim.

Tests assert the metric. Receipts are committed. The site reads them; it never retypes them. Drift fails the build.

CLAIM TEST RECEIPT PAGE one source of truth

05 / ABOUT

Systems engineer.
Allergic to hand-waving.

I'm Abhinav Chauhan. I work on the unglamorous layer between a model and its bill.

Caches, kernels, schedulers, and decoding loops. I like small, complete projects with sharp claims and honest measurements — the kind of work that survives a p99, a tight GPU budget, and a reviewer who asks where the number came from.

Twenty public repos; ten deep case files and a 20-card index on this page. Each case file is built around one inference problem and one committed receipt. The index keeps the rest of the inventory inspectable by layer, not by chronology.

QuantizationKV memorySchedulingKernelsSpeculative decoding

Open to MLSys & inference engineering roles.

GitHub All repositories

I make inference cheaper. And I show the receipts.

Selected systems

sdecode

pagemem

cplan

Seven more, still measured.

Twenty repos. Five layers.

Decode loop

Serving control

Memory & attention

Numerics & compression

Kernels & topology

What didn't work.

The receipt is the résumé.

Method, not magic.

Measure on the machine you have.

Make the claim runnable.

Let CI defend the claim.

Systems engineer.Allergic to hand-waving.

Have a system that needs to get cheaper?

Systems engineer.
Allergic to hand-waving.