Introduction

Meridian is an inference-time compute scheduler for reasoning-model serving. It treats think-decode and output-decode as separate scheduling domains with separate SLOs, separate KV eviction priorities, and real-time entropy-driven budget control.

Why this exists

Reasoning models (DeepSeek-R1, Qwen3, Qwen2.5, o3) emit two structurally different token sequences within a single request:

[prompt] → <think> ... N reasoning tokens ... </think> → [output tokens]

These two phases have opposite latency profiles:

Standard continuous-batching schedulers (including vLLM's default) process all decode tokens — thinking and output — from the same priority queue with the same TPOT target. Meridian is the scheduling layer that knows the difference.

What Meridian does

1. Dual-queue scheduling. Output-phase requests have absolute priority. Think-phase requests fill remaining capacity with a larger effective batch token budget.
2. Phase-aware KV block manager. Three-tier eviction: ThinkComplete < ThinkActive < OutputCritical. Blocks from a completed reasoning chain are demoted the moment </think> is emitted.
3. Entropy-driven budget forcing. EAT (arXiv:2509.26522) and RPDI (arXiv:2603.14251) signals inject </think> only when the model itself is signalling convergence or overthinking — not on a static token counter.
4. Drop-in vLLM plugin. No fork required. Wraps the existing scheduler via the plugin interface; exposes Prometheus + OpenTelemetry telemetry.
5. Disagg KV transfer. offload_block / ingest_block hooks on the block manager support prefill-decode disaggregation fabrics (NIXL, Mooncake-compatible). Documented in ADR-0006.

Scope and assumptions

• In scope: latency-differentiated scheduling for reasoning models served via vLLM on a single node or a disaggregated prefill-decode topology.
• Out of scope: model training, quantisation, speculative decoding, or serving systems other than vLLM. See Non-goals.
• Assumed: the serving stack emits per-request token IDs to a hook point that Meridian can intercept (the vLLM plugin interface satisfies this).
• Assumed: think/output phase boundaries are detectable from the decoded token stream via model-specific boundary token IDs (configurable per model in meridian.toml).

Threats to validity

• Synthetic benchmark results are directional, not absolute. The synthetic-replay harness simulates latency with a calibrated decoder; it does not model multi-tenant memory pressure or real CUDA kernel variance. Results should be reproduced on target hardware before drawing conclusions.
• Budget forcing can affect accuracy. Injecting </think> early may shorten correct reasoning chains. Meridian fires forcing only on entropy convergence signals, but the threshold is a tunable heuristic, not a guarantee. Accuracy measurement is the operator's responsibility.
• Phase detection depends on token IDs. If a model tokenises <think> / </think> differently than the configured boundary IDs, Meridian treats the entire request as single-phase. The models/*.toml files in the repo carry vetted IDs for supported models.

How to read this book

• Architecture — component map, per-component contracts, failure modes, and observability hooks.
• ADRs — every architectural decision with the rejected alternatives. Read these alongside the code to understand the why.
• Configuration — every meridian.toml field with type, default, valid range, and tuning guidance.
• API reference — Rust and Python surfaces with lifecycle and concurrency notes.
• Operations — metrics catalogue, alerting recommendations, troubleshooting runbooks, and benchmark methodology.
• Non-goals — what Meridian explicitly does not do.
• Glossary — definitions for TTFT, TTOT, ITL, EAT, RPDI, KV, disagg, NIXL, and other terms used throughout.

Architecture

Incoming requests
      │
      ▼
┌─────────────────────────────────────────────────────────┐
│                    Meridian Daemon                        │
│                                                           │
│  ┌──────────────┐    ┌────────────────────────────────┐  │
│  │   Prefill    │───▶│        Phase Router             │  │
│  │   Executor   │    │  (token stream state machine)   │  │
│  └──────────────┘    └───────────┬─────────────────────┘  │
│                                  │                         │
│                    ┌─────────────┴──────────────┐          │
│                    │                            │          │
│        ┌───────────▼──────────┐  ┌─────────────▼───────┐  │
│        │   Think-Decode       │  │   Output-Decode      │  │
│        │   Scheduler          │  │   Scheduler          │  │
│        │                      │  │                      │  │
│        │  TPOT: relaxed       │  │  TTOT: strict SLO    │  │
│        │  Batch: 2.5× larger  │  │  Batch: standard     │  │
│        │  Entropy probe live  │  │  Stream priority     │  │
│        │  Budget force ready  │  │                      │  │
│        └──────────┬───────────┘  └────────┬─────────────┘  │
│                   │                       │                 │
│                   └──────────┬────────────┘                 │
│                              │                              │
│               ┌──────────────▼─────────────┐               │
│               │   Phase-Aware KV Block Mgr  │               │
│               │                             │               │
│               │  Tier 0: ThinkComplete      │               │
│               │  Tier 1: ThinkActive        │               │
│               │  Tier 2: OutputCritical     │               │
│               └─────────────────────────────┘               │
└────────────────────────────┬────────────────────────────────┘
                             │
                        vLLM worker
                    (decode kernel, KV store)

Phase state machine

The Phase Router advances each request through this machine on every decoded token. ForceBudget is emitted as a side effect (the request stays in ThinkDecode until </think> is observed or injected).

stateDiagram-v2
    [*] --> Prefill
    Prefill --> ThinkDecode: think_start id  /  EnterThink
    ThinkDecode --> ThinkDecode: token  /  update EAT + RPDI
    ThinkDecode --> ThinkDecode: converged or overthinking  /  ForceBudget
    ThinkDecode --> OutputDecode: think_end id  /  ExitThink
    OutputDecode --> Complete: eos id  /  Complete
    Complete --> [*]

Disaggregated offload sequence

When a fabric is configured, ExitThink triggers a batched offload of the request's think-complete blocks. Each offloaded block is framed, pushed to the fabric, and its local slot is reclaimed (see ADR-0006).

sequenceDiagram
    participant R as PhaseRouter
    participant S as Scheduler / Plugin
    participant B as BlockManager
    participant F as Fabric (NIXL / Mooncake)
    R->>S: ExitThink(req, tokens_used)
    S->>B: demote_think_blocks(req)
    S->>B: blocks_for_request(req)
    B-->>S: [block_ids]
    loop batched at offload_threshold_blocks
        S->>B: offload_block(id)
        B->>F: push(encode(tier, body))
        F-->>B: handle
        B->>B: free_block_by_id(id)
    end
    Note over S,F: meridian_disagg_blocks_offloaded_total += n

Components

Phase Router

Inputs: raw token IDs emitted per step, per request ID.
Outputs: PhaseEvent stream (EnterThink, ExitThink, ForceBudget, BudgetForceReason).
Hot-path constraint: O(1) per token, zero heap allocation in the common case. Backed by DashMap with sharded locking — see ADR-0003.
Failure mode: if a request is never reaped, its entry leaks in the map. reap_stale_older_than(Duration) removes entries older than a wall-clock threshold; the vLLM plugin calls this on every batch step.
Observability: meridian.phase_router.tracked_requests gauge.

Source: crates/meridian-core/src/phase_router.rs.

Dual-Queue Scheduler

Inputs: a pool of pending requests tagged by their current phase.
Outputs: two ordered lists — one output-phase batch (drains first), one think-phase batch (fills remaining capacity).
Hot-path constraint: a single pass over both queues per schedule_batch call. No per-token work.
Invariant: output-phase requests are never starved. The think queue only receives tokens after the output queue is drained or SLO-budget-limited.
Failure mode: if think_batch_multiplier is set too high relative to GPU capacity, output ITL variance increases. meridian.queue_depth{queue=think} growing without accompanying budget_force_triggered activity is the signal.
Observability: meridian.schedule_batch.duration_ns, meridian.queue_depth.

See ADR-0001 for the design alternative this rejects.

Source: crates/meridian-core/src/scheduler.rs.

Phase-Aware Block Manager

Inputs: allocate(request_id, tier) and evict_for(required_blocks) calls from the vLLM KV allocator path.
Outputs: block IDs; eviction decisions ordered by tier.
Invariant: ThinkComplete blocks are always evicted before ThinkActive; OutputCritical blocks are evicted last and only under sustained pressure.
Failure mode: OutputCritical eviction is a user-visible degradation event (stream stutter). Every such event increments meridian.output_critical_eviction. Alert on any increment in a 5-minute window.
Disagg surface: offload_block(block_id) and ingest_block(bytes, tier) are available when a disagg fabric is configured — see ADR-0006.
Observability: meridian.output_critical_eviction counter.

Source: crates/meridian-core/src/block_manager.rs.

Entropy Probe

Inputs: raw logit vector (fp32, bf16, or fp16) from a completed forward pass.
Outputs: EntropySignal — per-token entropy (nats), EAT value, EAT EMA, EAT EMA variance, RPDI local/global ratio.
Hot-path constraint: designed to run on a dedicated secondary CUDA stream; must not stall the generation stream. In Sprint 0 both paths use the NumPy reference; python/meridian/_backends/cuda.py defines the CUDA backend interface and delegates to CPU until Sprint 1 wires it to the Rust kernels in crates/meridian-kernels/.
Invariant: CPU and CUDA backends must agree within atol=1e-5 on the same logit vector. Enforced by crates/meridian-kernels/tests/kernel_correctness.rs.
Failure mode: if the kernel returns Unavailable, the system falls back to count-only budget forcing (hard_cap on every termination). This is safe but loses entropy-driven adaptivity.
Observability: signals surface through meridian.budget_force_reason.

Sources:

• crates/meridian-kernels/ — CUDA kernels + C FFI.
• python/meridian/entropy_probe.py — Python facade + EMA state.
• python/meridian/_backends/ — CPU and CUDA backends.

vLLM Plugin

Inputs: vLLM Scheduler instance at attach time; schedule_batch calls at runtime.
Outputs: reordered batch with output-phase requests drained first; injected </think> tokens on budget-force events; disagg offload calls on ExitThink.
Constraint: no vLLM fork required. The plugin wraps the existing scheduler via attribute delegation; unknown attributes fall through to the wrapped scheduler so vLLM internals work unmodified. MeridianSchedulerPlugin.attach() is a classmethod that installs the plugin as engine.scheduler[0] — no separate detach() is provided in v0.1.x.
Failure mode: if the plugin raises during schedule_batch, it re-raises to the vLLM worker, which surfaces as a serving error for that batch. Errors in the disagg offload path are caught and logged; they do not block generation.
Observability: all Phase Router and Block Manager metrics (meridian.block_manager.*, meridian.queue_depth, meridian.schedule_batch.*), plus meridian_disagg_blocks_offloaded_total and meridian_vocab_fallback_total emitted by the plugin (Prometheus, and OTLP when [telemetry] is enabled).

Source: python/meridian/vllm_plugin.py.

Non-goals

Meridian has explicit scope boundaries. Documenting what it does not do is as important as documenting what it does — it prevents inflated expectations and makes the design auditable.

Not a serving engine

Meridian is a scheduling layer. It requires vLLM (or a compatible serving backend) underneath it. It does not implement:

• Model loading or weight management.
• Prefill execution (prompt processing).
• Decode kernel scheduling (CUDA streams, tensor parallelism).
• Tokenisation or detokenisation.
• HTTP/gRPC serving interfaces.

Not a throughput optimiser

Meridian optimises latency differentiation — protecting output-phase inter-token latency at the cost of slightly reduced think-phase throughput. It does not optimise:

• Raw tokens/sec/GPU throughput. For throughput, run vLLM's own scheduler.
• Batch filling efficiency in non-reasoning workloads.
• Speculative decoding, continuous batching variants, or chunked prefill.

Not an accuracy guarantee

Budget forcing injects </think> based on entropy signals, not correctness. This can shorten reasoning chains on hard problems. Meridian does not:

• Measure or guarantee reasoning accuracy.
• Validate that forced-short chains produce correct answers.
• Implement any feedback loop between output quality and forcing thresholds.

Operators running accuracy-sensitive workloads should validate that their eat_ema_variance_threshold and rpdi_threshold settings do not degrade task accuracy on representative prompts.

Not a multi-model router

Meridian schedules within a single vllm.AsyncLLMEngine instance. It does not:

• Route requests across multiple models.
• Implement A/B model experiments.
• Manage model replicas or horizontal scaling.

Not a vLLM fork

Meridian is a plugin, not a fork. It does not modify vLLM internals and does not ship a patched vLLM binary. Compatibility with a specific vLLM version is the operator's responsibility. See the Compatibility Matrix.

Not production-certified

v0.1.0 is an early release. It has not been validated under production traffic at scale. Known gaps:

• Real-vLLM end-to-end results exist only for Qwen/Qwen2.5-0.5B in the repo's GPU CI workflow.
• The disagg fabric integration uses a synthetic in-process mock when libnixl is not available; real NIXL interop has not been exercised in CI.
• No back-pressure mechanism exists if the disagg fabric becomes unavailable.

Compatibility Matrix

Runtime requirements

Build requirements

Model compatibility

Models that have been verified to work with Meridian's phase detection:

Models that are not verified to work:

• Models with non-standard <think> tokenisation not listed above — configure think_start_token_ids / think_end_token_ids manually and validate with a sample prompt before production use.
• Models served through streaming APIs (e.g. Claude via Anthropic API) — Meridian requires direct access to the logit vector, which API-served models do not expose.

Feature flags

CI coverage

Job	Platform	GPU	Status
rust-core	ubuntu-24.04	No
rust-kernels (stub)	ubuntu-24.04	No	Same badge
python	ubuntu-24.04	No	Same badge
docs	ubuntu-24.04	No	Same badge
supply-chain	ubuntu-24.04	No	Same badge
cuda-build	self-hosted gpu runner	Yes	Runs only on angelnicolasc org pushes

The GPU jobs are gated to prevent arbitrary code execution on the self-hosted runner from fork PRs. See GPU CI runner setup.

Known incompatibilities

• vLLM below 0.9.0: the dependency constraint is vllm>=0.9.0. Earlier versions are not supported and will be rejected at install time.
• Windows (native): the Rust workspace builds on Windows (tested in development), but the Python extension and benchmarks require Linux or WSL2 for the CUDA and maturin paths.
• macOS: not supported. CUDA is not available on macOS.

Deployment Model

Single-node deployment

The primary and most-tested deployment topology. One AsyncLLMEngine instance, one Meridian plugin, all on the same GPU node.

┌──────────────────────────────────────────────┐
│   GPU node                                    │
│                                               │
│  vLLM AsyncLLMEngine                         │
│   └── MeridianSchedulerPlugin (attached)     │
│        ├── PhaseRouter                       │
│        ├── MeridianScheduler                 │
│        └── PhaseAwareBlockManager            │
└──────────────────────────────────────────────┘

Prerequisites: Linux (or WSL2), NVIDIA driver 555+, CUDA 12.6, vLLM ≥ 0.9.0 (pip install "meridian[vllm]" resolves to 0.21.0 via uv.lock).

Configuration: standard meridian.toml with [disagg] enabled = false.

Disaggregated prefill-decode

Experimental. Requires NIXL-capable infrastructure. The block manager's offload_block / ingest_block hooks transfer ThinkComplete KV blocks to a remote decode node after </think> is emitted.

┌─────────────────┐         NIXL fabric          ┌─────────────────┐
│  Prefill node   │  ──── KV block transfer ────▶ │  Decode node    │
│                 │                               │                 │
│  vLLM prefill   │                               │  vLLM decode    │
│  Meridian plug  │                               │  Meridian plug  │
└─────────────────┘                               └─────────────────┘

Status: the disagg wire protocol and block manager hooks are implemented and verified with a synthetic in-process NIXL mock. Real NIXL interop requires cargo build --features nixl and libnixl.so on the deploy host.

Configuration:

[disagg]
enabled = true
fabric = "nixl"
offload_threshold_blocks = 4

See ADR-0006 for the protocol specification.

Installation

From source (recommended)

git clone https://github.com/angelnicolasc/meridian.git
cd meridian

# Build and install the Rust core + Python bindings.
uv sync --project python
maturin develop --release -m crates/meridian-python/Cargo.toml

# Optional: build with CUDA kernel support.
# Requires nvcc + CUDA 12.6 toolkit.
maturin develop --release \
  -m crates/meridian-python/Cargo.toml \
  --cargo-extra-args="--features cuda"

Devcontainer

The repo includes a devcontainer configuration with the full toolchain pre-installed:

# Open in VS Code with the Dev Containers extension, or:
./scripts/dev-up.sh

Configuration loading

The plugin looks for meridian.toml in the current working directory, then ~/.config/meridian/meridian.toml. Override with:

from meridian import load_config
from meridian.vllm_plugin import MeridianSchedulerPlugin

cfg = load_config("/path/to/meridian.toml")
plugin = MeridianSchedulerPlugin(scheduler=engine.scheduler, config=cfg)

Known limits

Rust API

The Rust API is generated by rustdoc. Build locally with:

cargo doc --workspace --no-deps --open

Top-level surface

dspark_bridge at a glance

Ships uncalibrated, and stays that way until a phase-segmented measurement exists. See ADR-0009 for why an uncalibrated hook can only ever shrink a draft budget, never grow it.

Object lifecycle

PhaseRouter

PhaseRouter is Send + Sync. Create once, share across threads via Arc. Call process_token(req_id, token_id) from any thread; it returns an Option<PhaseEvent> and is O(1) per call.

Call reap_stale_older_than(duration) periodically to free entries for completed requests. The vLLM plugin does this on every batch step.

#![allow(unused)]
fn main() {
use meridian_core::PhaseRouter;
use std::sync::Arc;
use std::time::Duration;

let router = Arc::new(PhaseRouter::new());

// Per-token — called from the decode loop.
if let Some(event) = router.process_token(req_id, token_id) {
    // Handle PhaseEvent::ExitThink, ForceBudget, etc.
}

// Periodic cleanup — call from the batch step hook.
let reaped = router.reap_stale_older_than(Duration::from_secs(60));
}

MeridianScheduler

MeridianScheduler is Send + Sync. The schedule_batch method takes a shared reference and returns owned Vec<RequestId> for each queue; it does not hold a lock across the call boundary.

BlockManager trait

The three required methods:

#![allow(unused)]
fn main() {
fn allocate(&mut self, request_id: RequestId, tier: BlockTier) -> Result<BlockId>;
fn evict_for(&mut self, required_blocks: usize) -> Vec<BlockId>;
fn block_location(&self, block_id: BlockId) -> BlockLocation;
}

Optional disagg methods (offload_block, ingest_block) have default implementations that return Err(BlockManagerError::FabricNotConfigured). Override them when wrapping with a NIXL-backed manager.

Error model

All errors implement std::error::Error and are defined in meridian_core::error::Error. Configuration errors carry the dotted field path and a human-readable message:

ConfigValidation { field: "entropy.ema_alpha", reason: "must be in (0, 1]" }

Kernel errors are defined in meridian_kernels::KernelError:

• Unavailable — built without the cuda feature or runtime missing.
• Launch(i32) — CUDA returned a non-zero error code.
• NullPointer(&'static str) — caller passed a null pointer.

Thread safety

Stability

Pre-1.0. Public API breakage is recorded under BREAKING CHANGE: in CHANGELOG.md. The C ABI (meridian_entropy_launch, meridian_eat_launch) is treated as stable from v0.1.0 — forks consuming the FFI directly will not see unexpected breakage on patch updates.

Python API

Installation

# From source — requires a Linux host with Rust 1.85+ and maturin.
uv sync --project python
maturin develop --release -m crates/meridian-python/Cargo.toml

The package exposes no CUDA dependency at import time. CUDA is lazily loaded when backend="cuda" is requested on EntropyProbe.

Top-level surface

Phase-conditioned speculative decoding

Both classes come from the native extension and require maturin develop.

from meridian._meridian import AcceptanceLedger, PhaseConditioningHook

# Ships uncalibrated: can only ever *reduce* the baseline draft depth.
hook = PhaseConditioningHook(baseline_proposal_len=7)
assert hook.is_calibrated is False
policy = hook.policy_for("think")           # -> {"proposal_len": 7, "basis": "baseline", ...}

# Offline analysis of a harness run.
ledger = AcceptanceLedger.measured(
    harness="DeepSpec@<sha>",
    draft_checkpoint="deepseek-ai/dspark_qwen3_4b_block7",
    target_model="Qwen/Qwen3-4B",
    thinking_mode=True,
    recorded_on="2026-08-07",
)
ledger.record("think", accepted_length=3, proposal_length=7)
ledger.record("output", accepted_length=6, proposal_length=7)
print(ledger.render())
verdict = ledger.verdict(between_domain_gap=0.5)

policy_for accepts either the short phase names ("think", "output") or the router's own phase_of_kind labels ("think_decode", "output_decode").

A ledger built with AcceptanceLedger.synthetic(fixture) computes the same statistics but raises ValueError from measured_claim() — synthetic traces cannot be promoted to published results. See ADR-0009.

Object lifecycle

EntropyProbe

One instance per request. Not thread-safe — do not share an instance across concurrent requests. Create, use through the token sequence, then discard.

from meridian import EntropyProbe, load_config
import numpy as np

cfg = load_config("meridian.toml")
probe = EntropyProbe(
    think_end_token_ids=cfg.model["qwen3"].think_end_token_ids,
    backend="cpu",          # "cpu" (NumPy) or "cuda" (CUDA kernel)
    ema_alpha=cfg.entropy.ema_alpha,
)

# Per-token call — call once per decoded token.
logits = np.random.randn(151_936).astype(np.float32)
sig = probe.compute(req_id=42, logits=logits)
print(sig.token_entropy, sig.eat, sig.eat_ema_variance)

# Batch path — more efficient for large batch sizes.
batch_logits = np.random.randn(8, 151_936).astype(np.float32)
signals = probe.compute_batch(req_ids=list(range(8)), logits_batch=batch_logits)

MeridianSchedulerPlugin

Wraps an existing vllm.core.scheduler.Scheduler at runtime. Safe to attach and detach. Holds no GPU resources; all GPU work goes through the underlying vLLM scheduler.

from vllm import AsyncLLMEngine
from vllm.engine.arg_utils import AsyncEngineArgs
from meridian import load_config
from meridian.vllm_plugin import MeridianSchedulerPlugin

engine = AsyncLLMEngine.from_engine_args(AsyncEngineArgs(model="Qwen/Qwen2.5-0.5B"))
cfg = load_config("meridian.toml")

# attach() is a classmethod — constructs the plugin, installs it as
# engine.scheduler[0], and returns the handle for metric access.
plugin = MeridianSchedulerPlugin.attach(engine, cfg, model_key="qwen3")

# ... serve requests ...

# v0.1.x has no detach(); the plugin runs for the engine's lifetime.

Error model

All configuration errors are raised at construction time as ValueError with a dotted field path. For example:

ValueError: entropy.ema_alpha must be in (0, 1]; got 1.5

Runtime errors from the CUDA kernel surface as meridian.KernelError (a subclass of RuntimeError). When the kernel returns Unavailable (built without the cuda feature, or missing runtime library), the probe falls back to the CPU backend automatically.

Concurrency notes

• EntropyProbe instances are not thread-safe. One instance per request.
• MeridianSchedulerPlugin is designed to be used from vLLM's single async event loop. Do not call schedule_batch concurrently.
• The Rust PhaseRouter and BlockManager bindings are thread-safe; they use interior mutability backed by DashMap.

Stability guarantees

Pre-1.0. Signatures may change on minor bumps. Breaking changes are listed under BREAKING CHANGE: in CHANGELOG.md and announced before merging.

Backends

EntropyProbe accepts backend="cpu" (default, pure NumPy) or backend="cuda". Both backends implement the same mathematical operations and agree within atol=1e-5. In Sprint 0 the cuda backend delegates to the CPU implementation; Sprint 1 will wire it to the Rust CUDA kernels in crates/meridian-kernels/ so the logit reduction runs on a dedicated secondary CUDA stream.

Configuration

Meridian is configured through a single TOML file consumed by both the Rust core (meridian-core::config::MeridianConfig) and the Python facade (meridian.config.MeridianConfig). Both parsers agree on field names; a round-trip test in crates/meridian-core/tests/config_parse.rs exercises every field.

The fully-annotated example lives at meridian.toml.example. Tune by overlay: keep the example as a reference and write a smaller meridian.toml containing only fields that differ from their defaults.

Validation

Both parsers reject unknown fields and out-of-range values. Cross-field violations (e.g. min_think_tokens >= max_think_tokens) are also caught. Errors carry the dotted field path and a human-readable message.

[scheduler]

Dual-queue scheduling policy. See ADR-0001.

think_tpot_budget_ms

TPOT budget for think-phase tokens. The user does not see inter-token latency during reasoning, so this can be set much higher than the output budget. Setting it 4× the output budget gives the batcher room to pack a larger effective batch during think. Raise if your GPU is underutilised during think; lower if think-phase requests monopolise capacity at the expense of output.

output_tpot_budget_ms

TPOT budget for output-phase tokens. This is the user-visible streaming latency floor. 20 ms keeps streams fluid on a 30–50 tok/s display target. Lower values produce tighter streams but reduce think-phase throughput.

think_batch_multiplier

The think-phase batch can fill this multiple of the output-phase token budget. 2.5× is conservative — empirically stable across H100-class hardware with MLA-aware allocation. Values above 3.5× risk output ITL variance spikes if think requests fail to yield promptly. Monitor meridian.queue_depth{queue=think}.

max_think_tokens

Hard cap on think tokens per request. Budget forcing fires unconditionally at this limit regardless of entropy signals. 32 768 matches DeepSeek-R1's documented maximum reasoning length and bounds the KV memory a single request can monopolise.

min_think_tokens

No budget forcing is allowed before this many think tokens. EAT/RPDI signals are noisy below 512 tokens; early forcing can prematurely terminate short-but-correct reasoning chains.

[entropy]

Entropy probe and convergence-detection thresholds.

enabled

When false, the entropy probe is disabled and all budget forcing uses hard_cap only (pure token-count limiting). Useful for A/B comparison or when the CUDA kernel is not available.

ema_alpha

EMA decay applied to all entropy signals. Smaller values give longer memory. α = 0.05 → ~95% mass within the last ~60 samples. Long enough to smooth single-token spikes; short enough to react within a reasoning chain.

rpdi_threshold

Overthinking is declared when rpdi_local / rpdi_global > threshold. The value 3.0 is the empirical threshold from arXiv:2603.14251. Raise to be more permissive (longer chains); lower to be more aggressive.

eat_ema_variance_threshold

Convergence is declared when EAT EMA variance drops below this threshold. 0.001 is approximately the noise floor of EAT in steady state. Lower values defer forcing; higher values fire earlier.

transition_entropy_threshold

A token counts as a "transition" for RPDI when its per-token entropy exceeds this value. 2.5 nats ≈ effective branching factor of 12 — a genuine decision point rather than low-entropy continuation.

eat_probe_interval_tokens

The EAT kernel runs every N think tokens. 1 = every token; higher values trade signal latency for reduced kernel-launch overhead. 32 is the sweet spot on H100-class hardware. Halving this on slower GPUs is safe.

[kv_memory]

Phase-aware KV block manager policy.

aggressive_think_eviction

When true, ThinkComplete blocks are freed immediately on phase transition. Leave false until cross-attention back-references are audited for your model; some reasoning-parser pipelines re-attend over the think segment when generating output and need those blocks resident.

think_phase_memory_fraction

Fraction of total KV budget reserved for think-phase blocks. 0.40 leaves 60% for output-phase blocks, which accommodates think_batch_multiplier = 2.5 without crowding output. Raise for workloads with very long reasoning chains; lower for workloads with long output sequences.

block_size_bytes

Must match the actual vLLM block layout for your model. The canonical vLLM layout is 16 KiB for bf16/fp16 KV at 16 tokens per block. MLA-aware models can run smaller blocks.

capacity_bytes

Total KV memory budget. "auto" queries the device at startup and uses 85% of torch.cuda.mem_get_info().total. Pin an integer for deterministic or multi-tenant deployments where you want to reserve GPU memory for other workloads.

[disagg]

Disaggregated KV transfer. Disabled by default. See ADR-0006.

enabled

Master switch. Leave false for single-node deployments.

fabric

Selects the disagg transport. nixl uses the NVIDIA NIXL library (requires cargo build --features nixl and libnixl on the deploy host). mooncake uses the Mooncake-compatible protocol adapter. none is only valid when enabled = false.

offload_threshold_blocks

Minimum ThinkComplete blocks to accumulate before flushing to the fabric. Larger values amortise transfer overhead; smaller values reduce latency.

[speculation]

Phase-conditioned speculative decoding. See ADR-0009 and the research note.

Read before enabling. The hypothesis this section exists to act on — that draft acceptance is lower during the think phase, because DeepSpec's released Qwen3 drafters were trained only on non-thinking-mode generations — has not been measured. The hook is therefore deliberately half-inert: without [speculation.acceptance_prior] it can only ever reduce the baseline draft depth, and only where the entropy probe proves a deeper draft cannot pay off. Only a measured prior unlocks upward adjustment and phase conditioning.

enabled

Master switch. When false the serving stack never consults the hook.

baseline_proposal_len

Draft depth the serving stack would use without the hook, and the ceiling on what an uncalibrated hook may return. Defaults to 7 because DeepSpec's released DSpark Qwen3 checkpoints are block7.

min_proposal_len / max_proposal_len

Bounds on the recommended draft depth. max_proposal_len must not exceed the drafter's block size — proposing past it is not something the drafter can do.

vocab_size

Target-model vocabulary size. Required for the entropy-derived acceptance ceiling, which depends on V as well as on entropy.

use_entropy_ceiling

Whether to derive a draft-depth ceiling from the entropy probe. Setting this false makes an uncalibrated hook a strict no-op — the most conservative posture available.

draft_token_us / verify_fixed_us / verify_token_us

Cycle cost model driving the throughput objective Θ(γ) = τ(a, γ) / (draft_token_us·γ + verify_fixed_us + verify_token_us·γ).

The defaults are placeholders. They produce sane relative ordering and are not a measurement of any hardware. Profile your deployment and replace them; if meridian.speculation.proposal_len sits pinned at min_proposal_len, these are the first thing to check.

[speculation.acceptance_prior]

Measured per-phase acceptance rates. Absent until a Phase 1 run exists — see the protocol.

Every field is required: think, output, harness, draft_checkpoint, target_model, thinking_mode, recorded_on. There is deliberately no way to express acceptance rates without naming the run that produced them, so phase conditioning cannot be switched on by a hunch. Partial or blank provenance is rejected by both parsers.

# ILLUSTRATIVE placeholders showing the shape of the table. These are not
# measurements — no phase-segmented run exists yet.
[speculation.acceptance_prior]
think            = 0.42   # <- the measured think-phase acceptance rate
output           = 0.88   # <- the measured output-phase acceptance rate
harness          = "DeepSpec@<commit-sha>"
draft_checkpoint = "deepseek-ai/dspark_qwen3_4b_block7"
target_model     = "Qwen/Qwen3-4B"
thinking_mode    = true
recorded_on      = "2026-08-07"

[model.<name>]

Per-model token-boundary configuration. One [model.*] table per model served. The phase router watches for boundary token IDs in the decoded stream.

See models/*.toml for the vetted token IDs for supported models.

think_start_token_ids

Token IDs that mark the start of a reasoning chain. Model- and tokenizer-specific. If empty, the router never enters think-phase for this model.

think_end_token_ids

Token IDs that mark the end of a reasoning chain. When the decoded stream contains any of these IDs, the router emits ExitThink.

reasoning_parser

Selects the reasoning-chain parser. Different models structure their think-output boundary differently; the parser handles model-specific normalization.

supports_think_disable

Whether the model supports a /no_think directive to suppress the reasoning phase entirely. When true and the prompt contains the directive, the router stays in output-phase for the entire request.

Glossary

Terms used throughout the Meridian documentation and codebase.

TTFT — Time to First Token. Wall-clock time from the moment a request is submitted to the moment the first token is returned to the client. Dominated by prefill latency for long prompts.

TTOT — Time to Output Token. Wall-clock time from the emission of the </think> boundary token to the first user-visible output token. The metric Meridian is specifically designed to protect.

TPOT — Time Per Output Token (also ITL: Inter-Token Latency). Time between consecutive output tokens during streaming. The user-perceived "speed" of the stream.

ITL — Inter-Token Latency. See TPOT.

EAT — Entropy-Aware Termination. A budget-forcing signal based on the variance of the EMA of per-token Shannon entropy over the reasoning chain. When EAT EMA variance drops below a threshold, the model is inferred to have converged on an answer. Defined in arXiv:2509.26522.

RPDI — Reasoning Phase Divergence Index. A signal based on the ratio of local transition-token frequency to global transition-token frequency. A high ratio indicates the model is cycling through redundant reasoning steps ("overthinking"). Defined in arXiv:2603.14251.

Entropy (Shannon) — Measure of uncertainty in the next-token distribution. Computed from the logit vector after softmax as -Σ p_i · log(p_i), in nats. High entropy = uncertain prediction; low entropy = confident prediction.

EMA — Exponential Moving Average. A smoothed average where recent values are weighted more heavily. Controlled by ema_alpha (smaller = longer memory).

KV — Key-Value cache. The GPU memory store holding the attention keys and values for each token in active requests. KV memory is the primary capacity constraint in serving systems.

KV block — The unit of KV cache allocation. A fixed-size chunk (default 16 KiB) covering a fixed number of tokens (default 16). Blocks are allocated at the request level and freed on eviction or request completion.

ThinkComplete — Block tier for KV blocks that belonged to a request's reasoning phase after </think> has been emitted. Lowest eviction priority — these blocks are freed first under memory pressure.

ThinkActive — Block tier for KV blocks belonging to a request currently in the reasoning phase.

OutputCritical — Block tier for KV blocks belonging to a request in the output phase. Highest eviction priority — evicting these causes user-visible stream disruption. Any eviction at this tier fires meridian.output_critical_eviction.

Disagg / disaggregated serving — Prefill-decode disaggregation: the model prefill (prompt processing) and decode (token generation) steps are executed on separate hardware. Disagg reduces head-of-line blocking by separating the two workloads, which have very different GPU utilisation profiles.

NIXL — NVIDIA Inference eXchange Layer. NVIDIA's reference fabric for transferring KV blocks between prefill and decode nodes in a disaggregated serving topology.

Mooncake — An open-source disaggregated serving framework with a KV-transfer protocol. Meridian's disagg surface is documented as Mooncake-compatible in ADR-0006.

vLLM — An open-source LLM serving framework. Meridian's primary integration target. Meridian wraps vLLM's scheduler without forking the codebase.

DashMap — A concurrent hash map crate used for the PhaseRouter's per-request state. Provides O(1) read and write with sharded locking. See ADR-0003.

EOS — End of Sequence. The special token that signals request completion. vLLM emits an EOS event that the Meridian plugin uses to trigger request teardown and router state reaping.

Conventional Commits — A commit message standard used throughout this repository. Format: <type>(<scope>): <summary>. See conventionalcommits.org.

DCO — Developer Certificate of Origin. A sign-off mechanism (git commit -s) that certifies the contributor has the right to submit the code under the project's license. Required for all contributions — see CONTRIBUTING.md.

SLSA — Supply-chain Levels for Software Artifacts. A framework for supply-chain security. Meridian attests Level 2 provenance on every tagged release via slsa-github-generator. See ADR-0007.

SBOM — Software Bill of Materials. A machine-readable inventory of software components and their licenses. Meridian generates a CycloneDX SBOM for each release, attached as a GitHub release asset.

Architectural Decision Records

Meridian uses Michael Nygard's ADR format to capture the why behind significant choices. ADRs are immutable once "Accepted"; we supersede them rather than edit them.

Lifecycle: Proposed → Accepted → (later) Superseded by ADR-NNNN / Deprecated.

Index

Writing a new ADR

1. Copy template.md to NNNN-short-kebab-title.md.
2. Open as Proposed; merge as Accepted after PR review.
3. If a later ADR supersedes this one, mark this one Superseded by ADR-NNNN in a new commit — never edit the body of an accepted ADR.

ADR-0001: Dual-queue scheduling vs. priority weights on a single queue

• Status: Accepted
• Date: 2026-05-20
• Authors: angelnicolasc
• Reviewers: sole-maintainer decision record

Context

Meridian's central thesis is that think-decode and output-decode are two structurally different workloads inside a single reasoning-model request:

• Output tokens are user-visible streaming. They must hit a tight TTOT (time-to-output-token) target — ~20 ms is the perceptual threshold for fluid streaming at typical display rates.
• Think tokens are user-invisible reasoning. The user is already waiting for the answer; inter-token latency during reasoning is irrelevant. Throughput (tokens/sec/GPU) is what matters here.

Given that, the scheduler needs to give output-phase requests absolute priority while letting think-phase requests fill any remaining batch capacity with a larger effective batch size to maximise GPU utilisation.

There are two plausible structural shapes to implement this:

1. Single queue with priority weights. One queue of all "decode-eligible" requests; each request carries a priority numeric. The scheduler picks the highest-weighted requests every iteration and the eviction policy reads block tier from the request's phase.
2. Two independent queues, one per phase. An output_queue drained first to its budget, then a think_queue drained to a larger budget capped by remaining KV memory.

Both can produce equivalent dispatch ordering. They diverge in observability, in the failure modes they expose, and in how cleanly they compose with KV tier management.

Decision

Meridian uses two independent queues — output_queue and think_queue — sharing the same GPU workers, with output drained first every iteration.

The scheduler exposes per-queue depth as a separate metric label, applies per-queue SLO budgets, and the block manager's eviction tiers are indexed on the block's phase membership rather than on the owning request's priority number.

Consequences

Positive

• SLO isolation by construction. TTOT and TPOT live on different queues and cannot interfere through priority arithmetic. We never need to reason about whether a priority of 5 vs. 7 is enough to keep an output token from being preempted by a think token — the queues are physically separate.
• Reasoning about starvation is local. With priority weights, you have to argue globally about the joint distribution of priorities under load to prove that think requests are not starved. With two queues, the worst-case is "output_queue saturates the budget → think_queue waits its turn." That is a one-line argument and a single bounded scalar (think_batch_multiplier × output_budget) to tune.
• Block manager tiering is structurally aligned. The eviction policy iterates BlockTier::ThinkComplete → ThinkActive → OutputCritical. The scheduler's queues map 1:1 onto two of those tiers (ThinkActive, OutputCritical), and the ThinkComplete tier appears precisely when a request transitions queues. The pipeline of state transitions is uniform end-to-end.
• Observability is honest. meridian.queue_depth{queue="output"} and …{queue="think"} are operationally meaningful — they correspond to things an oncall engineer can act on. A single meridian.queue_depth_p95_priority would obscure the failure mode.
• Future disaggregation is cheap. When we add a separate decode pool for think (a natural extension co-located with prefill-decode disagg systems like Mooncake / NIXL), the seam already exists.

Negative / risks

• Two queue data structures instead of one. Marginal memory cost (crossbeam::SegQueue is small) and a second O(log n) insert path. Not material against the per-token compute budget.
• Risk that think queue is permanently starved under sustained output pressure. Mitigation: the scheduler enforces a minimum think-batch reservation when output_queue.len() < output_budget. Detection: meridian.queue_depth{queue="think"} rising while meridian.budget_force_triggered stays flat — alert at p95 depth > 4× baseline for 5 minutes.
• Edge cases at phase transition. A request that emits </think> and the next token in the same decode step transitions queues mid-iteration. This is handled by MeridianScheduler::on_phase_event taking the request out of the think queue and pushing it into the output queue before the next schedule_batch call. Tests for this case live in tests/phase_router_state_machine.rs.

Neutral

• The number of tunables stays the same. A single-queue design with priority weights requires output_priority, think_priority, and a priority_gap_min; the dual-queue design requires output_tpot_budget_ms, think_tpot_budget_ms, and think_batch_multiplier. Both surfaces are three scalars.

Alternatives considered

Single queue with continuous priority weights

RequestSlot { priority: f32 }, dispatch is argmax(priority) with preemption. Output requests carry priority ≈ 10, think requests priority ≈ 1. Rejected because:

• The dispatch order under heavy load depends on the distribution of weighted requests, not on a per-class budget — you can no longer write down a one-line invariant like "output never waits more than K ms."
• The block manager would need to read priorities to decide eviction order, coupling two subsystems that we want orthogonal.
• Operators tuning the system find priority weights opaque — "is 8.5 enough?" is not a question with a principled answer.

Single queue with phase-stratified preemption

One queue, but a hard rule that any output-phase request preempts any think-phase one. This is structurally equivalent to two queues but expressed differently. Rejected for code-clarity reasons only: the dual-queue shape makes the invariant ("output drains first") the structure, rather than an invariant we have to police in the dispatcher.

Per-tenant queues with phase tags

Considered for multi-tenant SaaS deployments. Not rejected outright, but deferred — it is an orthogonal axis we can layer on top of the two-queue shape. Captured as a future ADR placeholder.

References

• Playbook §3.3 — Dual-Queue Scheduler.
• vLLM v0.9 scheduler internals: vllm/core/scheduler.py (single-queue priority-weighted implementation we are improving on).
• Mooncake disagg paper — separates prefill from decode; orthogonal axis.
• DUCHESS (arXiv:2509.24957) — intra-request branch orchestration; operates below the queue layer.

ADR-0002: Workspace tri-crate layout (core / kernels / python)

• Status: Accepted
• Date: 2026-05-20
• Authors: angelnicolasc

Context

Meridian spans three execution domains: a pure-Rust scheduler core, CUDA kernels behind an FFI boundary, and pyo3 bindings that the Python vLLM plugin consumes. The natural layouts are:

1. Single crate. All code in one place, gated by cfg(feature = ...).
2. Tri-crate Cargo workspace. meridian-core (Rust only), meridian-kernels (CUDA + FFI), meridian-python (pyo3, built via maturin). All members of one workspace, sharing lockfile and lints.
3. Polyrepo. Each layer in its own repository, glued by published versions.

Decision

Tri-crate workspace. All three crates live under crates/ in this repository.

Consequences

Positive

• cargo test -p meridian-core runs on any host — no CUDA, no Python, no nvcc. The CI matrix can validate the core invariants on the cheapest GitHub-hosted runner and only spin up a GPU runner for the CUDA layer.
• The unsafe surface area is visibly contained. Auditors looking at meridian-core see #![forbid(unsafe_code)] at the crate root. The unsafe code lives in exactly one crate (meridian-kernels) and at exactly one boundary (the FFI declarations in src/ffi.rs).
• Workspace [workspace.lints] is applied uniformly across all three crates — one place to change a clippy lint, no drift.
• A future fork that wants only the scheduler core (e.g. for a non-CUDA inference framework) can depend on meridian-core directly without pulling pyo3 or CUDA artifacts.

Negative / risks

• Three Cargo.tomls to keep in sync. Mitigated by workspace.dependencies and workspace.package inheritance — versioned dependencies are declared once.
• The meridian-kernels crate has links = "meridian_kernels". Cargo enforces uniqueness so we cannot accidentally link two implementations of the same native library — a small but real guard.

Neutral

• Build artifacts grow by one extra target/ directory per crate. Negligible in practice.

Alternatives considered

Single crate

Rejected because: any Python binding gate requires pyo3 in the dep graph, which pulls a non-trivial transitive closure. Anyone wanting to depend on just the scheduler core would be forced to compile it.

Polyrepo

Rejected because: in a pre-1.0 project where the three layers co-evolve, splitting them into separate repositories introduces version-skew bugs without a corresponding benefit. Once the public API stabilises post-1.0 this can be revisited.

References

• Rust API Guidelines — C-FEATURE.
• Cargo Book — Workspaces.

ADR-0003: DashMap for per-request phase state

• Status: Accepted
• Date: 2026-05-20
• Authors: angelnicolasc
• Reviewers: sole-maintainer decision record

Context

The PhaseRouter mutates per-request state (ThinkPhase) on every decoded token. A single decode worker processes a continuous-batch step that can touch dozens of requests; in a tensor-parallel deployment, multiple worker threads may invoke on_token for different requests in the same wall-clock window. We need:

• O(1) lookup keyed by req_id.
• Concurrent mutation of different keys without serialising the whole map.
• Cheap clone / sharing — the router is shared across the scheduler, the block manager touch path, and the vLLM plugin.
• No GC, no allocations on the hot path.

Candidates evaluated:

1. parking_lot::RwLock<HashMap<u64, ThinkPhase>>
2. DashMap<u64, ThinkPhase> (sharded HashMap behind per-shard RwLock)
3. papaya::HashMap<u64, ThinkPhase> (lock-free, June 2025)
4. scc::HashMap<u64, ThinkPhase> (lock-free, sharded)

Decision

DashMap 6.x. It is the boring, well-audited choice that hits the performance floor we need without introducing a less-battle-tested dependency on the hot path.

Consequences

Positive

• Sharded locking. Operations on different req_ids do not contend.
• Mature API. get_mut, entry, remove cover every access pattern in PhaseRouter. No need to invent abstractions on top.
• No unsafe. Internally backed by parking_lot; we keep the #![forbid(unsafe_code)] invariant in meridian-core intact.
• Crates.io top-100. Wide deployment, frequent security audits, stable semver.

Negative / risks

• Sharded lock is not truly lock-free. Under extreme contention (a single shard absorbing many requests because of hash skew), throughput degrades. Mitigation: monitor meridian.phase_router.tracked_requests; if the gauge approaches n_shards × shard_capacity (default ~512 per shard), revisit with papaya or shard-aware partitioning. In our workload (~1k concurrent requests max per worker) we are far from this ceiling.
• AHasher by default. Good for our integer keys; not a downside, but worth noting that we benefit from non-cryptographic hashing here.

Neutral

• Memory per entry is slightly higher than a flat HashMap because of the shard metadata. Negligible (~64 bytes overhead total).

Alternatives considered

RwLock<HashMap<...>>

Rejected: every on_token call needs a write lock to bump tokens_so_far, so the RwLock collapses to a Mutex in practice. Single-shard serialisation across all requests is unacceptable.

papaya::HashMap

Watched, not adopted. Genuinely lock-free, with better tail latency under contention than DashMap. Adoption deferred until: (a) it stabilises a 1.0 API, and (b) we can demonstrate a workload where DashMap is the bottleneck. Tracked as future work in the DEVLOG.

scc::HashMap

Comparable to papaya but more API surface; same deferral rationale.

References

• DashMap source: https://github.com/xacrimon/dashmap.
• The phase_router bench measures the steady-state cost of on_token under no contention.

ADR-0004: KV tier promotion is one-way; demotion is the only direction

• Status: Accepted
• Date: 2026-05-20
• Authors: angelnicolasc
• Reviewers: sole-maintainer decision record

Context

The block manager maintains three tiers (ThinkActive, ThinkComplete, OutputCritical). Tier transitions are emitted by the scheduler on phase events. We had to decide whether a block can promote (e.g. a ThinkComplete block re-attended during output generation gets restored to ThinkActive or even OutputCritical).

Two camps:

1. Bidirectional: any block currently accessed gets promoted to the highest applicable tier.
2. One-way demotion: blocks only move down the eviction order (OutputCritical → … → freed). Promotion is explicitly disallowed.

Decision

One-way demotion. A block's tier is set at allocation time (via BlockTier::ThinkActive or BlockTier::OutputCritical) and can only move toward eviction:

• ThinkActive → ThinkComplete via demote_think_blocks.
• Any tier → freed via evict_for or free.

There is no API to move a ThinkComplete block back to ThinkActive, or a ThinkActive block to OutputCritical.

Consequences

Positive

• Reasoning about eviction stability becomes trivial. A block's tier monotonically decreases. Cross-attention back-references over a reasoning span cannot accidentally "rescue" blocks the scheduler has already decided are evictable — operators can reason about KV pressure without tracking promotion races.
• The block manager API is smaller. No promote() method to test, no invariant to enforce ("you can only promote within the same request").
• Aligns with the playbook intent. kv_memory.aggressive_think_eviction is a one-way knob: think blocks either survive demotion (default) or are freed immediately (aggressive). Promotion would make this knob semantically incoherent.

Negative / risks

• A reasoning model that re-attends over its own think span during output generation pays the eviction cost twice. Cross-attention reads on a ThinkComplete block bring the block into the GPU's L1/L2 cache but do not promote its eviction tier. If that block is then evicted by memory pressure, the next cross-attention read forces a recompute. Mitigation: keep kv_memory.aggressive_think_eviction = false so blocks stay resident as ThinkComplete until pressure actually demands their eviction.
• No way to mark "this block is hot, please keep it" beyond keeping it in the lowest tier it was admitted to. The LRU within a tier is the only signal of recency.

Neutral

• The touch() API exists to update LRU position within a tier, not to promote across tiers. This is explicit in the trait documentation.

Alternatives considered

Bidirectional with promote_block(block_id, tier)

Rejected because:

• Adds a new invariant to police (can a ThinkComplete of request A promote to OutputCritical of request B? Obviously not, but the API must enforce that).
• Promotion races with eviction: a block that the eviction iterator has selected as next victim might be promoted mid-eviction. Resolving this needs either a lock around the whole eviction loop (kills throughput) or a generation counter (more complexity).
• The cross-attention rescue use case is real but rare; the simpler fallback (operator tunes think_phase_memory_fraction) handles it without architectural complexity.

Implicit promotion on touch()

Rejected: touch() is called on the hot path for every cache hit. Doing tier promotion there would dramatically slow the common case to optimise a rare one.

References

• ADR-0001 — dual-queue scheduling sits alongside this tier policy.
• Playbook §3.4 — original three-tier eviction design.

ADR-0005: Benchmark methodology and metric selection

• Status: Accepted
• Date: 2026-05-20
• Authors: angelnicolasc
• Reviewers: sole-maintainer decision record

Context

Meridian's value proposition is phase-differentiated scheduling. The benchmark harness has to surface that value — measuring overall throughput or aggregate TPOT will not distinguish Meridian from a stock vLLM scheduler running the same workload, even if Meridian is substantially better for the user-visible streaming experience.

We must choose:

1. What metrics to report.
2. What workload to drive the system with.
3. How to run the benchmark without a GPU in CI.

Decision

Metrics

The benchmark reports the following primary metrics (all per-request, aggregated to percentiles in the report):

The harness explicitly does NOT report aggregate throughput (tokens/sec/GPU). That is what every other benchmark already reports, and it does not distinguish Meridian from the baseline. Operators who only want a throughput number can run vLLM's own benchmark harness.

Workload

Reference workload: synthetic mix of two categories.

• Chat — short prompts, 40–240 output tokens, no think phase. Models the ShareGPT-style background traffic that should never stutter.
• Reasoning — math-style prompts with expected think token counts in [600, 6000]. Models a MATH-500-equivalent distribution.

Mix ratio is operator-configurable (--reasoning-ratio); default 0.4 is the realistic balance for a 2026 reasoning-model deployment.

Arrivals are Poisson (exponential inter-arrival) at a configurable rate. This matches how production traffic actually arrives and exercises the dual-queue policy under realistic burst conditions.

Two execution modes

• synthetic-replay — uses the native Meridian components (PhaseRouter, MeridianScheduler, BlockManager) over a synthetic decoder loop that does not require a GPU or a real vLLM. The phase events, scheduler queue transitions, KV allocations and eviction pressure are all real; only the per-token compute is simulated as a fixed-cost sleep. This mode runs in CI and detects regressions in the scheduler / block manager dynamics.

• real-vllm — drives a real AsyncLLMEngine with the MeridianSchedulerPlugin attached. Requires a CUDA-capable host and a model checkpoint. Runs on the GPU CI job and on demand for release validation.

Both modes emit the same BenchmarkReport JSON+Markdown shape so reports are directly comparable. CI uploads both as artefacts and the Markdown form can be posted as a PR comment for visual diff.

Consequences

Positive

• Reproducibility: synthetic-replay is deterministic given --seed and runs in seconds. Two PRs can be compared apples-to-apples without GPU access.
• Honest reporting: metrics call out exactly where Meridian wins (TTOT, output ITL variance) and acknowledge what we don't measure (raw throughput).
• Cross-mode parity: the same BenchmarkReport schema for both modes means a regression caught in synthetic-replay translates directly to expected behaviour under real-vllm.
• Honest about failures: the output_critical_eviction_events counter surfaces user-visible degradation immediately; the budget-force reason breakdown surfaces when the entropy probe is doing real work vs. just hitting the cap.

Negative / risks

• synthetic-replay does not exercise the CUDA kernels. A regression in the kernels will not be caught by CI; the GPU job's kernel_correctness test is the line of defence there.
• synthetic per-token latency is calibrated, not measured. The default sleeps (6 µs / 18 µs for think / output) are approximations of bf16-Qwen3-on-H100 wall-clock times. Operators tuning a different hardware target should override via the SyntheticDecoder constructor.
• Mix is synthetic. Real ShareGPT / MATH-500 replays are available via --workload sharegpt|math500 and use the offline HuggingFace dataset loader; they do not require a GPU.

Neutral

• Report artefacts are JSON + Markdown only. Operators who want an HTML dashboard can render the JSON externally.

Alternatives considered

"Just use vLLM's benchmark harness"

Rejected. vLLM's harness reports throughput and TTFT, neither of which shows Meridian's value. We would have to extend it to track TTOT and phase-differentiated latencies — at which point we have already built this harness, but with a tight coupling to vLLM's internal benchmark abstractions.

MLPerf-style reference workloads

Considered, deferred. MLPerf targets raw throughput and aggregate API-level latency, not phase-differentiated metrics. A MLPerf-compatible reporting mode could be added if there is downstream demand, but it does not fit the primary signal Meridian optimises for.

Per-token wall-clock tracing

Considered (would record every decode-step timestamp and reconstruct the queue depths after the fact). Rejected because it generates GiB of trace data per run for marginal additional signal — the aggregated percentiles are enough to identify regressions and the OpenTelemetry spans give the deep-dive when needed.

References

• Playbook §5 — target metrics table.
• ADR-0001 — dual-queue scheduling, the design this benchmark validates.
• ADR-0004 — KV tier policy, the design that output_critical_eviction_events directly measures.

ADR-0006: Disaggregated KV transfer protocol

• Status: Accepted
• Date: 2026-05-20
• Authors: angelnicolasc
• Reviewers: sole-maintainer decision record

Context

The current frontier of LLM serving infrastructure has settled on prefill-decode disaggregation: prompt-processing (compute-bound) and token-decoding (memory-bandwidth-bound) run on separate worker pools and exchange KV blocks across a high-bandwidth fabric. NVIDIA NIXL is the CUDA-blessed reference; Mooncake (Moonshot, ASPLOS '25) is the open-source protocol that the rest of the ecosystem implements.

Meridian's three-tier block manager already produces the exact signal a disaggregated pool wants — ThinkComplete blocks at ExitThink are known-cold. A scheduler with phase visibility is the natural producer of well-batched, well-timed offload events: no other layer in the stack knows that right now, this request just finished reasoning and won't re-read its think KV.

The remaining question is how a phase-aware scheduler talks to the fabric. We need a wire format and a set of trigger points that work across NIXL and Mooncake (and any future fabric) without forcing Meridian to depend on a specific runtime.

Constraints:

• The fabric layer cannot live inside meridian-core — that crate is forbid(unsafe_code) and has no transitive CUDA dependency. It must live in meridian-kernels behind a cargo feature so non-disagg deployments pay nothing.
• The wire format must be readable from Python and C++ NIXL agents, not just from Rust. NIXL's reference implementation is C/C++ + Python bindings; Mooncake's reference implementation is C++ + Python.
• A meridian deployment that wants disagg but doesn't have a libnixl runtime available must still be exercisable end-to-end — otherwise the integration is impossible to test without specialised hardware.

Decision

Meridian defines a small versioned wire protocol (MRDN v1) for disaggregated KV transfer, implements it behind the nixl cargo feature of meridian-kernels, and exposes the trigger points through the BlockManager trait.

Wire format

+---------------+---------------+---------------+---------------+
| magic (4)     | version (4)   | body_len (4)  | tier (1)+pad(3)|
+---------------+---------------+---------------+---------------+
| checksum (16, Blake3-128)                                     |
+---------------+---------------+---------------+---------------+
| body (body_len bytes — opaque to Meridian; NIXL/Mooncake      |
| interpret as raw KV bytes)                                    |
+---------------+---------------+---------------+---------------+

• magic = b"MRDN" — fail-fast on misrouted payloads.
• version = 1 — incremented on any breaking framing change. We commit to preserving v1 across all 0.x releases.
• tier — the producer's tier label (ThinkComplete | ThinkActive | OutputCritical). The consumer may ingest into a different tier; the field exists for telemetry and for fabric-side admission policy.
• checksum — Blake3 of the body, truncated to 128 bits. Detects bit flips on RDMA and silent corruption inside fabric staging buffers.

The header is exactly 32 bytes. Body is opaque to the protocol — NIXL treats it as a raw KV slab; Mooncake adds its own framing inside.

Trigger points

• ExitThink — the producer's natural offload window. The scheduler batches the request's ThinkComplete blocks and pushes them to the fabric in a single shot, amortised by disagg.offload_threshold_blocks.
• OutputCritical allocation pressure — if the local pool is thrashing OutputCritical (a user-visible degradation event), the scheduler may pull blocks back from the fabric to satisfy allocations. The ingest_block hook is implemented; the automatic pull policy under allocation pressure is deferred pending measured offload latency data to calibrate the threshold.

Fabric trait

#![allow(unused)]
fn main() {
pub trait Fabric: Send + Sync + std::fmt::Debug {
    fn push(&self, payload: Vec<u8>) -> Result<u64>;
    fn pull(&self, handle: u64) -> Result<Vec<u8>>;
    fn label(&self) -> &'static str;
}
}

Shipped implementations:

• SyntheticNixlFabric — in-process keyed map, wire-format-identical to a real NIXL agent on the host side. Used for integration tests and for portfolio deployments where libnixl isn't reachable.
• Real libnixl FFI — gated on nvidia-nixl-sys becoming available on crates.io. The call sites already speak the protocol; switching swaps the Fabric implementation only.

Mooncake compatibility is achieved by writing a MooncakeAdapter: Fabric that re-frames the v1 wire body inside Mooncake's transport. The header survives unchanged, so an end-to-end conversation between a Meridian producer and a Mooncake-only consumer is a one-adapter delta.

Consequences

Positive

• A single wire format covers two ecosystems (NIXL + Mooncake) and is forward-compatible because version is on the wire.
• Checksum-on-body catches silent corruption — the kind of incident that takes 48 h to diagnose in a heterogeneous fabric.
• The BlockManager trait gains offload_block / ingest_block / block_location as non-breaking additions: the default impls return DisaggUnavailable for offload/ingest and Local for location, so existing implementations keep working.
• Portfolio deployments can demonstrate the full disagg path without GPU hardware, via the synthetic fabric.

Negative / risks

• Adding Vec<u8> allocations on every offload contradicts the zero-allocation discipline of the router hot path. Mitigation: the offload path runs at ExitThink, which is at most once per request — well off the per-token critical path.
• The 32-byte header is overhead for blocks that may be only a few KiB each. At a typical 16 KiB block this is 0.2% — acceptable.
• The synthetic fabric is not a substitute for real NIXL benchmarks. Anyone reading a synthetic-fabric A/B chart should know what they're reading. We label the fabric as "nixl-synth" in all telemetry to make this unambiguous.

Neutral

• The version field commits us to a backward-compatibility plan once v2 ships. ADR-0007 documents the policy.

Alternatives considered

gRPC point-to-point. Considered briefly. Rejected because every KV transfer would pay HTTP/2 framing overhead on top of the actual payload, and the standard NIXL/Mooncake clients don't speak gRPC. Our producer would be the odd one out.

RDMA-only, no framing. Considered. Rejected because RDMA without framing requires every consumer to know the producer's exact tier and checksum convention out-of-band. The 32-byte header buys us self-describing payloads at a 0.2% overhead — strictly worth it.

Per-block streams over Mooncake without our own header. Considered. Rejected because we'd lose the tier and version fields, which makes mixed-version cluster rollouts dangerous: a producer at v1 talking to a consumer at v0 must fail fast, not silently mis-tier blocks.

References

• NIXL technical brief, NVIDIA Developer Blog, March 2026.
• Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot, ASPLOS '25.
• Blake3 specification, https://github.com/BLAKE3-team/BLAKE3-specs.
• Meridian playbook §6 (Disaggregation outlook).
• ADR-0004 (KV tier promotion policy) — describes why ThinkComplete is a natural offload candidate.

ADR-0007: Release and versioning policy

• Status: Accepted
• Date: 2026-05-20
• Authors: angelnicolasc
• Reviewers: sole-maintainer decision record

Context

Meridian ships three artefacts on three different cadences:

• The meridian-core and meridian-kernels crates → crates.io.
• The meridian Python package (the vLLM plugin) → PyPI.
• The mdBook → GitHub Pages.

A release that bumps any of them must keep the others coherent — breaking the Python plugin while leaving meridian-core stable would strand operators on a half-upgraded stack. We need an explicit policy for what triggers a release, which artefacts move together, and how breaking changes are signalled — both for the pre-1.0 phase (today) and for the post-1.0 phase (after a year of production use).

We also need to commit to a provenance and SBOM story so the project is auditable end-to-end. SLSA Level 2 is the bar reasonable consumers expect in 2026.

Decision

SemVer interpretation

• Post-1.0 — strict SemVer. Breaking changes require a major bump; additive changes require a minor bump; bugfixes are patches.
• Pre-1.0 — breaking changes ship as minor bumps. The CHANGELOG entry for every minor bump must list every breaking change under a BREAKING CHANGE: footer. Operators reading the CHANGELOG before upgrading get a complete diff with no surprises.

Release cadence

• Minor: every six weeks. The window is fixed; the content is whatever passes CI plus the ADRs accepted in that window.
• Patch: on demand for security fixes and high-severity bugs. No fixed cadence — patches ship within 48 h of a confirmed report for SEV-1 (CVSS ≥ 7.0), per the security policy.

Branch policy

• main is always-releasable. Nothing merges that breaks CI.
• No release branches. release-plz drives the changelog and tag from main directly.
• Hotfixes are commits on main followed by a patch tag. We do not back-port to N-1 minors during pre-1.0 — operators on an old pre-1.0 minor are expected to upgrade forward.

Artefact set per release

Every tagged release ships:

• Compiled Rust static libs and the meridian Python wheel — built and uploaded as GitHub release assets by release.yml.
• CycloneDX SBOM for the Rust workspace and the Python wheel, attached to the GitHub release.
• SLSA Level 2 provenance attestation, generated by slsa-github-generator and uploaded as a release asset.
• A signed Git tag (when GPG is configured) and an annotated tag otherwise.
• mdBook deploy → GitHub Pages (via docs.yml on push to main).

Note on crates.io and PyPI publishing: release.yml carries a manual (workflow_dispatch) publish job. It defaults to dry-run — cargo publish --dry-run for the crates plus twine check on the wheel — so packaging is validated on every invocation without pushing anything. Selecting publish_mode = live performs the real publish, but each live step is token-gated: it runs only when CARGO_REGISTRY_TOKEN (crates.io) and PYPI_API_TOKEN (PyPI) are present in repo secrets. Until those are configured the job is safe to run and simply validates. This keeps publish off the automatic tag path while the project stabilises pre-1.0.

The three artefacts (crates, wheel, mdBook) move together. A release with a partial set is a CI failure, not a partial release.

Version coupling

The policy is that meridian-core, meridian-kernels, and the meridian Python package carry the same version string. release-plz enforces this for the Rust side via Cargo workspace inheritance.

Current state of automation as of v0.1.0: python/pyproject.toml carries a hardcoded version = "0.1.0" that must be bumped manually in sync with the Cargo workspace. A build hook to read the workspace version automatically is planned but not yet wired. Operators upgrading the Rust workspace must also update python/pyproject.toml until that hook is in place.

Yanking

We will yank a crates.io publish only for a security incident or a correctness regression with no workaround. Style fixes, doc errors and "meh, the version should have been a minor" do not justify yanking — those get a forward fix in the next release.

Consequences

Positive

• Operators reading the CHANGELOG ahead of an upgrade get a complete picture of what's breaking — no surprises during pre-1.0.
• One number versions the entire stack. Cross-artefact compatibility questions ("does the wheel work with crate X.Y.Z?") become trivially answerable.
• SLSA L2 provenance + SBOM at every release make Meridian a credible citizen of supply-chain-conscious deployments.
• A six-week minor cadence is short enough to feel responsive but long enough that operators don't get release fatigue.

Negative / risks

• A monolithic version bump means even a docs-only release moves every artefact. Cargo wastes a publish; PyPI wastes a wheel. The cost is real but small — well below the cost of decoupled versions diverging during an incident.
• The fixed six-week cadence will sometimes ship "nothing material". We accept that; consistency beats hoarding.

Neutral

• The policy is enforced by release-plz config plus the CI release workflow. There is no human gate between a green CI on main and a tagged release.

Alternatives considered

Independent versions per artefact. Considered for the parity it gives with how Cargo and PyPI usually work in larger projects. Rejected because the cross-artefact compatibility matrix becomes a second README, and we are not large enough yet to justify the overhead.

Release on every merge to main. Considered as a continuous-release model. Rejected because it would publish to crates.io and PyPI dozens of times a week, polluting the index and triggering downstream Dependabot noise for every contributor.

Long-lived release branches per minor. Considered for the back-port story it enables. Rejected pre-1.0 because we explicitly do not support N-1; operators are expected to upgrade forward.

References

• SemVer 2.0.0.
• Keep a Changelog.
• SLSA Specification.
• release-plz — drives the changelog and tag.
• CycloneDX — SBOM format.
• ADR-0006 (Disagg KV transfer) — wire format version field references this policy.
• SECURITY.md — disclosure SLAs.

ADR-0008: Request preemption policy

• Status: Accepted
• Date: 2026-05-20
• Authors: angelnicolasc
• Reviewers: sole-maintainer decision record

Context

Meridian reorders the batch vLLM hands it so output-phase requests are dispatched ahead of think-phase requests (ADR-0001). Reordering is a non-destructive operation: it changes the rank of requests that vLLM has already decided to run this step, but it never removes a request from the running set and never reclaims KV from a request mid-flight.

The next lever — and the one every mature serving scheduler eventually reaches for — is preemption: evicting a request that is already dispatched so its KV blocks can be reused by a higher-priority request. vLLM has its own preemption (recompute / swap), but it is phase-blind: it preempts on global memory pressure without knowing that a think-phase request holding 14 GB of KV is a far better victim than an output-phase request streaming to a waiting user.

A phase-aware preemption policy is therefore a natural Meridian feature. The question this ADR answers is not "how do we build it" in isolation, but "do we build it before 1.0, and if not, what is the design and the risk that justify deferring it". Shipping preemption is the single highest-risk change available to the scheduler: it can deadlock, it can thrash, and it can corrupt a request's KV if the reconstruction path is wrong. A wrong preemption decision is user-visible as a stalled or restarted generation.

Decision

Meridian will not preempt already-dispatched requests before 1.0. The plugin's influence on vLLM remains advisory — reordering and budget forcing only. This ADR records the intended design and its risk analysis so the deferral is a deliberate, documented choice rather than a gap.

Intended design (post-1.0)

When preemption is implemented it will follow this shape:

• Victim selection by phase, then LRU. The victim search walks the block manager's tier order — ThinkComplete first, then ThinkActive, and OutputCritical only under the same severe-pressure warning that evict_for already emits. Within a tier, the least-recently-touched request is the victim. This reuses the existing eviction ordering rather than introducing a second policy.
• Recompute, not swap, as the default reclamation path. A preempted think-phase request is cheaper to recompute from its prompt than to swap its KV to host and back, because think-phase KV is exactly the data we are willing to discard (ADR-0004). Swap remains available for output-phase victims, which must never lose progress.
• A preemption budget. At most a configurable fraction of the running set may be preempted per scheduler step (preempt_max_fraction, default small). This caps thrash: a pressure spike cannot evict the whole batch in one step.
• A re-admission guard. A preempted request is parked with a monotonically increasing priority floor so it cannot be preempted again immediately after re-admission. This is the anti-livelock invariant.
• Disagg interaction. When a fabric is configured (ADR-0006), a ThinkComplete victim is offloaded rather than discarded, so its KV is recoverable from the fabric instead of recomputed. Preemption and offload share the same victim search.

API shape (sketch, not committed)

The scheduler would gain a single entry point that returns victims for the caller to actuate against vLLM, keeping Meridian advisory rather than reaching into vLLM's running set directly:

fn select_preemption_victims(
    &self,
    needed_bytes: u64,
    running: &[RequestId],
) -> Vec<PreemptionVictim>   // { req_id, reason, reclaim: Recompute | Swap | Offload }

The plugin translates each victim into the vLLM preemption call appropriate for that vLLM version, isolated in the same _extract_* / _reorder shim layer that already absorbs vLLM API drift.

Consequences

Positive

• The pre-1.0 scheduler stays advisory and therefore safe: the worst case of a wrong Meridian decision is a sub-optimal dispatch order, never a lost or corrupted generation.
• The design is written down, so when preemption lands it starts from a reviewed risk analysis rather than a blank page.
• Victim selection reuses the existing tier ordering and disagg offload path, so the eventual implementation is additive, not a rewrite.

Negative / risks (the reason for deferral)

• Without phase-aware preemption, Meridian leaves throughput on the table under heavy memory pressure: vLLM's phase-blind preemption will sometimes evict an output-phase request when a think-phase victim was available. This is the cost we accept pre-1.0.
• The risks that justify waiting:
  • Deadlock: a preempted request needs memory to be re-admitted that only its own preemption could free. Mitigated by the recompute default and the re-admission priority floor — both unproven until implemented.
  • Thrash / livelock: oscillating preempt/re-admit under sustained pressure. Mitigated by the per-step preemption budget.
  • KV correctness: a swap-and-restore bug silently corrupts a request's context. This needs a dedicated correctness harness on real hardware before it can ship — which is precisely the validation we do not yet have.

Neutral

• This ADR is revisited once Meridian has a real-hardware benchmark baseline. Preemption that cannot be measured against stock vLLM under memory pressure cannot be justified, so the work is gated on that measurement capability existing first.

Alternatives considered

Implement preemption now, behind a default-off flag. Considered, so the code path exists for early adopters. Rejected: a default-off feature with no real-hardware correctness harness is untested code that rots, and the risk analysis above shows the failure modes are the kind that only surface under real load.

Delegate entirely to vLLM's preemption forever. Considered as the permanent answer. Rejected as a permanent policy because phase-blind preemption contradicts Meridian's entire thesis; accepted as the pre-1.0 policy because the safety bar for taking over preemption is high.

References

• ADR-0001 (Dual-queue vs. priority weights) — the advisory-reordering baseline this ADR declines to extend pre-1.0.
• ADR-0004 (KV tier promotion policy) — the tier ordering victim selection reuses.
• ADR-0006 (Disagg KV transfer protocol) — the offload path a ThinkComplete victim takes when a fabric is configured.

ADR-0009: The phase-conditioning hook ships inert until measured

• Status: Accepted
• Date: 2026-08-07
• Authors: angelnicolasc
• Reviewers: sole-maintainer decision record

Context

DeepSpec, DeepSeek's open-source speculative-decoding toolkit, ships trained drafters for Qwen3. Its README states that each released checkpoint was trained on data its target model generated in non-thinking mode — while Qwen3 exposes an enable_thinking toggle that production reasoning traffic uses.

That is a documented distribution mismatch between how a drafter was trained and how it is used. It leads to a plausible hypothesis: draft acceptance is lower during a reasoning span than during the final answer, making speculative decoding's speedup silently phase-dependent for anyone using these checkpoints as shipped. Meridian already routes on exactly that phase boundary, so it is the natural place to act on the hypothesis.

Nobody has measured it. Doing so requires a GPU and is not currently scheduled. So we are implementing a scheduler behaviour whose motivating premise is unverified — which is a well-known way to ship a regression with a good story attached.

We also established something that is proved (see the companion note): for any deterministic drafter, single-step acceptance is bounded above by the target's mode mass, which is in turn bounded above by a function of the target's Shannon entropy — a quantity Meridian's probe already computes. Critically, that bound is one-sided: it can prove a draft is too deep, never that it is too shallow.

Three options were on the table:

1. Ship nothing until Phase 1 runs. Zero risk, zero progress; the code that would consume the measurement does not exist when the measurement arrives.
2. Ship the full phase-conditioning behaviour with plausible default rates. Maximum apparent completeness. The defaults would be invented, and an operator enabling the feature would be running an unvalidated policy that looks measured.
3. Ship the mechanism, gate the unproven half.

Decision

The hook ships with its unproven half gated, and the gate is enforced by the type system rather than by documentation.

Concretely:

• With an AcceptancePrior::Uncalibrated prior — the default, and the state the project ships in — the hook may only ever return a proposal length less than or equal to the operator's configured baseline. It acts solely on the proved, one-sided entropy ceiling.
• Raising draft depth, and conditioning on phase at all, requires an AcceptancePrior::Measured, which can only be constructed from a Provenance::Measured value.
• Every acceptance statistic carries a Provenance. The single function that promotes a statistic to a publishable claim, into_measured_claim, returns Error::SyntheticProvenance for anything synthetic.
• The TOML schema cannot express acceptance rates without the harness, checkpoint, target model, thinking-mode flag and date of the run that produced them. There is no field for a hand-tuned rate.

The invariant is pinned by tests at both levels: a unit test sweeping the full entropy range against both phases, and an integration test sweeping every synthetic fixture and baseline.

Consequences

Positive

• Merging cannot regress serving throughput on a hunch. The only behaviour change available to an uncalibrated deployment is a shallower draft, taken only where a theorem says the deeper draft cannot pay. Nothing about that depends on the hypothesis being true.
• The publication rule is mechanical. "Do not publish synthetic numbers as measurements" is a rule that gets broken under deadline pressure. Here it is a compile-time and run-time error. A synthetic trace cannot reach a paper without someone deleting code, which is a reviewable act.
• Phase 1 is a data-plug-in, not a project. The ledger, the statistic, the decision rule and the config path all exist and are tested. The experiment produces a JSONL file; everything downstream already runs.
• Enabling the feature is an honest signal. speculation.acceptance_prior being present in a config means someone measured something and said where. Its absence means nobody has.
• The entropy ceiling has standalone value. Corollary 5 holds regardless of how Phase 1 turns out. Even under a null result, capping draft depth at high-entropy steps remains correct.

Negative / risks

• The shipped default is close to a no-op. With use_entropy_ceiling = true it trims depth only where entropy is genuinely high; with it false it does nothing at all. Someone evaluating the module on behaviour alone will find little happening. That is the intended trade and it is documented at every entry point, but it is a real cost in perceived value.
• The cost model is unvalidated. γ* is chosen by maximising a throughput proxy whose three cost constants are placeholders. A deployment with a very different cost profile could see the ceiling bind at the wrong depth. Detection: meridian.speculation.proposal_len{phase} diverging from baseline_proposal_len far more or less often than an operator expects. Mitigation: the constants are configurable and documented as requiring profiling.
• τ(a, γ) assumes i.i.d. per-position acceptance, which is false. Real drafters decay along the block. The model will place γ* slightly too deep. Detection: Phase 1's per-position acceptance rates. Mitigation: the ceiling is one-sided, so the error is bounded by the baseline clamp while uncalibrated.
• A calibrated hook inherits its prior's staleness. Acceptance rates measured against one checkpoint and target do not transfer to another. Detection: the prior carries draft_checkpoint and target_model; a deployment serving something else is visibly mismatched. Mitigation: none automated. This is a known gap, not a solved problem.

Neutral

• SpeculationPhase duplicates part of ThinkPhase rather than reusing it. The speculation path needs the phase label and not the entropy accumulators, and From<&ThinkPhase> keeps them in sync at one site.
• The hook is stateless. Per-request state stays in PhaseRouter, which already owns it.

Alternatives considered

Ship nothing until Phase 1 runs

Rejected. The structural work — the formal comparison, the proved bounds, the statistic, the harness gap analysis — is complete, CPU-only, and independently useful. Withholding it buys no safety, because none of it changes runtime behaviour, and it guarantees that when the measurement arrives, nothing exists to consume it.

Ship full phase conditioning with plausible defaults

Rejected, and this was the tempting option. Numbers like "think 0.45, output 0.90" would have made the module look finished and demoed well. They would also have been invented, and an operator enabling the feature would have been running an unvalidated policy indistinguishable, from the outside, from a measured one.

The failure is not that the guesses might be wrong — it is that nothing downstream could tell they were guesses. Provenance tagging exists specifically so that difference is legible.

Gate by a runtime flag instead of the type system

Rejected. A boolean allow_uncalibrated_phase_conditioning is the same rule enforced by convention, and conventions decay. Making the calibrated path unreachable without a Provenance::Measured value means the gate cannot be bypassed by setting a flag under pressure — it has to be bypassed by editing code, in a diff a reviewer will see.

Let the ledger emit measurements regardless of provenance

Rejected for the same reason, one layer up. The ledger does compute the full statistic on synthetic data — hiding the number would make the fixtures untestable. What it refuses is the promotion step. Analysis stays cheap; publication stays gated.

References

• Phase-conditioned speculative decoding — the formal comparison and the five results this ADR rests on.
• DeepSpec harness instrumentation gap — MER-P0.4.
• Phase 1 protocol — the deferred experiment.
• DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation, arXiv:2607.05147.
• ADR-0007 — release and versioning policy; [speculation] is additive and defaults off, so this is a minor-version change.

ADR-NNNN: Short kebab title

• Status: Proposed | Accepted | Superseded by ADR-NNNN | Deprecated
• Date: YYYY-MM-DD
• Authors: name
• Reviewers: name, name

Context

What is the problem we are facing? What constraints apply? What evidence do we have (benchmarks, incident postmortems, prior art)?

Decision

What did we decide? State it as a single declarative sentence at the top, then expand.

Consequences

Positive

• Bullet list of what we gain.

Negative / risks

• Bullet list of what we lose, and how we will detect each risk if it materializes.

Neutral

• Things that change but are neither clearly positive nor negative.

Alternatives considered

For each alternative: one paragraph on what it would look like, and why we rejected it.

References

• Links to research papers, prior ADRs, incident reports, benchmarks.

Phase-Conditioned Speculative Decoding

Read this first. This note contains no measured acceptance rate, and the code it describes cannot produce one. It reports a documented mismatch, a proof relating two previously-unconnected signals, and a scheduler hook built so it cannot act on the unproven part. The empirical question is stated precisely and left open. See What I am not claiming.

1. The three-way split

Everything below sorts into exactly one of these rows. Keeping them apart is the point of the note.

2. The documented mismatch

DeepSpec is DeepSeek's open-source toolkit for training and evaluating speculative-decoding drafters. It ships ready-to-use checkpoints — Eagle3, DFlash and DSpark variants — for Qwen/Qwen3-4B, Qwen3-8B, Qwen3-14B and google/gemma-4-12B-it.

Its README states that each checkpoint was trained on open-perfectblend data generated by its corresponding target model in non-thinking mode.

Qwen3 exposes an explicit enable_thinking toggle that produces a delimited reasoning span before the final answer. Production traffic for a reasoning model runs with it on. The drafters were trained with it off.

That is a distribution mismatch between how a drafter was trained and how the target it drafts for is actually used. It is documented, not inferred — which is what makes it a usable foundation rather than a hunch.

It is also not yet a defect. Whether the mismatch costs anything depends on whether the two phases are statistically different enough for a drafter to notice, and nobody has published a phase-segmented measurement. That is the gap this work is aimed at.

3. What DSpark's confidence head computes (MER-P0.1)

DSpark pairs a parallel draft backbone with a lightweight sequential head, and attaches a confidence head that scores each drafted position's chance of surviving verification. For drafted position k:

c_k = σ( wᵀ [ h_k ; W₁[x_{k-1}] ] )                        (DSpark Eq. 7)

where h_k is the backbone hidden state and W₁[x_{k-1}] a Markov embedding of the previously drafted token. c_k ∈ (0,1) is the probability that position k survives given every preceding draft token was accepted, so prefix survival is the cumulative product S_k = ∏_{i ≤ k} c_i.

The interesting part is the supervision signal. The head is not trained against sampled accept/reject outcomes; it regresses an analytical target:

c*_k = 1 - ½ ‖ pᵈ_k - pᵗ_k ‖₁                              (DSpark Eq. 8)

½‖·‖₁ is total variation distance, so c* = 1 - TV(pᵈ, pᵗ) — exactly the single-step acceptance probability of speculative sampling under the standard accept/reject rule. Training is position-weighted binary cross-entropy against that target (Eq. 11), followed by sequential temperature scaling that minimises expected calibration error of the cumulative product, one position at a time.

Downstream, a prefix scheduler chooses verification depth per request by maximising throughput Θ = τ · SPS(B): accepted tokens per cycle times steps-per-second at the resulting batch size.

The one line that matters for what follows: the confidence head is a learned estimator of 1 - TV(pᵈ, pᵗ).

4. What Meridian's EAT/RPDI signal computes (MER-P0.2)

Meridian's entropy probe reads the target's predictive distribution pᵗ at each decode step and emits EntropySignal:

e_k = Σ_{v ∈ E} pᵗ_k(v)          EAT: mass on the think-end token set E
H_k = -Σ_v pᵗ_k(v) ln pᵗ_k(v)    Shannon entropy, nats

e_k is smoothed into μ_k = α e_k + (1-α) μ_{k-1}, with a companion second-moment EMA so Var[e] ≈ ν_k - μ_k² is available without keeping a sample window. RPDI compares a short-window EMA of the transition indicator 1[H_k > θ] against its running global mean. Both feed one decision: when to force </think>.

The one line that matters: EAT is the target's probability mass on a designated token set.

5. Where the two signals actually meet (MER-P0.3)

Both quantities are functionals of the same object. That much is obvious. The non-obvious part is that they are the same functional, evaluated for different reasons.

flowchart LR
    PT["target predictive<br/>distribution pᵗ"]
    PD["draft distribution pᵈ"]

    PT --> EAT["Meridian EAT<br/>e = Σ_E pᵗ(v)"]
    PT --> H["Meridian entropy<br/>H = -Σ pᵗ ln pᵗ"]
    PT --> CSTAR["DSpark target<br/>c* = 1 - TV(pᵈ, pᵗ)"]
    PD --> CSTAR

    EAT -->|"Prop. 1 + Cor. 2<br/>identity"| CSTAR
    H -->|"Prop. 4 + Cor. 5<br/>ceiling"| CSTAR

Every result below has a test in confidence_model.rs.

Proposition 1 — point-mass identity

For a deterministic drafter pᵈ = δ_x̂, the analytical acceptance rate is the target's mass on the drafted token: c* = pᵗ(x̂).

Proof. ‖δ_x̂ - pᵗ‖₁ = (1 - pᵗ(x̂)) + Σ_{v≠x̂} pᵗ(v) = 2(1 - pᵗ(x̂)). Halve and subtract from one. ∎

Corollary 2 — EAT is an acceptance rate

Meridian's e_k is precisely the DSpark supervision target c*_k for the drafter that always proposes the think-end token.

This is the result the whole project hangs on. Meridian computes EAT to decide when a model has finished reasoning. DSpark computes c* to decide how deep to verify. They are the same number, for a particular drafter, arrived at from opposite directions.

Proposition 3 — EAT ceilings any boundary-supported drafter

If supp(pᵈ) ⊆ E, then c* ≤ e, with equality iff pᵈ(v) ≥ pᵗ(v) for all v ∈ E — which holds automatically when E is a singleton.

Proof. TV(pᵈ,pᵗ) = sup_A |pᵈ(A) - pᵗ(A)| ≥ pᵈ(E) - pᵗ(E) = 1 - e. ∎

Proposition 4 — entropy brackets the target mode

Write M = max_v pᵗ(v) and let V be the vocabulary size. Then

e^{-H}  ≤  M  ≤  p*(H, V)

where p* is the unique root on [1/V, 1] of H = H_b(p) + (1-p)·ln(V-1).

Proof. Lower: H = -Σ p ln p ≥ -Σ p ln M = -ln M. Upper: the right-hand side is the maximum entropy attainable by any distribution over V symbols whose largest mass is p — a spike of p over a uniform tail — and it is strictly decreasing in p. For the observed H to be feasible, M ≤ p*. ∎

Corollary 5 — the actionable one

Every deterministic drafter satisfies c* = pᵗ(x̂) ≤ M ≤ p*(H, V).

High target entropy at a step is a proof that no deterministic drafter can be accepted often there — regardless of which drafter is deployed, computable from a signal Meridian already has, with no access to the drafter at all.

Note the direction carefully. This is a ceiling, so it can only ever justify drafting less. It can never justify drafting more. That asymmetry is the safety property the hook in §6 is built on.

The bracket, numerically

For Qwen3's vocabulary (V = 151,936, ln V = 11.93):

The ceiling is worth having: at H = 8 it proves no deterministic drafter exceeds 39 % single-step acceptance, which is enough to cut draft depth from seven to two on the cost model below. The floor is much weaker and is retained only as a diagnostic — it describes headroom for a hypothetical target-aware drafter, not a guarantee for a deployed one.

6. What this licenses in the scheduler

c* depends on both pᵈ and pᵗ. Every Meridian signal depends on pᵗ alone. So Meridian can bound draft acceptance from what it already computes, and can never predict it without running the drafter.

That single sentence determines the entire design of the shipped hook:

• Uncalibrated — the state it ships in, and the state it stays in until someone runs Phase 1 — the hook may only shrink the operator's configured draft depth, and only where Corollary 5 proves a deeper draft cannot pay off.
• Calibrated — only reachable by supplying measured per-phase acceptance rates together with the provenance of the run that produced them — the hook may also raise depth, and conditions on phase.

The depth choice maximises a throughput proxy with the same shape as DSpark's objective, minus the hardware cost table:

Θ(γ) = τ(a, γ) / (draft_us·γ + verify_fixed_us + verify_token_us·γ)
τ(a, γ) = (1 - a^{γ+1}) / (1 - a)

τ assumes i.i.d. per-position acceptance, which is the weakest link in the model: real drafters decay along the block, which is exactly why DSpark has a sequential head and why DeepSpec reports accept_rates_by_position. τ is a first-order planner, not a claim about measured behaviour.

Provenance is enforced by the type system rather than by discipline. A synthetic trace cannot become a published claim without deleting code:

#![allow(unused)]
fn main() {
let report = ledger.report();          // computes fine on synthetic data
assert!(report.verdict(0.0) == HypothesisVerdict::Supported);
assert!(report.into_measured_claim().is_err());   // ... but cannot be promoted
}

Configuration is gated the same way: there is no way to write acceptance rates into meridian.toml without also naming the harness, checkpoint, target model, thinking-mode flag and date of the run that produced them.

7. What shipped

8. What I am not claiming

Following the precedent this project set for itself:

• I have not measured a phase-dependent acceptance gap. No number in this note, the code, or the test suite is a measurement. The dspark_bridge module is deliberately incapable of emitting one without hardware in the loop.
• I have not shown the mismatch costs anything. A drafter trained on non-thinking text may generalise to thinking text perfectly well. The hypothesis is falsifiable and a null result is a publishable outcome.
• I have not shown Meridian's signals can predict DSpark acceptance. §6 says precisely the opposite: they bound it, one-sidedly, and the shipped hook is restricted to that one side.
• The τ model is not validated. The i.i.d. assumption is known to be wrong in the direction that matters (suffix decay). It plans; it does not describe.
• The cost-model defaults are placeholders. draft_token_us, verify_fixed_us and verify_token_us produce sane relative ordering. They are not a measurement of any hardware and are documented as such.
• The straddle problem is unresolved by data. A verification step commits several tokens, so a step can span the </think> boundary. The ledger makes the attribution policy explicit and reports the straddle rate, but which policy is right is an empirical question Phase 1 has to answer.

9. References

• Cheng, Yu, Shao et al. DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation. DeepSeek-AI and Peking University. arXiv:2607.05147
• deepseek-ai/DeepSpec — training and evaluation toolkit. MIT licensed; NOTICE records Apache-2.0 portions adapted from SpecForge and MIT-licensed design input from DFlash. See the instrumentation note for which files that affects.
• Leviathan, Kalman, Matias. Fast Inference from Transformers via Speculative Decoding. ICML 2023 — origin of the τ(a, γ) expected-accepted-length result.
• Qwen3 model family documentation — enable_thinking chat-template toggle.
• Meridian, "Reasoning models emit two workloads. Your scheduler sees one." — the thesis this note extends one level down the stack.

External facts were verified against live sources on 2026-08-07. Re-verify before republication.

DeepSpec Harness: Phase-Segmentation Gap Analysis

The question this note answers: what exactly has to change in DeepSpec's evaluation harness to measure draft acceptance separately for the think and output phases of the same response?

The answer is smaller than expected, and the reason is worth stating up front.

The decode loop already records everything needed. The information is destroyed at the aggregation boundary, one function later.

1. What the harness already does

Speculative decoding runs in generate_decoding_sample (deepspec/eval/base_evaluator.py). Each iteration of its while start < max_length loop appends to three parallel per-step lists:

The loop advances start by exactly acceptance_lengths[i] each iteration, and the terminal iteration appends accepted_draft_tokens without the bonus token and advances by that.

All three lists, plus the full output_ids tensor and num_input_tokens, are returned on the per-response SimpleNamespace.

Consequence: the absolute sequence position of every verification step is exactly reconstructible from the returned payload, with no new instrumentation inside the decode loop:

start_i = num_input_tokens + sum(acceptance_lengths[:i])
committed_span_i = output_ids[0, start_i + 1 : start_i + 1 + acceptance_lengths[i]]

Locate </think> once in output_ids and every step's phase follows.

2. Where the information is lost

BaseEvaluator.allreduce_response_metrics folds the per-step lists into five scalars and two per-position histograms:

metric_summary["proposal_count"]       += 1
metric_summary["acceptance_length_sum"] += int(acceptance_length)
metric_summary["proposal_length_sum"]   += int(proposal_length)
proposals_at_pos[pos_idx] += 1
accepted_at_pos[pos_idx]  += 1

These are what dist.all_reduce moves across ranks and what build_metrics_row turns into the reported acceptance_length, draft_tokens_per_proposal, verify_rate and accept_rates_by_position.

The per-step lists are never written to disk and never leave the process. By the time anything is persisted, position information is gone.

This is the gap. It is not a missing measurement — it is a discarded one.

3. The blocking issue: thinking mode is hard-coded off

BaseEvaluator.run_dataset builds its prompts with:

input_ids = encode_chat_messages(
    self.tokenizer,
    messages,
    add_generation_prompt=True,
    enable_thinking=False,
    # enable_thinking=True,
)

enable_thinking=False, with the True variant present as a commented-out line directly beneath it. encode_chat_messages (deepspec/data/parser.py) forwards the flag into tokenizer.apply_chat_template only when it is not None, so the plumbing is complete — the value is simply pinned, and not exposed as a CLI argument.

This is consistent with the README's statement that the released checkpoints were trained on non-thinking-mode generations: the evaluation matches the training distribution. It also means that as shipped, the harness cannot produce a thinking-mode trace at all, which has to be fixed before anything else matters.

4. The straddle problem

A verification step commits up to γ + 1 tokens at once. The released DSpark Qwen3 checkpoints are block7, so a step commits up to eight tokens. The </think> boundary will usually fall inside a committed span rather than on a step edge, making that step partly reasoning and partly output.

Such a step belongs to neither phase. There are three defensible policies — exclude it, attribute it to think, attribute it to output — and the choice is an empirical question, not a design one. The consumer side (meridian_core::dspark_bridge::ledger) therefore implements all three, defaults to excluding, and reports the straddle rate on every report, because a high straddle rate means the segmentation is too coarse relative to the block size to be trusted at all.

At most one step per response can straddle, since the phase transition happens once. With block7 and reasoning spans in the hundreds to thousands of tokens, the expected straddle rate is well under 1 % — but it must be reported, not assumed.

5. Required changes, in full

Four changes. Nothing else is needed.

Change 3 is the only one that needs care, and only in one respect: it must happen per rank before the all-reduce, since the all-reduce is where the data dies.

Recording the </think> index rather than the whole output_ids tensor keeps the record small and avoids persisting generated text — see §7.

6. What does not need to change

Worth stating explicitly, because each of these was a candidate and each turned out to be unnecessary:

• No change inside the decode loop. generate_decoding_sample already records per-step acceptance. Touching the hot loop was the expected cost of this work; it is not needed.
• No new metric definitions. mean accepted length and verify_rate are computed per phase using DeepSpec's exact formulas, so the resulting numbers are directly comparable to its published tables: verify_rate = acceptance_length_sum / (proposal_length_sum + proposal_count).
• No change to the confidence-head calibration harness. deepspec/eval/dspark/confidence_head.py already computes per-position ECE, AUROC and Brier scores over cumulative-product predictions. Running it separately over think-phase and output-phase steps yields a second hypothesis for free (see Phase 1 protocol, H2) with no new metric code — only segmentation.
• No new evaluation datasets. gsm8k, math500, aime25, humaneval, mbpp, livecodebench, mt-bench, alpaca and arena-hard-v2 ship with the repository, and holding the dataset fixed while varying only the thinking-mode flag is exactly the confound control the experiment needs.

7. Licensing and redistribution

DeepSpec is MIT licensed. Its NOTICE records that portions are adapted from third-party projects under their own terms, and this matters for a patch:

• Apache-2.0 portions adapted from SpecForge cover the Eagle3 modeling, loss, optimizer, attention and evaluation code. The listed files include deepspec/eval/eagle3/evaluator.py, and the list is explicitly non-exhaustive — files that incorporate adapted code carry an in-file comment pointing upstream.
• MIT design input from DFlash informs the DSpark/DFlash configurations and modeling.

deepspec/eval/base_evaluator.py and eval.py — the two files this patch touches — are not in the enumerated SpecForge list. Because that list is non-exhaustive, check both files for an upstream-pointer comment before redistributing a modified copy, and if either carries one, retain the Apache-2.0 notice and state the modification, as §4 of that licence requires.

The cleanest route, and the recommended one, is to publish the patch as a diff against an upstream commit rather than as a redistributed fork, which sidesteps the question entirely.

Data handling: the JSONL record in change 3 deliberately stores the </think> index rather than the token stream. Generated text is not needed for the measurement, and not persisting it avoids redistributing model output derived from third-party evaluation prompts.

8. Residual risks

9. Verification

Everything in §1–§3 was read from the repository at main on 2026-08-07: eval.py, deepspec/eval/base_evaluator.py, deepspec/data/parser.py, deepspec/eval/dspark/confidence_head.py, LICENSE and NOTICE. Line-level claims describe that revision; re-verify before implementing, since the enable_thinking line in particular is the kind of thing that changes.

Phase 1 Experiment Protocol

This document exists so that Phase 1 requires no design decisions — only hardware and time. Everything below is fixed in advance: hypotheses, models, datasets, sample sizes, statistic, decision rule, and what gets published under each outcome. Fixing the decision rule before seeing data is the point; it is what makes a null result publishable rather than embarrassing.

1. Hypotheses

H1 — primary

Using a DeepSpec-released, non-thinking-trained DSpark checkpoint against a Qwen3 target running in thinking mode, mean accepted draft length during the think-phase span is lower than during the output-phase span of the same responses, and the gap exceeds the accepted-length variation already observed between task domains at a fixed configuration.

Directional and pre-registered. The between-domain clause is what makes it a claim about phase rather than a rediscovery of the domain sensitivity DeepSpec already reports.

H2 — secondary, free

The confidence head's calibration error is higher on think-phase positions than on output-phase positions.

DeepSpec's confidence_head.py already computes per-position ECE, AUROC and Brier scores. Segmenting its existing accumulators by phase costs no new metric code. If H1 holds, H2 tells you why it matters operationally: a drafter that is merely worse can be scheduled around, but a drafter whose confidence is miscalibrated breaks DSpark's prefix scheduler, which trusts ∏ c_i to choose verification depth. H2 is the more consequential of the two.

2. Design

A within-response, between-condition design.

• Within-response (primary): compare think-phase steps against output-phase steps of the same responses, thinking mode ON. Controls for prompt, dataset, sampling parameters and checkpoint — everything except phase.
• Between-condition (control): the same prompts with thinking mode OFF. Establishes the baseline the checkpoint was trained for, and detects a confound in which think-phase text is merely longer or later rather than distributionally different.

Factors

Domain is held fixed rather than pooled because DeepSpec's own results show accepted length varies substantially by domain. Pooling would let a domain effect masquerade as a phase effect — the single largest threat to the result's validity, and the reason the risk register flags it.

Fixed across all runs

Checkpoint (deepseek-ai/dspark_qwen3_4b_block7), target (Qwen/Qwen3-4B), temperature, seed schedule, max_proposal_tokens, and confidence threshold. Anything not listed as a factor is pinned.

3. Procedure

1. Apply the four changes in §5 of the gap analysis.
2. For each domain, run the harness twice — thinking on, thinking off — over the same prompt set with the same seed schedule.
3. Reconstruct per-step spans from the JSONL records by cumulative sum of acceptance_lengths; attribute each step to a phase against the </think> index; mark straddling steps.
4. Feed the observations into AcceptanceLedger, tagged Provenance::Measured { .. } with the harness commit, checkpoint, target, thinking-mode flag and date.
5. Run the report. Apply the §5 decision rule. Publish.

Step 4 is the only place a mistake is irreversible: a run recorded with the wrong thinking_mode flag is uninterpretable and has to be repeated. Everything else can be recomputed from the JSONL.

Sample size

Target ≥ 10,000 verification steps per arm per domain. At block7, a 1,000-token response yields roughly 150–300 steps, so 100 responses per domain per condition is comfortably sufficient.

This is not a power calculation, because a power calculation is not the binding constraint here — at 10⁴ steps per arm, Welch's test resolves differences far smaller than anything operationally interesting. Statistical significance will be easy and nearly meaningless. The effect size and the between-domain comparison are what decide the outcome, which is why the decision rule in §5 is not a p-value threshold.

4. Statistic

Welch's unequal-variance t-test on mean accepted length, think minus output. Welch rather than Student's because the two arms have neither equal variance nor equal size, and thinking mode changes both.

Reported per comparison:

Implemented in meridian_core::dspark_bridge::stats, verified against closed forms exact for df = 1 and df = 2 and against published critical values.

5. Decision rule

Fixed in advance, and implemented as PhaseGapReport::verdict(between_domain_gap):

between_domain_gap is computed from this experiment's own runs: the spread in mean accepted length across the three domains at a fixed configuration. It is not taken from DeepSpec's published tables, since those use a different metric aggregation and a different confidence threshold.

All five outcomes are published. That is a rule this project has already committed to and it is not renegotiable after seeing the data.

6. Cost

The 8B follow-up doubles the memory requirement and remains single-GPU.

7. Threats to validity

8. Feeding the result back

A measured report closes the loop with no glue code:

#![allow(unused)]
fn main() {
let prior = report.to_acceptance_prior()?;   // fails on synthetic data
let hook = PhaseConditioningHook::new(PhaseConditioningConfig {
    prior,
    ..config
})?;
}

or equivalently in meridian.toml:

# Values below are ILLUSTRATIVE placeholders showing the shape of the table.
# They are not measurements — no phase-segmented run exists.
[speculation.acceptance_prior]
think            = 0.42   # <- replace with the measured think-phase rate
output           = 0.88   # <- replace with the measured output-phase rate
harness          = "DeepSpec@<commit-sha>"
draft_checkpoint = "deepseek-ai/dspark_qwen3_4b_block7"
target_model     = "Qwen/Qwen3-4B"
thinking_mode    = true
recorded_on      = "<date>"

Supplying that section is what switches the hook from "may only reduce draft depth" to "conditions on phase". Until it exists, the hook stays inert in the upward direction by construction — see ADR-0009.

9. If H1 is falsified

The most likely single outcome, and worth planning for rather than dreading.

A null result would say something genuinely useful: that speculative drafters trained on non-thinking-mode text generalise across the thinking-mode boundary well enough that the documented training mismatch does not cost throughput. That is a finding practitioners can act on — it retires a plausible worry about every released Qwen3 drafter — and it is worth publishing on its own terms.

The structural results in the companion note are unaffected either way. Corollary 5 is a theorem; it does not depend on how this experiment turns out.

Metrics

Meridian emits Prometheus metrics and OpenTelemetry traces. All metric names are stable contracts — renames or semantic changes trigger a minor-version bump per ADR-0007.

Metric catalog

meridian.think_tokens_per_request

Operator action: if P99 frequently hits max_think_tokens, lower max_think_tokens or tighten eat_ema_variance_threshold / rpdi_threshold to fire forcing earlier.

meridian.budget_force_triggered

meridian.budget_force_reason{reason=...}

Operator action: monitor the ratio hard_cap / (converged + overthinking) over a 1-hour window. A ratio above 0.5 is a signal to investigate probe thresholds or inspect sample EAT traces.

meridian.output_critical_eviction

Operator action: alert at rate(...) > 0 sustained for 5 minutes. Mitigate by lowering think_phase_memory_fraction, think_batch_multiplier, or max_think_tokens.

meridian.phase_router.tracked_requests

Operator action: if the gauge grows without a corresponding growth in active concurrent requests, check that post_step receives EOS and that the reap period (60 s default) is shorter than request lifetime.

meridian.schedule_batch.duration_ns

Operator action: P99 above 1 ms under steady load is unexpected. File an issue with a CPU profile.

meridian.queue_depth{queue=...}

Operator action: alert when queue_depth{queue="think"} P95 exceeds 4× its 1-hour baseline for 5 consecutive minutes without accompanying forcing activity.

meridian.block_manager.used_bytes

Operator action: alert when block_manager.used_bytes / capacity_bytes exceeds 0.90 for 10 minutes — this is the early-warning threshold before output_critical_eviction events begin.

meridian.block_manager.evictions{tier=...}

Operator action: alert on any tier=output_critical increment — use this series or meridian.output_critical_eviction, whichever is easier to route in your alerting stack.

meridian.scheduler.batch_size{phase=...}

Operator action: compare scheduler.batch_size{phase=output} P50 against queue_depth{queue=output} to verify output requests are being served promptly.

meridian_disagg_blocks_offloaded_total{fabric=...}

meridian_vocab_fallback_total

meridian.speculation.policy_basis{phase,basis}

Operator action: until Phase 1 of the phase-conditioned-speculation work has run, a healthy deployment shows zero samples with a measured_* basis. Any measured_* traffic means someone installed a [speculation.acceptance_prior] — check it names a real run. See ADR-0009.

meridian.speculation.proposal_len{phase}

Operator action: if the histogram sits at min_proposal_len across both phases, the cost model constants are likely wrong for your hardware — profile and set draft_token_us / verify_fixed_us / verify_token_us. They ship as placeholders, not measurements.

meridian.speculation.accepted_length{phase}

meridian.speculation.straddling_steps

Operator action: compare against the total recorded steps. Above ~5 %, phase attribution is too coarse relative to the draft block size and any think-vs-output comparison drawn from it should be discarded rather than caveated.

OTLP export

Prometheus is the primary metric surface. When [telemetry] otlp_enabled = true (requires the otel extra), the plugin additionally exports its counters to an OTLP/HTTP collector at [telemetry] otlp_endpoint, and the Rust core can wire its tracing spans to OTLP via the otel crate feature (meridian_core::telemetry::install). Both are off by default.

Trace spans

Each MeridianScheduler::schedule_batch call opens a meridian.schedule_batch OpenTelemetry span. PhaseEvents propagate meridian.phase_event{kind=...} events on the active request's span, allowing per-request phase timelines to be reconstructed from trace data.

Alerting summary

Troubleshooting

Each entry follows runbook format: Symptom → Likely cause → How to verify → Immediate mitigation → Longer-term fix.

Output streams stutter under load

Symptom: users report visible gaps in the output token stream; output ITL P99 spikes.

Likely cause: OutputCritical KV blocks are being evicted under memory pressure.

How to verify:

rate(meridian.output_critical_eviction[1m]) > 0

Any non-zero rate confirms the block manager is evicting user-visible KV.

Immediate mitigation: reduce load. Lower the arrival rate or raise kv_memory.capacity_bytes if headroom exists.

Longer-term fix (pick one or more):

1. Lower kv_memory.think_phase_memory_fraction (e.g. 0.40 → 0.30) to give output more room.
2. Lower scheduler.think_batch_multiplier to reduce think-phase KV pressure.
3. Lower scheduler.max_think_tokens so individual reasoning chains release blocks sooner.

Budget forcing never fires; chains always hit hard cap

Symptom: meridian.budget_force_reason{reason=hard_cap} increments constantly; converged and overthinking stay at zero.

Likely cause: the entropy probe is not detecting convergence. Either the probe is disabled, the thresholds are too tight, or the model is being asked to reason on prompts that produce genuinely non-converging chains.

How to verify:

1. Check meridian.toml: confirm entropy.enabled = true.
2. Capture a sample EAT trace: add logging.level = "debug" temporarily and inspect EAT EMA values in the logs for a known-good reasoning prompt.

Immediate mitigation: none — hard-cap forcing is safe, just less adaptive.

Longer-term fix:

• If probe is disabled: set entropy.enabled = true.
• If thresholds are too tight: relax eat_ema_variance_threshold (e.g. 0.001 → 0.005) or lower rpdi_threshold (e.g. 3.0 → 2.0).
• If prompts are genuinely non-converging: lower max_think_tokens to bound KV cost.

Phase router shows runaway tracked-requests gauge

Symptom: meridian.phase_router.tracked_requests grows monotonically without a corresponding growth in active requests.

Likely cause: completed requests are not being reaped from the router. Either post_step is not receiving EOS events, or reap_stale_older_than is not being called.

How to verify:

1. Confirm the vLLM plugin's post_step is wired to the engine's step callback.
2. Check the reap interval in the plugin config — default 60 s. If request lifetime is shorter than 60 s on average, the reaper may be lagging.

Immediate mitigation: no manual reap trigger is exposed in v0.1.x. The plugin calls router.reap_stale_older_than(60.0) automatically every 64 schedule() invocations; under active serving load this fires within seconds. To force an immediate reap at deploy time, temporarily lower _reap_interval_schedules in vllm_plugin.py to 1.

Longer-term fix: reduce the reap period in vllm_plugin.py or ensure post_step receives every EOS signal from the vLLM worker.

CUDA kernel returns Unavailable

Sprint 0 note: python/meridian/_backends/cuda.py currently delegates to CpuEntropyBackend; KernelError::Unavailable cannot be triggered through the Python probe in v0.1.x. The scenario below applies once Sprint 1 wires the Python backend to the Rust kernel path in crates/meridian-kernels/.

Symptom: logs show KernelError::Unavailable; entropy probe silently falls back to CPU; meridian.budget_force_reason{reason=hard_cap} is the only firing signal.

Likely cause: the meridian-kernels crate was built without --features cuda, or libcudart.so is not on the dynamic linker path at runtime.

How to verify:

# Confirm the Python extension loads and report the current backend behaviour.
python -c "
from meridian._backends.cuda import CudaEntropyBackend
b = CudaEntropyBackend()
print('backend name:', b.name)
# Sprint 0: always prints 'cuda' but delegates to CPU internally.
# Real CUDA dispatch requires a --features cuda build (Sprint 1).
"

# Confirm the Rust kernels extension is importable.
python -c "import meridian._meridian; print('native extension OK')"

Immediate mitigation: the CPU fallback is correct — entropy values are identical. The only cost is CPU cycles and higher schedule_batch.duration_ns latency.

Longer-term fix:

1. Rebuild with cargo build -p meridian-kernels --features cuda.
2. Verify libcudart.so is on LD_LIBRARY_PATH or in /usr/local/cuda/lib64.
3. Run maturin develop --release -m crates/meridian-python/Cargo.toml to regenerate the Python extension.

Disagg offload not firing

Symptom: meridian.disagg.blocks_offloaded counter never increments after [disagg] enabled = true is set.

Likely cause: the fabric is not reachable, the offload_threshold_blocks has not been reached, or the NIXL feature was not compiled in.

How to verify:

1. Confirm cargo build -p meridian-kernels --features nixl succeeds.
2. Confirm config.disagg.fabric != "none".
3. Check whether ThinkComplete block count per step is below offload_threshold_blocks.

Immediate mitigation: lower offload_threshold_blocks to 1 temporarily to force immediate offload on every ExitThink.

Longer-term fix: verify fabric connectivity (NIXL service running, network path open) and restore offload_threshold_blocks to the production value.

Plugin does not intercept schedule_batch

Symptom: no Meridian metrics appear; vLLM appears to schedule normally without phase-awareness.

Likely cause: MeridianSchedulerPlugin.attach() was never called, or the plugin was attached to a different scheduler instance than the one the engine uses.

How to verify:

print(type(engine.scheduler[0]))
# Should print: <class 'meridian.vllm_plugin.MeridianSchedulerPlugin'>
# Not: <class 'vllm.core.scheduler.Scheduler'>

Immediate mitigation: call MeridianSchedulerPlugin.attach(engine, cfg, model_key="...") explicitly after engine construction. attach is a classmethod — it constructs the plugin and replaces engine.scheduler[0] in one step.

Longer-term fix: verify the plugin initialisation order in the serving entrypoint. The plugin must be attached after AsyncLLMEngine is fully constructed but before the first request is submitted.

Benchmarks

The benchmark harness lives at benchmarks/. The methodology behind metric choice is recorded in ADR-0005.

Quick start

# CI-friendly: no GPU, no vLLM. Drives native Meridian components over a
# synthetic decoder loop. Finishes in seconds.
uv --project python run python -m benchmarks.meridian_bench synthetic-replay \
    --duration-s 30 --arrival-rate 8 --reasoning-ratio 0.4 \
    --out-dir bench-out/

# GPU-required: drives a real AsyncLLMEngine via the Meridian plugin.
uv --project python run python -m benchmarks.meridian_bench real-vllm \
    --model Qwen/Qwen2.5-0.5B --duration-s 30 --arrival-rate 4 \
    --out-dir bench-out/

Both modes produce identically-shaped artefacts in --out-dir:

• report.json — full structured report, diffable.
• report.md — Markdown summary suitable for PR comments.

Metric catalog

See benchmarks/metrics.py for the exact serialised shape.

A/B comparison mode

--baseline runs the same workload through one or more baseline schedulers alongside Meridian and writes ab-report.{json,md} to --out-dir:

• stock — StockSchedulerBaseline, a priority-weight single-queue scheduler equivalent to vLLM ≤0.8 (no phase awareness, never forces budget).
• static-budget — StaticBudgetBaseline, a fixed think-token cap equivalent to vLLM 0.9's thinking_token_budget (forces </think> on a counter, with no entropy signal). This is the prior art Meridian's EAT/RPDI forcing aims to beat.
• all — run every baseline; the report gets one value column per run and a Δ% vs <baseline> column per baseline with a WIN/win/FLAT/loss/LOSS flag.

Five-minute A/B (no GPU)

# Stock + static-budget + Meridian over a real prompt-length distribution.
python -m benchmarks.meridian_bench synthetic-replay \
    --workload sharegpt --baseline all \
    --duration-s 30 --arrival-rate 8 --out-dir bench-out/

# Read the comparison. Meridian should win TTOT P95 vs both baselines.
cat bench-out/ab-report.md

The synthetic-replay mode requires the native extension (maturin develop -m crates/meridian-python/Cargo.toml); the baseline and report logic alone are exercised by benchmarks/tests/test_baselines.py without it.

Dataset loaders

Pass --workload sharegpt or --workload math500 to load real traffic distributions from HuggingFace. Datasets are downloaded once and cached at ~/.cache/meridian/datasets/. Requires no GPU — the offline replay drives the synthetic decoder with the real prompt/response length distribution.

Test environment disclosure

When comparing numbers across runs:

Always report --seed and workload flag. Synthetic results are hardware-independent; real-vLLM results depend on GPU model, driver, and memory state — disclose all three.

How to compare two runs

# Run A (baseline config)
python -m benchmarks.meridian_bench synthetic-replay --seed 42 --out-dir bench-out/a/

# Run B (modified config)
python -m benchmarks.meridian_bench synthetic-replay --seed 42 --out-dir bench-out/b/

# Diff the JSON reports
diff <(jq -S . bench-out/a/report.json) <(jq -S . bench-out/b/report.json)

What this harness is, and what it isn't

• It is a phase-differentiated latency regression suite. It catches changes that move the TTOT or output-ITL distributions, the metrics Meridian was built to improve.
• It is reproducible: the synthetic-replay mode is deterministic given --seed. Two PRs can be diffed apples-to-apples.
• It is not a raw-throughput benchmark. vLLM's own harness already reports tokens/sec/GPU and that metric does not differentiate Meridian from the baseline. Operators who want a throughput number should run vLLM's benchmark.
• It is not an accuracy benchmark. Budget forcing can in principle hurt reasoning accuracy on hard problems. Accuracy measurement requires a separate ground-truth evaluation suite; this harness does not provide one.

GPU CI runner setup

The cuda.yml workflow targets a self-hosted runner labelled gpu because GitHub-hosted runners do not provide CUDA-capable GPUs in the free tier. This page documents how to provision and secure the runner.

Hardware requirements

Required secrets

The runner registration token is generated once from the GitHub Actions UI and is not stored as a persistent secret — it expires after one hour and is only used during ./config.sh.

Provisioning

# On the Linux host with the GPU
curl -O https://github.com/actions/runner/releases/download/v2.319.0/actions-runner-linux-x64-2.319.0.tar.gz
mkdir actions-runner && cd actions-runner
tar xzf ../actions-runner-linux-x64-2.319.0.tar.gz

./config.sh \
    --url https://github.com/angelnicolasc/meridian \
    --token <REGISTRATION_TOKEN> \
    --labels self-hosted,linux,x64,gpu \
    --unattended

sudo ./svc.sh install
sudo ./svc.sh start

Verification

Run ./run.sh once interactively. Then trigger the cuda.yml workflow from a branch and confirm nvidia-smi prints the expected device in the job logs.

Blast radius and fork safety

Self-hosted runners execute arbitrary code from the workflow YAML. Meridian mitigates this with a hard gate on every GPU job:

if: github.repository_owner == 'angelnicolasc'

PRs from forks never trigger the CUDA workflow. Only pushes and PRs from the angelnicolasc org are eligible.

Who can trigger: repository owners and collaborators with write access.
How to rotate the runner: generate a new registration token from the GitHub Actions UI, run ./config.sh --replace, restart the service.

See the GitHub documentation on self-hosted runner security for a full threat model.

Security & Supply Chain

Disclosure policy

Security vulnerabilities should be reported privately. See SECURITY.md for the full disclosure process, severity classification, and response SLAs.

Summary: critical severity (CVSS ≥ 7.0) issues receive a patch within 48 hours of confirmation. Do not open public GitHub issues for unpatched vulnerabilities.

Provenance attestation

Every tagged release (v*) generates a SLSA Level 2 provenance attestation via the slsa-github-generator reusable workflow. The attestation covers:

• The meridian-core and meridian-kernels static libraries.
• The meridian Python wheel.

The attestation is uploaded as a GitHub release asset alongside the release artefacts. To verify:

# Install the SLSA verifier
go install github.com/slsa-framework/slsa-verifier/v2/cli/slsa-verifier@latest

# Download the artefact and its provenance from the GitHub release.
# Then verify:
slsa-verifier verify-artifact meridian-*.whl \
    --provenance-path meridian.intoto.jsonl \
    --source-uri github.com/angelnicolasc/meridian \
    --source-tag v0.1.0

What is attested: the build provenance — that the artefact was built from the tagged source in the GitHub Actions environment. SLSA L2 does not attest to the security of the code itself.

Software Bill of Materials (SBOM)

Each release includes a CycloneDX SBOM covering:

• The Rust workspace (all transitive crate dependencies).
• The Python wheel (Python package dependencies from pyproject.toml).

The SBOM is attached as a .cdx.json asset on the GitHub release. Operators can use it with vulnerability scanning tools (Grype, Trivy, FOSSA).

Supply-chain controls

Dependency policy

New dependencies require:

1. A licence compatible with Apache-2.0 (verified by cargo deny).
2. No known CVEs at the time of merge (verified by cargo deny advisories check).
3. An entry in the SBOM at the next release.

CI workflow permissions

All CI workflows run with minimal permissions:

release.yml permission notes:

• id-token: write — required by slsa-github-generator to mint the OIDC-backed provenance token; scoped to the build-artifacts and provenance jobs.
• pull-requests: write — required by release-plz to open the automated release PR.
• attestations: write — required by slsa-github-generator to upload the attestation bundle as a release asset.

Contributor Covenant Code of Conduct

This project adopts the Contributor Covenant, version 2.1 as its code of conduct.

Our Pledge

We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.

Our Standards

Examples of behavior that contributes to a positive environment include:

• Demonstrating empathy and kindness toward other people.
• Being respectful of differing opinions, viewpoints, and experiences.
• Giving and gracefully accepting constructive feedback.
• Accepting responsibility and apologizing to those affected by our mistakes.
• Focusing on what is best for the overall community.

Unacceptable behavior includes:

• Sexualized language or imagery, and sexual attention or advances of any kind.
• Trolling, insulting or derogatory comments, and personal or political attacks.
• Public or private harassment.
• Publishing others' private information without explicit permission.
• Other conduct which could reasonably be considered inappropriate in a professional setting.

Enforcement

Instances of abusive, harassing, or otherwise unacceptable behavior may be reported confidentially to the project maintainer at nick.dicerutti@gmail.com. All complaints will be reviewed and investigated promptly and fairly.

For the full text of the Code of Conduct including enforcement guidelines, see https://www.contributor-covenant.org/version/2/1/code_of_conduct/.

Contributing to Meridian

Thank you for considering a contribution. Meridian is an inference-time compute scheduler for reasoning-model serving — correctness, performance and clarity of contracts all matter at the same time. This document describes the bar.

Code of Conduct

Participation is governed by the Contributor Covenant 2.1. Report violations to nick.dicerutti@gmail.com.

Developer Certificate of Origin (DCO)

Meridian uses the DCO instead of a CLA. Every commit must be signed off:

git commit -s -m "feat(core): add RPDI ratio computation"

This appends Signed-off-by: Your Name <you@example.com>, which certifies that you have the right to submit the change under the project license. PRs without sign-off will not merge — the DCO check blocks them.

Commit conventions

We use Conventional Commits with the following scopes:

Title must be under 72 characters. Breaking changes use feat!: / fix!: and include a BREAKING CHANGE: footer.

Branch protection

main is protected. PRs require:

1. Linear history (rebase, no merge commits).
2. CI green: cargo fmt --check, clippy -D warnings, cargo nextest, ruff, mypy --strict, pytest -m "not gpu", mdbook build.
3. DCO sign-off on every commit.
4. Conventional Commit title.
5. At least one approving review.

Local development

./scripts/dev-up.sh                  # devcontainer + sanity checks
./scripts/ci-local.sh                # mirrors CI matrix locally

Pre-commit hooks (rustfmt, clippy, ruff, mypy, commitlint) are configured in .pre-commit-config.yaml. Install with pre-commit install --install-hooks.

Test strategy

• Pure Rust logic lives in meridian-core and must be covered by unit tests plus, where state machines are involved, proptest-based property tests.
• CUDA correctness lives in meridian-kernels and is verified against the reference CPU implementation in Python.
• Anything that crosses the FFI boundary is exercised from Python tests (python/tests/).
• GPU-dependent tests are marked @pytest.mark.gpu; CI runs them only on the GPU job.

What lands in main

A change is mergeable when:

• Tests cover the new code path and existing tests still pass.
• Public API additions have rustdoc / docstrings with examples.
• Behavior changes that affect operators (config defaults, metric names, exposed traits) are recorded in an ADR.
• CHANGELOG.md is updated under ## [Unreleased] (handled automatically by release-plz for routine changes — manually for breaking ones).

Getting help

Open a GitHub Discussion for design questions, an issue for bugs.

Security Policy

Supported versions

Meridian is in pre-1.0 development. Only the latest tagged release receives security fixes. After 1.0, the latest minor of the latest two majors will be supported.

Reporting a vulnerability

Do not file a public issue for security problems. Use GitHub's Private Vulnerability Reporting or email nick.dicerutti@gmail.com with the subject line [meridian-security].

You should expect:

We credit reporters in the release notes unless anonymity is requested.

Scope

In scope for security reporting:

• Memory safety in meridian-core and meridian-kernels FFI boundary — any reachable UB, out-of-bounds access, double-free, use-after-free.
• CUDA kernel safety — buffer overruns, races on shared memory, illegal memory access reachable from sane inputs.
• Deserialization of meridian.toml, model configs, request payloads — panics on adversarial input, type confusion.
• Denial of service — pathological requests that crash the scheduler or exhaust KV memory irrecoverably.
• Supply chain — compromised crate/wheel that ships under the Meridian name.

Out of scope:

• Vulnerabilities in upstream dependencies (file with the upstream project; we will track and bump promptly).
• Misconfigurations of operator-controlled deployments (e.g. exposing the Prometheus endpoint publicly).
• Reasoning quality degradation when budget forcing is misconfigured — this is a correctness concern, not a security one.

Hardening notes

• meridian-core denies unsafe_op_in_unsafe_fn workspace-wide.
• The CUDA FFI boundary in meridian-kernels is the only unsafe surface and is reviewed for every change.
• Releases are signed via Sigstore cosign; artifact provenance is generated via SLSA Level 2.