DIY Self-Hosting 35B+ Reasoning LLMs with NVFP4 Quantization and vLLM

Executive Summary

The transition from standard instruction-tuned language models to test-time reasoning architectures has dramatically altered the economics of AI infrastructure. Cloud API providers frequently impose restrictive rate limits and high costs on reasoning tokens. For developers and researchers requiring high token throughput and full data privacy, self-hosting is no longer an enthusiast experiment; it is an architectural necessity.

Recent breakthroughs in sub-8-bit floating-point precision, particularly the NVFP4 micro-scaling format supported by modern tensor accelerators, have made it possible to run state-of-the-art 35-billion and 70-billion parameter reasoning models on single-node prosumer and workstation systems with 24 to 32 GB of VRAM. This guide outlines the complete hardware sizing, quantization format selection, vLLM engine orchestration, and performance tuning for a personal self-hosted inference rig.

flowchart TD
    Client["Client / Agent Prompt"] --> Proxy["Local Reverse Proxy (:11436)"]
    Proxy --> vLLM["vLLM High-Throughput Engine (:8000)"]
    subgraph GPU["Workstation GPU Memory (VRAM: 32 GB)"]
        NVFP4["NVFP4 Model Weights (Qwen 35B ~18.5 GB)"]
        KV["PagedAttention KV Cache Buffer (~10 GB)"]
        Act["Activation Buffer (~2 GB)"]
    end
    vLLM --> NVFP4
    vLLM --> KV
    vLLM --> Act

Implementation

Deploying a self-hosted reasoning server requires addressing memory bandwidth saturation, KV-cache allocation, and OpenAI-compatible API serving. The following configuration provides a production-grade Docker Compose stack for vLLM utilizing the official container image with flash-attention and NVFP4 kernel acceleration enabled.

1. Docker Compose Service Definition

Save the following specification to compose.yaml:

services:
  vllm-node:
    image: vllm/vllm-openai:latest
    container_name: local-vllm-node
    runtime: nvidia
    restart: unless-stopped
    ports:
      - "8000:8000"
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NCCL_DEBUG=WARN
    volumes:
      - /opt/models/cache:/root/.cache/huggingface
    ipc: host
    command: >
      --model Qwen/Qwen3.6-35B-A3B-Instruct-NVFP4
      --port 8000
      --max-model-len 32768
      --gpu-memory-utilization 0.92
      --kv-cache-dtype auto
      --trust-remote-code
      --disable-log-requests

2. High-Concurrency Client Benchmark & Verification

To verify inference latency and ensure that streaming completions respond within acceptable interactive thresholds, execute this Python test client:

#!/usr/bin/env python3
import json
import time
import urllib.request

API_URL = "http://127.0.0.1:8000/v1/chat/completions"

def benchmark_inference(prompt: str) -> None:
    payload = {
        "model": "Qwen/Qwen3.6-35B-A3B-Instruct-NVFP4",
        "messages": [
            {"role": "system", "content": "You are a concise mathematical reasoning assistant."},
            {"role": "user", "content": prompt}
        ],
        "temperature": 0.2,
        "max_tokens": 1024,
    }
    
    start_time = time.perf_counter()
    req = urllib.request.Request(
        API_URL,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST"
    )
    
    with urllib.request.urlopen(req, timeout=120) as resp:
        data = json.loads(resp.read().decode("utf-8"))
        elapsed = time.perf_counter() - start_time
        
    choice = data["choices"][0]["message"]["content"]
    usage = data["usage"]
    completion_tokens = usage.get("completion_tokens", 0)
    tok_per_sec = completion_tokens / elapsed if elapsed > 0 else 0
    
    print(f"Generated {completion_tokens} tokens in {elapsed:.2f}s ({tok_per_sec:.1f} tok/s)")
    print(f"Sample response:\n{choice[:250]}...")

if __name__ == "__main__":
    benchmark_inference("Derive the optimal policy gradient update under clipped surrogate objective.")

ShVaSa Intuition

When designing a self-hosted inference platform, the primary architectural hurdle is not compute throughput, but memory bus saturation. In transformer inference, autoregressive generation is bandwidth-bound during decode phases: every token generated requires streaming all model weights from High-Bandwidth Memory (HBM) into compute registers.

Standard 16-bit precision (FP16 or BF16) for a 35B model demands approximately 70 GB of VRAM solely for weights, necessitating multi-GPU setups linked via NVLink. While INT4 integer quantization historically solved the capacity problem, it introduced substantial perplexity degradation in mathematical reasoning tasks due to severe truncation of outlier activations.

The NVFP4 (4-bit Microscaling Floating Point) format solves this trade-off by dynamically scaling 4-bit mantissa and exponent representations across micro-blocks of tensor values. This preserves critical activation spikes in chain-of-thought tokens while shrinking weight footprint down to 18.5 GB. As a result, a workstation equipped with a single 32 GB accelerator can comfortably retain the full model alongside a 32,768-token KV cache without spilling into system host memory.

By combining NVFP4 weights with PagedAttention inside vLLM, self-hosted rigs achieve sustained generation speeds exceeding 35 tokens per second. This unlocks completely private, unmetered, high-velocity research and agentic coding execution directly on local home hardware.

Canonical Source: https://blog.shvasa.us/posts/diy-self-hosting-reasoning-llms-nvfp4-vllm