All work
Design doc · 01

nanoserve

A from-scratch LLM inference engine and distributed serving layer.

Lead projectLLM inferenceDistributed serving
Source on GitHub

Overview

nanoserve is a from-scratch LLM inference engine with a distributed serving layer.

More details coming soon

Architecture

requestsschedulerkv cacheworker 0worker 1worker n
More details coming soon

Key design decisions

More details coming soon

Benchmarks

More details coming soon

What I'd do next

More details coming soon