LLM serving engine
tinyorca
Minimal implementation of Orca-style LLM serving for understanding continuous batching behavior.
- Built an iteration-level scheduler that reselects active requests each model step, allowing finished requests to leave and queued requests to enter without waiting for a static batch.
- Modeled KV-slot reservation and FCFS admission control to make the memory tradeoff behind continuous batching explicit.
- Separated scheduling and execution responsibilities across endpoint, request pool, scheduler, and engine components.
- Documented selective batching and mixed prefill/decode execution in a detailed technical note.
- Added benchmark scenarios to compare equal-size workloads and mixed short/long request patterns.
Minimal serving system
microengine
Small readable serving loop built before tinyorca to isolate the core mechanics of request admission and static batching.
- Implemented request objects, FIFO queueing, endpoint tokenization, static batch scheduling, model runner, and engine orchestration.
- Separated prefill and iterative decode responsibilities in the model runner to make the serving data path explicit.
- Kept the design intentionally batch-to-completion, making it a clean contrast against tinyorca's continuous batching path.
Large-scale inference engine
vLLM
- Fixed false OOM errors in pipeline-parallel KV cache setup by projecting global KV cache groups onto each worker's assigned layers before memory validation.
- Removed ShellCheck baseline gating and cleaned CI/test scripts, including argv-splitting fixes and behavior-preserving shell cleanups.
Multimodal serving
vLLM-Omni
- Fixed KV cache extraction for NPU-style key/value tuple layouts and added checks for unsupported inputs.