Python, React, Vite, PyTorch
AirLLM — Ovari mode
Creator · AirLLM fork
A fork of AirLLM with a local desktop app on top. Upstream AirLLM streams model layers from disk so 70B-class models can run on a few gigabytes of VRAM — robust, but seconds per token. Ovari mode probes the machine, then picks a load strategy: keep a fitting model resident on the GPU, split it across VRAM and RAM, or fall back to layer-by-layer streaming. The Control UI is a React/Vite frontend served by a stdlib Python HTTP server — hardware diagnostics, Hugging Face download, Chat, Coding Agent, and a local knowledge store. Loopback-only bind; no Flask or FastAPI.
fits free VRAM → direct_gpu (resident transformers, ~16–22 tok/s on a 3–7B)
fits VRAM + RAM → direct_offload (Accelerate device_map=auto)
otherwise / OOM / quant → airllm (layer-by-layer disk stream)Deep dive
Load strategy
plan_load_strategy estimates weight size (MoE-aware, AWQ embeddings counted) and free VRAM with a ~1 GB reserve. Direct GPU uses resident Transformers with SDPA. Hybrid uses Accelerate device_map=auto. AirLLM streaming is the fallback, and a failed resident load falls back automatically instead of hard-failing. Default dtype is bfloat16 on capable GPUs so Qwen/Llama checkpoints do not overflow to NaN.
AirLLM Control
The UI reads CPU, RAM, CUDA or Apple MPS, PyTorch, bitsandbytes, Hugging Face cache disk, Windows power mode, and Wi-Fi. It applies hardware-aware defaults, then lets you tune device, dtype, 4/8-bit compression, prefetching, cleanup interval, sequence length, and layer-shard cache path. Chat streams over SSE. Stop is cooperative Transformers StoppingCriteria. Benchmark measures GPU/CPU matmul and cache-disk throughput without requiring a loaded model.
Coding agent and local memory
The agent reads the project tree, git status, and a small set of files, then returns a plan, proposed diffs, and test suggestions — it never runs a shell or writes files. Chat and agent tabs accept text/source attachments (2 MiB each, 10 files / 8 MiB cap). With learning enabled, excerpts land in data/learning.sqlite3 and later turns retrieve only relevant chunks. This is persistent RAG, not weight updates; JSONL export exists for a separate fine-tune. External OpenAI-compatible providers are per-request; stored memory is never forwarded to them.
CLI and runtime
airllm.cmd / airllm_cli.py talk to the same loopback API: status, model search, download with cancel, and agent runs from a terminal. Generated Triton/PyTorch kernels cache under .runtime-cache/ so Windows JIT lock files stay out of the user profile. Supported families follow AirLLM AutoModel: Qwen2/2.5, Llama, Mistral, Mixtral, ChatGLM, Baichuan, InternLM.