Documentation
Run open-weight language models directly inside your software
Nobodywho is a local-first inference engine for language models, with the same API across seven languages and frameworks. Streaming chat, tool calling, structured output, embeddings, text-to-speech, speech-to-text and RAG — entirely offline, GPU-accelerated, built on llama.cpp.
This page is a mirror of the essentials. The complete, always-current documentation lives at docs.nobodywho.ooo, and the code at GitHub.
What is Nobodywho's open source inference engine and SDK?
Nobodywho is a lightweight, open-source inference engine for running open-weights LLMs inside your software. We provide a simple, efficient, offline and privacy-forward way of interacting with LLMs. No infrastructure needed!
In short, if you want to run an LLM, integrate it with tools, configure its output, enable real-time streaming of tokens, create embeddings, or synthesize speech, Nobodywho makes it easy.
All of this is enabled by llama.cpp, while having a nice, simple API.
No need to mess around with docker containers, GPU servers, API keys, etc. We make it easy to run local models in Kotlin, Swift, Python, Flutter, React Native, Expo and Godot!
The runtime is free under EUPL 1.2, for proprietary and commercial projects alike. Open source and open weights mean no vendor lock-in: the capability stays with you, beyond any contract.
Pick your language
New to local LLMs?
Start with the two foundational guides: LLM Basics explains tokens, context, samplers and tools; Model Selection helps you choose a model that fits your hardware.
Running this in an organisation?
We deploy and look after Nobodywho for enterprises – on your servers or your devices.
Basics
LLM Basics
Our goal with Nobodywho is to make it easy to run local LLMs. For this reason we have made it possible to use Nobodywho with minimal knowledge of how LLMs work. However you still need to know some basic concepts, so for these we provide some brief explanations. The concepts covered are tokens, context, samplers, tools, and thinking/reasoning.
Tokens
Tokens are the fundamental units a language model processes. A single token is typically a word, a word fragment or a punctuation mark: "hello" is one token, whereas "understanding" may split into "understand" and "ing". Each model has its own token vocabulary, fixed during training.
The model generates text sequentially, one token at a time. That is why Nobodywho's default response streams tokens and lets you read the reply as it is produced.
Context
Context is all the text the model can see while generating a response: earlier messages in the conversation, the current user prompt and the system instructions. It works as the model's memory of the conversation; once a chat is initialised, most interaction is context manipulation.
Context size
Models have a maximum context size (context window or context length), measured in tokens and typically between 2,048 and 128,000. When it fills up you must start a new conversation, remove old messages or summarise earlier content. Nobodywho handles this automatically by removing older messages. Larger contexts allow longer conversations but reduce response speed.
CPU threads
During CPU inference, work is distributed across a thread pool. Because all threads synchronise at barriers, performance is bounded by the slowest thread: hyperthreads add no compute, and efficiency cores run slower than performance cores. Nobodywho defaults to performance cores only. On a 12-core Mac (8 P-cores, 4 E-cores), 8 threads is roughly twice as fast as all 12:
| threads | prompt processing | generation |
|---|
| 8 (performance cores) | 435 tok/s | 90 tok/s |
| 12 (all logical CPUs) | 327 tok/s | 48 tok/s |
Adjusting this is rarely necessary, except to keep CPU free for other work or to correct a misdetected topology.
Samplers
A language model does not output text directly; it produces a probability distribution over possible next tokens. Samplers decide which token is chosen, in two steps: shifting the distribution and sampling from it.
Shifting the distribution
- Temperature — higher values increase creativity, lower values increase focus.
- Top-k / Top-p — filter out unlikely tokens.
- Penalties — reduce the probability of tokens already present in the context.
These steps can be chained.
Sampling the distribution
- Dist — random sampling.
- Greedy — always pick the highest-probability token.
- Mirostat — an advanced technique from this paper.
These cannot be chained, since each directly selects the next token. Nobodywho also supports advanced configurations such as JSON Schema following.
Speculative decoding (MTP)
Speculative decoding speeds up inference without changing the model's output: a draft proposes tokens and the target model verifies them together. Nobodywho implements MTP (Multi-Token Prediction) from Gemma 4, which uses trained "heads" rather than a separate draft model. Download the MTP heads as a companion .gguf (e.g. mtp-gemma-4-E2B-it.gguf), pass it as the draft model path when loading, and enable it when constructing the chat.
Structured output (code, JSON, maths, tool calls) shows a substantial speed-up; prose shows modest to neutral gains. MTP adds roughly 5% VRAM and is off by default. Results vary by hardware — unified-memory systems may see little or negative change.
Tools
Tools (function calling) let the model request external actions instead of only generating text: database searches, calculations, API calls, custom code. You define the available tools; the model decides when to use them based on the conversation. After a tool runs, you hand its result back so the dialogue can continue.
Thinking / reasoning
Some models are trained to reason step by step before answering. Check the model's Hugging Face page for thinking support. Reasoning appears between model-specific tags in the raw output:
<think>
...reasoning steps...
</think>
...final answer...
- Qwen3:
<think>...</think> - Gemma 4:
<|channel>thought ...<channel|> - Ministral:
[THINK]…[/THINK]
Tags vary by model and template, so consult the model's chat template for parsing.
Basics
Model Selection
Quick start
A well-balanced ~2 GB chat model:
huggingface:NobodyWho/Qwen_Qwen3-4B-GGUF/Qwen_Qwen3-4B-Q4_K_M.gguf
A smaller, faster alternative:
huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf
Or pass auto and let Nobodywho pick a model based on available memory.
Ways to specify a model
- Hugging Face paths —
huggingface:owner/repo/filename.gguf (shorthand hf:)
- Direct URLs — any
https:// link to a hosted model
- llama.cpp format —
owner/repo:quantization
- Local files — an ordinary file path
Models download once and are cached locally for offline use.
Reading a model name
A filename like Qwen_Qwen3-0.6B-Q4_K_M.gguf breaks down as:
- Qwen — organisation / developer
- Qwen3 — model release
- 0.6B — parameter count (600 million)
- Q4 — quantisation level (bits per parameter)
- K_M — quantisation variant (S = faster, L = slower, M = balanced)
Key concepts
Quantisation reduces bits per weight to shrink the model and speed it up with minimal accuracy loss; Q4–Q5 generally works well.
Memory estimate is roughly parameter count × quantisation level: 2B @ Q8 ≈ 2 GB, 14B @ Q4 ≈ 7 GB.
Requirements for chat: the model must be instruction-tuned and carry a Jinja2 chat template in its metadata.
Resources
Python
Getting started
First, install nobodywho:
pip install nobodywho
Or preferably:
uv add nobodywho
Next, pick a model. Nobodywho can download GGUF models directly from Hugging Face — just pass a huggingface: path. See model selection for recommendations.
Then make a Chat object and call .ask():
from nobodywho import Chat
chat = Chat('huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf')
response = chat.ask('Is water wet?')
# print each token as it is generated
for token in response:
print(token, end="", flush=True)
# ...or get the entire response as a single string
full_response = response.completed()
print(full_response)
This is a super simple example, but we believe that examples which do simple things should be simple.
More in this section
These guides open on docs.nobodywho.ooo.
Kotlin
Getting started
First, add Nobodywho to your build.gradle.kts:
// Android
implementation("ai.nobodywho:nobodywho-android:2.2.0")
// Desktop JVM (Linux, macOS, Windows)
implementation("ai.nobodywho:nobodywho:2.2.0")
Next, pick a model. Nobodywho can download GGUF models directly from Hugging Face — just pass an hf:// path. See model selection for recommendations.
Then create a Chat and call .ask:
import ai.nobodywho.Chat
import kotlinx.coroutines.runBlocking
fun main() = runBlocking {
val chat = Chat.fromPath(
modelPath = "hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf"
)
// stream tokens as they are generated
chat.ask("Is water wet?").asFlow().collect { token ->
print(token)
}
// ...or get the entire response as a single string
val response = chat.ask("Is water wet?").completed()
println(response)
}
On Android, use lifecycleScope or viewModelScope instead of runBlocking.
More in this section
These guides open on docs.nobodywho.ooo.
Swift
Getting started
Add Nobodywho with Swift Package Manager. In Xcode, go to File → Add Package Dependencies and enter:
https://github.com/nobodywho-ooo/nobodywho-swift.git
Or add it to your Package.swift:
dependencies: [
.package(url: "https://github.com/nobodywho-ooo/nobodywho-swift.git", from: "2.1.0")
]
Models can be loaded from a local file, a Hugging Face repository using hf://, or any https:// URL. Read more about model selection.
import NobodyWho
// From a Hugging Face repository
let chat = try await Chat.fromPath(
modelPath: "hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf"
)
// From an HTTPS URL
let chat = try await Chat.fromPath(
modelPath: "https://huggingface.co/NobodyWho/Qwen_Qwen3-0.6B-GGUF/resolve/main/Qwen_Qwen3-0.6B-Q4_K_M.gguf"
)
// From a local file
let chat = try await Chat.fromPath(modelPath: "/path/to/model.gguf")
Once you have a Chat, call .ask to get a response:
let response = try await chat.ask("Is water wet?").completed()
print(response) // Yes, indeed, water is wet!
Platform requirements
- iOS — iPhone 11 or newer with at least 4 GB RAM. iOS 15+.
- macOS — Apple Silicon or Intel Mac with at least 8 GB RAM. macOS 13+.
- visionOS — Apple Vision Pro. visionOS 1.0+.
- watchOS — watchOS 10+. CPU-only (Metal is not available); only very small models are practical.
GPU acceleration is enabled by default using Metal on all Apple platforms.
More in this section
These guides open on docs.nobodywho.ooo.
React Native / Expo
Getting started
Installation
# React Native
npm install react-native-nobodywho
# Expo
npx expo install react-native-nobodywho
Setup
React Native
No additional initialisation step is required — the native module is loaded automatically when you first import from the package.
Expo
Nobodywho contains native code and cannot run in Expo Go; you need a development build.
CNG projects — for the first build, run the platform command:
npx expo run:ios
# or
npx expo run:android
After upgrading Nobodywho, regenerate the native folders:
npx expo prebuild --clean
npx expo run:ios
Bare projects — native folders are committed; autolinking registers the module on the next expo run:* build.
First chat
import { Chat } from "react-native-nobodywho";
const chat = await Chat.fromPath({
modelPath: "huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf",
});
const response = await chat.ask("Is water wet?").completed();
console.log(response);
Requirements
For x86_64 Android emulators set minSdkVersion to 31 or higher; ARM64 devices support any minSdkVersion.
- iOS — iPhone 11+ with 4 GB RAM minimum
- Android — Snapdragon 855 / Adreno 640 with 6 GB RAM or better
More in this section
These guides open on docs.nobodywho.ooo.
Flutter
Getting started
First, install nobodywho:
flutter pub add nobodywho
Import it under the nobodywho namespace, which avoids clashes with generic names like Model and Chat:
import 'package:nobodywho/nobodywho.dart' as nobodywho;
Initialise the package exactly once, as early as possible:
await nobodywho.NobodyWho.init();
Any Nobodywho call before .init() results in an error — and so does a second call to .init().
Select a model and create a Chat:
final chat = await nobodywho.Chat.fromPath(
modelPath: 'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
);
final msg = await chat.ask('Is water wet?').completed();
print(msg); // Yes, indeed, water is wet!
Minimum recommended specs
- iOS — iPhone 11 or newer with at least 4 GB RAM
- Android — Snapdragon 855 / Adreno 640 / 6 GB RAM or better
More in this section
These guides open on docs.nobodywho.ooo.
Godot
Installation & getting started
Via Asset Library
- Open Godot 4.5 (or any newer 4.x release) and switch to the Asset Library tab.
- Search for Nobodywho and select the entry.
- Click Download, tick Ignore asset root, then Install.
- Godot puts the plugin in
res://addons/nobodywho. Open Create Node and you should see NobodyWhoChat. If it is missing, restart Godot.
Via GitHub
- Download the latest ZIP from the GitHub releases.
- In Godot, open AssetLib ▸ Import and pick the ZIP.
- Tick Ignore asset root and finish the import.
Key terminology
| Term | Definition |
|---|
| Model (GGUF) | A *.gguf file containing the model weights |
| System prompt | Text establishing the ground rules for the model's behaviour |
| Token | Smallest unit of text emitted by the model |
| Chat | Node managing context, user input and conversation state |
| Context | Message history and metadata sent to the model each turn |
| Worker | Background task handling model readiness and communication |
Your first chat
Download a model — "Qwen3 0.6B Q4_K_M" is a lightweight, fast starting point. Add a NobodyWhoModel node and set its model_path (local path, Hugging Face reference or HTTPS URL). Then add a NobodyWhoChat node with a script:
extends NobodyWhoChat
func _ready():
# configure the node (feel free to do this in the UI)
self.system_prompt = "You are an evil wizard."
self.model_node = get_node("../ChatModel")
# connect signals to signal handlers
self.response_updated.connect(_on_response_updated)
self.response_finished.connect(_on_response_finished)
# Start the worker. Not required, but recommended early so the model
# is ready when the user prompts the first time. ask() calls it too.
self.start_worker()
self.ask("How are you?")
func _on_response_updated(token):
# this will print every time a new token is generated
print(token)
func _on_response_finished(response):
# this will print when the entire response is finished
print(response)
You can also call ask() straight away — prompts issued before the worker is ready are queued and dispatched as soon as loading completes.
chat.worker_started.connect(func():
print("Ready to chat!")
)
chat.worker_failed.connect(func(err):
push_error("Model load failed: " + err)
)
chat.start_worker()
When you run the scene, tokens should appear in the console as the model generates its response.
More in this section
These guides open on docs.nobodywho.ooo.