Local Inference Server
The Vi SDK ships an OpenAI-compatible inference server you can run on your own hardware. It serves any architecture the SDK's loaders support, applies LoRA adapter weights, and speaks the same /v1/chat/completions protocol as the OpenAI API, so existing OpenAI client code can point at it with only a base URL change.
- Vi SDK installed with the
deploymentextra - A model to serve: a HuggingFace ID, a local directory, or a Datature training run
- A GPU is strongly recommended; see GPU and compute
The local server runs any supported architecture with LoRA adapters applied, using the SDK's own loaders. NVIDIA NIM offers prebuilt GPU-optimized containers, but only for Cosmos-Reason1 and Cosmos-Reason2, and it does not apply PEFT adapters. Use the local server for everything else.
Installation
Start the server
python -m vi.deployment.local.server \
--pretrained-model-name-or-path nvidia/Cosmos-Reason2-2B \
--port 8000python -m vi.deployment.local.server \
--pretrained-model-name-or-path /path/to/model \
--port 8000python -m vi.deployment.local.server \
--run-id abc123 \
--secret-key your-key \
--organization-id your-org \
--port 8000python -m vi.deployment.local.server \
--pretrained-model-name-or-path nvidia/Cosmos-Reason2-2B \
--model-name my-vision-model \
--port 8000Weights load during startup, not on the first request. The server does not report healthy until the model is resident, so a large checkpoint means a slow first boot. This is expected.
Endpoints
curl http://localhost:8000/healthcurl http://localhost:8000/v1/modelsThe model field in a request must match an ID returned by /v1/models, or the server responds 404. When in doubt, query /v1/models first rather than guessing from the path you passed on the command line.
Request formats
The server accepts two input shapes. Send one or the other. A request carrying both returns 400.
OpenAI format
Standard messages, with images as content parts.
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "nvidia/Cosmos-Reason2-2B",
"messages": [{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "/path/to/image.jpg"}},
{"type": "text", "text": "Describe this image"}
]
}]
}'from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-used")
response = client.chat.completions.create(
model="nvidia/Cosmos-Reason2-2B",
messages=[{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "/path/to/image.jpg"}},
{"type": "text", "text": "Describe this image"},
],
}],
)
print(response.choices[0].message.content)Vi SDK format
A flatter shape for the common single-image case.
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "nvidia/Cosmos-Reason2-2B",
"source": "/path/to/image.jpg",
"user_prompt": "Describe this image"
}'Passing a plain string as content with no image. Vision models need the image in the multimodal content array. You can also use the Vi SDK source field, which is harder to get wrong.
Generation parameters
Streaming
Set stream: true to receive server-sent events.
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-used")
stream = client.chat.completions.create(
model="nvidia/Cosmos-Reason2-2B",
messages=[{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "/path/to/image.jpg"}},
{"type": "text", "text": "Describe this image"},
],
}],
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="", flush=True)curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "nvidia/Cosmos-Reason2-2B",
"source": "/path/to/image.jpg",
"user_prompt": "Describe this image",
"stream": true
}'Structured outputs
Pass a JSON schema in response_format to constrain generation to it.
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-used")
response = client.chat.completions.create(
model="nvidia/Cosmos-Reason2-2B",
messages=[{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": "/path/to/image.jpg"}},
{"type": "text", "text": "Extract the reading"},
]}],
response_format={
"type": "json_schema",
"json_schema": {
"name": "gauge_reading",
"schema": {
"type": "object",
"properties": {
"value": {"type": "number"},
"unit": {"type": "string"},
},
"required": ["value", "unit"],
},
},
},
)
print(response.choices[0].message.content)Embedding the server in Python
create_server() returns a configured FastAPI app you can run yourself or mount into a larger application.
import uvicorn
from vi.deployment.local.server import create_server
app = create_server(pretrained_model_name_or_path="nvidia/Cosmos-Reason2-2B")
uvicorn.run(app, host="0.0.0.0", port=8000)from vi.deployment.local.server import create_server
app = create_server(
run_id="abc123",
secret_key="your-key",
organization_id="your-org",
)Returns: a configured FastAPI application.
Custom backends
The server talks to models through the InferenceBackend interface, so you can swap in vLLM, TGI, a remote API, or your own stack. The bundled default is ViBackend.
from collections.abc import AsyncIterator
from typing import Any
from vi.deployment.local.backends.base import GenerationResult, InferenceBackend
class MyBackend(InferenceBackend):
def available_models(self) -> list[str]:
return ["my-model"]
async def generate(
self,
messages: list[dict] | None,
response_format: Any | None = None,
generation_config: dict | None = None,
**kwargs: Any,
) -> GenerationResult:
text = await self._run_one_shot(messages)
return GenerationResult(
text=text,
prompt_tokens=0,
completion_tokens=0,
finish_reason="stop",
)
async def generate_stream(
self,
messages: list[dict] | None,
response_format: Any | None = None,
generation_config: dict | None = None,
**kwargs: Any,
) -> AsyncIterator[str]:
async for token in self._stream(messages):
yield tokenfrom vi.deployment.local.server import create_server
app = create_server(backend=MyBackend())All four backend methods are async. If your runtime is synchronous and GPU-bound, as the bundled ViBackend is, wrap the call in asyncio.to_thread(...), or a long generation will stall every other in-flight request.
To serve Vi SDK format requests natively, also override generate_vi_format() and generate_stream_vi_format(). The defaults route through OpenAI-format conversion. generate_stream* must yield strings; the server skips None silently.
Troubleshooting
Related resources
Updated 1 day ago