Seongwoo Choi / Journal

Run Claude Code on your own vLLM server: what to expect

At a glancePoint Claude Code at a model you host with vLLM: the six settings, what Claude Code really sends, what stops working, and what Anthropic does not support.

Partly testedClaude Code was run against a stand-in server to record what it sends. No model was served: there was no GPU.

You need
A vLLM server with a model that supports tool calling, and Claude Code. Section 4's test needs only Python and Claude Code.
You end up with
Claude Code talking to a model on your own hardware, with a clear view of what works, what does not and what it costs in context.
In this article 7 sections
  1. 1. Why people do it, and when not to
  2. 2. How the connection works
  3. 3. The setup, from vLLM's documentation
  4. 4. What Claude Code actually sends
  5. 5. What does not work
  6. 6. What to expect from the model
  7. 7. Should you do this?

Claude Code is the terminal coding agent from Anthropic. vLLM is the open-source server for running models on your own GPUs. vLLM can speak the same API that Claude Code uses, so the two can be connected: the agent's interface, with a model on hardware you control. This guide shows how, what Claude Code actually sends to your server, and where the combination falls short. One thing first, because it decides whether you should do this at all.

Anthropic does not support thisIts documentation says it "doesn't support routing Claude Code to non-Claude models through any gateway". The settings below are documented by the vLLM project. They work at the level of the API, but Claude Code is built and tested for Claude models. Expect rough edges, and do not expect help from Anthropic when you hit one.

1. Why people do it, and when not to

A good reasonA poor reason
The code may not leave your network"It will be as good, for free." It will not be as good, and GPUs are not free
You already run a GPU server and want one more use for itYou have no GPU server yet. Start with the sizing guide
You want to see how an open model handles agent workYou need the web search and other tools that run on Anthropic's side

2. How the connection works

Claude Codesends requests in Anthropic's Messages format
→
ANTHROPIC_BASE_URLsays where to send them
→
vLLMaccepts that format at /v1/messages and translates it
→
Your modelmust be able to call tools
Nothing is patched or intercepted. One setting changes the address, and vLLM answers in the format Claude Code expects.

The last box is the demanding one. A coding agent works by calling tools: read a file, edit it, run a command. vLLM's page says the model needs "strong tool calling capabilities". A model that chats well and calls tools badly will read your request and then do nothing useful.

3. The setup, from vLLM's documentation

Start the server with tool calling switched on. This is vLLM's own example:

vllm serve openai/gpt-oss-120b --served-model-name my-model --enable-auto-tool-choice --tool-call-parser openai
FlagWhy it is there
--served-model-name my-modelA short name with no slash. vLLM's page warns that model names containing /, such as the Hugging Face name itself, cannot be used with Claude Code
--enable-auto-tool-choiceLets the model decide when to call a tool
--tool-call-parser openaiHow this model writes its tool calls. Other models need a different parser; vLLM's tool calling page lists them. For Qwen, see part 3 of our series

Start Claude Code with six settings:

ANTHROPIC_BASE_URL=http://localhost:8000 \
ANTHROPIC_API_KEY=dummy \
ANTHROPIC_AUTH_TOKEN=dummy \
ANTHROPIC_DEFAULT_OPUS_MODEL=my-model \
ANTHROPIC_DEFAULT_SONNET_MODEL=my-model \
ANTHROPIC_DEFAULT_HAIKU_MODEL=my-model \
claude
SettingWhat it does
ANTHROPIC_BASE_URLWhere requests go. No /v1 at the end; Claude Code adds the path
ANTHROPIC_API_KEYAny value, because vLLM asks for no key by default
ANTHROPIC_AUTH_TOKENvLLM's page says it is required and can be any value. If you started vLLM with --api-key, set both this and the line above to that key
The three …_MODEL settingsClaude Code asks for different models for different jobs. All three must point at the one name your server knows

On Windows PowerShell, set each one first and then start Claude Code:

$env:ANTHROPIC_BASE_URL = "http://localhost:8000"
$env:ANTHROPIC_API_KEY = "dummy"
$env:ANTHROPIC_AUTH_TOKEN = "dummy"
$env:ANTHROPIC_DEFAULT_OPUS_MODEL = "my-model"
$env:ANTHROPIC_DEFAULT_SONNET_MODEL = "my-model"
$env:ANTHROPIC_DEFAULT_HAIKU_MODEL = "my-model"
claude

To keep the settings, vLLM's page suggests your shell profile, a small wrapper script, or the env section of ~/.claude/settings.json. A wrapper script is the safest of the three: your normal claude command keeps talking to Anthropic, and a second command, say claude-local, talks to your server.

4. What Claude Code actually sends

We had no GPU for this article, so we tested the half that needs none. This small program pretends to be a model server: it records what arrives and replies with one fixed sentence. Save it as stand_in.py if you want to repeat the test:

"""stand_in.py: a stand-in for a model server. It records what a client asks for and streams back one fixed sentence.

It speaks just enough of the Anthropic Messages API to answer Claude Code. It is not a model.
"""
import json
from http.server import BaseHTTPRequestHandler, HTTPServer

REPLY = "Hello from your own server."


class StandIn(BaseHTTPRequestHandler):
    def do_POST(self):
        raw = self.rfile.read(int(self.headers.get("Content-Length", 0)))
        body = json.loads(raw or b"{}")
        print(json.dumps({"path": self.path.split("?")[0], "bytes": len(raw), "model": body.get("model"), "stream": body.get("stream"),
                          "max_tokens": body.get("max_tokens"), "tools": len(body.get("tools", [])),
                          "system_chars": len(json.dumps(body.get("system", ""))),
                          "auth": sorted(h for h in ("authorization", "x-api-key") if self.headers.get(h))}), flush=True)
        if self.path.startswith("/v1/messages/count_tokens"):
            return self.send_json({"input_tokens": 10})
        if not body.get("stream"):
            return self.send_json({"id": "msg_1", "type": "message", "role": "assistant", "model": body.get("model"),
                                   "content": [{"type": "text", "text": REPLY}], "stop_reason": "end_turn",
                                   "stop_sequence": None, "usage": {"input_tokens": 10, "output_tokens": 6}})
        self.send_response(200)
        self.send_header("Content-Type", "text/event-stream")
        self.end_headers()
        events = [
            ("message_start", {"type": "message_start", "message": {"id": "msg_1", "type": "message", "role": "assistant", "model": body.get("model"), "content": [], "stop_reason": None, "stop_sequence": None, "usage": {"input_tokens": 10, "output_tokens": 0}}}),
            ("content_block_start", {"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}}),
            ("content_block_delta", {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": REPLY}}),
            ("content_block_stop", {"type": "content_block_stop", "index": 0}),
            ("message_delta", {"type": "message_delta", "delta": {"stop_reason": "end_turn", "stop_sequence": None}, "usage": {"output_tokens": 6}}),
            ("message_stop", {"type": "message_stop"}),
        ]
        for name, data in events:
            self.wfile.write(f"event: {name}\ndata: {json.dumps(data)}\n\n".encode())
        self.wfile.flush()

    def send_json(self, data):
        payload = json.dumps(data).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(payload)))
        self.end_headers()
        self.wfile.write(payload)

    def log_message(self, *args):
        pass


HTTPServer(("127.0.0.1", 8000), StandIn).serve_forever()

With it running, we started Claude Code 2.1.322 with the six settings and one short question:

$ ANTHROPIC_BASE_URL=http://127.0.0.1:8000 ANTHROPIC_API_KEY=dummy ANTHROPIC_AUTH_TOKEN=dummy ANTHROPIC_DEFAULT_OPUS_MODEL=my-model ANTHROPIC_DEFAULT_SONNET_MODEL=my-model ANTHROPIC_DEFAULT_HAIKU_MODEL=my-model claude -p "Say hello"
[claude-code:unrecognized_model] {"model":"my-model[1m]","query_source":"sdk"}
Hello from your own server.
The sentence came from our stand-in, not from Anthropic. The grey line is Claude Code noting that it does not know this model name; the request went through regardless.

And this is what the stand-in recorded:

{"path": "/v1/messages", "bytes": 46224, "model": "my-model", "stream": true, "max_tokens": 32000, "tools": 12, "system_chars": 6132, "auth": ["authorization", "x-api-key"]}
What was recordedWhat it means for your server
"model": "my-model"The three model settings work. Your server must be serving exactly this name
"bytes": 46224, "tools": 12"Say hello" arrived wrapped in about 46 KB of instructions and twelve tool definitions. At a rough four characters per token, that is on the order of 11,000 tokens before Claude Code has read a single file of yours. That figure is our estimate
"max_tokens": 32000Claude Code asked for room to write up to 32,000 tokens
"stream": trueReplies are streamed. Any proxy in front of vLLM must not buffer them; the nginx settings in part 5 cover this
Both authorization and x-api-keyThe token and the key each travel in their own header. In a second run without ANTHROPIC_AUTH_TOKEN, only x-api-key was sent and the request still arrived. Set both, as vLLM's page says
The number to plan aroundThe instructions and the output allowance together need tens of thousands of tokens of context before any real work. A server started with a small --max-model-len will not be enough. Size the context first; part 6 shows what each extra token of context costs in GPU memory.

You can lower the output allowance. With CLAUDE_CODE_MAX_OUTPUT_TOKENS=4096 added, the stand-in recorded "max_tokens": 4096. That setting is in Claude Code's list of environment variables.

5. What does not work

SymptomCause, from vLLM's pageWhat to do
Web search fails with "tool type 'web_search_…' is not supported"Claude Code's web search runs on Anthropic's servers. vLLM does not run such tools and rejects the requestAdd an MCP server that provides search. See adding MCP servers
The model answers but never edits or runs anythingTool calling is off, or the parser does not match the modelCheck --enable-auto-tool-choice and the --tool-call-parser for your model
"Model not found"The name in the settings is not the served name, or it contains a /Use --served-model-name and the same short name in all three settings
Everything is slow, and each request seems to start from nothingClaude Code adds a per-request fingerprint to the start of its instructions, which defeats vLLM's prefix cache. vLLM handles this in versions after 0.17.1Upgrade vLLM, or on an older version add "CLAUDE_CODE_ATTRIBUTION_HEADER": "0" to the env section of ~/.claude/settings.json
A request with max_tokens: 0 is refusedAnthropic's API accepts that to warm its cache; vLLM always produces at least one tokenNothing, for normal use. A tool that sends it should send 1
"Connection refused"vLLM is not running, or the port is differentCheck the server and the address

Beyond that list, expect differences that no setting fixes. Claude Code's features were built around Claude models, and some will behave differently, or not at all, with another model behind the same interface.

6. What to expect from the model

We did not serve a model, so we have no results of our own to report, and we will not borrow anyone else's. What can be said from how the pieces work:

  • The instructions are written for Claude. Those 46 KB assume a model trained to follow them. Another model may ignore parts of them or follow them too literally.
  • Agent work punishes small mistakes. A chat answer that is 95 percent right is fine. A tool call with one wrong argument fails, and a task is many tool calls in a row.
  • Speed is yours to provide. Every step sends the whole conversation again. The prefix cache makes that affordable, which is why the fingerprint issue above matters so much.

The fair test is a task you know well. Give the same small job, in the same repository, to Claude Code on its normal model and on your server. Compare what was done, how many steps it took and how long you waited.

7. Should you do this?

Your situationSuggestion
Code must stay inside your network, and you have GPUsWorth trying. Go in knowing it is unsupported
You want the best results on hard tasksUse Claude Code as intended
You are curious and already run vLLMAn afternoon well spent. Use a wrapper script so your normal setup is untouched
You have no server yetBuild one first: the vLLM series starts from one command
You only want a model on your own machine to chat withThis is more than you need. See vLLM, Ollama or llama.cpp

If you do run it for a team, the rest of the series applies unchanged: keys and a proxy, dashboards and alerts.

Sources

This article reflects public information available when it was written. Versions, defaults and prices can change; check the linked official documentation.

The vLLM commands, settings and limits are from vLLM's own page on Claude Code, read on November 12, 2026; no GPU was available, so no model was served for this article. The Claude Code side was tested: Claude Code 2.1.322 was run with those settings against a small stand-in server on Linux, and the request details in section 4 are what that server recorded. That test shows what Claude Code sends, not how well any model answers. Anthropic's position on non-Claude models is quoted from its gateway documentation. The token estimate is ours and is marked as an estimate. The Windows commands follow the same documentation and were not run. Written for this site with AI assistance, from the sources above. For errors or additional information, please request a correction. Editorial policy