Hugging Face InferenceClient on a Custom OpenAI-Compatible API: Auto /v1, model="dummy" and Zero Retries

huggingface_hub.InferenceClient is mostly known for Hub models and HF Inference Providers. It also ships an OpenAI-shaped surface: pass base_url and api_key, then call client.chat.completions.create(...) or client.chat_completion(...). That is enough to hit any server that speaks POST /v1/chat/completions. The defaults are not the same as the official OpenAI Python SDK, and a few of them are easy to miss.

I installed huggingface_hub 2.2.0 on Python 3.13.5 and pointed InferenceClient at a local mock of POST /v1/chat/completions. The mock logged the path, Authorization, User-Agent and full JSON body of every call, and could return 429, 503, 401 or 400 on demand. Everything below is what went over the wire. The examples use APIClaw, an OpenAI-compatible gateway I build, so weigh my opinions with that in mind. Its base URL is https://apiclaw.biz/v1. Replace YOUR_MODEL_ID with an exact ID from your provider's model list — do not invent one.

The minimal client

import os
from huggingface_hub import InferenceClient

client = InferenceClient(
    base_url="https://apiclaw.biz/v1",
    api_key=os.environ["APICLAW_API_KEY"],
)

r = client.chat_completion(
    messages=[{"role": "user", "content": "Say hi"}],
    model="YOUR_MODEL_ID",
)
print(r.choices[0].message.content)

Install with pip install huggingface_hub. The OpenAI-compatible alias works the same way:

r = client.chat.completions.create(
    model="YOUR_MODEL_ID",
    messages=[{"role": "user", "content": "Say hi"}],
)

api_key is an alias for token. Passing both to __init__ raises ValueError. Same for model and base_url: you may pass only one. base_url is stored in self.model so the constructor stays OpenAI-shaped; the chat payload still needs its own model= argument (see below).

Authorization on the wire was Bearer sk-test (whatever string you passed). The User-Agent included hf_hub/2.2.0. If you omit api_key/token, InferenceClient falls back to HF_TOKEN / the Hub cache via get_token(). It does not read OPENAI_API_KEY.

URL joining: InferenceClient inserts /v1 for you

This is the biggest difference from the OpenAI SDK. InferenceClient's chat path goes through _build_chat_completion_url. Against my mock:

The OpenAI Python client (3.35.1, same box) does the opposite on a bare host: base_url="http://127.0.0.1:PORT" hit POST /chat/completions with no /v1. InferenceClient adds /v1/chat/completions when the path is empty, and appends only /chat/completions when the path already ends in /v1.

So for a normal OpenAI-compatible gateway you can pass either https://apiclaw.biz/v1 or https://apiclaw.biz and land on the right route. That is forgiving. The /api case is the footgun: any non-empty path that is not /v1 and not already /chat/completions gets /v1/chat/completions glued on. A reverse proxy mounted at /api will see /api/v1/chat/completions, which is rarely what you configured.

InferenceClient(model="http://.../v1") (URL as the positional model, no base_url) behaved the same for chat: the conversational helper still runs _build_chat_completion_url. The constructor docstring says a URL passed as model will not get a suffix; for chat_completion that is outdated — the suffix is added.

Forgetting model= sends model="dummy"

base_url in __init__ only sets the URL. The JSON body still needs a model id. The call-site model= argument fills the payload. If you leave it out, the helper sees that the only "model" it has is a URL and substitutes the string "dummy":

# base_url set, no model= in the call
client.chat_completion(messages=[{"role": "user", "content": "hi"}])
# mock body: {"model": "dummy", "messages": [...], "stream": false}

With model="YOUR_MODEL_ID" the body carried that id exactly. Always pass model= on every chat_completion / chat.completions.create when you constructed the client with base_url.

What the body actually contains

A minimal successful call sent three fields:

Unlike the OpenAI SDK, InferenceClient always includes "stream": false on non-streaming calls. filter_none drops None values but keeps False, and stream defaults to False in the signature. temperature, max_tokens, top_p and friends appeared only when I set them. The docstring mentions a default of 100 for max_tokens; that default is not sent on the wire — omitting the argument omits the field.

A system message went through unchanged as the first element of messages. Streaming set "stream": true and returned an iterable of chunk objects; my five-token mock produced five content deltas ("Hello", " ", "from", " ", "HF").

Retries: there are none

The OpenAI Python client defaults to max_retries=2 (up to three attempts) and honors Retry-After. InferenceClient's _inner_post does a single httpx POST and raises. Against the mock:

No x-stainless-retry-count header (that is an OpenAI SDK thing). No second try. If you want retries against a flaky gateway, wrap the call yourself or use the OpenAI client with the same base_url. Fail-fast is the InferenceClient default; that is useful when you already have an outer retry loop and do not want them to stack.

Errors and when to prefer the OpenAI client

Every failure I forced came back as huggingface_hub.errors.HfHubHTTPError, not the typed RateLimitError / AuthenticationError / BadRequestError hierarchy from openai. The HTTP status is on the exception (e.response.status_code when a response object is attached). That is fine for a script that logs and exits; it is awkward if your existing middleware already catches OpenAI SDK exceptions.

I also checked a wrong path on purpose. Pointing base_url at a host that only serves /v1/chat/completions but sending traffic to /api/v1/chat/completions (the /api footgun above) produced a 404 from the mock with a single attempt. There is no helpful "did you mean /v1?" message from InferenceClient — you have to log the URL yourself.

If you already depend on OpenAI retries, typed errors, or max_retries=0 for fail-fast with the same exception types elsewhere in the app, keep using openai.OpenAI(base_url=..., api_key=...). Use InferenceClient when you want one dependency that already talks to the Hub and also needs an occasional OpenAI-compatible chat call, and you are willing to add your own retry wrapper.

Checklist

Disclosure: I build APIClaw, the OpenAI-compatible gateway used in these examples (flat monthly plans; crypto on the site, other payment methods through support). Everything above applies to any compatible endpoint.