I'm Aman Kumar. A Rust question I get fairly often goes like this: "I added async-openai, pointed it at a non-OpenAI endpoint, and now either nothing compiles or every request fails." Usually it's one of three things. The chat API isn't enabled in Cargo, the base URL is a little off, or the model ID is a guess.
This post walks through using async-openai (version 0.42 at the time of writing) against any OpenAI-compatible API. It covers the setup, a normal request, streaming, timeouts and retries, and how I turn errors into something I can act on. My examples use APIClaw, an OpenAI-compatible gateway I build, so weigh that accordingly. Nothing below is specific to it. The same code works against vLLM, LiteLLM, Ollama's /v1 endpoint or any other server that implements /v1/chat/completions.
This is the one that surprises people. In current versions of async-openai, the default feature set only picks a TLS backend (rustls). The API groups are opt-in. If you run cargo add async-openai and then write client.chat(), the compiler will tell you that method doesn't exist.
Turn on what you use:
[dependencies]
async-openai = { version = "0.42", features = ["chat-completion", "model"] }
reqwest = "0.13"
tokio = { version = "1", features = ["full"] }
futures = "0.3"
chat-completion gives you Chat Completions plus streaming. model gives you client.models().list(), which I always use to check model IDs. There are separate flags for embedding, responses and the other API families if you need them.
I add reqwest directly only because I pass in my own HTTP client (Step 4). Keep it on the same major version that async-openai uses (0.13 for the 0.42 line). If the versions differ, Cargo pulls in two copies of reqwest and with_http_client won't accept your client. The resulting type error is confusing, but the fix is simply to match the versions.
The client reads OPENAI_API_KEY and OPENAI_BASE_URL from the environment by default. That's handy, but I'd rather be explicit, especially when one service talks to more than one provider:
use async_openai::{config::OpenAIConfig, Client};
fn make_config() -> OpenAIConfig {
let api_base = std::env::var("LLM_BASE_URL")
.unwrap_or_else(|_| "https://apiclaw.biz/v1".to_string());
let api_key = std::env::var("LLM_API_KEY").expect("LLM_API_KEY not set");
OpenAIConfig::new()
.with_api_base(api_base.trim_end_matches('/'))
.with_api_key(api_key)
}
There are two details here, and both cause 404s.
Include /v1, and nothing after it. The crate builds each URL by gluing the base onto a path such as /chat/completions, so the base has to end at the version segment. Don't use the bare domain (that drops /v1), and don't use the full .../v1/chat/completions endpoint (that doubles the path).
No trailing slash. Because the join is plain string concatenation, https://example.com/v1/ turns into https://example.com/v1//chat/completions. Some servers tolerate the double slash and many don't. The trim_end_matches('/') call is there so a slash that sneaks in through an environment variable can't break the URL.
use async_openai::types::chat::{
ChatCompletionRequestSystemMessageArgs, ChatCompletionRequestUserMessageArgs,
CreateChatCompletionRequestArgs,
};
async fn ask(client: &Client<OpenAIConfig>, model: &str) -> Result<(), Box<dyn std::error::Error>> {
let request = CreateChatCompletionRequestArgs::default()
.model(model)
.max_tokens(400u32)
.messages([
ChatCompletionRequestSystemMessageArgs::default()
.content("You are a concise assistant.")
.build()?
.into(),
ChatCompletionRequestUserMessageArgs::default()
.content("Explain what a base URL is in one paragraph.")
.build()?
.into(),
])
.build()?;
let response = client.chat().create(request).await?;
if let Some(choice) = response.choices.first() {
println!("{}", choice.message.content.as_deref().unwrap_or(""));
}
if let Some(usage) = response.usage {
println!("tokens: {} in, {} out", usage.prompt_tokens, usage.completion_tokens);
}
Ok(())
}
model takes a plain string, so you aren't limited to OpenAI's model names. Keep it in config rather than in code. Then switching models is an environment change, not a rebuild.
Get the model ID from the server. A wrong model ID is the most common failure I see with any gateway. Before writing anything else, I list what the key can actually see:
let models = client.models().list().await?;
for m in models.data {
println!("{}", m.id);
}
Copy the ID exactly as it prints, including any prefix or version suffix. For APIClaw, the same IDs are also listed on its public model hub.
The default client is built on a reqwest::Client with no timeouts at all. That's fine in a script and risky in a server: a stalled upstream connection can hold a task forever. I build my own HTTP client and pass it in:
use std::time::Duration;
fn make_client() -> Client<OpenAIConfig> {
let http = reqwest::Client::builder()
.connect_timeout(Duration::from_secs(10))
.read_timeout(Duration::from_secs(60))
.build()
.expect("failed to build HTTP client");
Client::with_config(make_config()).with_http_client(http)
}
I use read_timeout rather than timeout on purpose. In reqwest, timeout is a total deadline that runs until the response body has finished. For a streamed answer that takes two minutes to generate, that deadline kills a perfectly healthy stream. read_timeout resets after every successful read, so it catches a stalled connection without punishing a long one. If you only make short, non-streaming calls, a total timeout is fine too.
use futures::StreamExt;
use std::io::{stdout, Write};
let mut stream = client.chat().create_stream(request).await?;
let mut out = stdout().lock();
while let Some(item) = stream.next().await {
match item {
Ok(chunk) => {
for choice in &chunk.choices {
if let Some(text) = &choice.delta.content {
write!(out, "{text}")?;
}
}
out.flush()?;
}
Err(e) => {
eprintln!("\nstream error: {e}");
break;
}
}
}
A few things are worth knowing. create_stream sets stream: true for you, and calling create on a request that has stream set returns an InvalidArgument error, so build the request without it. Iterate over choices instead of indexing [0]. If you turn on usage reporting through the request's stream_options, the final usage chunk has an empty choices array, and indexing would panic on it. Stdout is line-buffered, so flush after each chunk if you want the text to appear as it arrives.
In the 0.42 line, the default client wraps every request in a retry layer. It retries up to three times on 429s, 5xx responses and connection errors, with exponential backoff starting at 100 ms and capped at 8 seconds. It also respects a Retry-After header. For streams, the retry covers getting the response started. A connection that drops halfway through an answer is not replayed, so handle the Err arm in your loop. It deliberately does not retry a 429 whose error code is insufficient_quota, because waiting won't fix a billing limit.
So don't wrap calls in a second retry loop of your own. Stacking retries multiplies the request count during an outage, which is exactly when the upstream is struggling. If you need different behaviour, the crate supports Tower middleware behind the middleware feature, and you can replace the policy there.
Every call returns OpenAIError. The variants I care about map neatly to fixes:
use async_openai::error::OpenAIError;
fn describe(err: &OpenAIError) -> String {
match err {
OpenAIError::ApiError(resp) => match resp.status_code.as_u16() {
401 => "bad or missing API key".into(),
404 => format!("wrong path or model ID: {}", resp.api_error.message),
429 => "rate limited or out of quota".into(),
code => format!("HTTP {code}: {}", resp.api_error.message),
},
OpenAIError::JSONDeserialize(e, body) => {
format!("response did not match the expected shape ({e}); body: {body}")
}
OpenAIError::Reqwest(e) if e.is_timeout() => "request timed out".into(),
other => other.to_string(),
}
}
JSONDeserialize deserves a note. Compatible servers sometimes add or omit fields compared with OpenAI's spec, and the crate's strongly typed responses can reject them. When that happens, the byot ("bring your own types") feature adds _byot versions of the methods, such as create_byot. Those accept and return your own types, including a plain serde_json::Value. I use that as an escape hatch for one odd endpoint rather than giving up on the typed API everywhere.
chat() is missing, add the chat-completion feature.https://host/v1 with no trailing slash and no /chat/completions?models().list() with the same key?status_code instead of reading strings.timeout for read_timeout.That covers nearly every Rust integration problem I've debugged against OpenAI-compatible servers. The crate does most of the work. You just have to give it the right features, a clean base URL and a real model ID.
Disclosure: I build APIClaw, the OpenAI-compatible gateway used in these examples. Everything here applies to any compatible endpoint.