Streamlit is still the fastest way I know to put a chat box in front of a model for a demo or an internal tool. The official chat tutorial is written for api.openai.com, though, and a few things behave differently once you point the client at another OpenAI-compatible endpoint: a vLLM server, a company proxy or a hosted gateway.
I built the app below with Streamlit 1.66.0, openai 3.35.1 and Python 3.13, and drove it with Streamlit's own AppTest harness against a small local mock server that logged every request and could fail on demand. Everything described here is what went over the wire. The examples point at APIClaw, an OpenAI-compatible gateway I build, so weigh my opinions accordingly. Its base URL is https://apiclaw.biz/v1; replace YOUR_MODEL_ID with an exact ID from your provider's model list.
This is the shape most tutorials give you, with the base URL added:
import streamlit as st
from openai import OpenAI
@st.cache_resource
def get_client():
return OpenAI(
base_url=st.secrets["OPENAI_BASE_URL"],
api_key=st.secrets["OPENAI_API_KEY"],
)
client = get_client()
MODEL = st.secrets["MODEL"]
if "messages" not in st.session_state:
st.session_state.messages = []
for m in st.session_state.messages:
with st.chat_message(m["role"]):
st.markdown(m["content"])
if prompt := st.chat_input("Ask something"):
st.session_state.messages.append({"role": "user", "content": prompt})
with st.chat_message("user"):
st.markdown(prompt)
with st.chat_message("assistant"):
stream = client.chat.completions.create(
model=MODEL, messages=st.session_state.messages, stream=True
)
reply = st.write_stream(stream)
st.session_state.messages.append({"role": "assistant", "content": reply})
And .streamlit/secrets.toml next to it:
OPENAI_API_KEY = "sk-..."
OPENAI_BASE_URL = "https://apiclaw.biz/v1"
MODEL = "YOUR_MODEL_ID"
It works. Requests go to POST /v1/chat/completions with Authorization: Bearer <key>, and st.write_stream accepts the SDK's stream object directly and returns the full reply as a plain str. The base URL must include /v1, because the SDK appends /chat/completions to whatever you give it. Add .streamlit/secrets.toml to .gitignore before your first commit.
Streamlit copies root-level keys from secrets.toml into os.environ. I checked when that happens: in a test app, os.environ.get("OPENAI_API_KEY") was None at the top of the script and "sk-..." right after the first st.secrets[...] lookup.
That has two practical effects. First, a bare OpenAI() with no arguments placed before any st.secrets access raised OpenAIError for the missing key, while the same call placed after one access picked up both OPENAI_API_KEY and OPENAI_BASE_URL and sent requests to the custom endpoint. So an app can work or fail depending on line order. Second, because both names are exported, any other library in the same process that reads OPENAI_BASE_URL will now send its traffic to your gateway too. That's usually what you want, but it surprised me when an unrelated helper started hitting the gateway.
I pass base_url and api_key explicitly, as above, so the client doesn't depend on that side effect. If you'd rather not export the values at all, nest them in a table, such as [llm] with st.secrets["llm"]["api_key"]. In my test, only root-level keys were exported.
Streamlit reruns the whole script on every interaction: each chat message, and also every button click or slider change. In my test, clicking an unrelated button reran the script but sent no request, because st.chat_input only returns a value on the run where the user submitted something. Without caching, the client constructor would run again on every one of those reruns. @st.cache_resource builds one client per process and shares it across reruns and sessions. That's fine here, because the OpenAI client is safe to share and holds no per-user state. Don't put a per-user key in a cached client.
st.session_state.messages grows by two entries per turn, and the full list goes out every time. My mock saw 1, then 3, then 5 messages on three consecutive turns. On a flat-rate plan that mostly affects latency and context limits. On per-token billing, it means a long session costs much more per turn than a short one. If your users keep one tab open all day, cap the list or summarize older turns before sending.
Streaming responses only include token counts if you ask for them with stream_options={"include_usage": True}. The server then sends one extra chunk at the end with an empty choices list and a usage object. st.write_stream handled that chunk without complaint in my test, but it consumes the stream and returns only the text, so the usage is gone.
The fix is to hand st.write_stream a small generator that yields text and stores the usage on the side:
def text_and_usage(stream, box):
for chunk in stream:
if chunk.usage:
box["usage"] = chunk.usage
if chunk.choices and chunk.choices[0].delta.content:
yield chunk.choices[0].delta.content
With box = {} passed in, box["usage"].prompt_tokens and .completion_tokens are available after st.write_stream returns. The if chunk.choices check matters: indexing choices[0] on the usage chunk raises an IndexError.
This is the bug most copies of the tutorial code have. The user's message is appended to history before the API call, and the assistant's reply is appended after it. If anything raises in between, the second append never runs.
I tested two failures. In the first, the mock sent a few tokens and then an error event in the middle of the stream, and the SDK raised openai.APIError: upstream died from inside st.write_stream. In the second, the mock returned 429 with Retry-After: 2. The SDK made three attempts (the first retry about 2 seconds later) and then raised RateLimitError. Either way, Streamlit showed the traceback and history was left as ["user"]. On the next turn, the request carried two user messages in a row, ["user", "user"]. Whether a backend accepts that varies, and even when it does, the model now answers a question the user thinks failed.
Wrap the call so a failed turn leaves history the way it was:
from openai import APIError
with st.chat_message("assistant"):
box = {}
try:
stream = client.chat.completions.create(
model=MODEL,
messages=st.session_state.messages,
stream=True,
stream_options={"include_usage": True},
)
reply = st.write_stream(text_and_usage(stream, box))
except APIError as e:
st.session_state.messages.pop()
st.error(f"The model call failed: {e.message}")
st.stop()
APIError is the base class for the connection, timeout and HTTP status errors the SDK raises, so this one clause covers all of them. With this in place, the mid-stream failure showed an error box, history went back to empty, and the next turn sent a single clean user message.
Some models served through compatible endpoints put their reasoning inline as <think>...</think> at the start of content. The tags came through st.write_stream as ordinary text and, in the original code, went straight into history, so every later turn resent the reasoning as part of the assistant's message. Stripping them before saving is enough:
import re
reply = re.sub(r"<think>.*?</think>\s*", "", reply, flags=re.S)
st.session_state.messages.append({"role": "assistant", "content": reply})
The user still sees the reasoning stream past once, which some people like. Hiding it while streaming needs a stateful filter in the generator, because a tag can be split across chunks.
All of the above came from streamlit.testing.v1.AppTest, which runs the script headlessly and lets you submit chat input:
from streamlit.testing.v1 import AppTest
at = AppTest.from_file("app.py", default_timeout=30)
at.run()
at.chat_input[0].set_value("hello").run()
assert not at.exception
assert [m["role"] for m in at.session_state.messages] == ["user", "assistant"]
It reads .streamlit/secrets.toml from the working directory, so pointing OPENAI_BASE_URL at a local mock in a test copy of that file gives you a repeatable test for the failure paths. That test is how I found the orphaned-message bug in my own app.
/v1. Pass it and the key to OpenAI() explicitly.os.environ on first access.@st.cache_resource, never a per-user one.st.write_stream in a generator if you want usage, and guard choices.Disclosure: I build APIClaw, the OpenAI-compatible gateway used in these examples (flat monthly plans; crypto on the site, other payment methods through support). Everything above applies to any compatible endpoint.