OpenAI API compatibility guide

Integrate vLLM workers with OpenAI client libraries and API-compatible tools.

Runpod’s vLLM workers implement OpenAI API compatibility, allowing you to use familiar OpenAI client libraries with your deployed models. This guide explains how to leverage this compatibility to integrate your models seamlessly with existing OpenAI-based applications.

Endpoint structure

When using the OpenAI-compatible API with Runpod, your requests are directed to this base URL pattern:

https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1

Replace ENDPOINT_ID with your Serverless endpoint ID.

Supported APIs

The vLLM worker implements these core OpenAI API endpoints:

Endpoint	Description	Status
/chat/completions	Generate chat model completions	Fully supported
/completions	Generate text completions	Fully supported
/models	List available models	Supported

Model naming

The MODEL_NAME environment variable is essential for all OpenAI-compatible API requests. This variable corresponds to either:

1. The Hugging Face model you’ve deployed (e.g., mistralai/Mistral-7B-Instruct-v0.2).
2. A custom name if you’ve set OPENAI_SERVED_MODEL_NAME_OVERRIDE as an environment variable.

This model name is used in chat and text completion API requests to identify which model should process your request.

Initialize the OpenAI client

Before you can send API requests, set up an OpenAI client with your Runpod API key and endpoint URL:

from openai import OpenAI

MODEL_NAME = "mistralai/Mistral-7B-Instruct-v0.2"  # Use your deployed model

client = OpenAI(
    api_key="<RUNPOD_API_KEY>",
    base_url="https://api.runpod.ai/v2/<endpoint_id>/openai/v1",
)

Send requests

You can use Runpod’s OpenAI-compatible API to send requests to your Runpod endpoint, enabling you to use the same client libraries and code that you use with OpenAI’s services. You only need to change the base URL to point to your Runpod endpoint.

You can also send requests using Runpod’s native API, which provides additional flexibility and control.

Chat completions

The /chat/completions endpoint is designed for instruction-tuned LLMs that follow a chat format.

Non-streaming request

Here’s how you can make a basic chat completion request:

from openai import OpenAI

MODEL_NAME = "mistralai/Mistral-7B-Instruct-v0.2"  # Use your deployed model

# Initialize the OpenAI client
client = OpenAI(
    api_key="<RUNPOD_API_KEY>",
    base_url="https://api.runpod.ai/v2/<endpoint_id>/openai/v1",
)

# Chat completion request (for instruction-tuned models)
response = client.chat.completions.create(
    model=MODEL_NAME,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hello, who are you?"}
    ],
    temperature=0.7,
    max_tokens=500
)

# Print the response
print(response.choices[0].message.content)

Response format

The API returns responses in this JSON format:

{
  "id": "cmpl-123abc",
  "object": "chat.completion",
  "created": 1677858242,
  "model": "mistralai/Mistral-7B-Instruct-v0.2",
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "I am Mistral, an AI assistant based on the Mistral-7B-Instruct model. How can I help you today?"
      },
      "index": 0,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 23,
    "completion_tokens": 24,
    "total_tokens": 47
  }
}

Streaming request

Streaming allows you to receive the model’s output incrementally as it’s generated, rather than waiting for the complete response. This real-time delivery enhances responsiveness, making it ideal for interactive applications like chatbots or for monitoring the progress of lengthy generation tasks.

# Create a streaming chat completion request
stream = client.chat.completions.create(
    model=MODEL_NAME,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write a short poem about stars."}
    ],
    temperature=0.7,
    max_tokens=200,
    stream=True  # Enable streaming
)

# Print the streaming response
print("Response: ", end="", flush=True)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()

Text completions

The /completions endpoint is designed for base LLMs and text completion tasks.

Non-streaming request

Here’s how you can make a text completion request:

# Text completion request
response = client.completions.create(
    model=MODEL_NAME,
    prompt="Write a poem about artificial intelligence:",
    temperature=0.7,
    max_tokens=150
)

# Print the response
print(response.choices[0].text)

Response format

The API returns responses in this JSON format:

{
  "id": "cmpl-456def",
  "object": "text_completion",
  "created": 1677858242,
  "model": "mistralai/Mistral-7B-Instruct-v0.2",
  "choices": [
    {
      "text": "In circuits of silicon and light,\nA new form of mind takes flight.\nNot born of flesh, but of human design,\nArtificial intelligence, a marvel divine.",
      "index": 0,
      "finish_reason": "stop",
      "logprobs": null
    }
  ],
  "usage": {
    "prompt_tokens": 8,
    "completion_tokens": 39,
    "total_tokens": 47
  }
}

Streaming request

# Create a completion stream
response_stream = client.completions.create(
    model=MODEL_NAME,
    prompt="Runpod is the best platform because",
    temperature=0,
    max_tokens=100,
    stream=True,
)

# Stream the response
for response in response_stream:
    print(response.choices[0].text or "", end="", flush=True)

List available models

The /models endpoint allows you to get a list of available models on your endpoint:

models_response = client.models.list()
list_of_models = [model.id for model in models_response]
print(list_of_models)

Response format

{
  "object": "list",
  "data": [
    {
      "id": "mistralai/Mistral-7B-Instruct-v0.2",
      "object": "model",
      "created": 1677858242,
      "owned_by": "runpod"
    }
  ]
}

Chat completion parameters

Here are all available parameters for the /chat/completions endpoint:

Parameter	Type	Default	Description
messages	list[dict[str, str]]	Required	List of messages with role and content keys. The model’s chat template will be applied automatically.
model	string	Required	The model repo that you’ve deployed on your Runpod Serverless endpoint.
temperature	float	0.7	Controls the randomness of sampling. Lower values make it more deterministic, higher values make it more random. Zero means greedy sampling.
top_p	float	1.0	Controls the cumulative probability of top tokens to consider. Must be in (0, 1]. Set to 1 to consider all tokens.
n	int	1	Number of output sequences to return for the given prompt.
max_tokens	int	None	Maximum number of tokens to generate per output sequence.
seed	int	None	Random seed to use for the generation.
stop	string or list[str]	list	String(s) that stop generation when produced. The returned output will not contain the stop strings.
stream	bool	false	Whether to stream the response.
presence_penalty	float	0.0	Penalizes new tokens based on whether they appear in the generated text so far. Values > 0 encourage new tokens, values < 0 encourage repetition.
frequency_penalty	float	0.0	Penalizes new tokens based on their frequency in the generated text so far. Values > 0 encourage new tokens, values < 0 encourage repetition.
logit_bias	dict[str, float]	None	Unsupported by vLLM.
user	string	None	Unsupported by vLLM.

Additional vLLM parameters

vLLM supports additional parameters beyond the standard OpenAI API:

Parameter	Type	Default	Description
best_of	int	None	Number of output sequences generated from the prompt. From these best_of sequences, the top n sequences are returned. Must be ≥ n. Treated as beam width when use_beam_search is true.
top_k	int	-1	Controls the number of top tokens to consider. Set to -1 to consider all tokens.
ignore_eos	bool	false	Whether to ignore the EOS token and continue generating tokens after EOS is generated.
use_beam_search	bool	false	Whether to use beam search instead of sampling.
stop_token_ids	list[int]	list	List of token IDs that stop generation when produced. The returned output will contain the stop tokens unless they are special tokens.
skip_special_tokens	bool	true	Whether to skip special tokens in the output.
spaces_between_special_tokens	bool	true	Whether to add spaces between special tokens in the output.
add_generation_prompt	bool	true	Whether to add generation prompt. Read more here.
echo	bool	false	Echo back the prompt in addition to the completion.
repetition_penalty	float	1.0	Penalizes new tokens based on whether they appear in the prompt and generated text so far. Values > 1 encourage new tokens, values < 1 encourage repetition.
min_p	float	0.0	Minimum probability for a token to be considered.
length_penalty	float	1.0	Penalizes sequences based on their length. Used in beam search.
include_stop_str_in_output	bool	false	Whether to include the stop strings in output text.

Text completion parameters

Here are all available parameters for the /completions endpoint:

Parameter	Type	Default	Description
prompt	string or list[str]	Required	The prompt(s) to generate completions for.
model	string	Required	The model repo that you’ve deployed on your Runpod Serverless endpoint.
temperature	float	0.7	Controls the randomness of sampling. Lower values make it more deterministic, higher values make it more random. Zero means greedy sampling.
top_p	float	1.0	Controls the cumulative probability of top tokens to consider. Must be in (0, 1]. Set to 1 to consider all tokens.
n	int	1	Number of output sequences to return for the given prompt.
max_tokens	int	16	Maximum number of tokens to generate per output sequence.
seed	int	None	Random seed to use for the generation.
stop	string or list[str]	list	String(s) that stop generation when produced. The returned output will not contain the stop strings.
stream	bool	false	Whether to stream the response.
presence_penalty	float	0.0	Penalizes new tokens based on whether they appear in the generated text so far. Values > 0 encourage new tokens, values < 0 encourage repetition.
frequency_penalty	float	0.0	Penalizes new tokens based on their frequency in the generated text so far. Values > 0 encourage new tokens, values < 0 encourage repetition.
logit_bias	dict[str, float]	None	Unsupported by vLLM.
user	string	None	Unsupported by vLLM.

Text completions support the same additional vLLM parameters as chat completions (see the Additional vLLM parameters section above).

Environment variables

Use these environment variables to customize the OpenAI compatibility:

Variable	Default	Description
RAW_OPENAI_OUTPUT	1 (true)	Enables raw OpenAI SSE format for streaming.
OPENAI_SERVED_MODEL_NAME_OVERRIDE	None	Override the model name in responses.
OPENAI_RESPONSE_ROLE	assistant	Role for responses in chat completions.

For a complete list of vLLM environment variables, see the environment variables documentation.

Client libraries

The OpenAI-compatible API works with standard OpenAI client libraries:

Python

from openai import OpenAI

client = OpenAI(
    api_key="<RUNPOD_API_KEY>",
    base_url="https://api.runpod.ai/v2/<endpoint_id>/openai/v1"
)

response = client.chat.completions.create(
    model="<MODEL_NAME>",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hello!"}
    ]
)

JavaScript

import { OpenAI } from "openai";

const openai = new OpenAI({
  apiKey: "<RUNPOD_API_KEY>",
  baseURL: "https://api.runpod.ai/v2/<endpoint_id>/openai/v1"
});

const response = await openai.chat.completions.create({
  model: "<MODEL_NAME>",
  messages: [
    { role: "system", content: "You are a helpful assistant." },
    { role: "user", content: "Hello!" }
  ]
});

Implementation differences

While the vLLM worker aims for high compatibility, there are some differences from OpenAI’s implementation:

Token counting may differ slightly from OpenAI models due to different tokenizers.

Streaming format follows OpenAI’s Server-Sent Events (SSE) format, but the exact chunking of streaming responses may vary.

Error responses follow a similar but not identical format to OpenAI’s error responses.

Rate limits follow Runpod’s endpoint policies rather than OpenAI’s rate limiting structure.

Current limitations

The vLLM worker has a few limitations:

• Function and tool calling APIs are not currently supported.
• Some OpenAI-specific features like moderation endpoints are not available.
• Vision models and multimodal capabilities depend on the underlying model support in vLLM.

Troubleshooting

Common issues and their solutions:

Issue	Solution
”Invalid model” error	Verify your model name matches what you deployed.
Authentication error	Check that you’re using your Runpod API key, not an OpenAI key.
Timeout errors	Increase client timeout settings for large models.
Incompatible responses	Set RAW_OPENAI_OUTPUT=1 in your environment variables.
Different response format	Some models may have different output formatting; use a chat template.

Next steps

• Learn how to send vLLM requests using Runpod’s native API.
• Explore environment variables for customization.
• Review all Serverless endpoint operations.
• Explore the OpenAI API documentation.