When you call the OpenAI API repeatedly, you may encounter error messages that say 429: 'Too Many Requests' or RateLimitError. These error messages come from exceeding the API's rate limits.
This guide shares tips for avoiding and handling rate limit errors.
Rate limits are a common practice for APIs, and they're put in place for a few different reasons.
First, they help protect against abuse or misuse of the API. For example, a malicious actor could flood the API with requests in an attempt to overload it or cause disruptions in service. By setting rate limits, OpenAI can prevent this kind of activity.
Second, rate limits help ensure that everyone has fair access to the API. If one person or organization makes an excessive number of requests, it could bog down the API for everyone else. By throttling the number of requests that a single user can make, OpenAI ensures that everyone has an opportunity to use the API without experiencing slowdowns.
Lastly, rate limits can help OpenAI manage the aggregate load on its infrastructure. If requests to the API increase dramatically, it could tax the servers and cause performance issues. By setting rate limits, OpenAI can help maintain a smooth and consistent experience for all users.
Although hitting rate limits can be frustrating, rate limits exist to protect the reliable operation of the API for its users.
A rate limit error will occur when API requests are sent too quickly. If using the OpenAI Python library, they will look something like:
RateLimitError: Rate limit reached for default-codex in organization org-{id} on requests per min. Limit: 20.000000 / min. Current: 24.000000 / min. Contact support@openai.com if you continue to have issues or if you’d like to request an increase.
Below is example code for triggering a rate limit error.
import openai # for making OpenAI API requests# request a bunch of completions in a loopfor _ inrange(100): openai.ChatCompletion.create(model="gpt-3.5-turbo",messages=[{"role": "user", "content": "Hello"}],max_tokens=10, )
One easy way to avoid rate limit errors is to automatically retry requests with a random exponential backoff. Retrying with exponential backoff means performing a short sleep when a rate limit error is hit, then retrying the unsuccessful request. If the request is still unsuccessful, the sleep length is increased and the process is repeated. This continues until the request is successful or until a maximum number of retries is reached.
This approach has many benefits:
Automatic retries means you can recover from rate limit errors without crashes or missing data
Exponential backoff means that your first retries can be tried quickly, while still benefiting from longer delays if your first few retries fail
Adding random jitter to the delay helps retries from all hitting at the same time
Note that unsuccessful requests contribute to your per-minute limit, so continuously resending a request won’t work.
Tenacity is an Apache 2.0 licensed general-purpose retrying library, written in Python, to simplify the task of adding retry behavior to just about anything.
To add exponential backoff to your requests, you can use the tenacity.retrydecorator. The following example uses the tenacity.wait_random_exponential function to add random exponential backoff to a request.
Note that the Tenacity library is a third-party tool, and OpenAI makes no guarantees about its reliability or security.
import openai # for OpenAI API callsfrom tenacity import ( retry, stop_after_attempt, wait_random_exponential,) # for exponential backoff@retry(wait=wait_random_exponential(min=1, max=60), stop=stop_after_attempt(6))defcompletion_with_backoff(**kwargs):return openai.Completion.create(**kwargs)completion_with_backoff(model="text-davinci-002", prompt="Once upon a time,")
<OpenAIObject text_completion id=cmpl-5oowO391reUW8RGVfFyzBM1uBs4A5 at 0x10d8cae00> JSON: {
"choices": [
{
"finish_reason": "length",
"index": 0,
"logprobs": null,
"text": " a little girl dreamed of becoming a model.\n\nNowadays, that dream"
}
],
"created": 1662793900,
"id": "cmpl-5oowO391reUW8RGVfFyzBM1uBs4A5",
"model": "text-davinci-002",
"object": "text_completion",
"usage": {
"completion_tokens": 16,
"prompt_tokens": 5,
"total_tokens": 21
}
}
Another library that provides function decorators for backoff and retry is backoff.
Like Tenacity, the backoff library is a third-party tool, and OpenAI makes no guarantees about its reliability or security.
import backoff # for exponential backoffimport openai # for OpenAI API calls@backoff.on_exception(backoff.expo, openai.error.RateLimitError)defcompletions_with_backoff(**kwargs):return openai.Completion.create(**kwargs)completions_with_backoff(model="text-davinci-002", prompt="Once upon a time,")
<OpenAIObject text_completion id=cmpl-5oowPhIdUvshEsF1rBhhwE9KFfI3M at 0x111043680> JSON: {
"choices": [
{
"finish_reason": "length",
"index": 0,
"logprobs": null,
"text": " two children lived in a poor country village. In the winter, the temperature would"
}
],
"created": 1662793901,
"id": "cmpl-5oowPhIdUvshEsF1rBhhwE9KFfI3M",
"model": "text-davinci-002",
"object": "text_completion",
"usage": {
"completion_tokens": 16,
"prompt_tokens": 5,
"total_tokens": 21
}
}
If you don't want to use third-party libraries, you can implement your own backoff logic.
# importsimport randomimport timeimport openai# define a retry decoratordefretry_with_exponential_backoff( func, initial_delay: float=1, exponential_base: float=2, jitter: bool=True, max_retries: int=10, errors: tuple= (openai.error.RateLimitError,),):"""Retry a function with exponential backoff."""defwrapper(*args, **kwargs):# Initialize variables num_retries =0 delay = initial_delay# Loop until a successful response or max_retries is hit or an exception is raisedwhileTrue:try:return func(*args, **kwargs)# Retry on specified errorsexcept errors as e:# Increment retries num_retries +=1# Check if max retries has been reachedif num_retries > max_retries:raiseException(f"Maximum number of retries ({max_retries}) exceeded." )# Increment the delay delay *= exponential_base * (1+ jitter * random.random())# Sleep for the delay time.sleep(delay)# Raise exceptions for any errors not specifiedexceptExceptionas e:raise ereturn wrapper@retry_with_exponential_backoffdefcompletions_with_backoff(**kwargs):return openai.Completion.create(**kwargs)completions_with_backoff(model="text-davinci-002", prompt="Once upon a time,")
<OpenAIObject text_completion id=cmpl-5oowRsCXv3AkUgVJyyo3TQrVq7hIT at 0x111024220> JSON: {
"choices": [
{
"finish_reason": "length",
"index": 0,
"logprobs": null,
"text": " a man decided to greatly improve his karma by turning his life around.\n\n"
}
],
"created": 1662793903,
"id": "cmpl-5oowRsCXv3AkUgVJyyo3TQrVq7hIT",
"model": "text-davinci-002",
"object": "text_completion",
"usage": {
"completion_tokens": 16,
"prompt_tokens": 5,
"total_tokens": 21
}
}
If you're processing real-time requests from users, backoff and retry is a great strategy to minimize latency while avoiding rate limit errors.
However, if you're processing large volumes of batch data, where throughput matters more than latency, there are a few other things you can do in addition to backoff and retry.
If you are constantly hitting the rate limit, then backing off, then hitting the rate limit again, then backing off again, it's possible that a good fraction of your request budget will be 'wasted' on requests that need to be retried. This limits your processing throughput, given a fixed rate limit.
Here, one potential solution is to calculate your rate limit and add a delay equal to its reciprocal (e.g., if your rate limit 20 requests per minute, add a delay of 3–6 seconds to each request). This can help you operate near the rate limit ceiling without hitting it and incurring wasted requests.
# importsimport timeimport openai# Define a function that adds a delay to a Completion API calldefdelayed_completion(delay_in_seconds: float=1, **kwargs):"""Delay a completion by a specified amount of time."""# Sleep for the delay time.sleep(delay_in_seconds)# Call the Completion API and return the resultreturn openai.Completion.create(**kwargs)# Calculate the delay based on your rate limitrate_limit_per_minute =20delay =60.0/ rate_limit_per_minutedelayed_completion(delay_in_seconds=delay,model="text-davinci-002",prompt="Once upon a time,")
<OpenAIObject text_completion id=cmpl-5oowVVZnAzdCPtUJ0rifeamtLcZRp at 0x11b2c7680> JSON: {
"choices": [
{
"finish_reason": "length",
"index": 0,
"logprobs": null,
"text": " there was an idyllic little farm that sat by a babbling brook"
}
],
"created": 1662793907,
"id": "cmpl-5oowVVZnAzdCPtUJ0rifeamtLcZRp",
"model": "text-davinci-002",
"object": "text_completion",
"usage": {
"completion_tokens": 16,
"prompt_tokens": 5,
"total_tokens": 21
}
}
The OpenAI API has separate limits for requests per minute and tokens per minute.
If you're hitting the limit on requests per minute, but have headroom on tokens per minute, you can increase your throughput by batching multiple tasks into each request. This will allow you to process more tokens per minute, especially with the smaller models.
Sending in a batch of prompts works exactly the same as a normal API call, except that pass in a list of strings to prompt parameter instead of a single string.
Warning: the response object may not return completions in the order of the prompts, so always remember to match responses back to prompts using the index field.
import openai # for making OpenAI API requestsnum_stories =10prompt ="Once upon a time,"# serial example, with one story completion per requestfor _ inrange(num_stories): response = openai.Completion.create(model="curie",prompt=prompt,max_tokens=20, )# print storyprint(prompt + response.choices[0].text)
Once upon a time, before there were grandiloquent tales of the massacre at Fort Mims, there were stories of
Once upon a time, a full-sized search and rescue was created. However, CIDIs are the addition of requiring
Once upon a time, Schubert was hot with the films. “Schubert sings of honey, flowers,
Once upon a time, you could watch these films on your VCR, sometimes years after their initial theatrical release, and there
Once upon a time, there was a forest. In that forest, the forest animals ruled. The forest animals had their homes
Once upon a time, there were two programs that complained about false positive scans. Peacock and Midnight Manager alike, only
Once upon a time, a long, long time ago, tragedy struck. it was the darkest of nights, and there was
Once upon a time, when Adam was a perfect little gentleman, he was presented at Court as a guarantee of good character.
Once upon a time, Adam and Eve made a mistake. They ate the fruit from the tree of immortality and split the consequences
Once upon a time, there was a set of programming fundamental principles known as the "X model." This is a set of
import openai # for making OpenAI API requestsnum_stories =10prompts = ["Once upon a time,"] * num_stories# batched example, with 10 stories completions per requestresponse = openai.Completion.create(model="curie",prompt=prompts,max_tokens=20,)# match completions to prompts by indexstories = [""] *len(prompts)for choice in response.choices: stories[choice.index] = prompts[choice.index] + choice.text# print storiesfor story in stories:print(story)
Once upon a time, there were two sisters, Eliza Pickering and Ariana 'Ari' Lucas. When these lovely
Once upon a time, Keene was stung by a worm — actually, probably a python — snaking through his leg
Once upon a time, there was a professor of physics during the depression. It was difficult, during this time, to get
Once upon a time, before you got sick, you told stories to all and sundry, and your listeners believed in you
Once upon a time, there was one very old nice donkey. He was incredibly smart, in a very old, kind of
Once upon a time, the property of a common lodging house was a common cup for all the inhabitants. Betimes a constant
Once upon a time, in an unspecified country, there was a witch who had an illegal product. It was highly effective,
Once upon a time, a long time ago, I turned 13, my beautiful dog Duncan swept me up into his jaws like
Once upon a time, as a thoroughly reformed creature from an army of Nazis, he took On Judgement Day myself and his
Once upon a time, Capcom made a game for the Atari VCS called Missile Command. While it was innovative at the time