Skip to content

[admission]

Queue limits for concurrent requests (off by default).

Key Type Default Description
admission.enabled bool false enabled turns on queue limits for incoming requests (default false). A request that finds its queue full gets 503 overload with Retry-After: 5.
admission.online_chat_depth integer 200 online_chat_depth is how many chat requests may wait at once (default 200).
admission.online_embed_depth integer 200 online_embed_depth is how many embedding requests may wait at once (default 200).
admission.online_rerank_depth integer 100 online_rerank_depth is how many rerank requests may wait at once (default 100).
admission.batch_embed_depth integer 5000 batch_embed_depth is how many batch embedding requests may wait at once (default 5000).
admission.online_max_wait_ms integer 30 OnlineMaxWaitMS is how long an online request may wait in the queue before it gets 504 timeout (default 30 ms).