[admission]
Queue limits for concurrent requests (off by default).
| Key | Type | Default | Description |
|---|---|---|---|
admission.enabled |
bool | false |
enabled turns on queue limits for incoming requests (default false). A request that finds its queue full gets 503 overload with Retry-After: 5. |
admission.online_chat_depth |
integer | 200 |
online_chat_depth is how many chat requests may wait at once (default 200). |
admission.online_embed_depth |
integer | 200 |
online_embed_depth is how many embedding requests may wait at once (default 200). |
admission.online_rerank_depth |
integer | 100 |
online_rerank_depth is how many rerank requests may wait at once (default 100). |
admission.batch_embed_depth |
integer | 5000 |
batch_embed_depth is how many batch embedding requests may wait at once (default 5000). |
admission.online_max_wait_ms |
integer | 30 |
OnlineMaxWaitMS is how long an online request may wait in the queue before it gets 504 timeout (default 30 ms). |