How we measured Prism under load
The Under load section of prism.tm shows four figures: 1.0 ms, 2,000 streams, ~250 MB and 2,000 requests per second. We measured them on 16 October 2026. They apply to Prism v0.44.0 and later; earlier releases do not have the changes they depend on. This post describes the setup, what each figure means, and what the test did not cover.
The setup
Machine. Linux on an Intel Core i5-13500 (20 hardware threads) with NVMe storage.
Prism's share. Prism was limited to 4 hardware threads (two performance cores with hyper-threading), comparable to a virtual machine with 4 vCPUs. The test provider and the load generator ran on other cores of the same machine. All connections were plain HTTP over loopback.
Shared machine. The machine ran other work during the tests. The Linux load average was 11 to 16 on 20 hardware threads.
Test provider. A small OpenAI-compatible server. For non-streaming requests it answers immediately (in about 0.1 ms when called directly), with 12 prompt tokens and 8 completion tokens. For streaming requests it sends 40 chunks 500 ms apart, so each answer takes 20 seconds. It reports the model as gpt-4o-mini, so Prism computes the cost of each answer from the gpt-4o-mini price in its model catalog. Prism had this one provider with one model, so it never had to choose between models.
Load generator. It works in one of two modes: a fixed number of clients, each sending its next request when the previous one is answered; or a fixed number of requests per second, using up to one connection per 20 requests per second (at least 50 connections; 100 at 2,000 per second). Latency is measured from the moment a connection sends a request to the end of the answer.
Requests with a personal key. Every request carried a Prism key issued to a person in a Prism Teams organization (Prism Teams is coming soon). For each such request Prism adds the cost and tokens to that person's spend and writes a request log entry; this takes about 15 to 17 KB in Prism's database per request. The tests used two keys of the same person: since v0.43.0 one person's key allows up to 1,024 requests in progress at the same time, and the stream test needed 2,000.
Empty database. Each test started Prism with an empty database, with no earlier requests stored. The memory test used one Prism process for all of its requests.
Background work off. The model catalog refresh, the price refresh and the provider health checks were turned off during the tests.
What each figure means
1.0 ms added to each request
One client sent 1,000 non-streaming requests, one after another. The median time per request was 0.98 ms; the 95th percentile was 2.74 ms and the 99th percentile 4.95 ms. The test provider takes about 0.1 ms of that, so almost all of it is Prism's own time. A real provider takes from several hundred milliseconds to several seconds to answer.
2,000 streams at the same time
2,000 clients, 1,000 per key, each opened a streaming request at the same moment, and each sent three 20-second streams one after another. We ran this test three times. All 6,000 streams completed in each run, 18,000 in total.
The median time from sending a request to receiving the first byte of the answer was 0.10 to 0.33 seconds, and the 99th percentile about 1 second. This is longer than for a single request because 2,000 requests arrive at the same time. While the streams were open, the Prism process used up to about 0.9 GB of memory, including the database pages described in the next section.
~250 MB of memory
We sent 80,000 non-streaming requests to one Prism process in four batches of 20,000, from 32 clients, without restarting it. Each batch took about 9 seconds, about 2,300 requests per second. Five seconds after each batch, Prism's own memory was 154, 182, 128 and 251 MB.
Prism's own memory is the memory it allocated for its data; Linux reports it as RssAnon. Prism also maps its database file into memory. Counting those pages, the process used between 255 and 930 MB (558 MB after the last batch). The operating system can drop those pages when another program needs memory and read them from disk again later.
Heavier load uses more memory: after the step at 2,500 requests per second, Prism's own memory was 536 MB. 80,000 requests are too few to show whether memory keeps growing over days of traffic.
2,000 requests per second
The generator sent 2,000 requests per second for 30 seconds. Prism answered all 59,999 requests with HTTP 200. The median latency was 23 ms and the 99th percentile 102 ms. Prism's counters of dropped records and failed record writes (prism_record_sink_dropped_total and prism_record_sink_write_errors_total) stayed at zero. Prism writes these records in the background; the answer does not wait for them.
| Requests per second | Median | 99th percentile | Refused |
|---|---|---|---|
| 300 | 1.1 ms | 4.7 ms | 0 |
| 600 | 1.0 ms | 7.2 ms | 0 |
| 900 | 1.3 ms | 15 ms | 0 |
| 1,200 | 2.7 ms | 30 ms | 0 |
| 1,500 | 6.0 ms | 56 ms | 0 |
| 2,000 | 23 ms | 102 ms | 0 |
| 2,500 | 46 ms | 141 ms | 873 (1%) |
At 2,500 requests per second the 30-second step took 32.4 seconds. Prism answered 74,127 requests, 2,288 per second, and refused 873 with HTTP 429 Too Many Requests. Prism sends this refusal before it forwards the request to the provider, so the provider does not receive or charge for a refused request, and the client can send it again.
Requests without a key
Without a key, Prism does less work per request. Measured the same day: a median of 0.63 ms per request with one client, 4,300 requests per second with 50 clients, and 3,000 requests per second for 60 seconds with every record kept (the only fixed rate we ran). In a longer run at about 3,500 requests per second, Prism dropped 1,292 records in the last batch of 130,000.
What changed since September
In September the Under load section showed 0.4 ms per request, a first token in 49 ms with 2,000 streams, ~260 MB of memory that did not grow over 500,000 requests, and 4,500 requests per second. Those figures were for requests without a key, measured on 27 and 28 September on Prism v0.14. The figures on prism.tm now describe requests with a personal key on Prism v0.44.0. The two sets are not directly comparable.
What the test did not cover
- One machine, shared with other work. Each fixed rate was sent for 30 seconds (60 seconds for the run without a key), and the memory test took about 35 seconds. We did not test hours of continuous load.
- The test provider answers immediately, over plain HTTP on loopback. With HTTPS to real providers, Prism also spends CPU on encryption. A real provider takes seconds per answer, so the same rate means many more requests in progress at the same time: at 2,000 requests per second with 20-second answers, 40,000 requests would be in progress. In the 2,000-per-second test there were at most 100; in the stream test, 2,000.
- One provider and one model, so Prism made no choice between models or providers.
- The generator sent the same short request every time. Real requests are larger and all different. Prism takes longer to read a larger request, and when the same request body arrives again it reuses part of the work, so identical requests cost less than real traffic.
- The model catalog refresh, the price refresh and the provider health checks were off.
- Each test started with an empty database. A database that already holds months of requests is larger; we did not test how that changes the figures.
- On our disk, appending 300 bytes to a file and flushing it to disk took a median of about 5.6 ms, and up to 17 ms. With Prism's database in memory (tmpfs), where flushing costs nothing, Prism also kept up with 2,000 requests per second (median 33 ms), and one client got 1.14 ms per request.
Comparing with your setup
- Prism had 4 hardware threads. Fewer threads mean fewer requests per second.
- Multiply your requests per second by your provider's answer time to get the number of requests in progress at the same time. That number, not the rate alone, decides memory and connection use.
- One person's key allows up to 1,024 requests in progress at the same time (Prism v0.43.0 and later).
- To check that no record is lost on your installation, watch prism_record_sink_dropped_total and prism_record_sink_write_errors_total in Prism's Prometheus metrics.
- To compare your disk with ours, time a small append followed by fsync. We got a median of 5.6 ms.