Skip to content

Experiments and experiment metrics

Experiments are off on a new installation, and nothing about them is sent anywhere until the installation owner turns them on. Metrics are a separate, second opt-in, and each person must agree to their own metrics.

Experiments become available in one of two ways:

  • Pull-in by the Prism team. For a paid subscription, the Prism team can make experiments available from its subscription admin. The decision arrives with the signed subscription grant at the next check, and it holds until the Prism team clears it. The owner sees this under Users and budgets → Members → Experiment access and cannot remove it there. Pull-in only makes experiments available. It does not turn on any experiment, does not choose where text goes, and sends nothing.
  • The owner’s own opt-in in the same place (Enable experiment access for this installation now). This is for installations without a subscription.

Once experiments are available, everything else is the owner’s decision:

  • which experiments run, in the Experiments section of the settings;
  • where the text of requests goes for the experiments that need it (Laya addresses, the verifier and judge models). Only the owner can change these settings; people the owner admitted to experiments see them read-only;
  • which of the installation’s people take part (Experiment access).

Nothing the Prism team does can change these choices.

Experiment What it does with a request Where the text goes
Routing probe (routing_probe) Asks a judge model whether the answer was good, and has cheaper or stronger models answer the same request to compare The judge and arm models the owner configured
Verifier (v2.verifier) Has a second model check a sample of answers The verifier model the owner configured
Secret detection (secret_shadow) Looks for secrets in requests and classifies the findings with Laya The Laya address the owner configured
Laya skill and reaction (laya_shadow) Classifies the kind of task and the user’s reaction to the previous answer The Laya addresses the owner configured
Meaning-based routing (semantic) Remembers which model served similar requests well; the embeddings come from a local model Prism knows (a connection or a discovered server) or from the installed one Stays on the installation
Local embeddings (provisioning.local_embeddings) Shows the embeddings model semantic memory uses. When there is none and the owner picks one, Prism reads the model list from Hugging Face, downloads the model from Hugging Face and the llama.cpp server from GitHub, and runs them on the installation Requests stay on the installation; Hugging Face and GitHub receive only the list and download requests, no request text

Requests under the private strategy or a local-only policy are never sent to a remote judge, verifier or Laya. Each experiment keeps its own log on the installation; see What Prism stores.

Off by default. Metrics send the results of experiments, never the requests themselves, to PostHog so the Prism team can see whether an experiment helps.

When they are sent. A person’s experiment results are sent only while all of these hold:

  1. experiments are available (pull-in or the owner’s opt-in);
  2. the owner turned metrics on (Experiment access → Turn metrics on);
  3. the owner admitted that person to experiments;
  4. the person agreed in their own profile (Profile → Experiment metrics → I consent).

A person can withdraw consent in their profile at any time, whether or not they still take part. Sending stops at once, and events already waiting to be sent are dropped. Turning off any of the other three conditions stops sending the same way. Requests made with the owner’s keys or on the ordinary listener count as the owner’s.

Where they go. PostHog Cloud EU (https://eu.i.posthog.com/batch/), project Prism_instances, directly over HTTPS through Prism’s outbound policy. Prism contains no PostHog library and loads nothing from PostHog.

What one event contains. One event per experiment result: routing_probe_result, verifier_result, secret_shadow_result, laya_skill_result, laya_reaction_result or semantic_route_result. Each carries:

  • distinct_id: the installation pseudonym and a person pseudonym (below);
  • installation: the first 32 hex digits of a SHA-256 of the installation fingerprint;
  • prism_version and experiment;
  • the result: outcome or verdict and its probabilities, which provider and model served, judged or classified, tier and effort, latency, cost, whether the input was truncated, and error codes.

What is never sent. Request or answer text; embeddings or prompt fingerprints; names, email addresses, local user IDs; session, application, integration, request or trace IDs; the fingerprint of a detected secret; error messages; host names, addresses, URLs or file paths. Events disable PostHog person profiles and GeoIP lookup.

Pseudonyms. The person pseudonym is an HMAC-SHA256 of the local user ID keyed by experiment-metrics.key, a random key created in the storage directory, readable only by Prism’s account, that never leaves the installation. Without the key a pseudonym cannot be tied to a person; the same person on another installation has a different pseudonym. Deleting the key starts new pseudonyms.

Delivery. Events are queued in memory and sent in batches about every 10 seconds. A failed batch is retried twice and then dropped; nothing is written to disk. The owner sees the counts (sent, dropped, failed, withdrawn) under Experiment access.

Delete sent data. Ask the Prism team to delete an installation’s events. Withdrawing consent or turning metrics off stops new events; it does not delete events already sent.