runs for this decision.No runs for this decision yet.
oldest runs were removed from the history.
Pick models and run it, or select another decision on the left.
Go to DefineProbabilities and answers.
How differently each pair spreads its probabilities: 0 = identical, 1 = no overlap.
| Model |
|---|
Largest gap: at .
For this question, including every repeated call.
| Model | Calls | Time | Tokens in / out | Cost |
|---|---|---|---|---|
| All questions |
of the probability fell outside the options and was left out.
How the probability of the chosen option shifts when each part of the input is removed.
Not run yet for this model and question.
Shifts inside ± are within noise at samples. Shaded rows can't be told apart from zero; raise the samples to resolve them.
sample items with known answers. Confidence is the probability of the chosen option.
Runs every enabled model over sample items with known answers, then shows how well their confidence matches their accuracy. The three default decisions come with known answers; other decisions have none yet, so there is nothing to calibrate.
Question · decision v · items · stopped before the end
oldest calibration runs were removed to make room.
In the busiest bin, , stated on average and was right : well calibrated there.overconfident by points.underconfident by points.
| true ↓ · predicted → |
|---|
Shade = share of the true row. Outlined = correct. Click a cell to list its items.
| Option | True | Picked | Precision | Recall |
|---|
Precision: share of its picks that were right. Recall: share of the items it should have picked that it did. — means undefined.
All models picked the same option on of the items they all answered. All right: · all wrong: · split: .
| same pick |
|---|
Share of items both models answered where they picked the same option.
| Model | Probabilities | Accuracy | Confidence | Gap | ECE | Brier | Answered | Failed |
|---|
Confidence: average probability of the chosen option. Gap: confidence minus accuracy, positive = overconfident. ECE: average gap per confidence bin. Brier: error of the full distribution. Lower is better for both.
| # | Input | Known | Answer | Confidence | Open |
|---|
No items match.
Scored as a probability for each option. P(yes) and P(no). a distribution over the scale and its expected value.
ID:
Version:
expected value
of the probability fell outside the options. Share of answers, unusable left out. Margin ±.