One taster, many nights.

Is the triangle test a dependable measure for one person scoring themselves?

Status: Open
Protocol: Version 1.0, 2026-10-18
Consent: S.one-taster.v2 (earlier data: S.one-taster.v2.past)
Investigator: John Schuster, Drunken Unicorn Productions
Preregistration: Being posted to the Open Science Framework; the link will appear here
Ethics review: Not yet reviewed; we intend independent review before any results are published
Target: 97 taster-pairs with at least 36 rounds
Runs until: The target is reached, or 2029-10-18

Why this study

The triangle test (ISO 4120) is designed for panels: many assessors, each tasting a few times, with sample sizes and significance tables computed for that design. Savorant uses it for one person tasting many times, calling a result at fixed points (18, 36, 54 rounds) with an exact binomial test against one in three. That use rests on an assumption no one has tested: that a taster’s chance of being right is roughly constant from night to night. If it isn’t, a single person’s verdicts will be less reproducible than the math says.

What taking part means

Nothing extra. Run triangle rounds the way you already do, with a pourer or the glass shuffle. Your triangle rounds: the coded pair being compared, the pour order, whether you were right, the blinding mode, and when.

To join: in the app, turn on research contribution in Settings → Research, then open One taster, many nights under Research studies, read it, and agree. You can leave with one tap.

Hypotheses

H1 (primary). For the same taster and the same pair, the verdict at the first fixed call (18 rounds) and the verdict at the second (rounds 19–36) agree less often than a binomial model with constant accuracy predicts.
H2 (secondary). Night-to-night variation in a taster’s accuracy is greater than binomial (overdispersion above zero).

Design

Observational, within-taster, repeated measures. Each taster-pair with at least 36 rounds provides two independent 18-round verdicts.

Who can take part. Adults 21 or older in the United States who use Savorant, have research contribution turned on, and have joined this study in the app.

Outcomes

Primary. Agreement between the first and second 18-round verdicts on the same pair (both “tells them apart” or both “no difference”).

Secondary and exploratory.

Overdispersion (beta-binomial ρ) of accuracy across nights, with 95% CI.
Agreement between the 18- and 36-round verdicts by blinding mode (pourer or glass shuffle), exploratory.

Sample size

97 taster-pairs with at least 36 rounds. This gives a 95% confidence interval no wider than ±0.10 on the observed agreement (n = 1.96² × 0.25 / 0.10² = 96.04, rounded up).

Analysis

For each taster-pair, the predicted agreement under a binomial model is computed from that taster’s accuracy over all 36 rounds. Observed agreement is compared with predicted agreement across taster-pairs by a one-sided paired permutation test (α = 0.05). Agreement is also reported as Cohen’s κ with a 95% CI. Overdispersion is estimated by a beta-binomial model with nights (session IDs) as the unit.

Data filed before a taster joined is used only if they also agree to the separate earlier-data question. It is labeled as earlier data. The primary analysis uses only data filed after joining; the same analysis including earlier data is reported as a sensitivity analysis.

When a study reports more than one secondary test, p-values are adjusted with the Holm method.

Limitations we know about

Tasters choose their own pairs, so pair difficulty varies. Rounds happen at home, not in a booth. Because there is no pair known to be identical, the study can measure reproducibility but not the false-positive rate.

Standards every Savorant study follows

Preregistration. This page is the protocol of record, versioned and dated, and is posted to the Open Science Framework (OSF) before any study data is analyzed. Hypotheses, outcomes, sample size and analysis are fixed here in advance.
Primary and secondary outcomes. Each study names one primary outcome, tested at the stated alpha. Everything else is labeled secondary or exploratory and reported as such, with effect sizes and 95% confidence intervals rather than bare p-values.
No peeking. No interim analysis of the outcomes. The only checks before the sample size is reached are for data quality (duplicates, impossible values), and they never look at results.
Repeated measures. One taster contributes many rounds, so analyses use mixed-effects models with the taster as a random effect, or per-taster summaries, never pooled rounds treated as independent.
Exact methods for small counts. Triangle results are analyzed against the exact binomial with chance at one in three, the way ISO 4120 sets out the triangle test, adapted here for repeated measures.
Exclusions. Rounds are excluded only for the reasons listed in advance: test accounts, the founder’s accounts, rounds filed in a preview build, and duplicates. Every exclusion is counted and reported.
Missing data. Tasters who stop early are kept for every analysis their data supports. The number who stop, and when, is reported.
Deviations. Any change after data collection starts is logged on this page as a dated amendment with its reason, and reported in the paper.
Reporting. Results are reported following the STROBE guideline for observational studies, with the analysis code published alongside.
Conflict of interest. The study is run by the company that sells the app. Preregistration, publishing regardless of outcome, and public analysis code are how that interest is kept out of the result.

The consent you agree to

This is the exact text shown in the app before you join.

Amendments

None. This is the first version of the protocol.

All open studies · The evidence review · Privacy policy · palate@savorant.com