Same whiskey, same words?.

Is describing the same whiskey the same way twice a stable, measurable skill?

Status: Open
Protocol: Version 1.0, 2026-10-18
Consent: S.repeat.v1 (earlier data: S.repeat.v1.past)
Investigator: John Schuster, Drunken Unicorn Productions
Preregistration: Being posted to the Open Science Framework; the link will appear here
Ethics review: Not yet reviewed; we intend independent review before any results are published
Target: 36 tasters with at least six repeat checks
Runs until: The target is reached, or 2029-10-18

Why this study

Sensory panels screen assessors for repeatability before trusting them. Consumers are never measured this way. Savorant’s repeat check gives every taster a consistency score, and the app treats 95 as the standard. Whether that score is a stable property of a taster, and whether 95 is the right line, has not been tested.

What taking part means

Nothing extra. Keep doing mystery pours and repeat checks the way you already do. Your repeat checks: the agreement between your two descriptions of the same whiskey (families, words, and measures), the descriptors and measures behind them, and when.

To join: in the app, turn on research contribution in Settings → Research, then open Same whiskey, same words? under Research studies, read it, and agree. You can leave with one tap.

Hypotheses

H1 (primary). The consistency score is reliable: the intraclass correlation between a taster’s mean score on their first three repeat checks and their next three is greater than 0.5.
H2 (secondary). Consistency scores rise as tasters complete more sessions.

Design

Observational test-retest reliability study, within-taster, repeated measures.

Who can take part. Adults 21 or older in the United States who use Savorant, have research contribution turned on, and have joined this study in the app.

Outcomes

Primary. Intraclass correlation, ICC(2,1), between block 1 (repeat checks 1–3) and block 2 (repeat checks 4–6) mean consistency.

Secondary and exploratory.

Change in consistency with sessions completed (linear mixed model, random intercept and slope for taster).
The distribution of consistency scores relative to the app’s 95 standard (descriptive).

Sample size

36 tasters with at least six repeat checks (Walter, Eliasziw & Donner, 1998: k = 2, ρ₀ = 0.5, ρ₁ = 0.75, α = 0.05, power 0.80, n = 35.4, rounded up). Recruitment target 50.

Analysis

ICC(2,1) with a one-sided test of H0: ρ ≤ 0.5 against ρ > 0.5 at α = 0.05, with a 95% CI.

Data filed before a taster joined is used only if they also agree to the separate earlier-data question. It is labeled as earlier data. The primary analysis uses only data filed after joining; the same analysis including earlier data is reported as a sensitivity analysis.

When a study reports more than one secondary test, p-values are adjusted with the Holm method.

Limitations we know about

The app chooses which whiskey repeats, from the taster’s own record, so the whiskeys differ between tasters. Six repeat checks take months to accumulate.

Standards every Savorant study follows

Preregistration. This page is the protocol of record, versioned and dated, and is posted to the Open Science Framework (OSF) before any study data is analyzed. Hypotheses, outcomes, sample size and analysis are fixed here in advance.
Primary and secondary outcomes. Each study names one primary outcome, tested at the stated alpha. Everything else is labeled secondary or exploratory and reported as such, with effect sizes and 95% confidence intervals rather than bare p-values.
No peeking. No interim analysis of the outcomes. The only checks before the sample size is reached are for data quality (duplicates, impossible values), and they never look at results.
Repeated measures. One taster contributes many rounds, so analyses use mixed-effects models with the taster as a random effect, or per-taster summaries, never pooled rounds treated as independent.
Exact methods for small counts. Triangle results are analyzed against the exact binomial with chance at one in three, the way ISO 4120 sets out the triangle test, adapted here for repeated measures.
Exclusions. Rounds are excluded only for the reasons listed in advance: test accounts, the founder’s accounts, rounds filed in a preview build, and duplicates. Every exclusion is counted and reported.
Missing data. Tasters who stop early are kept for every analysis their data supports. The number who stop, and when, is reported.
Deviations. Any change after data collection starts is logged on this page as a dated amendment with its reason, and reported in the paper.
Reporting. Results are reported following the STROBE guideline for observational studies, with the analysis code published alongside.
Conflict of interest. The study is run by the company that sells the app. Preregistration, publishing regardless of outcome, and public analysis code are how that interest is kept out of the result.

The consent you agree to

This is the exact text shown in the app before you join.

Amendments

None. This is the first version of the protocol.

All open studies · The evidence review · Privacy policy · palate@savorant.com