Study overview
A summary of the conditions, the sample, and what stood out most. All statistics below come from the official analysis CSV.
Participants
90
High school, ages 14–18
Implicitness groups
3
Low (25) · Moderate (37) · High (28)
Adaptability groups
2
No (52) · Yes (38)
Overall completion
46.7%
42 of 90 players finished
Sig. findings (p<.05)
2
Engagement & clarity (adaptability)
Marginal (p<.10)
3
Enjoyment adapt, completion impl, enjoyment interaction
Outliers removed
7–6
Deaths/time (N→83); Stars (N→84); BPS (N→87)
Interaction sig.
0
Enjoyment marginal p=0.063 †
Completion rate by implicitness
Players in the low-implicitness group finished at roughly twice the rate of those in the high-implicitness group.
χ²=5.796, p=0.055 †
Survey composites by adaptability
Players without adaptive difficulty scored higher on every single composite measure.
Engagement p=0.039 *
Clarity p=0.029 *
Enjoyment p=0.077 †
* p < 0.05 † p < 0.10 ns = not significant · Stars = sum of Level 4–7 boolean columns (max=4, N=84, 6 outliers removed) · Welch t-tests used for adaptability comparisons
Survey composites
Three composite measures, each built from related Likert items rated 1 to 5. No outliers turned up in any of the survey variables, so N=90 throughout.
By implicitness level (one-way ANOVA)
Low (n=25)
Moderate (n=37)
High (n=28)
Enjoyment composite
Enjoyed · Want to continue · Play again
F(2,87)=1.663, p=0.196 ns
Engagement composite
Felt engaged · Paid attention
F(2,87)=2.055, p=0.134 ns
Clarity composite
Easy to figure out · Challenge · Levels helped/prepared · Confident
F(2,87)=1.562, p=0.216 ns
By adaptability condition (Welch t-test)
No adaptability (n=52)
With adaptability (n=38)
Enjoyment composite
No (3.54) · Yes (3.10)
t(77.7)=1.792, p=0.077 †
Engagement composite
No (3.87) · Yes (3.41)
t(72.6)=2.107, p=0.039 *
Clarity composite
No (3.85) · Yes (3.38)
t(75.3)=2.222, p=0.029 *
All three composites side by side (grouped)
All composites by implicitness
Enjoyment, engagement, and clarity shown together. The high-implicitness group sits lowest on all three.
Low (n=25)
Moderate (n=37)
High (n=28)
Enjoyment F(2,87)=1.663, p=0.196 ns
Engagement F(2,87)=2.055, p=0.134 ns
Clarity F(2,87)=1.562, p=0.216 ns
All composites by adaptability
The same three composites by condition. Players without adaptive difficulty scored higher on every one.
No adaptability (n=52)
With adaptability (n=38)
Enjoyment t(77.7)=1.792, p=0.077 †
Engagement t(72.6)=2.107, p=0.039 *
Clarity t(75.3)=2.222, p=0.029 *
All Likert items are rated 1–5, and each composite is just the row-wise mean of its items. We used Welch's t-test throughout, since it doesn't assume equal variances between groups.
All 10 survey items
Individual item means, each on a 1–5 Likert scale. Items 1 through 5 feed into the Enjoyment and Engagement composites; items 6 through 10 feed into Clarity.
All items by implicitness
Low implicitness scores highest on almost every item, and high implicitness scores lowest.
Low
Moderate
High
All items by adaptability
Players without adaptive difficulty scored higher on every single item, no exceptions.
No adaptability
With adaptability
Performance metrics
Stars, deaths, total time, and button presses per second, shown before and after outlier removal where that applies. Stars are corrected to N=84 after 6 outliers were removed.
Stars earned: sum of L4–L7 boolean columns (max=4 · N=84, 6 outliers removed)
Low
Moderate
High
No adapt.
With adapt.
Stars by implicitness (N=84)
Low (0.88) · Moderate (1.30) · High (1.14)
F(2,81)=1.251, p=0.292 ns
Stars by adaptability (N=84)
No (1.27) · Yes (0.95)
t(81.4)=0.562, p=0.576 ns
Total deaths: before (N=90) vs. after outlier removal (N=83, 7 removed)
Deaths by implicitness (before / after)
Moderate's apparent high-death pattern disappears completely once the outliers are removed.
Before: p=0.119 ns
After: F(2,80)=0.498, p=0.610 ns
Deaths by adaptability (before / after)
No (11.3 → 6.9) · Yes (10.5 → 8.6): the direction actually flips after cleaning.
Before: p=0.792 ns
After: t(63.9)=−0.869, p=0.388 ns
Total time (seconds): before (N=90) vs. after outlier removal (N=83, 7 removed)
Time by implicitness (before / after)
A handful of extreme outliers inflated the means by 200–300 seconds; the gap narrows a lot once cleaned.
Before: p=0.512 ns
After: F(2,80)=0.999, p=0.373 ns
Time by adaptability (before / after)
No (418 → 248s) · Yes (509 → 224s)
Before: p=0.721 ns
After: t(79.2)=0.687, p=0.494 ns
Average button presses per second: before (N=90) vs. after (N=87, 3 removed)
BPS by implicitness (before / after)
Nearly identical across all three groups; there's just no meaningful variation here.
Before: p=0.899 ns
After: F(2,84)=0.024, p=0.976 ns
BPS by adaptability (before / after)
No (1.86 → 1.81) · Yes (1.81 → 1.69)
Before: p=0.793 ns
After: t(72.6)=0.721, p=0.473 ns
In the before/after charts, darker bars show the raw data and lighter bars show the cleaned data. Welch's t-test was used for every adaptability comparison.
Per-level analysis
Time and BPS broken down level by level. Outliers were removed independently at each level using IQR × 1.5. Significance tests for L6 and L7 come from the official analysis CSV; L1–L5 are from preliminary analysis. Sample size naturally shrinks as players progress.
Time per level (seconds · cleaned per level)
Low
Moderate
High
No adapt.
With adapt.
Time per level by implicitness
No significant implicitness effect on time shows up at any level, per the official analysis.
L6: F(2,57)=1.629, p=0.205 ns (N=60)
L7: F(2,37)=0.002, p=0.998 ns (N=40)
Time per level by adaptability
Adaptability doesn't move time at any level either.
L6: t(57.8)=1.189, p=0.239 ns
L7: t(37.1)=−0.853, p=0.399 ns
BPS per level (button presses ÷ time · cleaned per level)
BPS per level by implicitness
L1 is dramatically lower (around 0.2) since it's the introductory level; things settle down from L2 on.
L6: F(2,59)=0.845, p=0.435 ns (N=62)
L7: F(2,38)=0.020, p=0.980 ns (N=41)
BPS per level by adaptability
Adaptability has no effect on input rate at any level.
L6: t(40)=−0.543, p=0.590 ns
L7: t(25)=1.466, p=0.155 ns
Players with data per level
N per level (before per-level outlier removal)
The drop-off reflects players who simply didn't reach or finish the later levels.
N per level: L1=90, L2=83, L3=80, L4=77, L5=72, L6=65, L7=44. The official analysis only covers L6 and L7. BPS stands for button presses per second.
Game completion
Whether players finished the game or not, compared across conditions. Since this is a categorical outcome, we used chi-square tests.
Overall completion
46.7%
42 of 90
Low implicitness
60.0%
15 of 25 (highest)
High implicitness
28.6%
8 of 28 (lowest)
High impl. + adapt.
16.7%
2 of 12 (worst subgroup)
Completion by implicitness (stacked %)
Low (60%) · Moderate (51%) · High (29%)
χ²(2)=5.796, p=0.055 †
Completion by adaptability (stacked %)
No (52%) · Yes (40%)
χ²(1)=0.913, p=0.339 ns
Completion rate (implicitness × adaptability)
Adaptability helps a little at low implicitness, but it really hurts at high implicitness, dropping all the way to 16.7%.
No adaptability
With adaptability
Two-way interaction: F(2,84)=0.799, p=0.453 ns
Chi-square test for independence. Subgroup sample sizes: High/No=16, High/Yes=12, Low/No=16, Low/Yes=9, Moderate/No=20, Moderate/Yes=17.
Two-way ANOVA interactions
Interaction terms (implicitness × adaptability) for every metric in the official analysis CSV. Enjoyment was the only one with a marginally significant interaction.
Two-way ANOVA summary of all metrics
Adaptability main effects and interaction F-statistics. Implicitness main effects come from the one-way ANOVAs in the earlier section.
| Metric | N | Adaptability (main) | Interaction (Impl × Adapt) |
|---|---|---|---|
| Enjoyment | 90 | F(1,84)=3.478, p=0.066 † | F(2,84)=2.855, p=0.063 † |
| Engagement | 90 | F(1,84)=4.430, p=0.038 * | F(2,84)=0.815, p=0.446 ns |
| Clarity | 90 | F(1,84)=4.646, p=0.034 * | F(2,84)=1.015, p=0.367 ns |
| Stars (corrected) | 84 | F(1,78)=0.575, p=0.451 ns | F(2,78)=0.706, p=0.497 ns |
| Total deaths (cleaned) | 83 | F(1,77)=0.936, p=0.336 ns | F(2,77)=0.493, p=0.613 ns |
| Total time (cleaned) | 83 | F(1,77)=0.320, p=0.573 ns | F(2,77)=0.034, p=0.967 ns |
| BPS (cleaned) | 87 | F(1,81)=0.500, p=0.482 ns | F(2,81)=0.680, p=0.510 ns |
| Completion | 90 | F(1,84)=1.244, p=0.268 ns | F(2,84)=0.799, p=0.453 ns |
| L6 Time | 60 | F(1,54)=0.707, p=0.404 ns | F(2,54)=1.449, p=0.244 ns |
| L6 BPS | 62 | F(1,56)=0.216, p=0.644 ns | F(2,56)=0.205, p=0.816 ns |
| L7 Time | 40 | F(1,34)=0.614, p=0.439 ns | F(2,34)=1.680, p=0.202 ns |
| L7 BPS | 41 | F(1,35)=2.323, p=0.137 ns | F(2,35)=0.600, p=0.554 ns |
Enjoyment (p=0.063†)
No adaptability
With adaptability
Enjoyment cell means (grouped bar)
High/no=2.85 · High/yes=3.25 · Low/no=3.75 · Low/yes=2.96 · Mod/no=3.92 · Mod/yes=3.06
Interaction F(2,84)=2.855, p=0.063 †
Enjoyment interaction plot (line)
The lines cross at high implicitness: a disordinal interaction, where adaptability actually flips direction.
Engagement cell means
The lines run nearly parallel here; just additive effects, nothing more (p=0.446 ns).
Clarity cell means
Parallel lines, additive effects only (p=0.367 ns).
Two-way ANOVA values pulled from the official analysis CSV (S3-Interaction). Implicitness main effects weren't included in that CSV. See the one-way ANOVA section instead.
Demographic analysis
How gender, grade level, and age relate to learning outcomes. Worth noting: the official CSV doesn't include demographic tests, so everything below comes from a preliminary analysis and should be read with that in mind.
Male
49
54.4% of sample
Female
35
38.9% of sample
Ages
14–18
Modal age = 16 (n=30)
Grades
9–12
Modal grade = 10 (n=30)
Gender: engagement, clarity, and completion
Male (n=49)
Female (n=35)
Other (n=6)
Enjoyment by gender
Male (3.56) · Female (3.07) · Other (3.33)
ANOVA p=0.162; Male vs. Female p=0.063 †
Engagement by gender
Male (3.91) · Female (3.33) · Other (3.75)
ANOVA p=0.033 *; Male vs. Female p=0.010 *
Clarity by gender
Male (3.90) · Female (3.28) · Other (3.77)
ANOVA p=0.014 *; Male vs. Female p=0.004 *
Completion rate by gender
Male (61.2%) · Female (22.9%) · Other (66.7%)
ANOVA p=0.001 *; Male vs. Female χ²=10.632, p=0.001 *
Stars earned by gender
Male (1.29) · Female (1.00) · Other (0.67)
ANOVA p=0.208 ns
Grade level: significant differences in deaths and time (cleaned, N=83)
Grade 9 (n=18)
Grade 10 (n=30)
Grade 11 (n=26)
Grade 12 (n=16)
Total deaths by grade (cleaned, N=83)
Grade 12 students died substantially more often than grades 10 and 11.
ANOVA p=0.019 *
Gr 10 vs 12: p=0.003 *
Gr 11 vs 12: p=0.019 *
Total time by grade (cleaned, N=83)
Grade 12 also spent significantly more time in the game than grades 10 and 11.
ANOVA p=0.039 *
Gr 10 vs 12: p=0.007 *
Gr 11 vs 12: p=0.034 *
Enjoyment by grade
Gr 9 (3.52) · 10 (3.29) · 11 (3.30) · 12 (3.38)
ANOVA p=0.916 ns
Engagement by grade
Gr 9 (3.89) · 10 (3.50) · 11 (3.62) · 12 (3.84)
ANOVA p=0.534 ns
Completion rate by grade
Gr 9 (61%) · 10 (33%) · 11 (54%) · 12 (44%)
ANOVA p=0.243 ns
Age: marginal effect on deaths (cleaned); no effect on subjective outcomes
Total deaths by age (cleaned, N=83)
Ages 15 and 16 died the least; ages 14, 17, and 18 all died more often.
ANOVA p=0.044 *
Age 14 vs 16: p=0.046 *
Age 16 vs 17: p=0.024 *
Enjoyment & engagement by age
No consistent trend with age shows up for either subjective outcome.
Enjoyment p=0.592 ns
Engagement p=0.621 ns
Demographic tests aren't part of the official analysis CSV (treat these as preliminary). "Other" gender groups together Nonbinary, Transgender Woman, Prefer not to say, and two informal write-in responses (n=6), so interpret with caution given the small sample. Gaming experience wasn't collected, so the gender differences in completion are likely confounded by players' prior gaming background.