A chat before labelling: micro-narratives made the same posts read more divergently, not more alike

The annotation pipeline gains a stage before labelling: annotators read a scenario vignette matched to the themes of their items, talk with a conversational elicitor, and receive a first-person micro-narrative they can reopen while they work. Lived experience had been operationalised as a demographic form completed once at recruitment and treated as a covariate, while the interface asked annotators to apply a definition consistently and treated their own history as bias to suppress. A 2x2 between-subjects study on Prolific crossed elicitation against a matched control and lived-experience against non-lived-experience screeners, with 104 annotators producing 1,040 annotations over 30 ambiguous hate speech items.

Elicitation made annotators feel the content was more relevant to their identity and produced harsher harm judgements, while spreading the label distribution instead of collapsing it. Identity resonance averaged 2.94 and harm 2.78 under control, against 3.35 and above 2.78 under elicitation, with 25 of 30 items rated more resonant, 25 more harmful, and 20 higher on both. Mean per-item Shannon entropy was 2.05 bits under elicitation versus 1.88 under control, higher on 20 of 30 items, and pooled ordinal Krippendorff's alpha fell from 0.111 to 0.066 with the same direction in scenarios B and C.

What changed is the grounds annotators cite for a reading, shifting from the literal text toward their own situated knowledge, while how committed they remain to a single reading barely moves. Situated grounding accounted for 20.8% of control rationales and appeals to literal wording alone for 40.8%, against 50.8% and 12.5% under elicitation, while single-reading commitment went from 64.2% to 70.8% and acknowledged multiplicity stayed identical at 16.7%. A reflexive thematic analysis covered 276 units from a stratified subsample of 24 participants, comprising 240 rationales, 12 elicitation transcripts and 24 reflexivity responses, with near-perfect coder agreement on the construct carrying the central claim.

Annotators not screened for relevant lived experience invoked a lived-experience axis far more often under elicitation, and the axis they invoked was often not the one the screener tested. Axis invocation among non-lived-experience annotators rose from 10.0% under control to 48.3% under elicitation, against 36.7% to 56.7% among lived-experience annotators, with the gain carried largely by the sociocultural and geographic axis rising from 4.2% to 25.8% of all rationales. In scenario C, non-lived-experience participants grounded readings in class and regional upbringing rather than gender, a pattern a single-axis screener cannot locate.

This report is for researchers and engineers building subjective annotation datasets, and covers a preliminary 2x2 between-subjects study on Prolific: 104 annotators, 30 synthetic ambiguous hate speech items, 1,040 annotations, plus a qualitative analysis of 276 units from 24 participants. The corpus was built to satisfy four criteria requiring surface deniability, experiential divergence, embeddedness in exclusionary discourse and no explicit derogation, so it is not a random sample of hate speech but a concentration of the hardest cases, and effect sizes should be read as an upper bound. The authors propose reusing a worker's narrative within a domain to amortise the one-time overhead across hundreds of items, but that reuse was not measured here.

The authors frame this as preliminary: the design compares 60 against 44 participants over 30 synthetic items, cells are unbalanced with 56 of 104 participants in scenario A, domain-level comparisons are underpowered, and secondary analyses are uncorrected for multiple comparisons. Lived experience was operationalised through coarse Prolific screeners testing one axis per scenario, the assumption the non-lived-experience result is most sensitive to. Identity resonance is self-reported by an arm that knows the study concerns lived experience, so some of that gain is plausibly demand, though the harm and entropy results do not depend on self-report. Elicitation is model-mediated, so the micro-narrative may align annotators to a model's rendering of their experience rather than the experience itself, a risk fidelity coding bounds but cannot remove; 5 of 12 narratives contained at least one unsupported move, 6 of 12 conversations yielded mostly short general answers, and 2 entered a repair loop from which the elicitor never recovered. Participants are English-fluent North American residents and the discourse patterns encoded in the corpus are Anglophone.