Change.org’s AI Writing Tool Changed Petitions, Not Outcomes

Change.org’s in-platform AI writing tool changed the language of petitions and made them more alike, but did not improve petition outcomes, according to a study of 1.5 million petitions. The result exposes a gap product teams can miss: a feature can visibly reshape what users produce without helping them achieve their goal. The researchers found that AI access substantially altered petitions’ lexical features and increased homogeneity, while outcomes did not improve.

More uniform language, no better result

The study examines an online petition platform, where the desired result is not simply producing polished text. A writing assistant can change wording at scale, but that change matters to petitioners only if it helps their petitions succeed. In this case, the researchers found a shift in language and greater similarity between petitions, without better outcomes. Their analysis used data from 1.5 million petitions.

The abstract does not specify which individual language features changed or give a numerical estimate of the increase in homogeneity. So the finding supports a directional conclusion about the tool’s effect on petition text, not a claim about how large the shift was or what particular wording caused it.

A natural experiment, with a repeat-writer check

The researchers describe their work as a natural experiment and used a difference-in-differences analysis to estimate the effect of introducing the “write with AI” tool. They also examined repeat petition writers who had written petitions both before and after its introduction; that separate analysis produced consistent results. Both the main analysis and repeat-writer check point to changed text without improved outcomes.

That evidence makes the finding more informative than a simple comparison of people who chose to use AI with those who did not. Still, the available study abstract does not give the outcome definition, effect sizes or detailed assumptions behind the analysis. The result should be read as evidence about this tool and this platform, rather than a verdict on AI writing assistance in general.

Measure what the writing is for

For product teams, the distinction is practical: changes in generated or edited text are not the same as changes in user outcomes. A feature can influence content at platform scale while leaving the purpose of that content no better served. Here, the text also became more homogeneous, suggesting that an assistant’s effect on a workflow may include narrowing variation as well as helping users write.

The study does not establish that other writing tools will produce the same pattern. But it does make a persuasive case for evaluating an AI feature against the user’s actual goal, while keeping an eye on what happens to the range of content people create. On Change.org, more AI-shaped petition language did not translate into more successful outcomes.