Pew Research Center published a methodological experiment on 30 September testing whether AI models given personas of real survey panelists could reproduce what those panelists actually said.1 Pew Research Center 2026-09-30 About 12 point average error over nearly 300 questions; over 15 points on about 28 percent; Trump approval 46 versus 34 percent; data centre awareness 3 versus 25; Hispanic World Cup interest 97 versus 43; First Amendment correct 98 versus 52; humans chose not sure about four times as often; GPT-5.1 skewed more extreme, Opus 4.6 more moderate. Open source Across nearly 300 questions from three 2026 waves of its American Trends Panel, the best performing configuration, Anthropic's Claude Opus 4.6, missed the human answers by about 12 percentage points on average, and by more than 15 points on about 28 percent of questions.1 Pew Research Center 2026-09-30 About 12 point average error over nearly 300 questions; over 15 points on about 28 percent; Trump approval 46 versus 34 percent; data centre awareness 3 versus 25; Hispanic World Cup interest 97 versus 43; First Amendment correct 98 versus 52; humans chose not sure about four times as often; GPT-5.1 skewed more extreme, Opus 4.6 more moderate. Open source Pew concludes that AI is not an adequate replacement for rigorous probability based surveys on topics of broad public importance and says it has no plans to use models to generate survey results.1 Pew Research Center 2026-09-30 About 12 point average error over nearly 300 questions; over 15 points on about 28 percent; Trump approval 46 versus 34 percent; data centre awareness 3 versus 25; Hispanic World Cup interest 97 versus 43; First Amendment correct 98 versus 52; humans chose not sure about four times as often; GPT-5.1 skewed more extreme, Opus 4.6 more moderate. Open source We assess with high confidence that the study will become the reference citation against synthetic polling, and with moderate confidence that it will not slow commercial adoption of synthetic respondents in market research, where the errors are cheaper to tolerate.

How the test worked

Pew built digital twins. Each synthetic respondent received the self reported demographics of a real panelist plus that person's answers to Pew's 2025 political typology survey, then answered a later survey question by question with the same instructions and programming the human had seen.1 Pew Research Center 2026-09-30 About 12 point average error over nearly 300 questions; over 15 points on about 28 percent; Trump approval 46 versus 34 percent; data centre awareness 3 versus 25; Hispanic World Cup interest 97 versus 43; First Amendment correct 98 versus 52; humans chose not sure about four times as often; GPT-5.1 skewed more extreme, Opus 4.6 more moderate. Open source The design is close to a best case for synthetic polling: the model knew far more about each person than any commercial synthetic panel typically does.

The methodology shows how much tuning went in. Pew tested GPT-5.1, GPT-5 nano and Claude Opus 4.6.2 Pew Research Center 2026-09-30 Three waves, 119, 105 and 67 questions; 6,700 subsample; GPT-5.1 standard profile 16.0, extended 13.7, with reflection 13.1; GPT-5 nano 17.4; Opus 4.6 11.4, 14.7, 11.2; low reasoning chosen as higher settings did not help; Opus about five times GPT-5.1 price. Open source With only a standard demographic profile, GPT-5.1 averaged 16.0 points of error; adding 71 extended typology variables brought it to 13.7, and adding an expert reflection generated per respondent brought it to 13.1.2 Pew Research Center 2026-09-30 Three waves, 119, 105 and 67 questions; 6,700 subsample; GPT-5.1 standard profile 16.0, extended 13.7, with reflection 13.1; GPT-5 nano 17.4; Opus 4.6 11.4, 14.7, 11.2; low reasoning chosen as higher settings did not help; Opus about five times GPT-5.1 price. Open source Opus 4.6 posted 11.4, 14.7 and 11.2 points across the three waves.2 Pew Research Center 2026-09-30 Three waves, 119, 105 and 67 questions; 6,700 subsample; GPT-5.1 standard profile 16.0, extended 13.7, with reflection 13.1; GPT-5 nano 17.4; Opus 4.6 11.4, 14.7, 11.2; low reasoning chosen as higher settings did not help; Opus about five times GPT-5.1 price. Open source Higher reasoning settings did not improve accuracy, and Opus was run on a 6,700 person subsample of the first wave because it costs roughly five times as much as GPT-5.1.2 Pew Research Center 2026-09-30 Three waves, 119, 105 and 67 questions; 6,700 subsample; GPT-5.1 standard profile 16.0, extended 13.7, with reflection 13.1; GPT-5 nano 17.4; Opus 4.6 11.4, 14.7, 11.2; low reasoning chosen as higher settings did not help; Opus about five times GPT-5.1 price. Open source

The failure modes are the finding

The average error understates the problem because the errors have a shape. The model stereotyped: it estimated that 97 percent of Hispanic adults were at least somewhat likely to follow the World Cup, against 43 percent in reality, and that 95 percent of Republicans viewed Israel favourably, against 58 percent.1 Pew Research Center 2026-09-30 About 12 point average error over nearly 300 questions; over 15 points on about 28 percent; Trump approval 46 versus 34 percent; data centre awareness 3 versus 25; Hispanic World Cup interest 97 versus 43; First Amendment correct 98 versus 52; humans chose not sure about four times as often; GPT-5.1 skewed more extreme, Opus 4.6 more moderate. Open source It flattened distributions: on abortion it put the share saying legal in all cases at 13 percent against 23, and illegal in all cases at zero against 11.1 Pew Research Center 2026-09-30 About 12 point average error over nearly 300 questions; over 15 points on about 28 percent; Trump approval 46 versus 34 percent; data centre awareness 3 versus 25; Hispanic World Cup interest 97 versus 43; First Amendment correct 98 versus 52; humans chose not sure about four times as often; GPT-5.1 skewed more extreme, Opus 4.6 more moderate. Open source Nearly half of the questions had at least one answer choice that no synthetic respondent picked.1 Pew Research Center 2026-09-30 About 12 point average error over nearly 300 questions; over 15 points on about 28 percent; Trump approval 46 versus 34 percent; data centre awareness 3 versus 25; Hispanic World Cup interest 97 versus 43; First Amendment correct 98 versus 52; humans chose not sure about four times as often; GPT-5.1 skewed more extreme, Opus 4.6 more moderate. Open source

It knew too much. Synthetic respondents answered a First Amendment question correctly 98 percent of the time against 52 percent of humans.1 Pew Research Center 2026-09-30 About 12 point average error over nearly 300 questions; over 15 points on about 28 percent; Trump approval 46 versus 34 percent; data centre awareness 3 versus 25; Hispanic World Cup interest 97 versus 43; First Amendment correct 98 versus 52; humans chose not sure about four times as often; GPT-5.1 skewed more extreme, Opus 4.6 more moderate. Open source And it was too sure: humans chose not sure about four times as often.1 Pew Research Center 2026-09-30 About 12 point average error over nearly 300 questions; over 15 points on about 28 percent; Trump approval 46 versus 34 percent; data centre awareness 3 versus 25; Hispanic World Cup interest 97 versus 43; First Amendment correct 98 versus 52; humans chose not sure about four times as often; GPT-5.1 skewed more extreme, Opus 4.6 more moderate. Open source

The errors on current events are the most damaging for any political use. The synthetic panel put Trump's job approval at 46 percent against 34 percent among the humans, and the share who had heard a lot about data centres at 3 percent against 25.1 Pew Research Center 2026-09-30 About 12 point average error over nearly 300 questions; over 15 points on about 28 percent; Trump approval 46 versus 34 percent; data centre awareness 3 versus 25; Hispanic World Cup interest 97 versus 43; First Amendment correct 98 versus 52; humans chose not sure about four times as often; GPT-5.1 skewed more extreme, Opus 4.6 more moderate. Open source A model's picture of public opinion is built from its training data, and it lags events that move opinion after the training cutoff.

Model choice changed the answer. Pew found GPT-5.1 portrayed a more extreme public and Opus 4.6 a more moderate one, so each would mislead in a different direction.1 Pew Research Center 2026-09-30 About 12 point average error over nearly 300 questions; over 15 points on about 28 percent; Trump approval 46 versus 34 percent; data centre awareness 3 versus 25; Hispanic World Cup interest 97 versus 43; First Amendment correct 98 versus 52; humans chose not sure about four times as often; GPT-5.1 skewed more extreme, Opus 4.6 more moderate. Open source AI Weekly notes identical prompts produced substantially different results across the two.3 AI Weekly 2026-10-02 Summary published 2 Oct 2026; identical prompts gave substantially different answers across models; researcher reactions on synthetic data policy. Open source

Who gains and who loses

Probability based survey organisations gain a rigorous defence of their cost structure from the field's most cited nonpartisan pollster.1 Pew Research Center 2026-09-30 About 12 point average error over nearly 300 questions; over 15 points on about 28 percent; Trump approval 46 versus 34 percent; data centre awareness 3 versus 25; Hispanic World Cup interest 97 versus 43; First Amendment correct 98 versus 52; humans chose not sure about four times as often; GPT-5.1 skewed more extreme, Opus 4.6 more moderate. Open source Researchers arguing for disclosure rules on synthetic data gain evidence; AI Weekly records academics already discussing journal policies on the practice.3 AI Weekly 2026-10-02 Summary published 2 Oct 2026; identical prompts gave substantially different answers across models; researcher reactions on synthetic data policy. Open source

Vendors selling synthetic respondents as a substitute for polling lose the argument on public opinion measurement. Campaigns and advocacy groups tempted to test messages on synthetic voters lose cover for doing so, since the documented failure is precisely on subgroups and current events.1 Pew Research Center 2026-09-30 About 12 point average error over nearly 300 questions; over 15 points on about 28 percent; Trump approval 46 versus 34 percent; data centre awareness 3 versus 25; Hispanic World Cup interest 97 versus 43; First Amendment correct 98 versus 52; humans chose not sure about four times as often; GPT-5.1 skewed more extreme, Opus 4.6 more moderate. Open source

The counter case

The study tested the models of early 2026, and the error fell meaningfully as Pew added context, from 16.0 to 13.1 points for one model.2 Pew Research Center 2026-09-30 Three waves, 119, 105 and 67 questions; 6,700 subsample; GPT-5.1 standard profile 16.0, extended 13.7, with reflection 13.1; GPT-5 nano 17.4; Opus 4.6 11.4, 14.7, 11.2; low reasoning chosen as higher settings did not help; Opus about five times GPT-5.1 price. Open source A reasonable reader could conclude synthetic polling is improving, not failing. Pew's scope is also narrow: it tested AI as a respondent on topics of broad public importance and did not evaluate other uses such as imputing missing data.1 Pew Research Center 2026-09-30 About 12 point average error over nearly 300 questions; over 15 points on about 28 percent; Trump approval 46 versus 34 percent; data centre awareness 3 versus 25; Hispanic World Cup interest 97 versus 43; First Amendment correct 98 versus 52; humans chose not sure about four times as often; GPT-5.1 skewed more extreme, Opus 4.6 more moderate. Open source For product research, where a 12 point error on a concept test may be acceptable at a fraction of the cost, the conclusion may not transfer.

What to watch

  • A Pew rerun with newer models. A repeat within a year showing average error below 8 points would show the gap closing; error near 12 would suggest a structural limit.
  • Journal policies. Major survey research journals adopting disclosure requirements for synthetic respondents by mid 2027 would show the field drawing a line.3 AI Weekly 2026-10-02 Summary published 2 Oct 2026; identical prompts gave substantially different answers across models; researcher reactions on synthetic data policy. Open source
  • Campaign use in 2026 races. Reporting that campaigns relied on synthetic voter panels for message testing in the midterms would show the warning was ignored where the stakes are highest.
  • Vendor accuracy claims. Synthetic research firms publishing head to head error figures against probability panels would be a sign the market is accepting Pew's standard of proof.

What happened next (updated Oct 13, 2026)

Within two days the study was circulating among survey and data science researchers, with AI Weekly recording discussion of editorial policies on synthetic data in research.3 AI Weekly 2026-10-02 Summary published 2 Oct 2026; identical prompts gave substantially different answers across models; researcher reactions on synthetic data policy. Open source No rerun or vendor response had appeared in the coverage reviewed through 13 October.