Simulation · World Bank 2026
Better Data for Better Lives: Innovations in Survey Measurement in the Age of AI
Who Differs, and How Many? Assessing Language Models’ Potential to Complement Opinion Surveys
KAIST
We test 22 language models on US and African opinion surveys: not only whether they get each group’s percentages right, but whether they predict which groups differ, and by how much.
At a glance
Motivation
The question
Survey data are often too sparse to estimate opinion reliably within demographic groups. Even when a survey gives a precise national estimate, estimates for particular age groups, education levels, or other demographic categories may rely on only a small number of respondents. Collecting larger samples is costly, particularly in settings where regular, large-scale surveys are difficult to conduct, as in many African countries.
Language models have been proposed as a low-cost way to estimate opinions for groups that are sparsely represented in surveys. To assess this potential, we ask what information they can reliably provide. We examine not only how accurately they predict the share of people holding an opinion, but also how well they capture differences across groups. A model might get the percentages wrong for both younger and older people, for example, yet correctly predict that support is higher among the young.
Method
Data and analysis
We evaluate 22 language models against four survey programmes: Pew’s American Trends Panel through OpinionQA, Pew Global Attitudes, Afrobarometer Round 9, and the 2024 General Social Survey. The data cover the United States and Ghana, Kenya, Malawi, Nigeria, South Africa, Uganda, Zambia and Zimbabwe. This coverage allows us to assess whether performance extends beyond US surveys to African settings. The GSS was released after the models’ training cutoffs and contains 46 questions not asked in earlier waves. These features provide a check against reliance on previously published survey responses.
Our sample includes 13 open-weight models with 0.5 to 72 billion parameters and nine frontier models from the Claude, Gemini, GPT and Grok families. To obtain population shares, we use the probabilities a model assigns to answer letters or ask it to report percentages directly. To assess what the models add beyond simple alternatives, we compare their predictions with the average response pattern on other questions with the same number of options, a text model using question wording, and the previous survey wave for repeated items. For the second task, we examine group differences by party, age, education and religiosity in the United States, and by urban or rural residence, gender, age and education in Africa. We randomize answer order and measure agreement between predicted and observed group differences using rank correlations. We report 95% confidence intervals obtained by resampling whole questions.
Results
What we find
For population shares, model performance depends strongly on how predictions are obtained. Estimates based on answer-letter probabilities perform worse than the average-response benchmark in 37 of 38 model and survey combinations. By contrast, asking for percentages directly produces more accurate estimates. Under this approach, the two strongest frontier models outperform the benchmark in all four survey programmes, with confidence intervals for every gain above zero. Performance under direct elicitation also improves with model scale.
Turning to group differences, model predictions track observed variation across groups. Both frontier models outperform Llama 70B and the text benchmark in each dataset, with confidence intervals for all paired differences above zero.
| Model | OpinionQA | GSS 2024 (new) | Africa (within country) |
|---|---|---|---|
| Claude Opus | 0.64 [0.58, 0.70] | 0.80 [0.72, 0.86] | 0.52 [0.40, 0.62] |
| Gemini Flash | 0.68 [0.61, 0.73] | 0.79 [0.70, 0.85] | 0.48 [0.37, 0.58] |
| Llama 70B | 0.49 | 0.57 | 0.24 |
| Text benchmark | 0.21 | 0.11 | 0.06 |
- Not from memorized polls. Excluding every GSS topic with a documented pre-2024 poll, the correlations remain between 0.70 and 0.76.
- Gap sizes, not just rankings. Frontier models also approximate the size of the group gaps, whereas open-weight 70B models exaggerate them by a factor of 1.4 to 2.
- African settings. Within-country correlations are about 0.3 lower than on the new GSS questions for both frontier models. The difference from OpinionQA is small once we account for the greater sampling noise in African subgroup estimates.
- Across countries. Frontier models also predict differences across countries, with a correlation of 0.57 [0.52, 0.61], compared with 0.40 for a text model that includes country identity.
Implications
A different role for language models
Our findings suggest a different role for language models in survey research. Rather than treating model-generated responses as substitutes for survey respondents, researchers may be able to use model predictions of demographic differences to improve estimates for groups with limited survey data. This approach could be particularly valuable in African settings, where expanding survey samples can be costly and model predictions could help make better use of the data already collected. The critical next step is to test whether combining model predictions with small survey samples yields more accurate estimates than using those samples alone.