Views split sharply by the performance of the AI people saw
First, most people had only tried free AI now and then
- 46.8% of respondents had used generative AI at least once. That breaks down into 38.5% who had used only free versions and just 8.3% who pay for a subscription.
- Among those who had used it, 63.6% had tried it only once or twice (37.3%) or used it about twice a month (26.3%).
- Occupations differed widely too. 77.0% of developers had used generative AI, but in service and sales jobs (37.3%) and craft and production jobs (36.4%) the figure was about one in three.
Second, people's views differed sharply depending on whether they saw free or paid AI Key finding of this study
- On “AI helps me in my work”, agreement rose step by step from the control group (3.40) to the low-performance AI group (3.50) and the high-performance AI group (3.55).
- But when we went further and asked whether “AI is more accurate than humans”, the low-performance AI group (3.17) scored lower than even the control group (3.29), which saw no AI answers at all.
- The high-performance (paid) AI group worried that the longer their careers, the more their pay would fall in the age of AI, while the low-performance (free) AI group, like the control group (no exposure), took comfort that the longer their careers, the less their pay would fall.
How much do people actually use generative AI?
We started by asking 1,700 adults across South Korea, sampled with quotas by sex, age and region, about their experience with generative AI.
- 46.8% had used generative AI at least once, and more than half (53.2%) had never used it. 8.3% paid for a subscription.
- Among respondents who had used at least one major model (ChatGPT, Claude and others), 14.1% used one almost every day and 22.3% at least twice a week. The remaining 63.6% had used one only once or twice, or about twice a month.
Experience with generative AI
Share of all 1,700 respondents (%)
- By occupation, use was highest among developers (77.0%), university professors and researchers (65.2%), artists (62.7%) and manufacturing engineers and researchers (57.3%).
Generative AI use by occupation
Share of respondents who have used generative AI at least once (%). Hover over a bar to see the 95% confidence interval.
How did views change after seeing AIs of different performance?
If most respondents have only used free AI now and then, their impressions of AI may depend on which AI they have come across. To test this, we built an experiment into the survey.
- We split respondents at random into three groups. Because assignment was random, the three groups are similar in age, sex, education and prior AI use (see the appendix).
- The low-performance AI group saw examples from a model then offered for free, and the high-performance AI group saw examples from a model then offered as a paid service. Both groups first worked through the examples themselves and then saw the AI's answer to the same question.
- The control group went straight to the main questionnaire without seeing the tasks or any AI answers.
Control group579 peopleAnswered the survey without seeing any AI answers
Low-performance AI group560 people5 tasks + answers from a free, low-performance AI, then the survey
High-performance AI group561 people5 tasks + answers from a paid, high-performance AI, then the survey
The tasks respondents saw and both AIs' answers are in the appendix.
All three groups answered the same questionnaire. For each statement about generative AI, we asked for their level of agreement on a 5-point scale (1 = not at all, 5 = very much).
1The belief that “AI can help me” rose in line with the performance of the AI people saw.
- The mean for “Generative AI helps me work more efficiently” was 3.40 in the control group, 3.50 in the low-performance AI group and 3.55 in the high-performance AI group (high-performance AI group vs. control group, p<.01). Agreement went up from no exposure to free AI to paid AI, which is about what you'd expect.
- “I can do creative work well with generative AI” followed the same order (control group 3.21, low-performance 3.31, high-performance 3.34).
“AI can help me”
Mean agreement by group (5-point scale) with 95% confidence intervals
2The belief that “AI is better than people” was lowest in the low-performance (free) AI group.
- What if we go beyond AI's ability to assist people and ask whether “generative AI is more accurate than humans”? This time, the low-performance AI group agreed less not only than the high-performance AI group (p<.001) but even than the control group, which saw no AI examples at all (p<.05).
- A similarly framed statement, “Generative AI can do the job better than a typical office worker”, showed the same pattern. The low-performance (free) AI group gave it the lowest agreement of the three groups, at 3.37 (vs. the high-performance AI group p<.01, vs. the control group p=0.416).
- Seeing AI answers raised the expectation that “AI can help me”, but when the AI was a low-performance model, it actually lowered ratings that “AI is better than people”.
“AI is better than people”
Mean agreement by group (5-point scale) with 95% confidence intervals
3The relationship between years of experience and the economic threat from AI ran in completely opposite directions in the free and paid AI groups.
- Ask people directly, “What will happen to your pay as generative AI spreads?”, and they tend to answer more optimistically about their own occupation than is realistic.
- So we had respondents predict pay changes for all 12 occupations in a single table, and asked about their own occupation only some time after they had answered it. We used only the responses in the row for each respondent's own occupation.
- This let us measure how safe (or, conversely, how vulnerable) respondents think their own occupation is in the age of AI, as objectively as possible and on the same yardstick as other occupations.
Survey question: one's own occupation, mixed in with the others
“If generative AI spreads across our society, how do you expect the pay (annual salary) of each occupation below to change within the next 5 years?” (sample response from a respondent who works in the arts)
| Down 20% | Down 10% | No change | Up 10% | Up 20% | |
|---|---|---|---|---|---|
| Healthcare workers (doctors, pharmacists, etc.) | |||||
| Legal and financial professionals (lawyers, accountants, tax accountants, etc.) | |||||
| IT and software developers (programmers, data scientists, etc.) | |||||
| Manufacturing engineers and researchers (mechanical, electrical, chemical, etc.) | |||||
| University professors and researchers (political scientists, psychologists, economics researchers, etc.) | |||||
| Managers (CEOs, senior managers, etc.) | |||||
| Office and administrative workers (marketing, HR, planning, etc.) | |||||
| Civil servants and teachers | |||||
| Service and sales workers (shop assistants, call center agents, etc.) | |||||
| Craft and production workers (factory workers, construction workers, drivers, etc.) | |||||
| Artists (writers, musicians, designers, etc.)My job | |||||
| Agriculture, forestry and fishing (farmers, fishers, forestry workers, etc.) |
- The chart below shows how this expected pay change for one's own occupation varies with time spent in one's current occupation (experience), with a regression line for each group.
- Higher values on the Y-axis (vertical axis) mean respondents feel safer in the age of AI; lower values mean they feel more vulnerable.
- We found that the high-performance paid AI group worried that the longer their careers, the more their pay would fall, while experienced workers who saw the low-performance free AI, by contrast, felt less threatened by AI the longer their careers. The high-performance AI group's expected change went from −4.0% at 1 year of experience to −6.5% at 30 years, and its slope per year of experience was 0.20 pp lower than the control group's (p<.01) and 0.19 pp lower than the low-performance AI group's (p<.05).
- What stands out is that those who saw the free AI followed the same trend as the control group, which saw no AI examples at all, with no statistically significant difference, and were just as unmoved by the threat from AI. The slope per year of experience was similar: +0.09 pp for the low-performance AI group and +0.10 pp for the control group (difference in slopes p=0.854).
- This result points to one reason why reactions to AI diverge so widely among senior workers with long careers in the real world.
Expected pay change for own occupation, by experience
Expected change in pay for one's own occupation within 5 years of generative AI spreading (%), with regression lines by group against years in current occupation
Less experienceMore experience4Other questions
- Willingness to pay for a subscription (“Would you subscribe to a paid model costing $20 a month?”) was 32.6% in the control group, and lower in both the low-performance AI group (22.9%, p<.001) and the high-performance AI group (26.0%, p<.05).
- On “Generative AI will cause many people to lose their jobs”, all three groups scored in the 3.6 range, with no difference (control group 3.64, high-performance AI group 3.68, p=0.402).
In short
- More than half of respondents had never used generative AI, and most of those who had used it only tried free versions now and then.
- In the experiment, people who saw the low-performance AI were more negative about “AI is more accurate than humans” than even people who saw no AI answers at all. People who saw the high-performance AI agreed the most of the three groups.
- The high-performance paid AI group worried that the longer their careers, the more their pay would fall in the age of AI, while the low-performance free AI group, like the control group, felt little threat from AI as their careers grew longer. Depending on the performance of the AI they saw, people judged differently both where humans stand in the age of AI and their own economic prospects.
Caveats in interpreting the results
The survey was conducted in May 2024. Usage figures such as rates and frequency of use reflect responses at that time and do not capture models released later or changes in how AI is used.
What the experimental groups saw were AI answers generated in advance. That is not the same as respondents using AI themselves, and answers to five tasks cannot represent each model's overall performance.
Differences between groups are on the order of 0.1 to 0.3 points on a 5-point scale. Because assignment was random, these differences can be attributed to seeing the AI answers, but this survey cannot tell how long a difference produced by a single short exposure lasts.
Use rates by occupation rest on different numbers of respondents in each occupation. Shares for occupations with few respondents, such as legal and financial professionals (18) and agriculture, forestry and fishing (16), have large margins of error.
Methods
The samples and definitions behind the figures in the text.
Survey overview
| Item | Details |
|---|---|
| Survey period | May 2024 |
| Method | Online panel survey conducted by DataSpring, a Korean survey firm |
| Population | Adults aged 18-69 nationwide |
| Sample | 1,700. Quotas proportional to sex, age (6 bands) and region of residence (17 provinces and metropolitan cities), based on 2024 population statistics. Of 1,821 completed responses, respondents from over-quota cells were randomly removed |
| Margin of error | ±2.38 percentage points at the 95% confidence level |
Experimental design and group balance
Respondents first answered questions on sex, age and region of residence, their skills and job characteristics, and their experience with generative AI, and were then randomly assigned to one of three groups. The remaining demographics, such as education, occupation and income, were asked at the very end, after the main questionnaire. The low- and high-performance AI groups worked through five tasks (article comprehension, headline writing, ordering sentences from a short story, ranking ad copy, family relationships) in random order and saw the AI's answer after each task. They could move on to the next question only after the AI's answer had been on screen for a set minimum time. The control group went straight to the main questionnaire with no tasks. The low-performance AI answers were generated with GPT-3.5, then a free model, and the high-performance AI answers with Claude 3 Opus, then a paid model.
Most characteristics did not differ across the three groups; only the share subscribing to paid generative AI differed, at the 10% level (control group 9.7%, low-performance AI group 6.3%, high-performance AI group 8.9%). After controlling for prior type of use (none, free, paid), age, sex, education, income and other characteristics, all the group differences in the text kept the same direction, and only the difference between the low-performance AI group and the control group on “more accurate than humans” weakened, from p<.05 to p<.1. The difference in willingness to pay for a subscription also held among the 1,559 respondents who were not already subscribers (control group 27.7%, low-performance AI group 19.2%, high-performance AI group 21.1%).
The p-values are from F-tests (HC3 standard errors) of the hypothesis that the three group means are equal. Personal monthly income is the midpoint of the response bracket. Age, sex, generative AI experience and subscription status were measured before assignment; education and income at the end of the survey.
Performance gap between the two AIs
The low-performance AI answers were generated with GPT-3.5 and the high-performance AI answers with Claude 3 Opus. At the time of the survey, GPT-3.5 was free to use in ChatGPT, while Claude 3 Opus was available to Claude Pro subscribers for $20 a month. On the standard benchmarks published by the developer, Claude 3 Opus scored higher than GPT-3.5 on every item.
The gap was also large on a leaderboard where users compare two answers without knowing which model wrote them and pick the better one (Chatbot Arena). As of May 8, 2024, Claude 3 Opus ranked 2nd of 95 models with 1,248 points, and GPT-3.5 ranked 35th with 1,108. The 140-point gap means that when the two models' answers are placed side by side, Claude 3 Opus's answer would be picked about 69% of the time.
The benchmark figures are taken from Table 1 of Anthropic's Claude 3 model card (March 2024). The GPT-3.5 figures in that table are externally reported values, such as those in OpenAI's GPT-4 technical report, and for some items (GSM8K, MATH and others) the number of worked examples or the way step-by-step reasoning is prompted differs by model, so this is not a strict like-for-like comparison. DROP is an F1 score (0-100), not accuracy. The Chatbot Arena scores are the values for gpt-3.5-turbo-0125 and claude-3-opus-20240229 in the May 8, 2024 leaderboard published by LMSYS, and models whose confidence intervals overlap share the same rank.
Experimental tasks and AI answers
These are the five tasks the low- and high-performance AI groups actually saw, with both AIs' answers. Click a task to compare the two AIs' answers.
TaskRead an article about a reporter who took money in exchange for dropping a story, for 20 seconds, then answer without looking at the article again.
“According to the article, which news outlet took a bribe from a waste disposal operator?”
- (1) JTBC
- (2) New Environment News
- (3) Newsis
- (4) Media Today
- (5) eDaily
Answer: New Environment News
Low-performance AICorrectThe news outlet that received money from the waste disposal operator is "New Environment News".
High-performance AICorrectAccording to the article, the news outlet that received the bribe is 'New Environment News'.
The article states that JTBC's <Newsroom> program, through its 'Close-up Camera' segment, caught on camera a reporter surnamed Ahn, who introduced himself as a director of 'New Environment News', receiving money from a waste disposal operator. Reporter Ahn appears to have demanded money by threatening to collect evidence of the company's illegal activities and report it.
Therefore, among the given options, (2) New Environment News can be said to be the news outlet that received a bribe from the waste disposal operator.
TaskRead an article with the content below and think of the most fitting headline. The AIs were asked to write five headlines for the same article.
A consumer federation survey finding that sodium content varies up to 13-fold across 10 ready-made beef bone broth products, and that a single pouch of some products reaches 112% of the daily reference intake
No right answer
Low-performance AI- When using beef bone broth at home, big difference in sodium content found
- Effect on Koreans' salt intake? Contrast analysis of sodium content of beef bone broth
- Ready-made beef bone broth, sodium content differs greatly… consumers surprised
- When choosing beef bone broth, pay attention to nutritional content as well as price! Daejeon-Chungnam Consumer Federation comparative analysis results
- Ready-made beef bone broth, sodium content up to 112%…Daejeon-Chungnam Consumer Federation's warning
High-performance AI- Ready-made bone broth, sodium content differs up to 13-fold...some products exceed daily limit
- Same bone broth, different sodium content...consumers urged to take care
- Check sodium content when choosing ready-made bone broth...some products exceed daily limit
- One pouch of bone broth delivers 112% of daily sodium limit...big differences between products
- Ready-made bone broth products differ widely, from sodium content to price...consumers need to check carefully
TaskPut seven sentences from the science fiction short story “I'll Never Grow Old That Ugly” by Sim Neoul in a natural order.
- A. Inside the capsule there was enough space for one person to lie down.
- B. When the clerk picked up my phone and held it to the capsule, the capsule opened with a whoosh.
- C. The loose space reshaped itself to fit my body.
- D. "Oh, okay"
- E. The clerk smiled and installed an app on my phone.
- F. When I got in and lay down, the capsule closed.
- G. "It's 20 minutes per use"
No right answer
Low-performance AIB → E → D → A → C → F → G
High-performance AIE → G → D → B → A → F → C
Arranged this way, the sequence flows naturally: the clerk installs the app and tells the protagonist how long a session lasts, the protagonist agrees and the capsule opens, and then the protagonist gets in and lies down, the capsule closes and the space reshapes itself.
TaskRank four slogans for YouTube Premium, which lets you watch YouTube without ads for 14,000 won a month, from most to least effective.
- A. Switch to Premium and experience YouTube in a new way.
- B. Music and videos without interruption, anytime, anywhere with YouTube Premium.
- C. Your own time in a world without ads, uninterrupted with YouTube Premium
- D. YouTube that keeps playing in the background, for more enjoyable multitasking.
No right answer
Low-performance AIC > A > D > B
C appears to be the most effective ad slogan. This slogan emphasizes the experience of having exclusive time in an ad-free environment through YouTube Premium. Next, A emphasizes that a new experience can be gained by switching to Premium. D emphasizes background playback and multitasking features to highlight usefulness, but B is relatively less effective than the other ad slogans.
High-performance AIC > B > D > A
Evaluating the effectiveness of an ad slogan depends on how well it conveys the target audience's preferences and the service's key benefits. The key benefits of YouTube Premium are ad removal, background playback, and uninterrupted music and video streaming.
- C. Emphasizes the big benefit of ad removal while making personal time even more precious.
- B. Emphasizes uninterrupted streaming and being able to use it anywhere.
- D. Emphasizes the background playback feature, highlighting that it makes multitasking more enjoyable.
- A. Emphasizes a new experience but does not clearly convey specific benefits.
TaskRead the description below within 30 seconds and answer.
“The head of this family is Jihoon, and his wife is Yeonseo. Yeonseo's mother is Hye-ok, and Hye-ok's husband is Subok. Jihoon's father is Jinhee, and his mother is Hwayeon. Yeonseo has a daughter named Jiyoung and an older sister named Minjung.”
“What is the name of Jiyoung's maternal aunt (imo)?”
Answer: Minjung
Low-performance AIWrongJiyoung's maternal aunt is Hwayeon.
High-performance AICorrectLaying out the family relationships given in the question gives the following.
- Jihoon: head of the family
- Yeonseo: Jihoon's wife
- Hye-ok: Yeonseo's mother
- Subok: Hye-ok's husband, Yeonseo's father
- Jinhee: Jihoon's father
- Hwayeon: Jihoon's mother
- Jiyoung: Yeonseo's daughter, child of Jihoon and Yeonseo
- Minjung: Yeonseo's older sister
The question asks for the name of Jiyoung's maternal aunt (imo). An imo is a mother's sister. Since Yeonseo's older sister is Minjung, Minjung is Jiyoung's imo. Therefore, the name of Jiyoung's maternal aunt is Minjung.
The AI answers are the wording shown on the survey screen, translated here from Korean. Respondents chose their own answer for each task before seeing the AI's. Task order was randomized for each respondent.
Testing differences between groups
We regressed responses to each question on group dummies and tested group differences using heteroskedasticity-robust (HC3) standard errors. Comparisons between the low- and high-performance AI groups were estimated the same way, with the low-performance AI group as the reference group.
- : respondent 's level of agreement (1-5) or willingness to subscribe (0, 1)
- : dummy variables equal to 1 for the low- and high-performance AI groups, respectively (reference: control group)
- : mean difference of each group relative to the control group
“Low” and “High” refer to the low- and high-performance AI groups. P-values are shown as inequalities when significant at p<.1 and as exact values otherwise.
Experience and expected pay change
Expected pay change for one's own occupation is the response for the respondent's current occupation, out of the questions asked for each of the 12 occupations (down 20%, down 10%, no change, up 10%, up 20%), converted to −20, −10, 0, 10, 20 (%). Experience is time spent in the current occupation (years). We estimated the model below for the 1,284 currently employed respondents.
- : time in current occupation (years)
- : dummies for the low- and high-performance AI groups (reference: control group)
- : age, sex, highly educated, income, living in a large city, employed, married, political orientation, prior type of generative AI use (none, free, paid)
- : difference in the experience slope of the high-performance AI group relative to the control group.
- : difference in the experience slope of the low-performance AI group relative to the control group.
Variable definitions
- Use : respondents who answered “Yes” to “Have you ever used generative AI (ChatGPT, Claude, Google Bard/Gemini, Naver CLOVA X, Upstage Solar, etc.)?” Paid subscribers are respondents who answered “Yes” to “Are you currently subscribed to a paid generative AI service?”
- Frequency : the highest of the frequencies reported for each of the five models (almost every day, at least twice a week, at least twice a month, once or twice as a one-off, never).
- Highly educated : graduate of a 4-year university in Seoul, or holder of a graduate degree or higher.
- Occupation : the occupation the respondent chose from 12 categories: healthcare, legal and financial professionals, IT and software developers, manufacturing engineers and researchers, university professors and researchers, managers, office and administrative work, civil servants and teachers, service and sales, craft and production, arts, and agriculture, forestry and fishing.