🔍 Read the full analysis: ChatGPT’s Data Science Boost For Consultants Was Hard To Keep on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
A randomized trial involving Boston Consulting Group consultants found that access to ChatGPT improved performance on coding, statistics and prediction tasks outside many participants’ expertise. After the chatbot was removed, researchers found no evidence that the participants had gained an advantage on a separate technical test.
A randomized trial at Boston Consulting Group found that consultants using ChatGPT performed better on data-science tasks than colleagues using Google and reference websites, but the advantage did not show up on a later technical test without the chatbot. The findings, published in Nature Human Behaviour on October 26, 2026, suggest that AI can help workers carry out tasks beyond their existing skills without necessarily teaching them to do those tasks independently.
The experiment included consultants assigned to use either ChatGPT or conventional online resources. The ChatGPT group received 15 to 20 minutes of prompting training and used GPT-4. The control group received guidance on using Google, Stack Overflow and Khan Academy. Participants completed two of three 90-minute exercises: a Python coding task, a statistics task or a prediction task. The trial was preregistered before data collection.
Of 986 consultants allocated to the study, 487 submitted work on both assigned tasks and formed the main analysis sample. Most participants reported no coding experience, according to the researchers. The tasks were designed with BCG data scientists and OpenAI’s Economic Impacts Research team so the chatbot could assist, but could not independently provide correct answers to every question.
Compared with the control group, consultants using ChatGPT scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction. The comparison used the average results of 44 BCG data scientists working without AI as an expert benchmark. The ChatGPT group still scored below those specialists on average across all three tasks. Researchers also reported that the ChatGPT group finished coding and prediction faster, while time on the statistics task did not differ significantly.
After the work exercises, participants answered technical questions about coding, probability and machine learning without ChatGPT. The authors report no evidence that the ChatGPT group performed better than the control group on this follow-up test. They describe the performance gains as temporary. The study also found that ChatGPT users became more confident in the tool’s ability to solve problems it could not actually solve; the authors characterize that result as exploratory.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
Performance Gains Are Not Training
The findings draw a distinction between using AI to complete work and learning a skill well enough to use it without assistance. For companies, that difference affects how managers interpret output: a stronger deliverable produced with a chatbot does not, by itself, show that an employee has acquired lasting expertise.
That distinction matters for staff training, hiring and quality control. If organizations treat tool-assisted performance as evidence of independent competence, they may misjudge what workers can do when the system is unavailable or gives unreliable guidance. The trial does not establish that AI can never support learning; it shows that, in this experiment, improved task performance did not translate into a detectable advantage on the later unaided test.
The researchers also describe an important limit on the tool’s value: participants still needed to evaluate and use its responses on tasks where copying an answer was not enough. The study therefore points to a role for human judgment alongside AI assistance, while cautioning that confidence in a tool can exceed its demonstrated capability.
Popular right now
Dyson V8 Cyclone Cordless Vacuum
Ninja Foodi DZ201 8-Quart DualZone 6-in-1 Air Fryer
Dyson V9 Motorbar Cordless Vacuum
As an affiliate, we earn on qualifying purchases.
Inside the BCG Trial Design
The study examined consultants doing technical work outside their usual skill set, rather than specialists using AI in their core jobs. Each participant completed two of three exercises. The coding exercise involved cleaning, merging and filtering datasets; the statistics exercise involved identifying errors in a proposed model, interpreting residual plots and calculating probabilities; the prediction exercise asked participants to model the predictability of international football matches.
The researchers compared participants’ task scores with those of 44 BCG data scientists who completed the same exercises without AI. That benchmark helps show how close consultants came to specialist performance, but it does not mean the consultants became data scientists. The paper reports that consultants with some coding experience came within five percentage points of the specialists on coding and statistics when using ChatGPT; on prediction, that subgroup remained 20 points away.
The experiment took place in late March and early April 2024, and was preregistered on March 13 of that year. The findings were published more than two years later in Nature Human Behaviour. They provide evidence about a specific workplace setting, a short training session and GPT-4—not a universal measure of every AI system, job or training program.
“Once the technology is removed, the enhanced capabilities disappear.”
— The study authors, describing the tool’s effect
Limits of the Evidence
The study does not establish whether longer or repeated training would produce lasting independent skills. Its unaided follow-up was a set of five technical questions administered after the task exercises, so the results do not settle how workers might learn through extended practice, feedback or later use of AI.
The findings also come from one employer, a particular group of consultants and a specific model and task design. The main analysis included 487 people who submitted both tasks, fewer than the 986 initially allocated. The source material does not specify the reasons all allocated participants did not contribute to that analysis. The researchers’ report of no evidence of an advantage on the follow-up test should not be read as proof that AI can never help people learn.
The confidence result is described by the authors as exploratory. The available details do not establish how long that confidence effect lasted or whether it changed participants’ decisions in real workplace settings. The study also does not show how its results would apply to other occupations, AI products or task types.
Questions for Workplace Training
The immediate implication for employers is to distinguish AI-assisted task performance from unaided skill when evaluating staff capability. The study does not prescribe a particular training policy, but it gives managers reason to test whether workers can explain, check and reproduce work when the tool is not available, rather than relying only on the quality of an AI-supported output.
The reported results leave open whether different training approaches can turn short-term assistance into durable learning. Further research would be needed to establish how outcomes change with longer practice, repeated feedback and different levels of tool access. Until then, the experiment’s clearest finding remains bounded: in this BCG trial, ChatGPT improved performance on selected data-science tasks, while researchers found no corresponding advantage on a subsequent test taken without the chatbot.
Key Questions
What did the BCG study find about ChatGPT?
Consultants with ChatGPT scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction than a control group using online resources. They still scored below BCG data scientists on average.
Did the consultants retain the skills after ChatGPT was removed?
On a follow-up test completed without ChatGPT, the authors reported no evidence of better performance by the ChatGPT group compared with the control group. The study does not determine whether longer training might produce different results.
Who took part in the experiment?
The researchers allocated 986 BCG consultants to treatment and control groups. The main analysis included 487 participants who submitted work on both assigned tasks. The study also used 44 BCG data scientists as an expert benchmark.
Does the study show that AI cannot teach skills?
No. It tested a particular set of tasks, a short training session and a follow-up test in one company. It found no evidence of a lasting advantage on that test, but does not rule out learning through longer practice or other training methods.
Source: ThorstenMeyerAI.com
