TL;DR
A randomized trial of Boston Consulting Group consultants found that access to ChatGPT improved performance on coding, statistics and prediction tasks outside many participants’ skill sets. After ChatGPT was removed, researchers found no measurable advantage on a short technical knowledge test; the authors describe the upskilling as temporary.
The trial allocated 986 consultants between a ChatGPT group and a control group. Of those, 487 submitted work on both tasks, forming the main analysis sample. Participants in the treatment group received 15 to 20 minutes of prompting instruction and used GPT-4. The control group received instruction on using Google and resources including Stack Overflow and Khan Academy. The experiment took place in late March and early April 2024 and was preregistered before data collection.
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Participants completed two 90-minute tasks selected from coding, statistics and prediction. The ChatGPT group scored 49 percentage points higher on coding than the control group, along with gains of 20 points in statistics and 17 points in prediction. Scores were compared with the average performance of 44 BCG data scientists who completed the work without AI; consultants using ChatGPT narrowed the gap but remained below that expert benchmark on average.
After the tasks, participants answered five technical questions without ChatGPT. The authors report differences between the ChatGPT and control groups ranging from −7.7 to +4.8 percentage points, none statistically significant; the joint test had a p-value of 0.531. The authors say they found no evidence of a knowledge gain from the AI-assisted tasks. They also report that the ChatGPT group became more confident in the tool’s ability to solve problems it could not actually solve; the authors label that analysis exploratory.
Performance Gains Depend on AI Access
The findings distinguish task performance with an AI assistant from knowledge workers’ ability to complete similar work independently. For companies, that difference affects how managers evaluate productivity, training and the skills required for a role. A higher-quality deliverable produced with ChatGPT may reflect effective use of a tool, rather than a lasting increase in the worker’s technical expertise.
The trial suggests that short instruction can help consultants use AI to take on work outside their usual skill set. It does not establish that the same results apply across industries, occupations, longer training programs or newer AI systems. Nor does the knowledge test show whether repeated use over months would build skills; it measured performance shortly after the experiment under specific conditions.
For readers using AI at work, the practical distinction is whether the task requires a tool-assisted result or independent technical judgment. The study’s test of confidence also raises a question for quality control: users may overestimate what the system can do. The authors’ exploratory finding points to that possibility, but does not establish how widespread it is beyond this trial.
Popular right now
Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Bluetooth 5.3, USB-C Charging, IPX4, Black
Shark WANDVAC Power PET Cordless Handheld Vacuum, White, WV410WH
Bose QuietComfort Ultra Earbuds (2nd Gen), Wireless Noise-Cancelling Earbuds with Immersive Audio and Wireless Charging Case, Black
As an affiliate, we earn on qualifying purchases.
A Preregistered BCG Experiment
The researchers designed the tasks with BCG data scientists and OpenAI’s Economic Impacts Research team. The tasks were set up so ChatGPT could provide useful guidance but could not independently answer every question correctly, according to the study materials. This design aimed to test participants’ work with the tool, rather than whether they could simply copy a complete correct response.
The expert benchmark came from 44 BCG data scientists who performed the tasks without AI. Their average served as the reference point for task scores. The consultants’ later test also omitted ChatGPT, allowing the researchers to compare what participants could answer after the assisted work with what the control group could answer.
The paper, by Wiles, Krayer and co-authors, is titled “Generative AI and the temporary upskilling of knowledge workers.” Its results concern a particular trial using GPT-4 in 2024 and a short, five-question knowledge test. The source material identifies the study as a randomized controlled trial and says it was preregistered on March 13, 2024.
““We find no evidence” that the ChatGPT group did better on the technical questions after the tool was removed.”
— Wiles, Krayer and co-authors
Limits of the Knowledge Test
The post-task assessment covered five technical questions, so it cannot establish whether longer practice, different instruction or repeated AI use would produce lasting learning. The source material does not provide a longer-term follow-up. It also does not establish whether the results generalize beyond BCG consultants, the selected data science tasks or GPT-4 as used in 2024.
The sample narrowed from 986 people allocated to 487 who submitted both tasks. The supplied study summary does not explain in detail how the people who did not submit both tasks may have differed from those in the analysis sample. The confidence finding is described by the authors as exploratory, and the available information does not quantify how often misplaced confidence led to errors in real work.
Testing Skills Beyond the Trial
The paper’s findings leave open whether longer training or repeated use would change the results. Follow-up research could assess workers after longer periods, across more kinds of jobs, and with tests of both tool-assisted output and independent performance. The source material does not identify a specific next study or milestone.
For employers applying the findings, the immediate question is what a role requires: successful work with an AI system, or the ability to perform and check the work without one. The trial supports the conclusion that ChatGPT helped this group complete the assigned tasks; it does not show that the assistance produced measurable standalone technical knowledge on the test used.
Key Questions
What did the BCG trial find?
Consultants given ChatGPT scored higher than a control group on coding, statistics and prediction tasks. The reported gains were 49, 20 and 17 percentage points, respectively.
Did the consultants learn to do the tasks without ChatGPT?
The study found no statistically significant advantage for the ChatGPT group on a five-question technical test taken without the tool. The authors say they found no evidence of a knowledge gain in that test.
How many people took part?
986 consultants were allocated between the groups. The main analysis included 487 who submitted work on both tasks.
Were ChatGPT users as skilled as BCG data scientists?
No. ChatGPT users narrowed the performance gap on the tasks, but their average scores remained below those of 44 BCG data scientists who completed the work without AI.
Does the result prove AI cannot teach lasting skills?
No. The trial tested a specific intervention and a short assessment shortly after the tasks. It does not settle whether longer training or repeated use can build lasting skills.
Source: Thorsten Meyer AI
