The most useful idea I have found about AI at work comes from a 2023 experiment with consultants at Boston Consulting Group. Researchers from Harvard, MIT, the University of Warwick and the Wharton School gave some consultants GPT-4 and kept others from using AI, then compared their work.
On tasks within what the AI could do, the results were large. Ethan Mollick, a Wharton professor and one of the authors, summarized them: consultants using AI “finished 12.2% more tasks on average, completed tasks 25.1% more quickly, and produced 40% higher quality results than those without.”
Then the researchers gave them a task chosen to sit just outside what the AI could do well. Consultants without AI got it right 84% of the time. Consultants with AI got it right only 60 to 70% of the time. The AI’s answers sounded authoritative, and Mollick writes that the consultants “fell asleep at the wheel.”
Mollick and his co-authors call the edge between those two kinds of tasks the “jagged frontier.” It is jagged because two tasks can look almost the same and land on different sides of it. I have run into that edge at EG Capital, on both sides.
Where it helped
The clearest case was checking sources. A queue of claims about possible podcast guests had been parked for a person to review. The AI reopened every cited page and checked that each was about the same person, which cleared the whole queue. That is slow, careful reading, and it did it well.
It also helped by searching everything. A sweep of the whole prospect database, instead of only the week’s batch, found strong guests the weekly batch had missed.
And it drafts. The outreach system writes a first draft of each invite. A first draft is the kind of work that sits inside the frontier, and it is useful even when it needs edits.
Where it needed a person
The first place was judgment about fit. One drafting pass sorted prospects by how strong their story was and ignored an earlier check on whether they fit. It wrote invites to people already marked as not a fit. That is the jagged frontier in one step: writing the invite sat inside it, and deciding who should get one sat outside. In the Excel prospect register I built, every screening and ranking rule is written down, and every count is a live formula. I check each ranking against those rules.
The second was matching records. A set of email addresses was attached to the wrong people, because two lists numbered their rows differently and the match went by number. It was caught when the addresses were compared with the names on a lead list, and the addresses were matched again by name.
The third is numbers. In a report, a number has to come from a record. In the reporting system, every figure traces back to its record, and every report needs a person’s sign-off before it can publish.
The last is words in someone’s mouth. A caption that is almost right can change what a speaker meant, so the video system checks every caption on screen against the checked transcript.
Finding the edge
Nothing marks where the line falls, and I expect it to keep moving as the models improve. So I look at each step of a real task, ask which side it is on, and design the system around that answer.
Outreach shows how one task can cross the line partway through. Writing a clear first draft sits on the strong side. Deciding who gets one, and checking that the address belongs to them, sits on the weak side, so the draft stops there and waits for a person.
At EG, the goal is to let AI do the steps it does well, and to put a person exactly where the line falls.
Sources
Next post What I’d tell a business student starting with AI All posts