Navigating the Jagged Technological Frontier

The study

A study involving 758 management consultants across fields including healthcare, energy, and finance compared groups using GPT-4 with groups that did not. It examined how AI affected work efficiency and quality. The paper is available at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4573321

Creative tasks

On creative tasks, the group using AI completed 12% more work in 25% less time. Their outputs also received evaluation scores that were 40% higher.

When participants were divided into upper- and lower-performing halves, the lower-performing group benefited more. This equalizing effect—where AI raises the performance of lower-performing workers more strongly—has appeared consistently across a range of studies.

Analytical tasks

The result was different for analytical tasks. Rather than helping, AI use interfered with performance: accuracy fell by 19 percentage points.

Participants who received separate instruction on how to use AI were less negatively affected in analytical work than those who did not. The comparison between the pink and green groups in the study suggests that it is better not to use AI blindly. Training on its strengths, limitations, and best practices matters.

One amusing finding is that outputs containing recommendations based on incorrect analysis received higher scores when only the recommendation-writing result was evaluated.

The jagged frontier

The paper describes this as a “jagged technological frontier.” At the frontier, there are areas in which AI exceeds human capability, such as creative tasks, and areas in which using it can be counterproductive, such as analytical tasks.

We are not yet at a point where AI performs every kind of work well. We need to understand what it does well and poorly before deciding where to apply it.

Two ways to work across the frontier

• Centaur: Like a centaur, whose human and animal parts are clearly distinct, this approach separates the work. Humans handle the parts humans should handle, while AI is assigned the parts it does well.
• Cyborg: Like a cyborg, where it is difficult to distinguish human from machine, this approach integrates AI into all work and uses it continuously as though it were part of one’s own body.

My thoughts

The core messages of my lectures are: “Use AI extensively and learn through experience what it does well and poorly,” and “Even when asking for the same task, you become better at directing it by doing it repeatedly.” I was very glad to see empirical research that supports these ideas.

The study design also incorporates the importance of AI literacy training well. I was particularly impressed by the useful metaphors of the centaur and cyborg, and of tasks that fall inside or outside the frontier.