GPT-4 Ideas and Creative Writing: Doshi, 2023

Study overview

Doshi and Hauser’s August 2023 study, “Generative artificial intelligence enhances creativity,” examined whether writers could produce more creative work with help from GPT-4-generated ideas.

bullets

• Source: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4535536
• The study involved 293 writers, each asked to write an eight-sentence creative story.
• Writers were assigned to one of three conditions: writing alone, receiving one generative-AI idea, or receiving five generative-AI ideas.
• Writers evaluated their own stories, while 600 human evaluators also assessed them.
• The main measures were novelty—whether a story was novel, original, and rare—and usefulness—whether it was appropriate, feasible, and publishable.

How generative AI may help or hinder creativity

Generative AI may improve creativity by giving writers ideas from which they can branch out. It may also help writers overcome writer’s block.

At the same time, it may constrain the starting point of thought. AI-supported work may be no more than a variation on existing ideas. Both effects may occur at once: a story could be novel while still being low in usefulness.

Results

Both novelty and usefulness increased when writers received generative-AI assistance. The group that could receive up to five ideas performed best across the measured areas.

However, writers’ self-evaluations did not differ across the groups. External human evaluators rated AI-assisted stories more favorably on whether they were well written, enjoyable, surprising, and likely to change expectations of future reading; they also found them less boring. Stories written without AI assistance were rated funnier.

Before writing, participants completed the Divergent Association Task (DAT), a measure of creativity. Writers with higher DAT scores ultimately received higher ratings in every condition, but their ratings did not improve with generative AI. This suggests that lower-creativity writers benefited more, producing an equalizing effect rather than evidence that AI raised the upper limit of creativity.

Detectability, authorship, and similarity

Evaluators could predict whether generative AI had been used. When they were shown the AI ideas used in writing and asked how much of the resulting story belonged to the writer, they assigned the writer a lower share in AI-assisted conditions. The perceived human contribution was even lower when five ideas had been provided.

The estimated AI contribution was 25.4% when one idea was provided and 31.0% when five ideas were provided.

AI-assisted writing was also more similar, according to embedding cosine similarity, to generative-AI output and to writing by other participants in the same condition. If AI-influenced writing becomes widespread, there may be more overlap among published works. This concern becomes stronger if AI-influenced published work is subsequently used to train language models, creating a cycle.

Ethical views reported in the survey

The survey responses raised several ethical positions about generative-AI use in creative work.

• Using generative AI is not necessarily unethical.
• Work created with generative AI is no longer fully creative work.
• AI may be used to develop an idea, but its use should be disclosed.
• People whose work contributed to AI training data should be compensated.
• When AI is used, the AI output should also be disclosed.

Limitations

The study did not allow active, open-ended interaction with generative AI. It only gave writers an opportunity to receive ideas. About 90% of writers chose to receive them.

The study did not measure whether AI reduced the time required to write a story. It would also be interesting to know how a writer’s attitude toward AI affects the results.

My thoughts

If generative AI has a share in a creative work, who should own that share? Does it belong to the company operating the system, or should some portion also go to the unnamed original creators whose work supplied the model’s training data? This is closely connected to issues discussed in an earlier post: http://blog.mahler83.net/archives/3721

The finding that more capable people rely less on large language models resembles results from earlier research: http://blog.mahler83.net/archives/3545. This is the equalizing effect suggested here.

Creativity may ultimately be the creation of new combinations of concepts, or tokens, that already exist. A language model has learned relationships among an immense number of tokens in the world, so it may naturally be more creative than humans in this sense. The question is how to use it well.

Can large language models help humans break through the upper limit of human creativity? How could that be studied?

• Original tweet: https://twitter.com/mahler83/status/1691056916825612288