AI tests the top 1% for original creative thinking

MISSOULA – New research from the University of Montana and partners shows that artificial intelligence can match the top 1% of human thinkers on a standardized test for creativity.

This study was led by Dr. Erik Guzik, assistant clinical professor in UM’s College of Business. He and colleagues used the Torrance Tests of Creative Thinking, a well-known tool used for decades to assess human creativity.

The researchers submitted eight responses generated by ChatGPT, an application powered by the GPT-4 artificial intelligence engine. They also submitted answers from a control group of 24 UM students who took Guzik’s entrepreneurship and personal finance classes. These scores were compared with 2,700 students nationwide who took the TTCT in 2016. All submissions were assessed by the Scholastic Testing Service, which was unaware that AI was involved.

The results place ChatGPT in an elite company for creativity. AI applications are in the top percentile for fluency – the ability to generate a large number of ideas – and for originality – the ability to generate new ideas. AI slipped a bit – to the 97th percentile – for flexibility, the ability to generate different types and categories of ideas.

“For ChatGPT and GPT-4, we demonstrated for the first time that it performs in the top 1% for originality,” said Guzik. “That’s new.”

He is grateful because some of his UM students are also in the top 1%. However, ChatGTP outperformed most college students nationally.

Guzik puts AI and its students to the test during the spring semester. He was assisted in his work by Christian Gilde from UM Western and Christian Byrge from Vilnius University. The researchers presented their work in May at the University of Southern Oregon Creativity Conference.

“We were very careful at the conference not to over-interpret the data,” said Guzik. “We just delivered the results. But we share strong evidence that AI appears to develop creative abilities that match or even exceed those of humans.”

Guzik said he asked ChatGPT what it would show if it performed well on TTCT. AI provided a strong answer, which they shared at the conference:

“ChatGPT tells us that we may not fully understand human creativity, which I think is true,” he said. “It also suggests we may need more sophisticated assessment tools that can differentiate between human-generated and AI-generated ideas.”

He said TTCT is protected proprietary material, so ChatGPT cannot “cheat” by accessing information about tests on the internet or in public databases.

Guzik has long been interested in creativity. As a seventh grader growing up in the small town of Palmer, Massachusetts, he attended programs for gifted and gifted students. That experience introduced him to the Future Problem Solving process developed by Ellis Paul Torrance, the pioneering psychologist who also created TTCT. Guzik says he fell in love with brainstorming at the time and how it touches the human imagination, and he remains active with the Solve Future Problems organization – even meeting his wife at one of its conferences.

Guzik and his team decided to put ChatGPT’s creativity to the test after playing around with it for the past year.

“We’ve all been exploring with ChatGPT, and we realized that ChatGPT was doing some interesting things that we didn’t expect,” he said. “Some of the responses were new and surprising. That’s when we decided to put it to the test to see how creative it really is.

Guzik says the TTCT test uses cues that mimic real-life creative tasks. For example, can you think of new uses for a product or improvements to this product?

“Let’s say it’s basketball,” he said. “Think of as many uses for a basketball as possible. You can shoot it in the circle and use it on the screen. If you force yourself to think of a new use, maybe you cut it up and use it as a planter. Or with bricks you can build something, or it can be used as a paperweight. But maybe you grind it up and turn it into something completely new.

Guzik has hopes that ChatGPT will be good at generating lots of (fluent) ideas, because that’s what generative AI does. And it excels at responding to requests with lots of ideas that are relevant, useful, and valuable in the eyes of evaluators.

He was more surprised at how well it produced original ideas, which are the hallmarks of the human imagination. The test evaluator is provided with a list of common responses to prompts – which are almost expected to be submitted. However, AI lands in the top percentile for generating new responses.

“At the conference, we studied previous research on GPT-3 that was done a year ago,” said Guzik. “Back then, ChatGPT didn’t score as well as humans on tasks involving original thinking. Now with the more advanced GPT-4 it is in the top 1% of all human responses.”

With the accelerating advances of AI, he expects it to become a key tool for the business world going forward and a significant new driver of regional and national innovation.

“For me, creativity is doing things differently,” says Guzik. “One definition of entrepreneurship that I like is that being an entrepreneur is thinking differently. So AI can help us apply the world of creative thinking to business and innovation processes, and that’s really exciting to me.”

He said the UM College of Business is open to teaching about AI and incorporating it into courses.

“I think we know the future will include AI in some way,” said Guzik. “We have to be careful about how it is used and consider the necessary rules and regulations. But businesses already use them for many creative tasks. In terms of regional entrepreneurship and innovation, this is a game changer.”

###