Artificial intelligence: ChatGPT masters questions from a US medical exam

Enlarge image

ChatGPT: up to 75 percent in the test result

Photo: DADO RUVIC / REUTERS

The chatbot ChatGPT is considered a milestone in artificial intelligence.

The software, released last November by the US company OpenAI, can, for example, write meaningful texts, chat with others in a conversational tone, or do homework for students.

However, the learning machine solved some tasks surprisingly incompetently.

Nevertheless, there is a great fear that the software could one day replace some professions.

Scientists have now given the text robot another task.

The AI ​​software was able to achieve the required minimum number of points in three theoretical parts of the United States Medical Licensing Exam (USMLE) under certain conditions.

This is reported by US experts in the journal »PLOS Digital Health«.

However, the team led by Victor Tseng from the Californian startup AnsibleHealth left out a few questions.

The USMLE is a standardized three-part exam that medical students in the United States must take to become a doctor.

Knowledge from most medical disciplines – from biochemistry to diagnostic thinking to bioethics – is evaluated.

However, when testing with ChatGPT, the authors of the study had to take into account that the AI ​​software can only accept text input.

After removing image-based questions, the authors tested the software on 350 public-facing questions that were part of last summer's USMLE exam.

After ambiguous answers were also removed, ChatGPT achieved 52.4 to 75 percent of the points that could still be achieved in the three USMLE parts of the exam.

The results varied depending on the part of the test and the type of task – such as tick tests or open questions.

The threshold for passing is around 60 percent and changes slightly depending on the year.

If the ambiguous answers were included in the result, ChatGPT achieved 36.1 to 61.5 percent of the possible points.

ChatGPT can improve medical education

According to the authors, ChatGPT outperformed PubMedGPT, a counter-model trained exclusively on biomedical literature.

The team concluded that ChatGPT had the potential to improve medical education and, in turn, clinical practice.

“Achieving the pass mark on this notoriously difficult expert exam, and doing so without any human assistance, is a remarkable milestone in the development of clinical AI,” the authors wrote.

The text robot ChatGPT had previously asserted itself in other university subjects, even if it did not achieve top marks in the exams.

According to professors at the University of Minnesota, the AI ​​chatbot tool passed the law exams in four courses and in another course at the Wharton School of Business at the University of Pennsylvania.

After the public launch of ChatGPT by the Californian start-up company OpenAI in November 2022, a number of players in the tech industry were in a real race for the favor of users.

OpenAI is supported by Microsoft with billions of dollars.

In return, Microsoft may integrate the technology into its Bing search engine and other products.

The big opponent of OpenAI and Microsoft is Google.

The search engine giant launched its own chat bot Bard on Tuesday.

Smaller players such as You.com by German AI researcher Richard Socher are also taking part in the race.

Socher comes from Dresden, but has been living and working in the USA for years.

dpa/joe