NCT06457269

Evaluating the Potential of Large Language Models for Respiratory Disease Consultations

Evaluating the Potential of Large Language Models for Respiratory Disease Consultations: A Randomized Crossover Trial

• Status: Completed
• Phase: N/A
• Study type: Interventional
• Enrollment: 703 (actual)
• Sponsor: North Sichuan Medical College · Academic / Other
• Sex: All
• Age: —
• Healthy volunteers: Not accepted

Summary

The clinical trial aimes to evaluate multiple large language models in respiratory disease consultations by comparing their performance to that of human doctors across three major medical consultation scenarios. The main question aims to answer are: * How do large language models perform in comparison to human doctors in diagnosing and consulting on respiratory diseases across various clinical scenarios? In three clinical scenarios including the online query section, the disease diagnosis section and the medical explanation section, research assistants or volunteers will be asked to cross-question all LLMs or real doctors using predefined online questions and their own issues. After each questioning session, a short washout period is implemented to eliminate potential biases.

Conditions

• Hay Fever
• Asthma
• Acute Upper Respiratory Infection
• Pulmonary Embolism
• Pneumonia
• Tuberculosis
• Acute Bronchitis
• Pulmonary Fibrosis
• Lung Cancer
• Bronchiectasis

Interventions

Type	Name	Description
DIAGNOSTIC_TEST	Diagnosis by three human doctors	This intervention involves answering patient inquiries by different human doctors. Each patient is randomly assigned by the system to three doctors from different provinces in China selected from the database of doctors. The doctors all come from multiple online consultation platforms in China, and their diagnostic qualifications and medical licenses have undergone strict verification.
DIAGNOSTIC_TEST	Diagnosis by ChatGPT-3.5 (with search capabilities)	This intervention involves answering patient inquiries by ChatGPT-3.5 with search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.
DIAGNOSTIC_TEST	Diagnosis by ChatGPT-3.5 (without search capabilities)	This intervention involves answering patient inquiries by ChatGPT-3.5 without search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.
DIAGNOSTIC_TEST	Diagnosis by ChatGPT-4.0 (with search capabilities)	This intervention involves answering patient inquiries by ChatGPT-4.0 with search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.
DIAGNOSTIC_TEST	Diagnosis by ChatGPT-4.0 (without search capabilities)	This intervention involves answering patient inquiries by ChatGPT-4.0 without search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.
DIAGNOSTIC_TEST	Diagnosis by Claude instant (with search capabilities)	This intervention involves answering patient inquiries by Claude instant with search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.
DIAGNOSTIC_TEST	Diagnosis by Claude instant (without search capabilities)	This intervention involves answering patient inquiries by Claude instant without search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.
DIAGNOSTIC_TEST	Diagnosis by Claude 2 (with search capabilities)	This intervention involves answering patient inquiries by Claude 2 with search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.
DIAGNOSTIC_TEST	Diagnosis by Claude 2 (without search capabilities)	This intervention involves answering patient inquiries by Claude 2 without search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.
DIAGNOSTIC_TEST	Diagnosis by Gemini Pro (with search capabilities)	This intervention involves answering patient inquiries by Gemini Pro with search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.
DIAGNOSTIC_TEST	Diagnosis by Gemini Pro (without search capabilities)	This intervention involves answering patient inquiries by Gemini Pro without search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.

Timeline

• Start date: 2023-10-01
• Primary completion: 2023-12-12
• Completion: 2024-10-12
• First posted: 2024-06-13
• Last updated: 2024-11-27

Locations

1 site across 1 country: China