The Power of Prompting: Through prompting alone, GPT-4 can be guided to become a specific expert in multiple fields.

The Power of Prompting: Through prompting alone, GPT-4 can be guided to become a specific expert in multiple fields.

Microsoft Research released a study showing how to make GPT 4 perform like an expert on medical benchmarks using only the mention policy.

Research shows that GPT-4 outperforms Med-PaLM 2, a leading model fine-tuned specifically for medical applications, by a significant margin on the same benchmark.

Research shows that domain-specific expertise can be effectively elicited from a general underlying model through prompting strategies alone.

Previously, stimulating these capabilities required language models to be fine-tuned using specially curated data to achieve optimal performance in a specific domain.

Now GPT-4 can be guided to become a specific expert in multiple fields just by prompting it.

Not only has Medprompt made significant advances in the medical field, it has also demonstrated its versatility in assessments in fields such as electrical engineering, machine learning, philosophy, accounting, law, nursing, and clinical psychology.

Research methods:

Medprompt strategy: A method called “Medprompt” was proposed in the study, which combines several different prompt strategies to guide GPT-4.

Medprompt uses three main technologies: dynamic few-sample selection, automatically generated Chain of Thought (CoT), and Choice Shuffle Ensembling.

The Medprompt method includes the following key aspects:

1. Diversified prompts: Medprompt uses a variety of different types of prompts to improve the model’s performance on problems in the medical field. These prompts may include different formulations of the question, relevant background information, explanations of technical terms, etc.

2. Context learning: In order to allow the model to better understand the specific context in the medical field, Medprompt uses context learning technology. This means adding relevant information before and after a given question to help the model build a more comprehensive understanding.

3. Thinking chain method: This method encourages the model to simulate a series of thinking steps before making an answer, similar to the thinking process of professional doctors when diagnosing problems. This can help the model identify key information more accurately and come up with more reasonable answers.

4. Selective Shuffling Ensemble: This is a technique to improve model performance by combining responses generated by multiple different prompts to improve overall accuracy. This way, even if some prompts don’t yield the best answer, others may still be able to provide valuable information.

5. Cross-dataset application: Medprompt is designed to operate effectively on multiple different medical data sets, thereby increasing its applicability and flexibility.

The success of this method demonstrates that using innovative prompting techniques can significantly improve the capabilities of basic models in specialized fields, thereby providing new ways to solve complex problems.

Benchmarks:

These techniques are combined and applied to different datasets, including MedQA, MedMCQA, PubMedQA and multiple subsets of MMLU. In a study called MedQA, GPT-4 using Medprompt outperformed expert-produced CoT prompts by 3.1 percentage points without integration alone.

The study used the MedQA dataset and nine benchmark datasets in the MultiMedQA suite to test the performance of GPT-4 in the medical field.

Through these tests, the researchers evaluated GPT-4’s performance on medical knowledge and compared it with models fine-tuned specifically for medical applications.

Performance evaluation:

Research results show that GPT-4 using Medprompt

- Performance on the MedQA dataset exceeded 90% for the first time

- Achieved the best reported results on all nine benchmark datasets of the MultiMedQA suite.

- On MedQA, GPT-4 reduces the error rate by 27% compared to MedPaLM 2.

Medprompt has performed well on multiple benchmarks, not only achieving significant advances in the medical field, but also demonstrating its versatility in assessments in fields such as electrical engineering, machine learning, philosophy, accounting, law, nursing, and clinical psychology​ ​.

In addition, the research also conducted an Ablation Study to evaluate the contribution of each component of Medprompt, and found that GPT-4’s automatically generated CoT, dynamic small sample prompts, and selection rearrangement integration significantly improved performance. contribute

Significance of the study:

1. Demonstrate the domain specialization of general models: This study proves that general models such as GPT-4 can demonstrate expert-level capabilities in specific fields (such as medicine) through prompt strategies without domain-specific fine-tuning.

This is an important advance for the field of natural language processing (NLP) because it shows that general models can be adapted to specific application scenarios through appropriate prompting strategies rather than through expensive specialized training.

2. Reduce resources and costs: Traditionally, making a model perform well in a specific domain requires specialized fine-tuning, which often involves using expert-labeled datasets and extensive computing resources. With an effective prompting strategy, this need can be reduced, opening up the possibility of using advanced AI technologies for small and medium-sized organizations.

3. Cross-field application potential: Research also shows that this prompting method has shown value in professional competency examinations in multiple fields, which means that its application potential is not limited to a single field.

Official introduction

Paper

More AI News

Artificial Intelligence Article

New AI Technology