In Short:
Researchers from MIT and other institutions have developed a new technique called natural language embedded programs (NLEPs) that allows large language models to solve reasoning tasks by generating and executing Python programs. NLEPs improve accuracy and transparency by creating step-by-step programs within the model, enabling users to easily check and fix errors. This approach also offers efficiency and privacy benefits, especially for small language models.
Large language models like those that power ChatGPT have demonstrated strong performance in various tasks such as drafting legal briefs, analyzing customer reviews sentiment, and translating documents into different languages.
However, these machine-learning models often struggle with tasks requiring numerical or symbolic reasoning since they rely solely on natural language processing.
Researchers from MIT and other institutions have introduced a new approach known as natural language embedded programs (NLEPs) to enable large language models to tackle natural language, math, data analysis, and symbolic reasoning tasks by generating programs.
Enhancing Reasoning Capabilities
The implementation of NLEPs has shown to significantly improve the accuracy of large language models across a wide range of reasoning tasks. This approach not only enhances transparency but also allows users to review and correct the generated programs in case of errors.
Dr. Hongyin Luo, an MIT postdoc and co-lead author of the research paper on NLEPs, emphasizes the importance of AI performing complex reasoning in a transparent and trustworthy manner.
Utilizing Programs for Problem-Solving
Unlike traditional language models that embed programs within natural language, the researchers prompt models to generate step-by-step programs entirely in Python code, incorporating necessary natural language elements within the program.
NLEPs follow a structured problem-solving template involving four key steps to ensure accurate computation and reasoning.
Leonid Karlinsky, principal scientist at the MIT-IBM Watson AI Lab, highlights the possibilities unlocked by having language models reason with code.
Key Findings and Future Prospects
NLEPs have exhibited over 90 percent accuracy in prompting large language models like GPT-4 to solve symbolic reasoning tasks, delivering improved results compared to task-specific prompting methods. The approach also enhances data privacy and performance of smaller language models without the need for extensive retraining.
While NLEPs rely on the program generation capabilities of models, future research aims to explore methods to make smaller language models generate effective NLEPs and investigate prompt variations for enhanced reasoning robustness.
This research received partial support from the Center for Perceptual and Interactive Intelligence of Hong Kong.
