In Short:
OpenAI’s success with ChatGPT was due in part to human trainers guiding the AI model. Now, OpenAI aims to integrate AI to assist human trainers in making AI helpers smarter and more reliable. They developed a new model called CriticGPT, fine-tuned from GPT-4, to help assess code more accurately. This technique aims to improve large language models and ensure AI behaves acceptably as it becomes more capable.
OpenAI has announced a new technique that involves integrating more AI into the process of training human trainers. This new method aims to make AI helpers smarter and more reliable by assisting human trainers in assessing code and other complex outputs.
RLHF Technique for Developing ChatGPT
In developing ChatGPT, OpenAI used the pioneering technique called reinforcement learning with human feedback (RLHF). This technique involves fine-tuning an AI model based on feedback from human testers to ensure more coherent, less objectionable, and accurate outputs. The ratings provided by human trainers help drive the behavior of the model, making chatbots more reliable and useful while preventing misbehavior.
Introduction of CriticGPT
OpenAI has introduced a new model, CriticGPT, which is based on fine-tuning its powerful offering, GPT-4. This new model has shown the ability to catch bugs missed by humans and provide better critiques of code, outperforming human judges 63% of the time. OpenAI plans to expand this approach to other areas beyond code in the future.
Potential Benefits and Challenges
While the new technique shows promising results, there are challenges to overcome. Nat McAleese, a researcher at OpenAI, mentions that human feedback can be inconsistent and rating complex outputs can be challenging. Despite its imperfections, the technique could help enhance the accuracy of AI models like ChatGPT and aid in training AI models beyond human capabilities.
Implications for AI Development
This new technique is part of ongoing efforts to improve large language models and ensure that AI behaves in desirable ways as it becomes more capable. It could potentially lead to the development of more powerful and trustworthy AI models aligned with human values.
Industry Response and Future Directions
Anthropic, a competitor to OpenAI, has also made advancements in AI models, showcasing a more capable chatbot named Claude. Both companies are exploring ways to understand and prevent unwanted behaviors in AI models. OpenAI is committed to deploying this new technique in various areas to train its next major AI model and demonstrate its dedication to responsible AI development.
Dylan Hadfield-Menell, a researcher at MIT, acknowledges the significance of using AI models to train more powerful ones and sees it as a natural progression in AI development.
Overall, the integration of AI into human training processes presents opportunities for enhancing AI capabilities while ensuring their responsible use in various applications.
