From OpenAI to Google DeepMind: Who’s Leading the AI Game?
AI&Future |
Artificial intelligence (AI), as one of the most disruptive technologies of the 21st century, is profoundly impacting the global economic and social landscape. Globally, several tech giants have demonstrated exceptional technological capabilities and commercial prospects in the AI field.
In large language models, we expect the model to understand the general rules of human language, thereby producing expressions similar to humans. This is achieved through training with large amounts of data to learn the underlying patterns. When training pre-trained models, there are typically two options to improve the performance of large language models: increasing the dataset size and increasing the number of parameters in the model. Furthermore, there is a constraint during training: training cost, such as the number of GPUs and the available training time. Therefore, the pre-training of large language models usually involves a trade-off between model capacity, data volume, and training cost.
What are Scaling Laws?
This triangular tug-of-war often involves some trilemmas, such as the CAP theorem in distributed computing: consistency, availability, and partition tolerance cannot be satisfied simultaneously; at most, only two of these conditions can be satisfied at the same time. Similar explorations of this ternary relationship exist in the training of large language models, known as Scaling Laws.
In the pre-training process of large language models, cross-entropy loss is a commonly used performance metric to evaluate the difference between the model's predicted output and the actual output. Lower cross-entropy loss means the model's predictions are more accurate. The training process is also a process of minimizing the loss value.
The significance of Scaling Laws lies in the fact that AI professionals can use them to predict how the loss value of a large model will change when the number of parameters, the amount of data, and the computational cost of training vary. This prediction can help with some key design decisions, such as matching the optimal model size and data size within a fixed resource budget, without resorting to extremely costly trial and error.
OpenAI V.S Deep
- Mind DeepMind
We’re a team of scientists, engineers, ethicists and more, committed to solving intelligence, to advance science and benefit humanity.
— DeepMind DeepMind, founded in 2010 and acquired by Google in 2015, is a subsidiary of Alphabet Inc. The company focuses on developing AI systems that can mimic human learning and problem-solving abilities. As part of Alphabet Inc., DeepMind maintains a high degree of independence while leveraging Google's powerful capabilities to advance AI research.
DeepMind has achieved remarkable technological successes, including developing AlphaGo, the AI system that defeated world Go champion Lee Sedol, demonstrating the potential of deep reinforcement learning and neural networks and ushering in a new era of AI. Another significant achievement is AlphaFold, a revolutionary tool for accurately predicting protein folding, which has had a profound impact on the bioinformatics community. DeepMind's breakthroughs in AI-based protein folding prediction will help us better understand the most fundamental basis of life and assist researchers in tackling new and more challenging problems, including disease and environmental sustainability.
- OpenAI
“Our mission is to ensure that artificial general intelligence—AI systems that are generally smarter than humans—benefits all of humanity.”
—February 14, 2023, Planning for AGI and Beyond
Following Google's acquisition of DeepMind, to prevent Google from forming a monopoly in the AI field, Elon Musk and other figures in the tech industry decided to create OpenAI in 2015. As a prestigious non-profit organization, it is dedicated to developing AI technologies that can drive social progress. Unlike DeepMind, which is like a master at solving complex chessboard tactics, focusing on solving problems with clear rules and goals, OpenAI is more like a poet skilled in the art of language, dedicated to enabling machines to understand and generate natural human language.
From its initial adherence to the GPT route, which was difficult for outsiders to understand, to the advent of GPT-3 with 175 billion parameters, OpenAI has demonstrated its unparalleled capabilities in generative models, leading to a new era of AI. Similar to the relationship between Deepmind and Google, OpenAI has joined hands with technology giant Microsoft to launch in-depth strategic cooperation to further promote the development of AI technology.
