The research arm of SemiAnalysis has done surveys of many startups and enterprises and arrived at ~$1.5 per SXM A100 GPU per hour as a baseline cost for large clusters of 256 GPUs with NVLink and 1.6T networking. Some companies have better deals with AWS, Azure, Oracle Cloud, CoreWeave, etc., but this is a baseline. For example, the list price at Azure is only $1.36 per hour, but signing up for the 2020 released A100 for three years is not something most want to do. On-premises will also be cheaper over multiple years if utilization rates are high, but that is very difficult for most enterprises and startups to commit to/achieve.
Affordable AI Training: MosaicML Claims GPT-3 Level Models at a Fraction of the Cost
MosaicML, a company in the AI field, claims they can train models that are as good as GPT-3 (which is a super impressive model) for less than $500,000! They can even train smaller models for about $2,500,000. That sounds like a lot, but in the AI world, it’s actually quite affordable.
According to some research, the baseline cost for a big cluster of 256 GPUs (that’s a lot!) with all the fancy networking stuff is around $1.5 per GPU per hour.
You know, training AI models is not just about having a powerful machine and pressing a button. There are many other things to consider that can be quite costly – the people involved, ML Ops tools (which help manage the whole process), gathering and preparing the data, dealing with failures and restoring the process, and even handling situations where we want the model to learn from just a few examples. All of these components can add up to be quite expensive.
The Power of Smaller Models: Chinchilla Scaling Observations Redefine Cost-Effective AI Training
Nowadays, many people are following something called the Chinchilla scaling observations. These observations tell us that it’s more cost-effective to train smaller models with more data rather than training the biggest and most advanced models. While there are some criticisms of these observations, the overall idea is that we can get good results without spending too much money by focusing on smaller models with lots of data.
Imagine we have a really big model with lots and lots of parameters. These parameters help the model learn and make smart decisions. Right now, we can train a model with about 1 trillion parameters, and it would cost around $300 million to do so. That’s a huge amount of money!
To train such a big model, we would need a lot of powerful machines called A100s. If we had 100,000 of these machines working together, it would take about three months to finish training the model. This is something that big tech companies like Meta, Microsoft, Amazon, and others can afford because they have lots of money to spend on these machines.
Scaling Limits: The Challenges of Training Mega-Models with Trillions of Parameters
But what if we wanted to go even bigger? Let’s say we wanted to train a model with 10 trillion parameters. The training costs would go up to around $30 billion! That’s an enormous amount of money, even for the big tech companies. We would need one million of those A100 machines working together, and it would take more than two years to complete the training. It’s a really long time!
What’s more, the amount of power needed for all those machines and the networking required would be like having a nuclear reactor generating electricity. It’s not something we can easily handle with our current technology.
So, training models with trillions and trillions of parameters is not practical right now. It would cost too much money and take too long. Plus, there are limitations in terms of how accurate these models can be and how well they can be compressed.
Affordable Alternatives: Training AI Models on a Budget
For organizations that don’t have the financial resources to train large-scale AI models with billions or trillions of parameters, there are still options available. Here are a few recommendations:
- Start small: Instead of aiming for the largest and most complex models, begin with smaller models that have fewer parameters. While they may not have the same level of performance as the massive models, they can still provide useful insights and results.
- Use pre-trained models: Many pre-trained models are available that have been trained by big tech companies or research institutions. These models have already undergone extensive training and can be fine-tuned or adapted to suit specific tasks or domains. By using pre-trained models, organizations can save on the cost of training from scratch.
- Focus on data quality and preprocessing: Data plays a crucial role in training AI models. Ensuring high-quality data and investing in data preprocessing techniques can have a significant impact on model performance. By optimizing data collection, cleaning, and labeling processes, organizations can improve their models’ accuracy and effectiveness.
Optimizing Large Language Models: Unleashing Task-Specific Efficiency with Parameter Selection and Pruning
To determine which parameters of a large language model are most suitable for your specific task, you can follow a process called “parameter selection” or “parameter pruning.” Here’s a step-by-step guide on how to do it:
- Pretrained Model Selection: Start by selecting a large pretrained language model that aligns with your task. Models like GPT-3 or BERT are commonly used as a starting point due to their extensive pretraining on diverse datasets.
- Task-Specific Dataset: Gather a dataset that is relevant to your specific task. This dataset should consist of labeled examples that represent the input-output pairs you want the model to learn.
- Parameter Importance Calculation: Apply a method to calculate the importance or relevance of each parameter based on the extracted activations. One popular technique is “magnitude-based pruning” which ranks parameters based on their absolute values.
- Parameter Pruning: Set a threshold value to determine which parameters to keep and which ones to discard. Parameters with values below the threshold are considered less important and can be pruned (removed) from the model.
- Fine-tuning: Once you have pruned the unnecessary parameters, you can fine-tune the remaining parameters using your task-specific dataset. During fine-tuning, only the remaining parameters are updated, while the pruned parameters stay fixed.
- Evaluation and Iteration: Evaluate the performance of the fine-tuned model on your task-specific validation or test dataset. If the results are satisfactory, you can proceed with deploying the model. Otherwise, you may need to iterate the process by adjusting the pruning threshold or exploring different model architectures or hyperparameters.
Conclusion
By starting with smaller models, leveraging pre-trained models, and focusing on data quality and preprocessing, organizations can make significant progress. In the end, it’s not always about having the biggest and most complex model, but rather finding the right balance between model size, data quality, and computational resources.