Nowadays the LLMs are getting larger and larger. While larger models generally have better performance, there are many scenarios where a smaller model can perform as good. If you goal is to training and run LLMs for specific tasks, there is likely redundancies in large models. This means that you don’t have to always use larger models for better performance, especially when the budget is limited.

Here are few scenarios where it is possible to training and run smaller models to achieve results that are as good as bigger models.

Downsizing the models through quantization for your task.

If the model is trained and stored in 32-bit floating point values, you may try mapping it to a 16-bit floating point and evaluate the performance and if it can achieve the performance you require.

Pruning parameters.

Based on the Lottery Ticket Hypothesis, if you choose the right subset of parameters and initialize them correctly, you can achieve the same level of performance as the fully trained model.

Model Distillation.

You can train a smaller model to mimic the behavior of a bigger model.

Training through LoRA or QLoRA.

When retraining a model and updating earlier layers of parameters, you don’t have to retraining all the parameters, which may lead to catastrophic forgetting. You can instead freeze the original weights and training a set of delta weights. The delta weights can be much smaller in size through SVD. Basically what you need to training is no longer the same size of the original parameters but the size of the decomposed matrices which is much smaller.

Model training with domain specific data.

If you have high quality domain specific data, you can train a model that is much smaller in size but still delivers good performance as larger models. For example, BloombergGPT, with 50 billion parameters, can achieve similar performance and even outperforms other bigger models in certain tasks. This shows that the number of parameters is not the only important factor. Note that training a model like BloombergGPT still takes a significant amount of resources and cost.