A team of researchers has made a breakthrough in the field of natural language processing (NLP) by introducing a novel post-pretraining technique for Large Language Models (LLMs). This new technique, called block expansion, allows for the incorporation of domain-specific knowledge without compromising the overall capabilities of the models.
The challenge with LLMs is that while they excel in various tasks, their performance is limited in domains such as programming, mathematics, biomedical sciences, and finance. The current method of domain-adaptive pretraining improves the models, but it leads to catastrophic forgetting, causing the model’s general abilities to deteriorate.
To overcome this limitation, the researchers proposed block expansion, which involves extending Transformer blocks in the LLMs. By adding duplicate Transformer blocks, domain-specific information can be effectively integrated into the pre-trained models. The existing blocks remain frozen, while the newly inserted blocks are fine-tuned using domain-specific corpora.
This technique ensures that the model retains its general capabilities while also acquiring specific knowledge relevant to the domain. The researchers demonstrated the effectiveness of block expansion by developing the LLAMA PRO-8.3B model, which performs exceptionally well in general tasks, programming, and mathematics.
The LLAMA PRO family, including the instruction-following variant LLAMA PRO – INSTRUCT, demonstrated superior performance compared to existing models in the LLaMA family. These models showed great potential in reasoning and handling a variety of tasks as intelligent agents.
The primary contributions of this study include the introduction of the block expansion technique for LLMs, which allows for the incorporation of new information without sacrificing existing capabilities. Additionally, the flexible LLAMA PRO models seamlessly combine programming and natural languages, excelling in both general and domain-specific tasks.
The researchers thoroughly benchmarked the LLAMA PRO family on various datasets, showcasing their adaptability and potential in handling complex applications. This study provides valuable insights into the interplay between programming and natural languages and paves the way for developing more flexible and powerful language models.
In conclusion, the block expansion technique revolutionizes the capabilities of LLMs, allowing them to become powerful language agents that can effectively function across different domains. The findings of this research highlight the importance of overcoming the limitations of LLMs and open up exciting possibilities for the future of natural language processing.
The source of the article is from the blog revistatenerife.com