Retrieval-Augmented Generation (RAG) represents a paradigm shift in natural language processing (NLP), integrating large language models (LLMs) with dynamic information retrieval systems to produce responses that are both contextually enriched and factually grounded (Lewis et al. 2020). At its core, the RAG architecture couples a conventional generative model—one that
Most contemporary LLMs employ a decoder‑only transformer architecture, which processes sequences in parallel via self‑attention. However, scaling dense transformers linearly in size increases computation and cost. Mixture‑of‑experts (MoE) approaches address this by activating only a subset of parameters per token. In the Switch Transformer, MoE routing
Small Language Models (SLMs) are compact neural networks designed to perform natural language processing (NLP) tasks with significantly fewer parameters and lower computational requirements than their larger counterparts. SLMs aim to deliver robust performance in resource-constrained environments, such as mobile devices or edge computing systems, where efficiency is paramount. The
A defining characteristic of LLMs is their scale, measured by the number of parameters, which has grown exponentially in recent years. Models such as GPT-3, with 175 billion parameters, and its successors have demonstrated remarkable capabilities, raising questions about the relationship between model size and performance (Brown et al. 2020)
A large language model can be defined as a computational model, typically based on deep neural networks, trained on vast datasets of text to perform a wide range of language-related tasks. According to Vaswani et al. (2017), the advent of transformer architectures marked a pivotal shift in NLP, enabling models
