A defining characteristic of LLMs is their scale, measured by the number of parameters, which has grown exponentially in recent years. Models such as GPT-3, with 175 billion parameters, and its successors have demonstrated remarkable capabilities, raising questions about the relationship between model size and performance (Brown et al. 2020)
A large language model can be defined as a computational model, typically based on deep neural networks, trained on vast datasets of text to perform a wide range of language-related tasks. According to Vaswani et al. (2017), the advent of transformer architectures marked a pivotal shift in NLP, enabling models
Having explored the technical foundations, architectures, and modalities of generative AI, a critical question remains regarding its real-world impact and long-term viability. The unprecedented technological progress has been met with equally unprecedented public and investor attention, leading to a debate about whether the current boom is a sustainable reality or
Beyond their underlying architectures, another crucial way to classify generative systems is by the type of data—or modality—they process and generate. Previously, such models were typically limited to a single modality (unimodal), meaning they were designed to process and generate only one type of content. A prominent example
Generative models represent a fundamental paradigm in machine learning, enabling computers to create new data samples that closely mirror real-world examples. These models have become indispensable tools across diverse fields including image creation, natural language processing, and scientific research. Three principal architectures have emerged as dominant approaches: Generative Adversarial Networks
While the concept of generative models has existed for decades, the field began to accelerate in the mid-to-late 2010s, building upon foundational advances in deep learning. According to Bill Gates, founder of Microsoft, these tools represent the most important technological advance since the advent of the graphical user interface in
Generative AI relies on a specialised branch of machine learning (ML), namely deep learning (DL) algorithms, which employ neural networks to detect and exploit patterns embedded within data. By processing vast volumes of information, these algorithms are capable of synthesising existing knowledge and applying it creatively. As a result, generative
Benchmarking is the practice of evaluating artificial intelligence models on a standard suite of tasks under controlled conditions. In the context of large language models (LLMs), benchmarks provide a common yardstick for measuring capabilities such as factual knowledge, reasoning, and conversational coherence. They emerged because the proliferation of new models
Creating effective large language models (LLMs) involves two critical stages: pre-training and fine-tuning. These stages enable models to progress from capturing broad linguistic knowledge to excelling in specific tasks, powering applications such as automated translation, sentiment analysis, and conversational agents. Rigorous evaluation and performance measurement ensure LLMs meet general and
The evolution of machine learning has led to the development of powerful general models, such as BERT, GPT-3, and Vision Transformers, which have transformed artificial intelligence applications across diverse domains. These models, pre-trained on extensive datasets like Common Crawl for natural language processing or ImageNet for computer vision, demonstrate exceptional
Pre-training underpins the capabilities of large-scale language models like BERT and GPT, enabling them to capture linguistic patterns from extensive text corpora. This process equips models with versatile language understanding and adaptability through fine-tuning for tasks such as translation or sentiment analysis. The principles, methods, and mechanisms of pre-training reveal
Building upon the foundational principles of the attention mechanism discussed in the previous section, the Transformer architecture represents a paradigm shift by leveraging attention exclusively, completely replacing the recurrent structures that once dominated sequence modeling. This architectural innovation, first unveiled by Vaswani et al. (2017), has since catalysed a seismic
