The emergence of large language models (LLMs) and generative AI applications has ushered in a new era of artificial intelligence capabilities, fundamentally altering the landscape of computational requirements and associated costs. Generative AI systems, built upon transformer architectures and trained on vast datasets, have demonstrated remarkable scalability and adaptability across diverse applications. However, the exponential growth in model size and complexity has significantly outpaced advancements in compute capacity, memory bandwidth, and cost efficiency, creating substantial challenges for organisations seeking to develop and deploy these technologies (Guo et al. 2025). The financial implications of generative AI development extend far beyond initial research and development costs. From the issuer's perspective, the deployment of generative AI applications requires comprehensive consideration of hardware acquisition costs, infrastructure development, energy consumption, and ongoing operational expenses. These costs have become increasingly prohibitive, with recent analysis indicating that 15% of generative AI projects have been placed on hold and 21% of initiatives have failed to scale due to computational cost concerns (IBM 2024).
As Strubell et al. (2019) demonstrated in their foundational work, the substantial energy consumption required for training neural networks carries both financial and environmental implications. Their research brought critical attention to the need for quantifying the approximate financial and environmental costs of training neural network models, establishing a framework for understanding the broader implications of AI development. The hardware requirements for generative AI applications represent a fundamental shift in computational infrastructure needs. Unlike traditional software applications, generative AI systems demand specialised hardware configurations optimised for parallel processing, high-bandwidth memory access, and sustained computational throughput. These requirements translate into specific hardware costs that organisations must carefully evaluate when considering generative AI deployment strategies.
The cornerstone of generative AI infrastructure lies in specialised graphics processing units (GPUs) designed for parallel computation and high-throughput processing. Contemporary generative AI applications require enterprise-grade GPUs such as NVIDIA's H100 and A100 series, which command substantial acquisition costs. Individual NVIDIA H100 GPUs are priced between $25,000 and $40,000 per unit, whilst A100 GPUs, though slightly less expensive, still represent significant capital investments. For organisations requiring substantial computational capacity, the acquisition of GPU clusters can reach extraordinary levels, with a pod of 1,000 H100 GPUs representing a hardware investment of $25-40 million before considering supporting infrastructure. The computational requirements for training large-scale generative models necessitate extensive GPU deployments. Recent analysis of the GPT-MoE-1.8T model revealed that training requires either 25,000 Ampere-based GPUs for 3-5 months or 8,000 H100 GPUs for 90 days. These figures illustrate the substantial hardware requirements and the trade-offs between hardware generation and training duration. The efficiency improvements offered by newer GPU generations, such as the H100's 2-3 times performance advantage over A100 units for training workloads, can significantly impact both training time and overall computational costs (Ohiri & Poole 2025).
Beyond GPU acquisition costs, organisations must consider the comprehensive infrastructure requirements that support generative AI operations. Memory requirements are particularly demanding, with large-scale models requiring hundreds of gigabytes of high-bandwidth memory (HBM) for optimal performance. The FlashAttention technique, designed to alleviate memory bandwidth constraints, demonstrates the critical importance of memory architecture in generative AI systems by reducing data movement between HBM and on-chip SRAM through advanced tiling strategies (Guo et al. 2025). Storage infrastructure represents another significant cost component, as generative AI applications require substantial capacity for dataset storage, model checkpoints, and intermediate computational results. Training datasets for large language models can encompass hundreds of gigabytes to several terabytes, whilst model checkpoints themselves can reach hundreds of gigabytes for large-scale models (Ohiri & Poole 2025). The storage requirements extend beyond capacity to include high-performance storage systems capable of sustaining the data throughput demands of distributed training operations.
The computational demands of generative AI applications have grown exponentially, with model sizes increasing approximately 750 times every two years (Guo et al. 2025). Contemporary models range from billions to trillions of parameters, with corresponding increases in computational requirements measured in floating-point operations (FLOPs). GPT-4's training consumed an estimated 2.1 × 10²⁵ FLOPs, whilst Google's Gemini Ultra model required approximately 5.0 × 10²⁵ FLOPs. These computational requirements translate directly into hardware costs and energy consumption. The relationship between computational requirements and financial costs is exemplified by recent training cost estimates. The original 2017 Transformer model cost approximately $900 to train, whilst GPT-3, with 175 billion parameters, required between $500,000 and $4.6 million in computational costs. More recent models have pushed costs substantially higher, with GPT-4 training reportedly costing over $100 million and Google's Gemini Ultra estimated at $191 million in training compute (Ohiri & Poole 2025).
Cloud computing infrastructure has become the predominant approach for managing these computational requirements, driven partly by the limited availability of GPUs and the substantial capital requirements for on-premises infrastructure. Major cloud providers have constructed massive supercomputers specifically for LLM training, such as Microsoft's Azure supercomputer with over 10,000 GPUs designed for OpenAI's model training. However, cloud-based training introduces ongoing operational costs that can accumulate rapidly during extended training periods. The operational costs of cloud-based generative AI development are substantial and ongoing. Current pricing for NVIDIA A100 GPUs through cloud providers such as CUDO Compute starts from $1.50 per hour, with monthly commitment options available at $1,125.95 per GPU. For large-scale training operations requiring multiple GPUs over extended periods, these costs accumulate rapidly. A configuration suitable for training a Falcon 180B model, requiring 8 A100 GPUs along with supporting computational resources, totals approximately $12,401.52 per month (Ohiri & Poole 2025).
Energy consumption represents a critical component of generative AI operational costs, encompassing both direct electricity costs and associated cooling requirements. The substantial computational demands of generative AI training and inference operations translate into significant power consumption, with large training runs consuming megawatt-hours of energy (Ohiri & Poole 2025). The electricity and cooling requirements for operating GPU clusters at full capacity 24/7 create substantial ongoing operational expenses that organisations must factor into their total cost of ownership calculations. The energy implications extend beyond immediate operational costs to encompass broader infrastructure requirements. Data centres supporting generative AI operations require substantial electrical capacity and sophisticated cooling systems to maintain optimal operating conditions for high-performance computing equipment. These infrastructure requirements often necessitate significant capital investments in electrical and cooling infrastructure, particularly for organisations developing on-premises capabilities. The environmental implications of energy consumption have become increasingly important considerations for organisations deploying generative AI systems. The carbon footprint associated with training large-scale models has prompted research into more efficient training methodologies and hardware optimisations. The development of techniques such as Mixture-of-Experts (MoE) models represents one approach to mitigating computational costs by allowing increased model complexity without corresponding increases in computational requirements (Guo et al. 2025).
The substantial costs associated with generative AI development have created significant market dynamics that influence the competitive landscape and accessibility of AI technologies. The high barriers to entry created by hardware and computational costs have concentrated advanced AI development capabilities among well-funded organisations and technology companies. Only a handful of companies and well-funded academic laboratories can afford to train the largest models, creating potential concerns about market concentration and equitable access to AI technologies. The cost escalation in generative AI development has prompted organisations to reconsider their development strategies. Rather than training models from scratch, many organisations are adopting approaches that leverage pre-trained models provided by AI laboratories or open-source communities, subsequently adapting these models to specific applications through fine-tuning processes (Ohiri & Poole 2025). This approach avoids the substantial computational costs associated with initial training whilst still enabling customisation for specific use cases.
The economic pressures associated with generative AI costs have also driven innovation in cost optimisation strategies. Cloud cost governance has become increasingly important, with 53% of organisations currently managing their cost of compute governance centrally, and 73% expected to implement centralised governance by 2026. The recognition that cloud costs associated with deploying generative AI are now twice as high as the cost of the models themselves has prompted organisations to develop more sophisticated cost management approaches (IBM 2024).
References:
1. Guo, Wenzhe, Joyjit Kundu, Uras Tos, Weijiang Kong, Giuliano Sisto, Timon Evenblij, and Manu Perumkunnil. 2025. ‘System-Performance and Cost Modeling of Large Language Model Training and Inference.’ arXiv preprint arXiv:2507.02456. ^ Back
2. IBM Institute for Business Value. 2024. ‘The CEO's Guide to Generative AI: Cost of Compute.’ IBM Corporation. Available at: ibm.com. ^ Back
3. Ohiri, Emmanuel, and Richard Poole. 2025. ‘What Is the Cost of Training Large Language Models?’ CUDO Compute Blog. [Online] ^ Back
4. Strubell, Emma, Ananya Ganesh, and Andrew McCallum. 2019. ‘Energy and Policy Considerations for Deep Learning in NLP.’ Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 3645–3650. Florence, Italy: Association for Computational Linguistics. ^ Back
