Elon Musk says the next-generation Grok 3 model will require 100,000 Nvidia H100 GPUs to train


 


Elon Musk, CEO of Tesla and founder of xAI, has made striking predictions regarding the timeline for artificial general intelligence (AGI), suggesting it could exceed human intelligence as early as next year or by 2026. However, he notes that achieving this milestone will require an enormous number of processors, leading to substantial electricity demands, as reported by Reuters.


Musk's company, xAI, is currently working on the second version of its Grok large language model and expects to complete the next training phase by May. Training for Grok's version 2 has utilized approximately 20,000 Nvidia H100 GPUs, and Musk forecasts that future iterations, specifically Grok 3, could require around 100,000 H100 chips.


He highlighted two primary challenges currently hindering AI advancement: supply shortages of advanced processors like Nvidia's H100 and the availability of electricity. The H100 GPU consumes about 700 watts at full capacity, meaning that a deployment of 100,000 GPUs could consume around 70 megawatts of power. Including the additional requirements for servers and cooling, a data center with this number of GPUs could use about 100 megawatts, comparable to the energy consumption of a small city.


Musk emphasized that while the GPU supply has been a significant challenge, the availability of electricity will become increasingly critical in the next couple of years. This dual constraint highlights the difficulties of scaling AI technologies to meet rising computational demands.


Despite these challenges, advancements in computing and memory architectures are expected to enable the training of larger language models in the future. At GTC 2024, Nvidia unveiled its Blackwell B200, a new GPU architecture designed to support the scaling of large language models effectively.



#ElonMusk

 #AGI

#xAI

 #ArtificialIntelligence

#Nvidia

 #GPUs

 #AIChallenges

#EnergyDemand

#TechPredictions

#LargeLanguageModels

#ComputePower

#AIRevolution

 #ProcessorShortages

 #H100



Comments

Post a Comment

Popular posts from this blog

Venod Khosla on AI, the Future of Programming, and a World of Abundance

Image
"The only thing we know about the future is it'll be uncertain. We have to figure it out in increments." - Vinod Khosla Visionary venture capitalist and entrepreneur Vinod Khosla, from his early days at Sun Microsystems to his current billion-dollar bets, offers a compelling glimpse into the future of technology, particularly AI, clean energy, and biotech. In a recent insightful conversation at the Abundance Summit, Khosla shared his bold predictions and unique perspectives, challenging conventional wisdom and painting a picture of a world brimming with abundance. For those unfamiliar, Khosla isn't just an investor; he's a company builder with a remarkable track record. His aversion to the traditional "investor" label speaks volumes about his hands-on, entrepreneurial approach. And his predictions are anything but ordinary. A Billion Bipedal Robots by 2040: Bigger Than the Auto Industry? Khosla stands firm on his December 2023 prediction: a billion biped...
Read more

The AI Upset: How a Chinese Villager Shook Silicon Valley with DeepSeek

Image
  Introduction: In a world dominated by tech giants, one name has emerged from obscurity tmo challenge the status quo: Liang Fen Wang. His creation, DeepSeek, an AI system released for free, has sent shockwaves through Silicon Valley and Wall Street, sparking a global AI race. But who is this man, and how did he achieve the impossible? The Rise of Liang Fen Wang: Liang Fen Wang's story is one of humble beginnings and relentless innovation. Born in a small village in Guangdong, China, in 1985, Liang grew up in a rapidly changing China. His parents, both primary school teachers, instilled in him the value of education. Liang excelled in mathematics and attended Xiangtan University, a breeding ground for tech giants like Alibaba. From Financial Crisis to AI Pioneer: The 2007 financial crisis sparked Liang's curiosity. He formed a team to explore whether machine learning could predict market trends better than human traders. Their findings were groundbreaking, leading Liang into th...
Read more

How scientists are creating real life invisibility: Can We Ever Truly Disappear?

Image
From spy novels to fantasy epics, invisibility has captivated the human imagination for centuries. The idea of becoming unseen, of moving through the world undetected, holds a powerful allure. But is this fantastical concept destined to remain firmly in the realm of fiction? While true invisibility, like vanishing entirely from sight, might still be a distant dream, scientists are making remarkable strides towards technologies that blur the lines between reality and illusion. The Science of Seeing (and Not Seeing) To understand how invisibility might be achieved, we must first understand how we see. Our eyes perceive objects by detecting the light waves that reflect off them. To make something truly invisible, we need to manipulate these light waves, essentially tricking our eyes into "seeing through" the object. Current Approaches:  * Metamaterials: These artificially engineered materials can manipulate light in unprecedented ways. By altering their structure at the microsco...
Read more