Dr. Omar Florez earned his Ph.D. in Machine Learning from Utah State University, USA, and his undergraduate degree in Computer Science from the National University of San Agustín in Arequipa (UNSA), Peru. He has authored over 20 academic publications and holds 10 patents in machine learning, multimedia indexing, and generative AI.
Over the past 10 years, he has worked as a research scientist at Silicon Valley companies such as Twitter Cortex, IBM Research, Intel Labs, and Capital One.
He received the IBM Research Innovation Award for his doctoral thesis: “Real-time Prediction of Dangerous Activities in Traffic Videos.”
In 2021, he was invited by the Presidency of the Council of Ministers of Peru to serve on the technical committee that authored Peru’s National Artificial Intelligence Strategy.
He has served on the industry advisory board of startups such as Udacity in California and is a co-founder of LatinX in AI.
He currently leads the pretraining of LatamGPT, the first large language model (LLM) trained in Latin America.
Efficient Pretraining of Large Language Models (LLMs)
This talk will delve into the technical foundations and recent advances in the efficient pretraining of large language models (LLMs), with a focus on the technical challenges involved in scaling to billions of parameters. We will examine core components such as large-scale data curation, flash attention, and the application of neural scaling laws to guide architectural and training decisions. The discussion will include parallelization strategies –Fully Sharded Data Parallel (FSDP), Tensor Parallelism, Pipeline Parallelism, and hybrid approaches– to maximize throughput across GPU clusters.
A case study will highlight LatamGPT, the first LLM trained in Latin America, currently under development on a 256 GPU H200 cluster. We will detail the system architecture, curriculum design, and optimization strategies used to process over 300 billion tokens in Spanish, Portuguese, and English. LatamGPT represents a milestone in building culturally grounded AI systems and showcases how Latin American research institutions can lead in producing open, regional foundation models for diverse applications across language, law, health, and education.