From weeks to hours: Accelerated computation on anonymized mobility datasets of up to 259 billion rows
From 60 hours to less than three: Dramatically reduced time required to investigate missing data
New benchmarks for large-scale understanding of human movement patterns at various spatio-temporal specificities to aid in disaster response and mitigation
Fast, scalable mobility data analysis helps researchers better predict human behavior to aid in disaster response.
Fast, scalable mobility data analysis helps researchers better predict human behavior to aid in disaster response.
Scaling up to manage petabytes of population data
We efficiently scaled up thousands of nodes within minutes. This rapid processing allowed our research team to significantly speed up the research process.
Debraj De
Staff Scientist, Oak Ridge National Laboratory
Every ten years, the United States Census provides a static snapshot of the American population – who we are and where we live. In turn, policymakers and government agencies use this data to design and implement community initiatives and plan infrastructure improvements. A key component is detailed analysis of human mobility – specifically where and how people might move around in their communities everyday – is a much more nuanced and complex challenge.
Dr. Gautam Malviya-Thakur has made this his life's work. As a senior staff scientist and group leader for location intelligence at Oak Ridge National Laboratory (ORNL), Malviya-Thakur simulates daily human activity to create solutions that address a range of important and complex social problems – from evacuation planning, to epidemic modeling, to everyday traffic management.
A part of the US Department of Energy, ORNL delivers scientific discoveries and technical breakthroughs needed to realize solutions in energy and national security and provide economic benefit to the nation.
As population data volumes rapidly expanded beyond petabyte scale, Malviya-Thakur and his colleagues at ORNL added Google Cloud Storage to store their datasets, and BigQuery and scalable cluster computing to analyze and query them. By using Google Kubernetes Engine, their workflow can auto-scale to available virtual machines for added flexibility. Introducing AI to analyze billions of rows of simulation data to discern large scale, macro patterns in human mobility, these solutions represented advanced backend technology that first seemed all but incomprehensible.
Initial datasets comprise up to 50,000 user agents, or automated software entities that simulate up to 60 days of human activities at specific times and locations. The scale of these datasets grew considerably over stages. To manage these massive datasets while maintaining data security and privacy standards, the team designed a workflow on Google Cloud called DICER, a Data Intensive Computing Environment and Runtime. In their published results, DICER completed complex computations on mobility datasets of 6.048 billion, 48.384 billion, and 259.2 billion rows of geospatial-temporal data points in about four hours, 11 hours, and 23.5 hours respectively. For the field of human mobility, this technical achievement required a combination of data scale, spatial precision, and hyperscaled computing infrastructure. In a later optimized trial, the team was able to dramatically accelerate this computation even further.
Advancing human mobility modeling to support disaster mitigation and response
To accurately model what they call "patterns of life," Malviya-Thakur and his colleagues designed a Human Mobility Network simulation framework (HumoNet), which draws on transportation networks, regional landmarks, synthesized population data, and daily activities predicted by generative AI. The team documented how their simulations captured key features of human mobility, creating a useful tool for exploring scenarios where researchers and planners need to predict how people will move through transportation networks under different conditions.
To study resilience in mobility systems, the team also developed a suite of Mobility Benchmarks, a framework that evaluates large-scale movement patterns of simulated populations through multiple criteria, including accessibility, availability, and efficiency.
The team also deployed other Google Cloud services, which reduced the time required to investigate, cutting it down from approximately 60 hours to less than three hours.
Xiuling Nie
Backend Developer, Software Engineer at Oak Ridge National Laboratory
By leveraging AI and cloud computing, this framework enables planners to simulate and assess the potential impacts of policy decisions, urban design changes, and disruptions such as power outages or natural disasters. Beyond that, large-scale human mobility modeling offers numerous prospective applications, including wireless network planning, public health, disaster response, retail site selection, smart city development, and epidemiology.
For Malviya-Thakur, this field presents myriad potential: "The dynamics of human mobility is such a complex research problem. We need more research and the development of first principles thinking to really understand how to predict and plan how people will move around on any given day. My work is attempting to do that, and Google Cloud is among the tools that help us achieve that mission."
Oak Ridge National Laboratory is managed by UT-Battelle for the Department of Energy's Office of Science, the single largest supporter of basic research in the physical sciences in the United States. The Office of Science is working to address some of the most pressing challenges of our time. For more information, please visit energy.gov/science.
Industry: Technology
Location: United States
Products: Google Cloud, BigQuery, Cloud Storage, Google Kubernetes Engine, Cloud SQL