Apache Spark on Google Cloud
The new way to Spark easier, smarter, and faster.
Run Apache Spark workloads on Google Cloud with less operational overhead, more AI-powered assistance, and better price-performance. Focus on your code, not your cluster.
Benefits
A better experience for Apache Spark on Google Cloud
Easier - Eliminate the operational burden of Spark
Easier - Eliminate the operational burden of Spark
Choose between zero-ops Google Cloud Serverless for Apache Spark or managed Dataproc clusters. Both automate away infrastructure complexity so you can accelerate your development life cycle.
Smarter - AI assisted Spark development
Smarter - AI assisted Spark development
Accelerate your entire workflow with Gemini in Dataproc and Google Cloud Serverless for Apache Spark. Get Gemini-powered assistance to generate and debug code, and troubleshoot failed jobs.
Faster - Accelerate Spark performance
Faster - Accelerate Spark performance
Get industry-leading price-performance, automatically. For your most demanding jobs, unlock over 4.3x faster performance with Lightning Engine. This reduces TCO and accelerates time-to-insight.
Key features
Choose the right Spark for your workload
Select from Serverless for Apache Spark for zero-ops simplicity or Dataproc for managed clusters with deep customizations.
Google Cloud Serverless for Apache Spark
Focus solely on your code and accelerate development. With tiers for both cost-effective batch processing and high-performance AI/ML, it’s ideal for new Apache Spark pipelines, interactive analysis, and workloads with unpredictable demand where a "NoOps" model is preferred.
Best for: Data scientists & ML engineers, ad-hoc queries, new applications, developer productivity.
Dataproc
Get maximum control over your cluster environment. Perfect for migrating existing Apache Hadoop/Spark workloads, running long-lived persistent clusters, or using a diverse open source ecosystem.
Best for: Enterprise engineering and operations, on-prem migrations, long-running jobs, deep customization.
Customers
Delivering proven business outcomes
Related services
The engine for your open cloud data lakehouse
Spark on Google Cloud is more than a processing engine; it's a core component of a modern data and AI foundation. Unify all your data, choose the best engine for the job, and maintain consistent governance — all without vendor lock-in.
- BigQueryGoogle Cloud’s fully managed and serverless enterprise data to AI platform. Run and write Apache Spark jobs directly from the BigQuery interface for unified analytics.
- BigLakeBuild an open, managed, and high-performance Iceberg lakehouse to enable advanced analytics and data science, with automated data management and built-in governance.
- Vertex AIGoogle's unified AI platform with pre-trained and custom tooling is a one-stop, secure, fully managed interface for Spark runtime and code.
- Dataplex Universal CatalogCentrally discover, manage, monitor, and govern data and AI artifacts across your data platform, providing access to trusted data, and powering analytics, and AI at scale.
Solution
Data science on Google CloudUnified solution for the entire data science and machine learning life cycle built on a multimodal data foundation ensuring unified governance. Leverage powerful analytical engines like BigQuery SQL and Spark, then build models using BigQuery ML or Vertex AI.Solution
Data CloudWith Google’s Data Cloud, you can use built-in generative AI capabilities to activate your enterprise data across BigQuery, AlloyDB for PostgreSQL, Spanner, Cloud SQL, and more, and use built-in features to apply AI/ML directly to your data.
Documentation
Documentation
Unify your analytics: SQL and Spark on a single copy of data
Stop choosing between the power of SQL and the flexibility of Spark. BigLake lets you use both engines on the same governed data. It's a unified experience that lets you use the best tool for every job.
Accelerate your entire AI and ML life cycle
Go from data preparation to model training and inference, faster. Our Premium tiers are designed for AI/ML, letting you use pre-configured ML Runtimes with built-in GPU support, such as NVIDIA RAPIDS, to eliminate complex setup.
What's new
Get the latest Spark on Google Cloud
Apache Spark is a trademark of The Apache Software Foundation.
** The queries are derived from the TPC-DS standard and TPC-H standard and as such are not comparable to published TPC-DS standard and TPC-H standard results, as these runs do not comply with all requirements of the TPC-DS standard and TPC-H standard specification.
Take the next step
Tell us what you’re solving for. A Google Cloud expert will help you find the best solution.
Start building
Try the interactive tutorialStart using Google Cloud today
Get $300 in creditsDive into the technical details
View documentation