Google Cloud Serverless for Apache Spark

Experience industry-leading price-performance. Google Cloud Serverless for Apache Spark is powered by our next-generation native query engine, Lightning Engine , in Preview. It delivers significantly faster Spark query and data processing performance, over 4.3x faster** than open source Apache Spark , through its advanced vectorized execution, in-built intelligent caching, and optimized storage I/O, helping you get insights faster and reduce costs.

** The queries are derived from the TPC-DS standard and TPC-H standard and as such are not comparable to published TPC-DS standard and TPC-H standard results, as these runs do not comply with all requirements of the TPC-DS standard and TPC-H standard specification.

Run your production Spark workloads with confidence. Google Cloud Serverless for Apache Spark optimizes resources, provides job isolation, and supports Google Cloud’s enterprise security capabilities (including VPC-SC, CMEK, personal authentication, and custom organization policies). It ensures a secure execution environment with capabilities like secure subnets, encryption by default for data at rest and in transit, and no direct VM or root access, minimizing your operational security burden. While built for automation, expert users retain full access to Spark configurations for fine-grained control.

Infuse generative AI into your Spark development life cycle. Leverage Gemini for context-aware PySpark code generation in notebooks with intelligent context of your data to supercharge productivity. Get AI-assisted troubleshooting recommendations with Gemini Cloud Assist Investigate to quickly resolve issues, deeper operational insights, and optimize performance.

Seamlessly run distributed training or batch inference workloads. Google Cloud Serverless for Apache Spark offers built-in support for GPU acceleration and comes with pre-packaged popular ML libraries like XGBoost, PyTorch, and Transformers. This leads to significantly faster startup times for AI/ML environments and improves reliability since the images are Google-certified.