Autotuning Spark batch workloads

Console

To enable Serverless for Apache Spark autotuning on each submission of a recurring batch workload, perform the following steps:

1. In the Google Cloud console, go to the Dataproc Batches page.

   Go to Dataproc Batches

2. To create a batch workload, click Create.

3. In the Autotuning section:

   • Toggle the Enable button to enable autotuning for the Spark workload.

   • Cohort: Fill in the cohort name, which identifies the batch as one of a series of recurring workloads. Autotuning is applied to the second and subsequent workloads that are submitted with this cohort name. For example, specify daily_sales_aggregation as the cohort name for a scheduled batch workload that runs a daily sales aggregation task.

4. Fill in other sections of the Create batch page as needed, then click Submit. For more information about these fields, see Submit a batch workload.

gcloud

To enable Serverless for Apache Spark autotuning on each submission of a recurring batch workload, run the following gcloud CLI gcloud dataproc batches submit command locally in a terminal window or in Cloud Shell.

gcloud dataproc batches submit COMMAND \
    --region=REGION \
    --cohort=COHORT \
    --autotuning-scenarios=auto  \
    other arguments ...

Replace the following:

• COMMAND: the Spark workload type, such as Spark, PySpark, Spark-Sql, or Spark-R.
• REGION: the region where your batch workload will run.
• COHORT: the cohort name, which identifies the batch as one of a series of recurring workloads. Autotuning is applied to the second and subsequent workloads that are submitted with this cohort name. For example, specify daily_sales_aggregation as the cohort name for a scheduled batch workload that runs a daily sales aggregation task.

• --autotuning-scenarios=auto: Enable autotuning.

API

To enable Serverless for Apache Spark autotuning on each submission of a recurring batch workload, submit a batches.create request that includes the following fields:

• RuntimeConfig.cohort: the cohort name, which identifies the batch as one of a series of recurring workloads. Autotuning is applied to the second and subsequent workloads submitted with this cohort name. For example, specify daily_sales_aggregation as the cohort name for a scheduled batch workload that runs a daily sales aggregation task.

• AutotuningConfig.scenarios: Specify AUTO to enable autotuning on the Spark batch workload.

Example:

...
runtimeConfig:
  cohort: COHORT_NAME
  autotuningConfig:
    scenarios:
    - AUTO
...

Java

Before trying this sample, follow the Java setup instructions in the Serverless for Apache Spark quickstart using client libraries. For more information, see the Serverless for Apache Spark Java API reference documentation.

To authenticate to Serverless for Apache Spark, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

To enable Serverless for Apache Spark autotuning on each submission of a recurring batch workload, call BatchControllerClient.createBatch with a CreateBatchRequest that includes the following fields:

• Batch.RuntimeConfig.cohort: The cohort name, which identifies the batch as one of a series of recurring workloads. Autotuning is applied to the second and subsequent workloads submitted with this cohort name. For example, you might specify daily_sales_aggregation as the cohort name for a scheduled batch workload that runs a daily sales aggregation task.

• Batch.RuntimeConfig.AutotuningConfig.scenarios: Specify AUTO to enable autotuning on the Spark batch workload.

Example:

...
Batch batch =
  Batch.newBuilder()
    .setRuntimeConfig(
      RuntimeConfig.newBuilder()
        .setCohort("daily_sales_aggregation")
        .setAutotuningConfig(
          AutotuningConfig.newBuilder()
            .addScenarios(Scenario.AUTO))
    ...
  .build();

batchControllerClient.createBatch(
    CreateBatchRequest.newBuilder()
        .setParent(parent)
        .setBatchId(batchId)
        .setBatch(batch)
        .build());
...

To use the API, you must use google-cloud-dataproc client library version 4.43.0 or later. You can use one of the following configurations to add the library to your project.

Maven

<dependencies>
 <dependency>
   <groupId>com.google.cloud</groupId>
   <artifactId>google-cloud-dataproc</artifactId>
   <version>4.43.0</version>
 </dependency>
</dependencies>

Gradle

implementation 'com.google.cloud:google-cloud-dataproc:4.43.0'

SBT

libraryDependencies += "com.google.cloud" % "google-cloud-dataproc" % "4.43.0"

Python

Before trying this sample, follow the Python setup instructions in the Serverless for Apache Spark quickstart using client libraries. For more information, see the Serverless for Apache Spark Python API reference documentation.

To authenticate to Serverless for Apache Spark, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

To enable Serverless for Apache Spark autotuning on each submission of a recurring batch workload, call BatchControllerClient.create_batch with a Batch that includes the following fields:

• batch.runtime_config.cohort: The cohort name, which identifies the batch as one of a series of recurring workloads. Autotuning is applied to the second and subsequent workloads submitted with this cohort name. For example, you might specify daily_sales_aggregation as the cohort name for a scheduled batch workload that runs a daily sales aggregation task.

• batch.runtime_config.autotuning_config.scenarios: Specify AUTO to enable autotuning on the Spark batch workload.

Example:

# Create a client
client = dataproc_v1.BatchControllerClient()

# Initialize request argument(s)
batch = dataproc_v1.Batch()
batch.pyspark_batch.main_python_file_uri = "gs://bucket/run_tpcds.py"
batch.runtime_config.cohort = "daily_sales_aggregation"
batch.runtime_config.autotuning_config.scenarios = [
    Scenario.AUTO
]

request = dataproc_v1.CreateBatchRequest(
    parent="parent_value",
    batch=batch,
)

# Make the request
operation = client.create_batch(request=request)

To use the API, you must use google-cloud-dataproc client library version 5.10.1 or later. To add it to your project, you can use the following requirement:

google-cloud-dataproc>=5.10.1

Airflow

Instead of submitting each autotuned batch cohort manually, you can use Airflow to schedule the submission of each recurring batch workload. To do this, call BatchControllerClient.create_batch with a Batch that includes the following fields:

• batch.runtime_config.cohort: The cohort name, which identifies the batch as one of a series of recurring workloads. Autotuning is applied to the second and subsequent workloads submitted with this cohort name. For example, you might specify daily_sales_aggregation as the cohort name for a scheduled batch workload that runs a daily sales aggregation task.

• batch.runtime_config.autotuning_config.scenarios: Specify AUTO to enable autotuning on the Spark batch workload.

Example:

create_batch = DataprocCreateBatchOperator(
    task_id="batch_create",
    batch={
        "pyspark_batch": {
            "main_python_file_uri": PYTHON_FILE_LOCATION,
        },
        "environment_config": {
            "peripherals_config": {
                "spark_history_server_config": {
                    "dataproc_cluster": PHS_CLUSTER_PATH,
                },
            },
        },
        "runtime_config": {
            "cohort": "daily_sales_aggregation",
            "autotuning_config": {
                "scenarios": [
                    Scenario.AUTO,
                ]
            }
        },
    },
    batch_id="BATCH_ID",
)

To use the API, you must use google-cloud-dataproc client library version 5.10.1 or later. You can use the following Airflow environment requirement:

google-cloud-dataproc>=5.10.1

To update the package in Cloud Composer, see Install Python dependencies for Cloud Composer .