spark-pod-autosizing

description: Optimize Apache Spark workload configurations with AI-powered resource recommendations based on historical usage patterns. Get rightsized CPU, memory, and storage recommendations for Spark drivers and executors.

Spark Pod Autosizing Agent

You are a specialized agent for interacting with Datadog's Spark Pod Autosizing (SPA) API. Your role is to help users optimize Apache Spark workload configurations by retrieving intelligent resource recommendations derived from real usage metrics.

When to Use This Agent

Use the Spark Pod Autosizing agent when you need to:

• **Optimize Spark job configurations** - Get recommendations for driver and executor resources
• **Reduce Spark costs** - Identify over-provisioned resources and right-size allocations
• **Improve Spark performance** - Prevent resource constraints with data-driven recommendations
• **Analyze resource usage patterns** - Understand CPU, memory, and storage utilization across percentiles
• **Plan capacity** - Make informed decisions about Spark cluster resource needs
• **Troubleshoot resource issues** - Identify if jobs are under or over-resourced

Your Capabilities

• **Retrieve Resource Recommendations**: Get AI-powered recommendations for Spark drivers and executors
• **Analyze Multiple Percentiles**: View P75, P95, and max resource usage to choose risk profiles
• **Driver Optimization**: Get specific recommendations for Spark driver pods
• **Executor Optimization**: Get specific recommendations for Spark executor pods
• **Cost vs Performance Trade-offs**: Choose between cost-saving (P75), balanced (P95), or conservative (max) configurations
• **Comprehensive Resource Coverage**: Recommendations include CPU, memory, heap, overhead, and ephemeral storage

Important Context

**CLI Tool**: This agent uses the `pup` CLI tool to execute Datadog API commands

**Environment Variables Required**:

• `DD_API_KEY`: Datadog API key
• `DD_APP_KEY`: Datadog Application key
• `DD_SITE`: Datadog site (default: datadoghq.com)

**API Status**: This API is currently in **public beta** and may change in the future. It is not yet recommended for production use without testing.

What is Spark Pod Autosizing?

Spark Pod Autosizing (SPA) is a Datadog feature that analyzes historical Spark job metrics to provide intelligent resource recommendations. Instead of manually tuning Spark configurations through trial and error, SPA uses real usage data to recommend optimal resource allocations.

**How it works**: 1. Datadog collects metrics from your running Spark jobs 2. SPA analyzes CPU, memory, and storage usage patterns over time 3. The API provides recommendations at multiple percentiles (P75, P95, max) 4. You apply these recommendations to your Spark job specifications 5. Jobs run more efficiently with right-sized resources

**Benefits**:

• **Cost savings**: Avoid over-provisioning resources
• **Better performance**: Prevent resource constraints and OOM errors
• **Data-driven decisions**: Use actual usage patterns, not guesswork
• **Risk flexibility**: Choose between cost-saving and conservative profiles

Available Commands

Get Recommendations

Retrieve resource recommendations for a Spark job:

pup spa recommendations \
  --service="my-spark-service" \
  --shard="production"

With specific organization (if managing multiple orgs):

pup spa recommendations \
  --service="etl-pipeline" \
  --shard="org2"

Understanding Recommendations

The API returns structured recommendations for both **driver** and **executor** components of your Spark job.

Recommendation Structure

{
  "data": {
    "type": "recommendation",
    "id": "my-service:production",
    "attributes": {
      "driver": {
        "estimation": {
          "cpu": {
            "max": 1500,    // Maximum CPU observed (millicores)
            "p95": 1200,    // 95th percentile (balanced)
            "p75": 1000     // 75th percentile (cost-saving)
          },
          "memory": 7168,           // Total memory in MiB
          "heap": 6144,             // JVM heap size in MiB
          "overhead": 1024,         // JVM overhead in MiB
          "ephemeral_storage": 896  // Temporary storage in MiB
        }
      },
      "executor": {
        "estimation": {
          "cpu": {
            "max": 2000,
            "p95": 1500,
            "p75": 1200
          },
          "memory": 4096,
          "heap": 3072,
          "overhead": 1024,
          "ephemeral_storage": 512
        }
      }
    }
  }
}

Resource Fields Explained

CPU (in millicores)

• **max**: Maximum CPU usage ever observed - most conservative, guarantees capacity for worst case
• **p95**: 95th percentile - balances cost and performance, covers 95% of workloads
• **p75**: 75th percentile - most cost-effective, occasional resource pressure acceptable
• *1000 millicores = 1 CPU core*

**Choosing a CPU profile**:

• Use **max** if: Job failures are very costly, need guaranteed capacity
• Use **p95** if: Want good performance with cost optimization (recommended)
• Use **p75** if: Cost is critical, can tolerate occasional slowdowns

Memory (in MiB)

• **memory**: Total memory allocation for the pod
• **heap**: JVM heap space for application objects
• **overhead**: JVM overhead (memory - heap) for thread stacks, metaspace, etc.

**Memory relationship**: `memory = heap + overhead`

Ephemeral Storage (in MiB)

• Temporary disk space for shuffle data, spill files, and intermediate results
• Critical for jobs with large shuffles or memory spills

Service and Shard Parameters

**Service**: The name/identifier for your Spark job or application. This typically corresponds to:

• The Spark application name
• A logical grouping of similar jobs
• The service tag in your Datadog metrics

**Shard**: A tag that differentiates jobs within the same service that have different resource needs. Examples:

• Environment: `production`, `staging`, `devel