Add Monitoring & Alerting for Agent Deployments

Overview

Pertaining #142. This PR adds monitoring and alerting infrastructure to the agent-starter-pack, giving users production-grade observability out of the box. The implementation is platform-aware, agent-aware, and fully configurable through user prompts during project creation.

What's New

User-Facing Features

1. Optional Email Alert Notifications

• Interactive prompt during agent-starter-pack create asks for an email address
• If provided, alerts are delivered via email + Cloud Console
• If skipped, alerts are console-only (no email noise for dev environments)
• User sees clear confirmation of their choice with visual feedback

2. Configurable Alert Thresholds
All thresholds are exposed as Terraform variables with sensible defaults:

• Latency alerts: P95 threshold (default: 3000ms)
• Error rate alerts: Error count per 5-min window (default: 10 errors)
• Retriever latency alerts (Agentic RAG only): P99 threshold (default: 10000ms)
• Agent error rate (Cloud Run): Errors per second (default: 0.5/sec)

Users can customise these in deployment/terraform/dev/vars/env.tfvars after project creation.

Infrastructure Added

Universal Log-Based Metrics (All Agents, All Platforms)

1. Agent Operation Count: Tracks all agent operations with operation type labels
2. Agent Error Count by Category: Categorised errors (LLM_FAILURE, TOOL_FAILURE, RETRIEVER_FAILURE, etc.)

Agentic RAG-Specific Metrics

1. Retriever Latency Distribution: P50/P95/P99 retrieval performance with histogram buckets
2. Document Count Distribution: Number of documents retrieved per call
3. Retriever Latency Alert: Fires when P99 > threshold (default 10s)

Agent Engine (Reasoning Engine) Platform

• Latency Alert: P95 request latency monitoring using native platform metrics
• Error Rate Alert: Fires when log-based error count exceeds threshold in 5-min window
• Dashboard: 5-7 chart dashboard including:
  • Request count (requests/sec)
  • Request latency (P50/P95/P99)
  • CPU allocation
  • Memory allocation
  • Agent errors by category
  • Retriever latency (Agentic RAG only)
  • Documents retrieved per call (Agentic RAG only)

Cloud Run Platform

• Latency Alert: P95 request latency using Cloud Run native metrics
• 5xx Error Rate Alert: Monitors 5xx response codes
• Agent Error Alert: Log-based agent errors with rate threshold

Technical Details

Terraform Structure

• New file: deployment/terraform/dev/monitoring.tf (757 lines)
• New file: deployment/terraform/monitoring.tf (prod equivalent)
• Modified: deployment/terraform/dev/variables.tf (added 4 monitoring variables)
• Modified: deployment/terraform/dev/vars/env.tfvars (added default threshold values)
• Modified: deployment/terraform/dev/apis.tf (added monitoring.googleapis.com)

Python CLI Integration

• agent_starter_pack/cli/commands/create.py: Added interactive email prompt
• agent_starter_pack/cli/utils/template.py: Thread alert_notification_email through template processing
• tests/cli/commands/test_create.py: Updated test mocks to handle new prompt

Smart Templating

• Uses Jinja2 conditionals to render appropriate resources based on:
  • cookiecutter.deployment_target (agent_engine vs cloud_run)
  • cookiecutter.agent_name (agentic_rag gets extra retriever metrics)
• Notification channel only created if email provided (using Terraform count)

Metric Design Decisions

Why log-based metrics for agent telemetry?

• Platform-agnostic: Works on both Agent Engine and Cloud Run
• Flexible: Can extract any JSON payload attribute from structured logs
• Extensible: Users can add custom agent metrics by logging with the right labels

Why native metrics for platform SLOs?

• Accuracy: Platform-provided metrics are the source of truth
• Performance: No additional overhead from log processing
• Consistency: Aligns with Google Cloud best practices

Alert auto-close: 30 minutes

• Prevents alert fatigue from transient issues
• Long enough to investigate without losing context
• Configurable via alert_strategy.auto_close if users want different behaviour

Test Coverage

All tests pass:

• ✅ 95/95 CLI tests (including new email prompt flow)
• ✅ Ruff linting
• ✅ Mypy type checking
• ✅ Import ordering fixed

Migration Notes

Existing Projects

• This is a template change only - existing deployed agents are unaffected
• Users can retrofit monitoring by:
  1. Copying the new monitoring.tf files
  2. Adding the monitoring variables
  3. Running terraform apply

New Projects

• Zero additional effort required
• Users just need to answer the email prompt during creation
• Monitoring deploys automatically with the agent infrastructure

Example Usage

$ agent-starter-pack create my-agent

# ... after other prompts ...

Monitoring & Alerting Setup
Configure email notifications for production alerts (optional).
Email for alert notifications: ops-team@example.com
✓ Alerts will be sent to: ops-team@example.com

# Or skip it:
Email for alert notifications:
⚠ Email notifications disabled. Alerts will only appear in Cloud Console.

After deployment, users get:

• Real-time dashboards in Cloud Monitoring
• Automatic alerts when thresholds are breached
• Structured logs for debugging with labels.service_name filtering

Related Documentation

The monitoring infrastructure automatically creates:

• Cloud Monitoring dashboard (named "Reasoning Engine - {project_name}")
• Alert policies with descriptive documentation
• Notification channel (email) if configured

Users can find their dashboard at:

https://console.cloud.google.com/monitoring/dashboards

Checklist

• Added user-facing email prompt with clear feedback
• Created comprehensive monitoring.tf for both dev and prod
• Added configurable threshold variables with sensible defaults
• Platform-specific alerts (Agent Engine vs Cloud Run)
• Agent-specific metrics (Agentic RAG retriever monitoring)
• Updated CLI tests with new prompt flow
• All linting and type checking passes
• Tested with both email provided and skipped scenarios