Description
The Retry-After header is a standard part of the HTTP spec which can be returned with 429 and 503 responses in order to specify a duration, or timestamp, which subsequent retries should wait before attempting to resend. It provides downstream event recipients with a mechanism to provide back-pressure when requests are arriving too frequently. The event retry mechanism in Knative eventing currently does not respect the Retry-After header. The intent of this new experimental-feature is to expose the ability for users to opt-in to respecting the Retry-After header.

Design Overview
Following the pattern established in the experimental-features process, and closely mirroring the implementation in the similar Delivery Timeout experimental-feature, the plan is to enhance the DeliverySpec to include a new optional retryAfter component. Use of this new component will be gated by the delivery-retryafter experimental feature flag in the config-features ConfigMap and enforced via WebHook validation.

Example DeliverySpec with new retryAfter component...

delivery:
  backoffDelay: PT0.5S
  backoffPolicy: exponential
  retry: 3
  retryAfter:
    enabled: true
    maxDuration: "PT30S"

The new retryAfter component will only take effect if the retry value is specified and is at least 1. The optional maxDuration field provides an override to prevent excessive backoff durations as might be desirable in certain use cases.

Exit Criteria
DeliverySpec allows optional configuration of retry behavior for 429 and 503 Retry-After headers.

Experimental Feature Flag Name
delivery-retryafter

Experimental Feature Stages
The following is the proposed plan for moving through the stages of the experimental feature process...

• Alpha: (Target 1.1 Release)
  • Initial Implementation, Unit Tests, e2e Test [Experimental-Feature] DeliverySpec.RetryAfter #5813
  • User Documentation DeliverySpec.RetryAfter Experimental-Feature docs#4361
  • Refactor Flaky Unit Tests Fix flaky Retry-After Header Test #5943, Refactor Retry-After backoff tests #5981
  • Fix Channel Backed MT Broker To Proxy Response Headers Enhance Broker Filter to proxy Response Headers #5946
  • Adjust KafkaChannel CRD to allow experimental-features in delivery spec PR template should have examples of good release notes #1025.
• Beta: (minimum of 1 release after Alpha)
  • Changes / Fixes based on Alpha ?
  • Documentation Enhancements ?
  • Conformance Tests ?
• Stable: (minimum of 2 releases after Beta)
  • Remove delivery-retryafter feature flag.
  • Remove eventing WebHook validation enforcement.
  • Add requirements to support DeliverySpec.RetryAfter to knative/specs repo

Affected WG

• Eventing WG
• Eventing-Kafka WG (previously Event-Delivery)

Additional Context

• Resources
  Experimental Features Process
  Retry-After HTTP 1.1 RFC
  Retry-After Mozilla Docs
  CloudEvent Webhook Spec

• History
  An initial attempt at supporting Retry-After headers was made in March 2021 and is mostly documented in Knative Eventing Discussion #5011. This second attempt has been briefly discussed at the Eventing WG in the past few weeks.