GuidesAPI Reference
Log In

Progressive Delivery

Suggest Edits

Progressive delivery is a controlled, slow rollout of Service Instances within a single Release Channel, usually with metric analysis as part of the rollout. Progressive delivery can be useful for large, high QPS production workloads, however, it is quite limited for low QPS systems. We recommend using the Protections feature with our Built-in Protections alert check for low QPS services.

Prodvana supports progressive delivery in Kubernetes via Argo Rollouts.

Install Argo Rollouts

Follow the instructions in https://argo-rollouts.readthedocs.io/en/latest/installation/#controller-installation to install the Rollouts controller in your Kubernetes Runtime (copied below for convenience). This must be done once per each runtime you plan to perform progressive delivery.

$ kubectl create namespace argo-rollouts
$ kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

Deploying a Service Gradually

Progressive delivery can be used to deploy a Service Instance slowly to reduce the scale of an outage in case something goes wrong.

A human or external automation can validate that there are no issues while it goes through various steps in the deployment.

In the example below, the Rollout object (which replaces the Deployment object) is set to roll out to 10%, wait for manual approval, roll out to 25%, wait 30 seconds, roll out to 50%, wait 30 seconds, then roll out to 100%.

The Prodvana config is unchanged from how it would look without progressive delivery - Prodvana will automatically detect that the Kubernetes object being deployed is an Argo Rollout object.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: gradual-rollout
spec:
  replicas: {{.Params.replicas}}
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      app: gradual-rollout
  strategy:
    canary:
      canaryService: gradual-rollout-preview
      steps:
      - setWeight: 10
      - pause: {} # Block indefinitely waiting for manual promotion
      - setWeight: 25
      - pause:
          duration: 30 # seconds
      - setWeight: 50
      - pause:
          duration: 30
  template:
    metadata:
      labels:
        app: gradual-rollout
    spec:
      containers:
      - image: "{{.Params.image}}"
        imagePullPolicy: Always
        name: gradual-rollout
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        resources:
          requests:
            cpu: 5m
            memory: 32Mi

apiVersion: v1
kind: Service
metadata:
  name: gradual-rollout
spec:
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: gradual-rollout
---
apiVersion: v1
kind: Service
metadata:
  name: gradual-rollout-preview
spec:
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: gradual-rollout

service:
  name: gradual-rollout
  application: external-configs
  externalConfig:
    type: KUBERNETES
    local:
      path: .

  parameters:
    - name: image
      required: true
      dockerImage:
        imageRegistryInfo:
          containerRegistry: dockerhub-prodvana
          imageRepository: argoproj/rollouts-demo
    - name: replicas
      description: replica count
      int:
        defaultValue: '3'

Prodvana will now display rollout progress in the Service Instance convergence view and provide Promote buttons to promote pauses. Note that all pauses can be manually promoted, regardless of whether they are configured to advance after a certain amount of time or not automatically.

You can also promote the Rollout object as you would without Prodvana via the Kubectl CLI. For more information, please see Argo Rollout documentation.

Initial deployment

Deploying a Service With Automated Canary Analysis

A common use of progressive delivery is to deploy a Service Instance while maintaining an invariant throughout the deployment process. For example, if the Service starts showing unacceptably high errors, abort the deployment and safely roll back to the previous state.

The following example describes deploying a service while ensuring that a Prometheus query is within spec.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: canary-demo
spec:
  replicas: {{.Params.replicas}}
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      app: canary-demo
  strategy:
    canary:
      analysis:
        templates:
        - templateName: success-rate
      canaryService: canary-demo-preview
      steps:
      - setWeight: 40
      - pause:
          duration: 30
      - setWeight: 60
      - pause:
          duration: 10
      - setWeight: 80
      - pause:
          duration: 10
  template:
    metadata:
      labels:
        app: canary-demo
    spec:
      containers:
      - image: "{{.Params.image}}"
        imagePullPolicy: Always
        name: canary-demo
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        resources:
          requests:
            cpu: 5m
            memory: 32Mi

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  metrics:
  - name: success-rate
    interval: 10s
    successCondition: result[0] >= 0
    provider:
      # To use other metrics providers like datadog or influxdb, use the appropriate
      # section from https://argo-rollouts.readthedocs.io/en/latest/features/analysis/
      prometheus:
        address: http://prom-prometheus-server.default.svc.cluster.local
        # Replace query below with desired metric to query
        query: >+
          sum(rate(apiserver_request_total{group="coordination.k8s.io", verb="PUT"}[1h]))

apiVersion: v1
kind: Service
metadata:
  name: canary-demo
spec:
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: canary-demo
---
apiVersion: v1
kind: Service
metadata:
  name: canary-demo-preview
spec:
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: canary-demo 

service:
  name: rollout
  application: external-configs
  externalConfig:
    type: KUBERNETES
    local:
      path: .

  parameters:
    - name: image
      required: true
      dockerImage:
        imageRegistryInfo:
          containerRegistry: dockerhub-prodvana
          imageRepository: argoproj/rollouts-demo
    - name: replicas
      description: replica count
      int:
        defaultValue: '3'

Argo Rollouts supports polling other metrics providers and also custom plugins to supply metrics.

Deploying a Service with the bluegreen Strategy

Bluegreen strategy can be used to deploy an identical copy of Service Instance at the new release. A human or external automation can validate the new release. Once validated, live traffic can be routed to the new release instantly.

In the example below, the Rollout object creates a new preview release which needs to be manually promoted, and once promoted, the old release is spun down after a delay of 120s (2 minutes).

The Prodvana config is unchanged from how it would look without progressive delivery - Prodvana will automatically detect that the Kubernetes object being deployed is an Argo Rollout object.

🚧

Autorollback must be disabled to allow Rollouts to handle rollback on failure

service:
  name: gradual-rollout
  autoRollback:
    disabled: true

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: bluegreen-demo
spec:
  replicas: {{.Params.replicas}}
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      app: bluegreen-demo
  template:
    metadata:
      labels:
        app: bluegreen-demo
    spec:
      containers:
      - name: rollouts-demo
        image: "{{.Params.image}}"
        ports:
        - containerPort: 8080
  strategy:
    blueGreen:
      # activeService specifies the service to update with the new template hash at time of promotion.
      # This field is mandatory for the blueGreen update strategy.
      activeService: bluegreen-demo-active
      # previewService specifies the service to update with the new template hash before promotion.
      # This allows the preview stack to be reachable without serving production traffic.
      # This field is optional.
      previewService: bluegreen-demo-preview
      # autoPromotionEnabled disables automated promotion of the new stack by pausing the rollout
      # immediately before the promotion. If omitted, the default behavior is to promote the new
      # stack as soon as the ReplicaSet are completely ready/available.
      # Rollouts can be resumed using: `kubectl argo rollouts promote ROLLOUT`
      autoPromotionEnabled: false

kind: Service
apiVersion: v1
metadata:
  name: bluegreen-demo-active
spec:
  selector:
    app: bluegreen-demo
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

---
kind: Service
apiVersion: v1
metadata:
  name: bluegreen-demo-preview
spec:
  selector:
    app: bluegreen-demo
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

service:
  name: bluegreen
  application: external-configs
  externalConfig:
    type: KUBERNETES
    local:
      path: .
  autoRollback:
    disabled: true # Autorollback must be disabled to allow Rollouts to handle rollback on failure

  parameters:
    - name: image
      dockerImage:
        defaultTag: blue
        imageRegistryInfo:
          containerRegistry: dockerhub-prodvana
          imageRepository: argoproj/rollouts-demo
    - name: replicas
      description: replica count
      int:
        defaultValue: '3'

Caveats

  • Only setWeight and pause steps are fully supported. Using other steps may result in broken UI (but expected to be otherwise functional).
  • No support for aborting a canary rollout currently.

Updated 5 months ago