Progressive Delivery

Suggest Edits

Progressive delivery is a controlled, slow rollout of services within a single release channel, usually with metric analysis as part of the rollout. Progressive delivery can be useful for large, high QPS production workloads, however, it is quite limited for low QPS systems. We recommend using the Protections feature with our Built-in Protections alert check for low QPS services.

Prodvana supports progressive delivery in Kubernetes via Argo Rollouts.

Install Argo Rollouts

Follow the instructions in https://argo-rollouts.readthedocs.io/en/latest/installation/#controller-installation to install the Rollouts controller in your Kubernetes runtime (copied below for convenience). This must be done once per each runtime you plan to perform progressive delivery.

$ kubectl create namespace argo-rollouts
$ kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

Deploying a Service Gradually

Progressive delivery can be used to deploy a service slowly to reduce the scale of an outage in case something goes wrong.

A human or external automation can validate that there are no issues while it goes through various steps in the deployment.

In the example below, the Rollout object (which replaces the Deployment object) is set to roll out to 10%, wait for manual approval, roll out to 25%, wait 30 seconds, roll out to 50%, wait 30 seconds, then roll out to 100%.

The Prodvana config is unchanged from how it would look without progressive delivery - Prodvana will automatically detect that the Kubernetes object being deployed is an Argo Rollout object.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: gradual-rollout
spec:
  replicas: {{.Params.replicas}}
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      app: gradual-rollout
  strategy:
    canary:
      canaryService: gradual-rollout-preview
      steps:
      - setWeight: 10
      - pause: {} # Block indefinitely waiting for manual promotion
      - setWeight: 25
      - pause:
          duration: 30 # seconds
      - setWeight: 50
      - pause:
          duration: 30
  template:
    metadata:
      labels:
        app: gradual-rollout
    spec:
      containers:
      - image: "{{.Params.image}}"
        imagePullPolicy: Always
        name: gradual-rollout
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        resources:
          requests:
            cpu: 5m
            memory: 32Mi

apiVersion: v1
kind: Service
metadata:
  name: gradual-rollout
spec:
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: gradual-rollout
---
apiVersion: v1
kind: Service
metadata:
  name: gradual-rollout-preview
spec:
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: gradual-rollout

service:
  name: gradual-rollout
  application: external-configs
  externalConfig:
    type: KUBERNETES
    local:
      path: .

  parameters:
    - name: image
      dockerImage:
        defaultTag: blue
        imageRegistryInfo:
          containerRegistry: dockerhub-prodvana
          imageRepository: argoproj/rollouts-demo
    - name: replicas
      description: replica count
      int:
        defaultValue: '3'

Prodvana will now display rollout progress in the release channel convergence view and provide Promote buttons to promote pauses. Note that all pauses can be manually promoted, regardless of whether they are configured to advance after a certain amount of time or not automatically.

You can also promote the Rollout object as you would without Prodvana via the Kubectl CLI. For more information, please see Argo Rollout documentation.

Initial deployment

Deploying a Service With Automated Canary Analysis

A common use of progressive delivery is to deploy a service while maintaining an invariant throughout the deployment process. For example, if the service starts showing an unacceptably high errors, abort the deployment and safely roll back to the previous state.

The following example describes deploying a service while ensuring that a Prometheus query is within spec.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: canary-demo
spec:
  replicas: {{.Params.replicas}}
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      app: canary-demo
  strategy:
    canary:
      analysis:
        templates:
        - templateName: success-rate
      canaryService: canary-demo-preview
      steps:
      - setWeight: 40
      - pause:
          duration: 30
      - setWeight: 60
      - pause:
          duration: 10
      - setWeight: 80
      - pause:
          duration: 10
  template:
    metadata:
      labels:
        app: canary-demo
    spec:
      containers:
      - image: "{{.Params.image}}"
        imagePullPolicy: Always
        name: canary-demo
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        resources:
          requests:
            cpu: 5m
            memory: 32Mi

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  metrics:
  - name: success-rate
    interval: 10s
    successCondition: result[0] >= 0
    provider:
      # To use other metrics providers like datadog or influxdb, use the appropriate
      # section from https://argo-rollouts.readthedocs.io/en/latest/features/analysis/
      prometheus:
        address: http://prom-prometheus-server.default.svc.cluster.local
        # Replace query below with desired metric to query
        query: >+
          sum(rate(apiserver_request_total{group="coordination.k8s.io", verb="PUT"}[1h]))

apiVersion: v1
kind: Service
metadata:
  name: canary-demo
spec:
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: canary-demo
---
apiVersion: v1
kind: Service
metadata:
  name: canary-demo-preview
spec:
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: canary-demo 

service:
  name: rollout
  application: external-configs
  externalConfig:
    type: KUBERNETES
    local:
      path: .

  parameters:
    - name: image
      dockerImage:
        defaultTag: blue
        imageRegistryInfo:
          containerRegistry: dockerhub-prodvana
          imageRepository: argoproj/rollouts-demo
    - name: replicas
      description: replica count
      int:
        defaultValue: '3'

Argo Rollouts supports polling other metrics providers and also custom plugins to supply metrics.

(Coming Soon) Deploying a Service With The bluegreen Strategy

Caveats

  • Prodvana only supportscanary deployment strategy currently
  • Only setWeight and pause steps are fully supported. Using other steps may result in broken UI (but expected to be otherwise functional).
  • No support for aborting a rollout currently.

Updated 3 months ago