Motivation

Breaking this work up into multiple parts to hopefully make this easier to review.

• Part 1: Mobilecoind python tests (should match what's in master now) Docker and helper scripts.
• Part 2: Helm charts
• Part 3: Github Actions and test wrappers.

The main goal for refactoring the release workflow was to enable block_version 0 -> block_version 1 testing.

1. Build rust and go binaries.
2. Build/Publish Docker images.
3. Build/Publish Helm Charts
4. Deploy "previous" release (v1.1.3)
5. Run integration tests against current release.
6. Upgrade to current release block_version=0.
7. Run integrations tests.
8. Upgrade consensus to block_version=1
9. Run integration tests.

CI/CD "improvements"

• Updated OS and utilities for build and runtime.
• Slimmed down build and runtime images.
• Using new build image will provide a static, versioned, verifiable and repeatable build process.
• pre-install of rust/cargo targets.
• Versioned helm charts published to S3 repo for each release tied to the versioned docker images.
• More consistent runtime environment setup through helm configuration sub-charts.
• GitHub Actions with private runners.
• Dynamically built dev environments for feature/* branches.
• Ability to retry failed steps.
• Skip build/ci with head commit messages.
• Build -dev release on release/*
• reduce chance of "shared" secret leaks, generate unique dev env secrets and keys.
• Manual ad-hoc dev environment actions: deploy, reset, delete, test

In this PR

Add Github actions workflows to do block version upgrade testing.

.github/workflows

Naming

Since we have multiple projects in the monorepo, I've done a simple namespacing of the workflows to try and keep things organized. Names follow this pattern.

• mobilecoin - these are workflows for testing core mobilecoin funtions.
• dev - these workflows interact with CD for the dynamic "dev" environments used for integration testing.
• workflow|dispatch - type of workflow.
• <description>

The workflows here take advantage of workflow_call (reusable workflows) and workflow_dispatch (manual workflows) to keep things DRYish and allow us to run steps manually, like reset and redeploy to a dev namespace. In general the main workflows triggered by pushes call the "dispatch" workflows that and they use the "workflow" events.

Base workflows triggered by github events.

• .github/workflows/mobilecoin-dev-cd.yaml - Main workflow triggered on pushes, drives build and publishing of artifacts.
• .github/workflows/mobilecoin-dev-delete.yaml - Workflow run on branch delete events to clean up old dev environments.

Dispatch (manual) events.

• .github/workflows/mobilecoin-dispatch-dev-delete.yaml - Delete a dev environment (cluster namespace)
• .github/workflows/mobilecoin-dispatch-dev-deploy.yaml - Deploy a dev environment (apps to the cluster namespace)
• .github/workflows/mobilecoin-dispatch-dev-reset.yaml - Reset an existing dev environment (delete all data in a namespace)
• .github/workflows/mobilecoin-dispatch-dev-test.yaml - Run Integration tests (fog-distribution, fog-test-client, mobilecoind python)
• .github/workflows/mobilecoin-dispatch-dev-update-consensus.yaml - Update consensus config and restart pods.

Reusable Workflows.

• .github/workflows/mobilecoin-workflow-dev-delete.yaml - Delete a dev environment (cluster namespace)
• .github/workflows/mobilecoin-workflow-dev-deploy.yaml - Deploy a dev environment (apps to the cluster namespace)
• .github/workflows/mobilecoin-workflow-dev-reset.yaml - Reset an existing dev environment (delete all data in a namespace)
• .github/workflows/mobilecoin-workflow-dev-test.yaml - Run Integration tests (fog-distribution, fog-test-client, mobilecoind python)
• .github/workflows/mobilecoin-workflow-dev-update-block-version.yaml - Update consensus config and restart pods.

Build image

The build is using a dockerhub based image that includes rust and all the usual targets pre-installed. We reference this with the sha256 identifier so we can guarantee the image and build environment is consistent. I worked to create a minimal image needed to build our apps, but I'm sure we will need more tools as we integrate other CI actions.

This image is automatically built and published when we update https://github.com/mobilecoinofficial/docker-rust-sgx-base

Caching

This build takes advantage of cargo and binary caching for rebuilds. It generates the cache keys based on hashes of .proto, Cargo., .rs or .go files as appropriate. Cache should be shared with the default branch.

Custom Actions

Most of the Kubernetes interactions are powered by a custom action here: https://github.com/mobilecoinofficial/gha-k8s-toolbox

Artifacts

• Docker images will be published to the mobilecoin org.
• Helm charts will be published to the S3 repository.
• Binaries and measurements will be published as "artifacts" attached to the specific build.
• Environment access information can be found in the 👾 Environment Info 👾 job/step in the main CD workflow.

Caveat :)

I think I have all the required repo config set up, but there's a pretty good chance this will need a follow up or two to make everything functional. It was tested and working on the fork, but its really hard to test the CI/CD system without committing code.

Future Work

• manual dispatch workflow to build artifacts for TestNet and MainNet deployments.
• refactor of entrypoint scripts to align internal deployment configuration with partner deployments.
• generate fog-report signing keys, instead of using shared key.