Development Env and Release CI/CD - Part 2 #1829

jgreat · 2022-04-20T16:56:55Z

Motivation

Breaking this work up into multiple parts to hopefully make this easier to review.

Part 1: Mobilecoind python tests (should match what's in master now) Docker and helper scripts.
Part 2: Helm charts
Part 3: Github Actions and test wrappers.

The main goal for refactoring the release workflow was to enable block_version 0 -> block_version 1 testing.

Build rust and go binaries.
Build/Publish Docker images.
Build/Publish Helm Charts
Deploy "previous" release (v1.1.3)
Run integration tests against current release.
Upgrade to current release block_version=0.
Run integrations tests.
Upgrade consensus to block_version=1
Run integration tests.

CI/CD "improvements"

Updated OS and utilities for build and runtime.
Slimmed down build and runtime images.
Using new build image will provide a static, versioned, verifiable and repeatable build process.
pre-install of rust/cargo targets.
Versioned helm charts published to S3 repo for each release tied to the versioned docker images.
More consistent runtime environment setup through helm configuration sub-charts.
GitHub Actions with private runners.
Dynamically built dev environments for feature/* branches.
Ability to retry failed steps.
Skip build/ci with head commit messages.
Build -dev release on release/*
reduce chance of "shared" secret leaks, generate unique dev env secrets and keys.
Manual ad-hoc dev environment actions: deploy, reset, delete, test

In this PR

Refactor of helm charts used for consensus and fog deployments.

.internal-ci/helm

One of the goals was to abstract shared and local configuration values from the runtime of the charts and do "Zero Config" deployments and upgrades of the applications. My intention is to allow environment specific (config and secrets for TestNet vs MainNet...) configuration to be defined and set externally from the application deployments.

Assuming configuration is relatively static, this should minimize the human error factor when we deploy Consensus and Fog services. Of course the real world may have other ideas 🤷

To achieve this, the helm charts are now split into "config" charts and application charts. Although most of the time we would run config separately from the app charts, the app charts set the appropriate config charts as optional dependencies for a more convenient install outside our systems.

A secondary goal was having a repository of static versioned charts (configuration and deployment instructions) that is tested with and directly correlates with a specific build. This should eliminate the pain point of build/deployment config drift from the time we deployed the core apps. The trade off is that updates to the deployment and or tooling now have to follow the application lifecycle, i.e. at least a point patch and follow up to future release and edge branches.

Charts and descriptions

mc-core-dev-env-setup - Chart that takes all the various config charts and bundles them into a convenient single step deployment for our dev environments.
mc-core-common-config - common config elements needed for any of the core apps. client auth, ias, network for monitoring, mobilecoind configs.
consensus-node[-config] - Deploy a single consensus node.
fog-ingest[-config] - Deploy a blue or green fog-ingest.
fog-services[-config] - Deploy fog services (report, view, ledger).
mobilecoind - Deploy a standalone mobilecoind with api endpoint enabled.
watcher - Deploy a copy of the AVR Watcher service.
fog-test-client - Deploy fog-test-client in canary mode.

Future Work

manual dispatch workflow to build artifacts for TestNet and MainNet deployments.
refactor of entrypoint scripts to align internal deployment configuration with partner deployments.
generate fog-report signing keys, instead of using shared key.

.internal-ci/helm/fog-ingest-config/templates/fog-recovery-postgresql-configmap.yaml

joekottke

Having talked with @jgreat often through the process of this, as well as going through the walk-through, I'm giving an LGTM. Will continue to consume the chart code, but didn't want to hold up the review process,

MCrank

LGTM - This was a ton of work and is somewhat of a Modern Marvel

wjuan-mob · 2022-04-22T16:47:44Z

.internal-ci/helm/fog-ingest/README.md

+
+`fog-ingest` is only designed to have one active instance. We should run at-least 2 in order to have a hot standby incase the active instance fails. Scaling the replicas doesn't improve performance.
+
+The peer list generation happens when the chart is generated.  In order to scale the fog-ingest service you should adjust the `fogIngest.replicaCount` value and upgrade the fogIngest.  The peer list is added to the ConfigMap additional pods will be added, but existing pods will not automatically update.  Either destroy and re-create the pods or execute a restart of the fog services with supervisord.


Suggested change

The peer list generation happens when the chart is generated. In order to scale the fog-ingest service you should adjust the `fogIngest.replicaCount` value and upgrade the fogIngest. The peer list is added to the ConfigMap additional pods will be added, but existing pods will not automatically update. Either destroy and re-create the pods or execute a restart of the fog services with supervisord.

The peer list generation happens when the chart is generated. In order to scale the fog-ingest service you should adjust the `fogIngest.replicaCount` value and upgrade the fogIngest. When the peer list is added to the ConfigMap additional pods will be added, but existing pods will not automatically update. Either destroy and re-create the pods or execute a restart of the fog services with supervisord.

wjuan-mob · 2022-04-22T16:48:28Z

.internal-ci/helm/fog-ingest/README.md

+
+- `supervisord-mobilecoind`
+
+    `mobilecoind` configuration for in container supervisord.  Example values are for MobileCoin MainNet.


Suggested change

`mobilecoind` configuration for in container supervisord. Example values are for MobileCoin MainNet.

`mobilecoind` configuration in container supervisord. Example values are for MobileCoin MainNet.

wjuan-mob · 2022-04-22T16:52:24Z

.internal-ci/helm/fog-ingest/README.md

+
+- `fog-ingest` ConfigMap
+
+    Database connection configuration for fog-ingest


I know you noted this in the file itself, but is it worth also noting that: "For helm deployed postgres, set configMap.enabled and secret.enabled true" here?

wjuan-mob · 2022-04-22T17:08:28Z

.internal-ci/helm/fog-services-config/templates/_helpers.tpl

+{{- $salt }}
+{{- end }}
+
+{{/* fogViewHTTPCookieSalt - reuse existing password */}}


Could you add a bit more comment on how these salt functions are supposed to work?

wjuan-mob · 2022-04-22T21:27:38Z

.internal-ci/helm/fog-services/templates/supervisord-fog-view-configmap.yaml

+      --client-responder-id "%(ENV_CLIENT_RESPONDER_ID)s"
+{{- if (include "fogServices.clientAuth" .) }}
+      --client-auth-token-secret "%(ENV_CLIENT_AUTH_TOKEN_SECRET)s"
+      --client-auth-token-max-lifetime 31536000


Is this number arbitrary?

jgreat added 9 commits April 13, 2022 09:26

update mobilecoind test_client requirements and add log levels

4c438ff

update ignore files for new CD env

05e85c4

add empty sample_data for docker build

e9a66e8

internal-ci - docker files, utility and test scripts.

cd8de57

update workflow in README, util update scripts to latest version

c19ea87

add/enhance script descriptions

4eb45bd

remove workflow description

132282c

refreshed helm charts

a53b801

Merge branch upstream 'release-1.2.0' into release/v1.2.0

41cccb2

jgreat requested review from a team, MCrank and joekottke April 20, 2022 16:56

wjuan-mob self-requested a review April 21, 2022 17:47

jgreat commented Apr 21, 2022

View reviewed changes

.internal-ci/helm/fog-ingest-config/templates/fog-recovery-postgresql-configmap.yaml Show resolved Hide resolved

joekottke approved these changes Apr 22, 2022

View reviewed changes

MCrank approved these changes Apr 22, 2022

View reviewed changes

add notes about duplicate settings

cb3304d

jgreat merged commit d25894a into mobilecoinfoundation:release-1.2.0 Apr 22, 2022

wjuan-mob reviewed Apr 22, 2022

View reviewed changes

jgreat mentioned this pull request May 25, 2022

CD for dynamic development environments #2054

Merged

jgreat mentioned this pull request Aug 19, 2022

[Added] CI/CD - Add GHA CD to master - part 1 #2414

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Development Env and Release CI/CD - Part 2 #1829

Development Env and Release CI/CD - Part 2 #1829

Uh oh!

jgreat commented Apr 20, 2022

Uh oh!

Uh oh!

joekottke left a comment

Uh oh!

MCrank left a comment

Uh oh!

wjuan-mob Apr 22, 2022

Uh oh!

wjuan-mob Apr 22, 2022

Uh oh!

wjuan-mob Apr 22, 2022

Uh oh!

wjuan-mob Apr 22, 2022

Uh oh!

wjuan-mob Apr 22, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants


		`fog-ingest` is only designed to have one active instance. We should run at-least 2 in order to have a hot standby incase the active instance fails. Scaling the replicas doesn't improve performance.

		The peer list generation happens when the chart is generated. In order to scale the fog-ingest service you should adjust the `fogIngest.replicaCount` value and upgrade the fogIngest. The peer list is added to the ConfigMap additional pods will be added, but existing pods will not automatically update. Either destroy and re-create the pods or execute a restart of the fog services with supervisord.


		- `supervisord-mobilecoind`

		`mobilecoind` configuration for in container supervisord. Example values are for MobileCoin MainNet.

	`mobilecoind` configuration for in container supervisord. Example values are for MobileCoin MainNet.
	`mobilecoind` configuration in container supervisord. Example values are for MobileCoin MainNet.


		- `fog-ingest` ConfigMap

		Database connection configuration for fog-ingest

Development Env and Release CI/CD - Part 2 #1829

Development Env and Release CI/CD - Part 2 #1829

Uh oh!

Conversation

jgreat commented Apr 20, 2022

Motivation

In this PR

.internal-ci/helm

Charts and descriptions

Future Work

Uh oh!

Uh oh!

joekottke left a comment

Choose a reason for hiding this comment

Uh oh!

MCrank left a comment

Choose a reason for hiding this comment

Uh oh!

wjuan-mob Apr 22, 2022

Choose a reason for hiding this comment

Uh oh!

wjuan-mob Apr 22, 2022

Choose a reason for hiding this comment

Uh oh!

wjuan-mob Apr 22, 2022

Choose a reason for hiding this comment

Uh oh!

wjuan-mob Apr 22, 2022

Choose a reason for hiding this comment

Uh oh!

wjuan-mob Apr 22, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants