diff --git a/docs/advanced-usage/persistent-storage.md b/docs/advanced-usage/persistent-storage.md index 8cfd9e3173a..96b5f0f4af9 100644 --- a/docs/advanced-usage/persistent-storage.md +++ b/docs/advanced-usage/persistent-storage.md @@ -5,10 +5,11 @@ The preferred method to mount external containers to a Dokku managed container, is to use the Dokku storage plugin. ``` -storage:list # List bind mounts for app's container(s) (host:container) -storage:mount # Create a new bind mount -storage:report [] [] # Displays a checks report for one or more apps -storage:unmount # Remove an existing bind mount +storage:ensure-directory [--chown option] # Creates a persistent storage directory in the recommended storage path +storage:list # List bind mounts for app's container(s) (host:container) +storage:mount # Create a new bind mount +storage:report [] [] # Displays a checks report for one or more apps +storage:unmount # Remove an existing bind mount ``` > The storage plugin is compatible with storage mounts created with the docker-options. The storage plugin will only list mounts from the deploy/run phase. @@ -20,50 +21,77 @@ The storage plugin supports the following mount points: ## Usage -This example demonstrates how to mount the recommended directory to `/storage` inside an application called `node-js-app`. For simplicity, the Dokku project recommends using the directory `/var/lib/dokku/data/storage` directory as the root host path for mounts. This directory is created on Dokku installation. +### Creating storage directories + +> New as of 0.25.5 + +A storage directory can be created with the `storage:ensure-directory` command. This command will create a subdirectory in the recommended `/var/lib/doku/data/storage` path - created during Dokku installation - and prepare it for use with an app. ```shell -# we use a subdirectory inside of the host directory to scope it to just the app -dokku storage:mount node-js-app /var/lib/dokku/data/storage/node-js-app:/storage +dokku storage:ensure-directory lollipop +``` + +``` +-----> Ensuring /var/lib/dokku/data/storage/lollipop exists + Setting directory ownership to 32767:32767 + Directory ready for mounting ``` -Dokku will then mount the shared contents of `/var/lib/dokku/data/storage/node-js-app` to `/storage` inside the container. Mounts are only available for containers crated via `run` and by the deploy process, and not during the build process. In addition, the host path is never auto-created by either Dokku or Docker, and should be an explicit path, not one relative to the current working directory. +By default, permissions are set for usage with Herokuish buildpacks. These permissions can be changed via the `--chown` option according to the following table: -> If the `/storage` path within the container had pre-existing content, the container files will be overrwritten. This may be an issue for users that create assets at build time but then mount a directory at the same place during runtime. Files are not merged. +- `--chown herokuish` (default): Use `32767:32767` as the folder permissions. + - This is used for apps deployed with Buildpacks via Herokuish. +- `--chown heroku`: Use `1000:1000` as the folder permissions. + - This is used for apps deployed with Cloud Native Buildpacks using the `heroku/buildpacks` builder. +- `--chown packeto`: Use `2000:2000` as the folder permissions. + - This is used for apps deployed with Cloud Native Buildpacks using the `cloudfoundry/cnb` or `packeto` builders. +- `--chown false`: Skips the `chown` call. -Once you have mounted persistent storage, you will also need to restart the application. See the -[process scaling documentation](/docs/processes/process-management.md) for more information. +Users deploying via Dockerfile will want to specify `--chown false` and manually `chown` the created directory if the user and/or group id of the runnning process in the deployed container do not correspond to any of the above options. + +> Warning: Failing to set the correct directory ownership may result in issues in persisting files written to the mounted storage directory. + +### Mounting storage into apps + +Dokku supports mounting both explicit host paths as well as docker volumes via the `storage:mount` command. This takes two arguments, an app name and a `host-path:container-path` or `docker-volume:container-path` combination. ```shell -dokku ps:restart app-name +# mount the directory into your container's /app/storage directory, relative to the container root (/) +# explicit host paths _must_ exist prior to usage. +dokku storage:mount node-js-app /var/lib/dokku/data/storage/node-js-app:/app/storage + +# mount the docker volume into your container's /app/storage directory, relative to the container root (/) +# docker volumes _must_ exist prior to usage. +dokku storage:mount node-js-app some-docker-volume:/app/storage ``` -A more complete workflow may require making a custom directory for your application and mounting it within your `/app/storage` directory instead. The mount point is *not* relative to your application's working directory, and is instead relative to the root (`/`) of the container. +In the first example, Dokku will then mount the shared contents of `/var/lib/dokku/data/storage/node-js-app` to `/app/storage` inside the container. The mount point is *not* relative to your app's working directory, and is instead relative to the root (`/`) of the container. Mounts are only available for containers created via `run` and by the deploy process, and not during the build process. In addition, the host path is never auto-created by either Dokku or Docker, and should be an explicit path, not one relative to the current working directory. -```shell -# creating storage for the app 'node-js-app' -mkdir -p /var/lib/dokku/data/storage/node-js-app +> If the `/storage` path within the container had pre-existing content, the container files will be over-written. This may be an issue for users that create assets at build time but then mount a directory at the same place during runtime. Files are not merged. -# set the directory ownership. Below is an example for herokuish -# but see the `Directory Permissions` section for more details -chown -R 32767:32767 /var/lib/dokku/data/storage/node-js-app +Once persistent storage is mounted, the app requires a restart. See the [process scaling documentation](/docs/processes/process-management.md) for more information. -# mount the directory into your container's /app/storage directory, relative to root -dokku storage:mount app-name /var/lib/dokku/data/storage/node-js-app:/app/storage +```shell +dokku ps:restart app-name ``` -You can mount one or more directories as desired by following the above pattern. +### Unmounting storage + +If an app no longer requires a mounted volume or directory, the `storage:unmount` command can be called. This takes the same arguments as the `storage:mount` command, an app name and a `host-path:container-path` or `docker-volume:container-path` combination. + +```shell +# unmount the directory from your container's /app/storage directory, relative to the container root (/) +dokku storage:unmount node-js-app /var/lib/dokku/data/storage/node-js-app:/app/storage -### Directory Permissions +# unmount the docker volume from your container's /app/storage directory, relative to the container root (/) +dokku storage:unmount node-js-app some-docker-volume:/app/storage +``` -The host directory should always be owned by the container user and group id. If this is not the case, files may not persist when written to mounted storage. +Once persistent storage is unmounted, the app requires a restart. See the [process scaling documentation](/docs/processes/process-management.md) for more information. -- Buildpacks via Herokuish: Use `32767:32767` as the file permissions -- For Cloud Native Buildpacks: This will depend on the builder in question, but can be retrieved via the `CNB_USER_ID` and `CNB_GROUP_ID` environment variables on the builder image. Common builders are as follows: - - heroku/buildpacks: `1000:1000` - - cloudfoundry/cnb: `2000:2000` - - packeto: `2000:2000` -- Dockerfile and Docker Image: Use the user and group id which corresponds to the one running the process within the container. +```shell +dokku ps:restart app-name +``` ### Displaying storage reports for an app @@ -113,7 +141,7 @@ dokku storage:report node-js-app --storage-deploy-mounts ### Sharing storage across deploys -Dokku is powered by Docker containers, which recommends in their [best practices](https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#containers-should-be-ephemeral) that containers be treated as ephemeral. In order to manage persistent storage for web applications, like user uploads or large binary assets like images, a directory outside the container should be mounted. +Dokku is powered by Docker containers, which recommends in their [best practices](https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#containers-should-be-ephemeral) that containers be treated as ephemeral. In order to manage persistent storage for web apps, like user uploads or large binary assets like images, a directory outside the container should be mounted. ### Shared storage between containers @@ -139,11 +167,11 @@ You cannot use mounted volumes during the build phase of a Dockerfile deploy. Th > Note: **This can cause data loss** if you bind a mount under `/app` in buildpack apps as herokuish will attempt to remove the original app path during the build phase. -## Application User and Persistent Storage file ownership (buildpack apps only) +## App User and Persistent Storage file ownership (buildpack apps only) > New as of 0.7.1 -By default, Dokku will execute your buildpack application processes as the `herokuishuser` user. You may override this by setting the `DOKKU_APP_USER` config variable. +By default, Dokku will execute your buildpack app processes as the `herokuishuser` user. You may override this by setting the `DOKKU_APP_USER` config variable. > NOTE: this user must exist in your herokuish image. diff --git a/plugins/storage/bin/chown-storage-dir b/plugins/storage/bin/chown-storage-dir new file mode 100755 index 00000000000..abb6ba0328c --- /dev/null +++ b/plugins/storage/bin/chown-storage-dir @@ -0,0 +1,27 @@ +#!/usr/bin/env bash +set -eo pipefail +[[ $DOKKU_TRACE ]] && set -x + +main() { + declare desc="chowns a storage directory" + declare DIRECTORY="$1" CHOWN_ID="$2" + + if [[ -z "$DIRECTORY" ]]; then + echo " ! Please specify a directory to create" 1>&2 + exit 1 + fi + + if [[ ! "$DIRECTORY" =~ ^[A-Za-z0-9\\_-]+$ ]]; then + echo " ! Directory can only contain the following set of characters: [A-Za-z0-9_-]" 1>&2 + exit 1 + fi + + if [[ "$CHOWN_ID" != "32767" ]] && [[ "$CHOWN_ID" != "1000" ]] && [[ "$CHOWN_ID" != "2000" ]]; then + echo " ! Unsupported chown permissions. Supported values: 32767, 1000, 2000" + exit 1 + fi + + chown -R "$CHOWN_ID:$CHOWN_ID" "${DOKKU_LIB_ROOT}/data/storage/$DIRECTORY" +} + +main "$@" diff --git a/plugins/storage/help-functions b/plugins/storage/help-functions index ac324d47ba2..dacb089a1c9 100755 --- a/plugins/storage/help-functions +++ b/plugins/storage/help-functions @@ -27,6 +27,7 @@ help_desc fn-help-content() { declare desc="return help content" cat <, Creates a persistent storage directory in the recommended storage path storage:list , List bind mounts for app's container(s) (host:container) storage:mount , Create a new bind mount storage:report [] [], Displays a checks report for one or more apps diff --git a/plugins/storage/install b/plugins/storage/install index 9c9ce83bdab..28629b92582 100755 --- a/plugins/storage/install +++ b/plugins/storage/install @@ -5,9 +5,40 @@ set -eo pipefail trigger-storage-install() { declare desc="storage install trigger" declare trigger="install" + local storage_directory="${DOKKU_LIB_ROOT}/data/storage" + mkdir -p "${storage_directory}" + chown dokku:dokku "${storage_directory}" - mkdir -p "${DOKKU_LIB_ROOT}/data/storage" - chown dokku:dokku "${DOKKU_LIB_ROOT}/data/storage" + STORAGE_SUDOERS_FILE="/etc/sudoers.d/dokku-storage" + + case "$DOKKU_DISTRO" in + debian) + echo "%dokku ALL=(ALL) NOPASSWD:$PLUGIN_AVAILABLE_PATH/storage/bin/chown-storage-dir *" >"$STORAGE_SUDOERS_FILE" + echo "Defaults env_keep += \"DOKKU_LIB_ROOT\"" >>"$STORAGE_SUDOERS_FILE" + ;; + + ubuntu) + echo "%dokku ALL=(ALL) NOPASSWD:$PLUGIN_AVAILABLE_PATH/storage/bin/chown-storage-dir *" >"$STORAGE_SUDOERS_FILE" + echo "Defaults env_keep += \"DOKKU_LIB_ROOT\"" >>"$STORAGE_SUDOERS_FILE" + ;; + + opensuse) + echo "%dokku ALL=(ALL) NOPASSWD:$PLUGIN_AVAILABLE_PATH/storage/bin/chown-storage-dir *" >"$STORAGE_SUDOERS_FILE" + echo "Defaults env_keep += \"DOKKU_LIB_ROOT\"" >>"$STORAGE_SUDOERS_FILE" + ;; + + arch) + echo "%dokku ALL=(ALL) NOPASSWD:$PLUGIN_AVAILABLE_PATH/storage/bin/chown-storage-dir *" >"$STORAGE_SUDOERS_FILE" + echo "Defaults env_keep += \"DOKKU_LIB_ROOT\"" >>"$STORAGE_SUDOERS_FILE" + ;; + + centos | fedora | rhel) + echo "%dokku ALL=(ALL) NOPASSWD:$PLUGIN_AVAILABLE_PATH/storage/bin/chown-storage-dir *" >"$STORAGE_SUDOERS_FILE" + echo "Defaults:dokku !requiretty" >>"$STORAGE_SUDOERS_FILE" + echo "Defaults env_keep += \"DOKKU_LIB_ROOT\"" >>"$STORAGE_SUDOERS_FILE" + + ;; + esac } trigger-storage-install "$@" diff --git a/plugins/storage/internal-functions b/plugins/storage/internal-functions index d69566208ae..70be5e16fe9 100755 --- a/plugins/storage/internal-functions +++ b/plugins/storage/internal-functions @@ -5,6 +5,62 @@ source "$PLUGIN_AVAILABLE_PATH/storage/functions" set -eo pipefail [[ $DOKKU_TRACE ]] && set -x +cmd-storage-ensure-directory() { + declare desc="creates a persistent storage directory in the recommended storage path" + declare cmd="storage:ensure-directory" + [[ "$1" == "$cmd" ]] && shift 1 + declare DIRECTORY CHOWN_FLAG + + CHOWN_FLAG=herokuish + skip=false + for arg in "$@"; do + if [[ "$arg" == "--chown" ]]; then + skip=true + continue + fi + + if [[ "$skip" == "true" ]]; then + CHOWN_FLAG="$arg" + skip=false + continue + fi + + if [[ -z "$DIRECTORY" ]]; then + DIRECTORY="$arg" + fi + done + + if [[ -z "$DIRECTORY" ]]; then + dokku_log_fail "Please specify a directory to create" + fi + + if [[ ! "$DIRECTORY" =~ ^[A-Za-z0-9\\_-]+$ ]]; then + dokku_log_fail "Directory can only contain the following set of characters: [A-Za-z0-9_-]" + fi + + if [[ "$CHOWN_FLAG" == "herokuish" ]]; then + CHOWN_FLAG="32767" + elif [[ "$CHOWN_FLAG" == "heroku" ]]; then + CHOWN_FLAG="1000" + elif [[ "$CHOWN_FLAG" == "packeto" ]]; then + CHOWN_FLAG="2000" + elif [[ "$CHOWN_FLAG" == "false" ]]; then + CHOWN_FLAG="false" + else + dokku_log_fail "Unsupported chown permissions" + fi + + local storage_directory="${DOKKU_LIB_ROOT}/data/storage/$DIRECTORY" + dokku_log_info1 "Ensuring ${storage_directory} exists" + mkdir -p "${storage_directory}" + if [[ "$CHOWN_FLAG" != "false" ]]; then + dokku_log_verbose_quiet "Setting directory ownership to $CHOWN_FLAG:$CHOWN_FLAG" + sudo "$PLUGIN_AVAILABLE_PATH/storage/bin/chown-storage-dir" "$DIRECTORY" "$CHOWN_FLAG" + fi + + dokku_log_verbose_quiet "Directory ready for mounting" +} + cmd-storage-report() { declare desc="displays a storage report for one or more apps" declare cmd="storage:report" diff --git a/plugins/storage/subcommands/ensure-directory b/plugins/storage/subcommands/ensure-directory new file mode 100755 index 00000000000..bbce7325d61 --- /dev/null +++ b/plugins/storage/subcommands/ensure-directory @@ -0,0 +1,6 @@ +#!/usr/bin/env bash +set -eo pipefail +[[ $DOKKU_TRACE ]] && set -x +source "$PLUGIN_AVAILABLE_PATH/storage/internal-functions" + +cmd-storage-ensure-directory "$@" diff --git a/tests/unit/storage.bats b/tests/unit/storage.bats index 8a20f0be8f7..87ae535821f 100644 --- a/tests/unit/storage.bats +++ b/tests/unit/storage.bats @@ -5,6 +5,7 @@ load test_helper setup() { global_setup create_app + rm -rf "$DOKKU_LIB_ROOT/data/storage/rdmtestapp*" } teardown() { @@ -26,6 +27,86 @@ teardown() { assert_output "$help_output" } +@test "(storage) storage:ensure-directory" { + run /bin/bash -c "test -d $DOKKU_LIB_ROOT/data/storage/$TEST_APP" + echo "output: $output" + echo "status: $status" + assert_failure + + run /bin/bash -c "dokku storage:ensure-directory @" + echo "output: $output" + echo "status: $status" + assert_failure + + run /bin/bash -c "dokku storage:ensure-directory $TEST_APP/" + echo "output: $output" + echo "status: $status" + assert_failure + + run /bin/bash -c "dokku storage:ensure-directory $TEST_APP/$TEST_APP" + echo "output: $output" + echo "status: $status" + assert_failure + + run /bin/bash -c "dokku storage:ensure-directory $TEST_APP" + echo "output: $output" + echo "status: $status" + assert_success + assert_output_contains "Setting directory ownership to 1000:1000" 0 + assert_output_contains "Setting directory ownership to 2000:2000" 0 + assert_output_contains "Setting directory ownership to 32767:32767" 1 + + run /bin/bash -c "test -d $DOKKU_LIB_ROOT/data/storage/$TEST_APP" + echo "output: $output" + echo "status: $status" + assert_success + + run /bin/bash -c "dokku storage:ensure-directory $TEST_APP" + echo "output: $output" + echo "status: $status" + assert_success + + run /bin/bash -c "dokku storage:ensure-directory --chown false $TEST_APP" + echo "output: $output" + echo "status: $status" + assert_success + assert_output_contains "Setting directory ownership to 1000:1000" 0 + assert_output_contains "Setting directory ownership to 2000:2000" 0 + assert_output_contains "Setting directory ownership to 32767:32767" 0 + + run /bin/bash -c "dokku storage:ensure-directory $TEST_APP --chown false" + echo "output: $output" + echo "status: $status" + assert_success + assert_output_contains "Setting directory ownership to 1000:1000" 0 + assert_output_contains "Setting directory ownership to 2000:2000" 0 + assert_output_contains "Setting directory ownership to 32767:32767" 0 + + run /bin/bash -c "dokku storage:ensure-directory --chown heroku $TEST_APP" + echo "output: $output" + echo "status: $status" + assert_success + assert_output_contains "Setting directory ownership to 1000:1000" 1 + assert_output_contains "Setting directory ownership to 2000:2000" 0 + assert_output_contains "Setting directory ownership to 32767:32767" 0 + + run /bin/bash -c "dokku storage:ensure-directory --chown packeto $TEST_APP" + echo "output: $output" + echo "status: $status" + assert_success + assert_output_contains "Setting directory ownership to 1000:1000" 0 + assert_output_contains "Setting directory ownership to 2000:2000" 1 + assert_output_contains "Setting directory ownership to 32767:32767" 0 + + run /bin/bash -c "dokku storage:ensure-directory --chown herokuish $TEST_APP" + echo "output: $output" + echo "status: $status" + assert_success + assert_output_contains "Setting directory ownership to 1000:1000" 0 + assert_output_contains "Setting directory ownership to 2000:2000" 0 + assert_output_contains "Setting directory ownership to 32767:32767" 1 +} + @test "(storage) storage:mount, storage:list, storage:umount" { run /bin/bash -c "dokku storage:mount $TEST_APP /tmp/mount:/mount" echo "output: $output"