Skip to content

[WIP] unleash import/export script and job definition #49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
maknop wants to merge 8 commits into
base: main
Choose a base branch
from
Draft

[WIP] unleash import/export script and job definition #49

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
bb3740b
adding script and job definition
maknop Feb 28, 2024
53d8ac2
refactoring script
maknop Mar 11, 2024
70961c6
refactored script
maknop Mar 14, 2024
e059fd4
deleted unused file
maknop Mar 14, 2024
d62ec4e
log cleanup
maknop Mar 14, 2024
4ea9233
fixing logic in export/import script
maknop Mar 14, 2024
e73e2a8
cleanup script
maknop Mar 21, 2024
a524d11
updates to job template
maknop Mar 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 97 additions & 0 deletions featureflags/unleash_config_export_import.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
#!/bin/bash

set -x
Comment thread
maknop marked this conversation as resolved.

echo "setting required environment variables"
EPHEMERAL_NAMESPACE="${EPHEMERAL_NAMESPACE:-}"
EXPORT_UNLEASH_URL="${EXPORT_UNLEASH_URL:-}"
EXPORT_NAMESPACE="${EXPORT_NAMESPACE:-}"
EXPORT_ADMIN_SECRET="${EXPORT_ADMIN_SECRET:-}"
IMPORT_PAT="${IMPORT_PAT:-}"
Copy link
Copy Markdown

@awood awood May 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than require this be specified, it might be worth it to issue a PAT via this script. I say this because for the end user, the most sensible way to get a PAT is via the web UI, but that's not very useful in a CI/CD context. Issuing a PAT requires the password for an admin user. Here is a script I wrote that issues a PAT, uses it for the bulk import, and then deletes it afterwards (only 10 PATs can exist for a user at once).

#!/usr/bin/env bash
# https://github.com/olivergondza/bash-strict-mode
set -eEuo pipefail
trap 's=$?; echo >&2 "$0: Error on line "$LINENO": $BASH_COMMAND"; exit $s' ERR
trap cleanup EXIT

# Note that Unleash will only issue up to a maximum of 10 tokens, so we'll clean up after ourselves
function cleanup() {
  # So we don't get a warning about an undefined variable if the trap runs before AUTH_ID is defined
  AUTH_ID=${AUTH_ID:-}
  if [ -n "${AUTH_ID}" ]; then
    curl -v -L -X DELETE "${UNLEASH_URL}/api/admin/user/tokens/${AUTH_ID}" \
      --header "Authorization: $AUTH_TOKEN"
  fi

  COOKIE_JAR=${COOKIE_JAR:-}
  if [ -n "${COOKIE_JAR}" ]; then
    rm "${COOKIE_JAR}"
  fi
}

# Here's how resolution works:
# * If UNLEASH_HOST is set in the environment, that's the value that's going to be used, period.
# * If not and the CLOWDER_CONFIG file has featureFlags.hostname, that's the value that will be used
# * If the CLOWDER_CONFIG file isn't readable or doesn't have featureFlags.hostname, and
#   UNLEASH_HOST isn't in the environment, default to localhost.

: "${CLOWDER_CONFIG:=/cdapp/cdappconfig.json}"
if [ -r "${CLOWDER_CONFIG}" ]; then
  # jq returns "null" instead of the empty string if something isn't there.  We want the empty
  # string so that later down we can still use parameter substitution to set a default value.  Use
  # the "//" operator and "empty" operator to effect this.
  # See https://github.com/jqlang/jq/issues/354#issuecomment-43147898
  : "${UNLEASH_SCHEME:=$(jq -r '.featureFlags.scheme // empty' "${CLOWDER_CONFIG}")}"
  : "${UNLEASH_HOST:=$(jq -r '.featureFlags.hostname // empty' "${CLOWDER_CONFIG}")}"
  : "${UNLEASH_PORT:=$(jq -r '.featureFlags.port // empty' "${CLOWDER_CONFIG}")}"
fi

: "${UNLEASH_USER:=admin}"
: "${UNLEASH_PASSWORD:=unleash4all}"
: "${UNLEASH_SCHEME:=http}"
: "${UNLEASH_HOST:=localhost}"
: "${UNLEASH_PORT:=4242}"
: "${UNLEASH_ADMIN_TOKEN:=}"

UNLEASH_URL="${UNLEASH_SCHEME}://${UNLEASH_HOST}:${UNLEASH_PORT}"

IMPORT_FILE=${1:-}
if [ -z "$IMPORT_FILE" ]; then
  echo "Usage: $(basename "$0") JSON_FILE"
  exit 1
fi

# Unleash offers multiple types of authentication tokens: admin tokens (deprecated), personal
# access tokens (PATs), and client tokens.  The tokens we care about are the first two.  The
# problem is that some APIs require PATs and some do not.  Notably the environment import endpoint
# /api/admin/features-batch/import (see
# https://docs.getunleash.io/reference/api/unleash/import-toggles and
# https://docs.getunleash.io/how-to/how-to-environment-import-export#import) *requires* a PAT.
#
# We are using an older, deprecated import API /api/admin/state/import (see
# https://docs.getunleash.io/reference/api/unleash/import and
# https://docs.getunleash.io/how-to/how-to-import-export) which accepts admin tokens.  We're
# using this API since it uses the same JSON format as the start-up import (see
# https://docs.getunleash.io/how-to/how-to-import-export#startup-import) that we use for the local
# container during development.
#
# The expected flow for using PATs seems to be that tokens are created via the web UI.  This won't
# work for us since we need the whole process to be automated.  PATs can be created via the
# /api/admin/user/tokens endpoint, but it requires a session cookie.  We can acquire one of
# those by logging in at /api/admin/user/tokens. The bad part is it requires a user name and
# password which are *not* provided via clowder!  Nevertheless, I'm adding the code to login and
# acquire a PAT for the future case when we need to move off of the deprecated import API.

# See https://stackoverflow.com/a/13864829
if [ -z "${UNLEASH_ADMIN_TOKEN}" ]; then
  COOKIE_JAR=$(mktemp -t cookie.XXXXX)

  curl -s -S -c "$COOKIE_JAR" -L "${UNLEASH_URL}/auth/simple/login" \
    --header "Content-Type: application/json" \
    --data @- > /dev/null <<EOF
    {
    "username": "$UNLEASH_USER",
    "password": "$UNLEASH_PASSWORD"
    }
EOF

  AUTH_JSON=$(curl -s -S -b "$COOKIE_JAR" -L "${UNLEASH_URL}/api/admin/user/tokens" \
    --header "Content-Type: application/json" \
    --data @- <<EOF
    {
    "description": "Admin PAT $(date +%s.%N)",
    "expiresAt": "$(date -d '+100 years' --utc +%Y-%m-%dT%H:%M:%S)Z"
    }
EOF
  )
  AUTH_TOKEN="$(echo $AUTH_JSON | jq -r '.secret')"
  AUTH_ID="$(echo $AUTH_JSON | jq -r '.id')"
else
  AUTH_TOKEN="${UNLEASH_ADMIN_TOKEN}"
fi

curl -L -X POST "${UNLEASH_URL}/api/admin/state/import" \
  --header "Content-Type: application/json" \
  --header "Authorization: ${AUTH_TOKEN}" \
  --data @"${IMPORT_FILE}"

IMPORT_LOCALLY="${IMPORT_LOCALLY:-false}"
JSON_TOGGLES_PATH="${JSON_TOGGLES_PATH:-featureflags/feature_flags_exported_toggles.json}"

echo "JSON_TOGGLES_PATH: $JSON_TOGGLES_PATH"
echo "EXPORT_UNLEASH_URL: $EXPORT_UNLEASH_URL"
echo "IMPORT_LOCALLY: $IMPORT_LOCALLY"

export_toggles() {
echo "Retrieve a list of toggle names to export from environment"
curl -L -X GET "${EXPORT_UNLEASH_URL}/api/admin/projects/default/features" \
-H "Accept: application/json" \
-H "Authorization: $EXPORT_ADMIN_SECRET" > "featureflags/feature_flags_toggle_names.json"

echo "JSON_TOGGLES_PATH: $JSON_TOGGLES_PATH"
UNLEASH_PROJECT_TOGGLE_NAMES=$(jq -r [.features[].name] featureflags/feature_flags_toggle_names.json)
echo "list of toggle names: $UNLEASH_PROJECT_TOGGLE_NAMES"

echo -e "Exporting toggles from environment"
curl -L -X POST "${EXPORT_UNLEASH_URL}/api/admin/features-batch/export" \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "Authorization: $EXPORT_ADMIN_SECRET" \
--data-raw '{
"environment": "development",
"downloadFile": true,
"features": '"${UNLEASH_PROJECT_TOGGLE_NAMES}"'
}' > "$JSON_TOGGLES_PATH"
}

validate_environment_variables() {
echo "Validating that environment variables are set"
if [ -z "${EXPORT_UNLEASH_URL}" ]; then
echo "environment variable [EXPORT_UNLEASH_URL] was not set"
return 1
fi

if [ -z "${EXPORT_NAMESPACE}" ]; then
echo "environment variable [EXPORT_NAMESPACE] was not set"
return 1
fi

if [ -z "${EXPORT_ADMIN_SECRET}" ]; then
echo "environment variable [EXPORT_ADMIN_SECRET] was not set"
return 1
fi

if [ -z "${EPHEMERAL_NAMESPACE}" ]; then
echo "environment variable [EPHEMERAL_NAMESPACE] was not set"
return 1
fi

if [ -z "${IMPORT_PAT}" ]; then
echo "environment variable [IMPORT_PAT] was not set"
return 1
fi

echo "\nAll required environment variables are set"
}

# if ! validate_environment_variables; then
# exit 1
# fi

echo "Making directory for temp json files"
mkdir featureflags

if [ "${IMPORT_LOCALLY}" == false ]; then
echo "environment variable [JSON_TOGGLES_PATH] was not set, exporting out of environment"
export_toggles
fi

echo -e "\nImport toggles into ephemeral environment"
EXPORTED_UNLEASH_TOGGLES=$(cat $JSON_TOGGLES_PATH)

curl -L -X POST 'http://localhost:4243/api/admin/features-batch/import' -H 'Content-Type: application/json' \
-H 'Accept: application/json' \
-H "Authorization: ${IMPORT_PAT}" \
--data-raw '{
"project": "default",
"environment": "development",
"data": '"$EXPORTED_UNLEASH_TOGGLES"'
}'

echo "Toggles imported into ephemeral environment: $EXPORTED_UNLEASH_TOGGLES"

echo "Cleanup json files"
rm -rf featureflags
28 changes: 28 additions & 0 deletions featureflags/unleash_import_job.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
apiVersion: template.openshift.io/v1
kind: Template
metadata:
name: unleash-export-import-template
annotations:
description: "The purpose of this job is to import Unleash toggles into ephemeral namespaces"
objects:
- apiVersion: batch/v1
kind: Job
metadata:
name: "unleash-import-config-${RANDOM_STRING}"
spec:
template:
spec:
containers:
- name: ubi8
image: registry.access.redhat.com/ubi8/ubi:8.9-1107.1706791207
command: ["dnf", "install", "jq", "-y", "&&", "source", "<(https://github.com/RedHatInsights/cicd-tools/blob/437db2c2c738b16dc567accc45f4254b21ae6f69/featureflags/unleash_config_export_import.sh)"]
volumeMounts:
- mountPath: /tmp
name: unleash-export-import-script-volume
restartPolicy: Never
backoffLimit: 4
parameters:
- description: generates a random string to uniquely identify the name of an object
name: RANDOM_STRING
value: "$(head -c 6 /dev/random | base64)"
required: true
15 changes: 0 additions & 15 deletions helpers/vars.groovy
View file
Open in desktop

This file was deleted.