Skip to main content
Version: 26.2

Air-gapped deployment

The standard deployment cannot be used in air-gapped environments, because it downloads the distribution with the AWS CLI and pulls the container images from the Axiomatics registry.

For air-gapped environments, you download the Authorization Hub distribution and pull the container images on a machine that has internet access, and then transfer them into your environment. You push the images to your own container registry and install the Helm chart from that registry. No component needs internet access after the files are transferred.

The procedure involves two machines:

  • Transfer machine: Has internet access. It is used only to download the distribution, pull the container images, and move the files into your environment, for example on removable media.
  • Air-gapped machine: Has no internet access. It has access to your container registry and the Kubernetes cluster, and is used for all remaining steps.

Prerequisites​

Before you begin, ensure the prerequisites listed at the beginning of the Deployment page are met, with the following differences.

  • Private container registry

    A registry that the cluster nodes and the air-gapped machine can reach. It must be able to store images that are referenced by digest, which any registry that supports the Docker Registry HTTP API V2 does. Allow enough storage for the nine images listed in Container images. If your registry uses a certificate from an internal CA, or plain HTTP, configure the cluster nodes to trust it. Configure the Docker daemon on the air-gapped machine in the same way before step 4, because that step runs docker push from that machine. To do so, install the CA certificate, or, for plain HTTP, add the registry to insecure-registries in daemon.json.

  • Docker

    Installed on both machines. The transfer machine uses it to pull and save the images. The air-gapped machine uses it to load the images and push them to your registry.

  • AWS CLI and Axiomatics credentials

    Installed and configured on the transfer machine only, as described in the Preparation section. They are not needed on the air-gapped machine.

  • Command-line tools

    On Linux and macOS, unzip and sha256sum. On macOS, use shasum -a 256 in place of sha256sum. On Windows, Windows PowerShell 5.1 or later is used and no additional tools are needed. The commands on Windows use curl.exe, which is part of Windows 10 and later. Do not use curl in Windows PowerShell 5.1, because it is an alias of Invoke-WebRequest.

  • Correct system clock

    The date of the machines that run Authorization Hub must be correct. A machine that has never been online can have a wrong clock, which causes license validation to fail.

What you need from Axiomatics​

The distribution and the container images are the same as for the standard deployment. Contact Axiomatics Customer SupportOpens in a new tab to obtain the following.

  • The Access key ID and Secret access key for the Axiomatics AWS account. They give access to the distribution and to the Amazon ECR registry. Read the Preparation section for details.
  • The license file, axiomatics_HUB.license.

The distribution, hub-<version>.zip, contains the Helm chart in kubernetes/charts/hub. It does not include the license or the container images.

Container images​

Authorization Hub uses the following images. The tags are those of version 26.2.0.

ImageTagSourceRequired
hub-frontend1.5.2Amazon ECRYes
hub-keycloak1.3.3Amazon ECRYes
hub-service1.3.2Amazon ECRYes
hub-service-aggregator3.34.1Amazon ECRYes
adm1.5.1Amazon ECRYes
console-api-gateway2.4.2Amazon ECRYes
redis8.8-alpineDocker HubYes
hub-ai-service1.0.0Amazon ECROnly to use the AI assistant, which is disabled by default
postgres17.6Docker HubOnly if you deploy the local database instead of using an external one

The Helm chart references the Authorization Hub images by digest. Each tag in the table corresponds to the digest that the chart requires. You verify this in step 6.

note

The Redis and PostgreSQL images are public tags and can change over time. The chart references them by tag, so step 6 cannot check them. Step 2 records the digests of the images you ship. Keep them with the archive.

Deployment​

warning

Replace all $CHANGE_ME placeholders in the commands below with your actual values before running them.

Step 1: Download the distribution​

On the transfer machine, follow steps 1 to 3 of the Preparation section to configure the AWS CLI, download the distribution, and log in to the Amazon ECR registry.

Then download the checksum of the distribution and verify it.

aws s3api get-object --bucket axiomatics-customer-artifacts --key releases/com/axiomatics/hub/26.2.0/hub-26.2.0.zip.sha256 hub-26.2.0.zip.sha256
echo "$(cut -d' ' -f1 hub-26.2.0.zip.sha256)  hub-26.2.0.zip" | sha256sum -c -

The output must be hub-26.2.0.zip: OK.

If the checksums do not match, download the distribution again.

Step 2: Pull and save the images​

On the transfer machine, pull the images from Container images, tag them for your registry, and save them to a single archive. $REGISTRY is the hostname of your registry, optionally followed by a path, without a trailing slash. For example, registry.example.com/hub.

note

The axiomatics/ path of the Amazon ECR images is not kept.

Omit hub-ai-service and postgres from the lists if you do not need them.

export REGISTRY=$CHANGE_ME
ECR=748131003707.dkr.ecr.eu-central-1.amazonaws.com/axiomatics
AXIOMATICS_IMAGES="hub-frontend:1.5.2 hub-keycloak:1.3.3 hub-service:1.3.2 hub-service-aggregator:3.34.1 adm:1.5.1 console-api-gateway:2.4.2 hub-ai-service:1.0.0"
THIRD_PARTY_IMAGES="redis:8.8-alpine postgres:17.6"

for i in $AXIOMATICS_IMAGES; do
  docker pull --platform linux/amd64 "$ECR/$i"
  docker tag "$ECR/$i" "$REGISTRY/$i"
done

for i in $THIRD_PARTY_IMAGES; do
  docker pull --platform linux/amd64 "$i"
  docker tag "$i" "$REGISTRY/$i"
done

The --platform linux/amd64 option pulls the images for x86-64 cluster nodes. Replace it if your nodes use a different architecture.

Check the pulled images before you save the archive. Extract the distribution from step 1 on this machine, in the same shell session, and compare the digest of every Authorization Hub image that you pulled with the digests that the chart requires. This finds a moved tag or a wrong image now, instead of after the transfer.

unzip hub-26.2.0.zip
for i in $AXIOMATICS_IMAGES; do
  pulled=$(docker inspect --format '{{index .RepoDigests 0}}' "$ECR/$i" | sed 's/.*@//')
  grep -qF "$pulled" hub-26.2.0/kubernetes/charts/hub/values.yaml && echo "MATCH    $i" || echo "DIFFERS  $i"
done

Every image must be reported as MATCH. If an image is reported as DIFFERS, do not save the archive. Pull the image again, and if it still differs, contact Axiomatics Customer SupportOpens in a new tab.

Record the digests of the third-party images that you are shipping, and keep them with the archive. The Redis and PostgreSQL tags can change, as described in Container images. Docker Hub also limits anonymous pulls. If a pull is refused, wait, or log in to Docker Hub on the transfer machine.

docker inspect --format '{{index .RepoDigests 0}}' redis:8.8-alpine postgres:17.6

Save the images to a single archive.

docker save -o hub-images.tar $(for i in $AXIOMATICS_IMAGES $THIRD_PARTY_IMAGES; do echo "$REGISTRY/$i"; done)

Step 3: Transfer the files​

  1. On the transfer machine, put the following files into one directory.

    • hub-26.2.0.zip
    • hub-images.tar
    • axiomatics_HUB.license

    Plan the space: hub-images.tar is approximately 1.6 GB with a current Docker that uses the containerd image store. With the classic image store, docker save writes uncompressed layers and the archive can be several times larger. Ensure that the transfer medium and both machines have enough free disk space.

  2. Create a checksum for each file.

    sha256sum hub-26.2.0.zip hub-images.tar axiomatics_HUB.license > SHA256SUMS.txt

  3. Copy the three files and SHA256SUMS.txt to the air-gapped machine, and run the following commands from the directory that contains them. This verifies that the files were not damaged during the transfer.

    sha256sum -c SHA256SUMS.txt

    Every line must end with OK.

    If a file fails the check, copy it again from the transfer machine.

  4. Extract the distribution on the air-gapped machine.

    unzip hub-26.2.0.zip

    The Helm chart is in hub-26.2.0/kubernetes/charts/hub. In the remaining steps, kubernetes/charts is the directory that contains the hub directory.

Step 4: Load the images and push them to your registry​

On the air-gapped machine, load the images into Docker and push them to your registry. Run the commands from the directory that contains hub-images.tar. Use the same $REGISTRY value as in step 2.

If your registry requires authentication, run docker login first.

export REGISTRY=$CHANGE_ME

docker load -i hub-images.tar
docker image ls --format '{{.Repository}}:{{.Tag}}' | grep -F "$REGISTRY/" | xargs -n 1 docker push

Verify that all images have arrived in your registry. Under your chosen path, the list should contain the images that you pulled in step 2. For example:

- adm
- console-api-gateway
- hub-ai-service
- hub-frontend
- hub-keycloak
- hub-service
- hub-service-aggregator
- postgres
- redis

For a registry that implements the Docker Registry HTTP API V2, you can list them with the following command. Add -u <user> if the registry requires authentication.

curl -s https://$CHANGE_ME/v2/_catalog
note

The push command in this step pushes every local image whose name starts with $REGISTRY/. On an upgrade, older versions that are still on the machine are pushed again. Remove them first if you do not want that.

warning

The Authorization Hub images are referenced by digest in the Helm chart. Push the images as they are, and do not rebuild or convert them. If you copy the images with another tool, use a method that preserves their digests. You verify this in step 6.

Step 5: Configure the Helm chart​

  1. Copy axiomatics_HUB.license into the kubernetes/charts/hub/configurations directory. The chart reads the license when Helm renders it, so this must be done before the next step.

  2. Create a file named values-airgap.yaml in kubernetes/charts, outside the chart directory, with the location of your images. Use the same file for every installation and upgrade.

    hubHostname: "$CHANGE_ME"
    registry: "$CHANGE_ME/"
    services:
      redis:
        image:
          repository: "$CHANGE_ME/redis"
      postgres:
        image:
          repository: "$CHANGE_ME/postgres"
    • hubHostname: the DNS name users use to access Authorization Hub.
    • registry: the same value as $REGISTRY in step 2, with a trailing slash. The chart adds it as a prefix to the name of each Authorization Hub image, including the AI assistant image, so without the trailing slash the image names are invalid.
    • services.redis.image.repository and services.postgres.image.repository: the same value as $REGISTRY, followed by /redis and /postgres. Redis and PostgreSQL do not use the registry value, so both must be overridden. Redis is always deployed, so without this override the installation fails. The postgres value is used only when you deploy the local database. Read step 7.
    warning

    Order of the values files. The local database file, hub/values-local-database.yaml, sets the PostgreSQL image to the public postgres image. Pass it before values-airgap.yaml in every helm install, helm upgrade, and helm template command. The last file wins, so if the local database file comes after values-airgap.yaml, it silently replaces your registry value, and the PostgreSQL pod is pulled from Docker Hub, which fails without internet access.

  3. If your registry requires authentication, create a secret that the cluster uses to pull the images and reference it in the file.

    kubectl create secret docker-registry regcred \
      --docker-server=$CHANGE_ME \
      --docker-username=$CHANGE_ME \
      --docker-password=$CHANGE_ME
    imagePullSecrets:
      - name: regcred

    Skip this step if your registry does not require authentication.

Step 6: Verify the images​

Before you install, verify that every image that the chart uses is taken from your registry, and that your registry holds the digests that the chart requires. Run the commands from the kubernetes/charts directory, which contains the hub directory and values-airgap.yaml.

  1. Render the chart and list the images.

    helm template hub hub -f hub/values.yaml -f values-airgap.yaml | grep "image:" | sort -u

    If you deploy the local database, add -f hub/values-local-database.yaml between -f hub/values.yaml and -f values-airgap.yaml (on Windows, -f .\hub\values-local-database.yaml) in the helm template commands of this step, exactly as in step 7. Without it, the render shows no PostgreSQL image, and a missing or overridden PostgreSQL value is not detected.

    Every line must refer to your registry. The Authorization Hub images are referenced by @sha256 digest, and the Redis and PostgreSQL images by tag. A line that refers to docker.io, to the Amazon ECR registry, or to an unqualified redis or postgres image means that a value is missing from values-airgap.yaml. Correct it before you continue.

    note

    If this command reports a license error, the license file was not copied to hub/configurations.

  2. Check that your registry holds each digest.

    ACCEPT="application/vnd.docker.distribution.manifest.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.index.v1+json"

    helm template hub hub -f hub/values.yaml -f values-airgap.yaml \
      | grep "image:" | grep -oE '[^"[:space:]]+@sha256:[0-9a-f]{64}' | sort -u \
      | while read -r ref; do
          repo="${ref%@*}"; digest="${ref#*@}"
          if curl -fsSI -H "Accept: $ACCEPT" "https://${repo%%/*}/v2/${repo#*/}/manifests/${digest}" > /dev/null; then
            echo "FOUND    $ref"
          else
            echo "MISSING  $ref"
          fi
        done

    To use a registry over plain HTTP, replace https:// with http://. If your registry requires authentication, add -u <user> to the curl command.

    Every digest must be reported as FOUND. The Authorization Hub images that are disabled by default, such as hub-ai-service, are not listed.

    If a digest is reported as MISSING, first confirm that the registry is reachable and that you are authorized to read from it, because an unreachable registry also reports MISSING. If the registry is reachable, the images in the registry do not have the digests that the chart requires. This can happen if an image was pushed under a different digest, or if a tag in Container images does not match the chart. Do not install, and do not change the chart to use tags instead. Contact Axiomatics Customer SupportOpens in a new tab and provide the output of the check.

Step 7: Create the secrets and install​

  1. Create the secrets that Authorization Hub requires following steps 3 (Keycloak secret) to 7 (Redis secret) of the Deployment section. Skip step 4, because email notifications are not supported in air-gapped environments. In other words, create the secrets keycloak, db-connection, principal, and redis.

    If your registry requires authentication, do not create the ECR pull secret (step 2). Use the regcred secret from step 5 of this page instead.

  2. Install the Helm chart from the kubernetes/charts directory.

    helm install hub \
      -f hub/values.yaml \
      -f values-airgap.yaml \
      hub

    If you don't have an external database, add -f hub/values-local-database.yaml before the -f values-airgap.yaml line to deploy a PostgreSQL pod automatically. The order matters, as described in step 5.

    helm install hub \
      -f hub/values.yaml \
      -f hub/values-local-database.yaml \
      -f values-airgap.yaml \
      hub

  3. Wait for the pods to become ready.

    kubectl wait --for=condition=ready pod --all --timeout=900s
    kubectl get pods

    All pods must be Running:

    • adm
    • api-gateway
    • frontend
    • hub
    • keycloak
    • service-aggregator
    • the three redis pods
    • postgres if you deployed the local database
    note

    The hub and adm pods may restart once or twice during the first minutes, because they start before the database accepts connections. This is expected. A pod that keeps restarting requires investigation.

    If a pod is in the ImagePullBackOff state, run kubectl describe pod <pod-name>. The Events section names the image reference that cannot be pulled. Compare it with the images in your registry.

  4. Expose Authorization Hub.

    Read the Cloud deployment section for the ingress requirements, or use a local test deployment. Authorization Hub is then available at hubHostname.

Considerations for air-gapped environments​

  • TLS certificates

    Certificates that are issued by public authorities, such as Let's Encrypt, and cloud-managed certificates cannot be used without internet access. Use a certificate that your internal certificate authority has issued for the Authorization Hub hostname, and make sure that browsers and the services that connect to Authorization Hub trust that authority.

  • User interface

    The Authorization Hub user interface does not load resources from the internet, so users do not need internet access to sign in and work. Only the Help link, which opens docs.axiomatics.comOpens in a new tab, require internet access when they are clicked.

  • Email notifications

    Email notifications are not supported in air-gapped environments. Features that depend on email, such as password reset, user invitations, and email verification, are therefore not available.

  • Attribute connectors

    Attribute connectors can reach only the data sources that are accessible from your network.

  • Authentication

    Configure user authentication with an identity provider that is reachable from the cluster.

AI assistant​

The AI assistant is disabled by default. It requires a language model that the cluster can reach through an OpenAI-compatible API. In an air-gapped environment, this requires a model that is hosted inside your network. Read the AI assistant deployment section for the requirements.

Upgrade​

To upgrade an air-gapped deployment, repeat step 1 and step 2 on the transfer machine for the new version, and then:

  1. Repeat step 3 and step 4 of this page to transfer the new files and push the new images to your registry.

  2. Copy your axiomatics_HUB.license into the hub/configurations directory of the new chart. Copy your values-airgap.yaml into the kubernetes/charts directory of the new version, next to the new hub directory.

  3. Repeat step 6 to verify the images.

  4. Back up your PostgreSQL database and read the Upgrade section for the version-specific steps, such as creating new secrets. Then upgrade the Helm chart with the same values-airgap.yaml file.

    helm upgrade hub \
      -f hub/values.yaml \
      -f values-airgap.yaml \
      hub
    warning

    If you deployed the local PostgreSQL pod, which means that you installed with -f hub/values-local-database.yaml, keep that file in every upgrade, before values-airgap.yaml. Without it, the upgrade removes the database and its data. Omit the file only if you use an external database.

    helm upgrade hub \
      -f hub/values.yaml \
      -f hub/values-local-database.yaml \
      -f values-airgap.yaml \
      hub
    note

    During the upgrade, the new PostgreSQL pod can restart once, because it starts while the old pod is still shutting down on the same volume. Wait until kubectl get pods shows it Running again.

Applying a renewed license, or any other change to the chart configuration, uses the same helm upgrade command. Read the License section for details. On this page, use the command above: the commands on the License page use the Amazon ECR registry and are for the standard deployment. Add settings with --set, for example --set services.apiGateway.env.warningThresholdDays=45.

Test in a local minikube cluster​

For testing purposes only, you can verify an air-gapped installation on a single machine with minikube and a local registry. The registry in this procedure stands in for your own registry.

note

A local cluster that is created without internet access needs the minikube node image and the Kubernetes images to be available locally. Prepare them on a machine with internet access, using the same minikube version as on the offline machine. This is not needed for a cluster that your organization already operates.

This procedure was verified with minikube v1.38.1, Kubernetes v1.35.1, and the node image kicbase:v0.0.50. Docker, minikube, kubectl, and Helm must already be installed on the air-gapped machine.

Before you begin, ensure that Docker has at least 4 CPUs and 11 GB of memory available, because the cluster in this procedure is given 10 GB. Ports 80 and 5000 must be free: port 80 is used by the minikube tunnel and port 5000 by the registry. The local test on the Deployment page uses 8 GB. This procedure needs more because it also runs the registry and the local database, and on Windows the service-aggregator needs fork: 2 in the values below.

  1. On a machine with internet access, download the minikube cache and save the registry image.

    minikube start --download-only --driver=docker --kubernetes-version=$CHANGE_ME
    docker pull registry:2
    docker save registry:2 -o registry-2.tar

    Copy the kic and preloaded-tarball directories from the minikube cache, ~/.minikube/cache (on Windows, %USERPROFILE%\.minikube\cache), together with registry-2.tar, to the air-gapped machine.

  2. On the air-gapped machine, put the two cache directories into the same location, and load the images into Docker.

    docker load -i registry-2.tar
    docker load -i <path-to>/kicbase.tar

    <path-to>/kicbase.tar is the file in the kic directory of the cache.

  3. Create the cluster. The --base-image option is required: the node image is loaded under its tag only, and minikube's default reference, which includes a digest, would make Docker try to download it. The --insecure-registry option allows the cluster to pull from the local registry, which uses plain HTTP.

    minikube start --driver=docker --kubernetes-version=$CHANGE_ME \
      --memory=10000 --cpus=4 \
      --insecure-registry="192.168.49.0/24" \
      --base-image=gcr.io/k8s-minikube/kicbase:$CHANGE_ME

    Replace the kicbase tag with the one in the name of the file in the kic directory, for example v0.0.50. Verify that the minikube network uses the subnet in --insecure-registry. If it does not, delete the cluster and start it again with that subnet.

    docker network inspect minikube -f "{{(index .IPAM.Config 0).Subnet}}"

  4. Start the registry and connect it to the cluster network.

    docker run -d --name registry --restart=always -p 5000:5000 registry:2
    docker network connect minikube registry
    docker inspect -f "{{.NetworkSettings.Networks.minikube.IPAddress}}" registry

    Note the IP address in the output of the last command, for example 192.168.49.3. Verify that the registry answers, both from your machine and from the cluster node. Replace <registry-ip> with the IP address.

    curl http://localhost:5000/v2/_catalog
    minikube ssh -- curl -s http://<registry-ip>:5000/v2/_catalog

    Both commands must return {"repositories":[]}. If the node cannot reach the registry, the cluster cannot pull any image. The IP address can change when Docker restarts, so look it up again if it does.

  5. Follow step 1 to step 4, with localhost:5000 as $REGISTRY. Complete steps 1 and 2 while the machine still has internet access. Because a single machine is used, the transfer and the checksums in step 3 are not needed, but you must still extract the distribution.

    Before step 4, remove the images from local Docker, so that step 4 really loads them from the archive. On a single machine, the images are still there from step 2. Without this, docker load restores nothing and the push sends the pulled images, so a damaged or wrong archive is not detected.

    docker image ls --format '{{.Repository}}:{{.Tag}} {{.ID}}' | grep -E "^($REGISTRY|748131003707|redis|postgres)" | awk '{print $2}' | sort -u | xargs docker rmi -f

    The cluster pulls from the registry through its IP address, so use the following values in values-airgap.yaml, and localhost as the hostname.

    hubHostname: "localhost"
    hubProtocol: "http"
    registry: "192.168.49.3:5000/"
    services:
      apiGateway:
        service:
          type: LoadBalancer
      serviceAggregator:
        env:
          fork: 2
      redis:
        image:
          repository: "192.168.49.3:5000/redis"
      postgres:
        image:
          repository: "192.168.49.3:5000/postgres"

    Replace 192.168.49.3 with the IP address from the previous step.

  6. Continue with

    StepNote
    5Skip the regcred secret.
    6The second check cannot reach the registry through the IP address from the chart. Replace the registry host in its URL with localhost:5000, and use http://. Include -f hub/values-local-database.yaml in the render commands, before values-airgap.yaml.
    7Add -f hub/values-local-database.yaml to the install command, before -f values-airgap.yaml, as shown in step 7.

  7. Start the minikube tunnel in a separate terminal window and keep it running.

    minikube tunnel

    Authorization Hub is available at http://localhost.

Next step​

Once Authorization Hub is deployed, configure user authentication before you begin inviting users. Read the User authentication section for details.