Skip to main content

External KMS (KMIP) Configuration

This guide explains how to connect your DS3 Gateway to an external KMS using the KMIP protocol (1.4). It covers the configuration that the Kubernetes gateway operator (gateway-operator) applies to enable per-tenant encryption with your own KMS in production.

Experimental feature

External KMS encryption is currently experimental. When enabled, objects uploaded to the Swarm are encrypted with keys stored in the KMS. Once encrypted:

  • All Gateways that serve the same tenant must share the same KMS configuration (same KMIP server, same tenant key seeds). A Gateway without the correct keys cannot decrypt objects uploaded by another Gateway.
  • If you decommission a Gateway or migrate to a new one, the new Gateway must be configured with the same KMS and the same tenant keys — otherwise existing objects become unreadable.
  • If you lose access to the KMS or misconfigure any Gateway, objects encrypted under this mode become permanently unreadable.

Do not enable this feature unless you have a robust KMS backup and disaster recovery plan in place, and all Gateways serving the tenant are configured identically.

Encryption Modes Overview​

The DS3 Gateway encrypts objects before uploading them to the Swarm. Three key-source modes are available:

ModeDescriptionUse case
single-keyOne encryption key for all tenantsDev, test, simple deployments
local-multi-keysPer-tenant keys read from a local file inside the SecretSmall-scale, air-gapped
remote-multi-keysKeys fetched from the Coordinator, with a KMIP KMS fallback for private tenantsProduction with external KMS

This document covers remote-multi-keys with an external KMIP-compliant KMS.

Architecture​

The DS3 Gateway loads its encryption configuration from two Kubernetes resources that the gateway operator injects as volumes. The configuration lives under the crypto JSON key in both:

  • ConfigMap (/opt/config/application.json) — typically the encryption mode (mode) and the refresh interval (update_period).
  • Secret (/opt/secret/application.json) — typically the KMS connection parameters (kms_config), TLS certificates, and key blobs.

There is no strict rule enforced by the code — gods3 merges both files at startup, so you can split fields by sensitivity. The examples below follow the common convention: ConfigMap for non-sensitive settings, Secret for credentials and keys.

Key resolution flow in remote-multi-keys mode:

  1. The DS3 Gateway polls the Coordinator for per-tenant configurations.
  2. Public tenants (zero_knowledge: false) carry their key seed directly in the Coordinator response — no external KMS call.
  3. Private tenants (zero_knowledge: true) go through this chain:
    • The Gateway checks a local file in the Secret (tenant_keys).
    • If no key found there, it queries the external KMS via KMIP 1.4.
    • If the KMS has no key for that tenant either, HTTP 500 is returned and the upload/download fails.

Prerequisites​

  • A KMS server supporting KMIP 1.4.
  • A TLS client certificate + private key for mTLS authentication to the KMS.
  • The KMS must contain one Managed Object (type Secret Data) per tenant, keyed by tenant UUID as the object's Name attribute.
  • A Coordinator instance serving tenant configurations (required for remote-multi-keys mode).

Step 1: Configure the ConfigMap (non-sensitive)​

Add the encryption mode and refresh interval to your Gateway's ConfigMap. The gateway operator mounts this file at /opt/config/application.json — the default name is gods3-config, referenced by spec.gods3.applicationConfigMapName in the Gateway CR.

# gods3-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: gods3-config
  namespace: <namespace>
data:
  application.json: |
    {
      "crypto": {
        "mode": "remote-multi-keys",
        "update_period": "1h"
      }
    }

Merge with your existing config: the YAML above shows only the new crypto section. Your real file must include all your existing sections (cache, redis, etc.) alongside this one — add the crypto block to your current application.json content, do not replace it.

JSON vs YAML

The gateway reads application.json as raw JSON. If you deploy via the Cubbit Helm chart, you write YAML values (e.g. config.crypto.mode: remote-multi-keys) and the chart converts them to JSON automatically. When managing the ConfigMap directly, the file content must be valid JSON.

kubectl apply -f gods3-config.yaml

Or, for a quick setup:

kubectl create configmap gods3-config \
  --namespace <namespace> \
  --from-literal=application.json='{
    "crypto": {
      "mode": "remote-multi-keys",
      "update_period": "1h"
    }
  }'
FieldAlways requiredDescription
modeno — defaults to single-keySet to "remote-multi-keys"
update_periodyes for multi-keys (local-multi-keys and remote-multi-keys)How often the DS3 Gateway refreshes keys from the Coordinator or KMS. Format: Go duration string (e.g. "30s", "5m", "1h"). Without this value the gateway panics at startup in multi-keys mode.

Step 2: Configure the Secret (sensitive)​

Add the KMS connection parameters and the encryption key blob to your Gateway's Secret. The gateway operator mounts this file at /opt/secret/application.json — the default name is gods3-service, referenced by spec.gods3.applicationSecretName in the Gateway CR.

The Secret holds the KMS connection details. Both live under the crypto JSON key — the DS3 Gateway reads this file and merges it with the ConfigMap values at startup.

# gods3-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: gods3-service
  namespace: <namespace>
stringData:
  application.json: |
    {
      "crypto": {
        "kms_config": {
          "kmip": {
            "address": "kms.example.com:5696",
            "major_version": 1,
            "minor_version": 4
          },
          "tls": {
            "key": "-----BEGIN PRIVATE KEY-----\n<your-private-key>\n-----END PRIVATE KEY-----",
            "cert": "-----BEGIN CERTIFICATE-----\n<your-cert>\n-----END CERTIFICATE-----",
            "cipher_suites": [4865, 4866, 4867],
            "insecure_skip_verify": false
          }
        }
      }
    }
caution

stringData is write-only — Kubernetes does not return the values on read. Store the source file securely; never commit plain-text secrets to git.

Apply it:

kubectl apply -f gods3-secret.yaml

Or, for a quick setup:

kubectl create secret generic gods3-service \
  --namespace <namespace> \
  --from-literal=application.json='{
    "crypto": {
      "kms_config": {
        "kmip": { "address": "kms.example.com:5696", "major_version": 1, "minor_version": 4 },
        "tls": {
          "key": "-----BEGIN PRIVATE KEY-----\n<your-private-key>\n-----END PRIVATE KEY-----",
          "cert": "-----BEGIN CERTIFICATE-----\n<your-cert>\n-----END CERTIFICATE-----",
          "cipher_suites": [4865, 4866, 4867],
          "insecure_skip_verify": false
        }
      }
    }
  }'

Field reference​

FieldRequiredDescription
kms_config.kmip.addressyesHostname and port of the KMIP server (e.g. "kms.example.com:5696").
kms_config.kmip.addressyesHostname and port of the KMIP server (e.g. "kms.example.com:5696").
kms_config.kmip.major_versionyesMust be 1.
kms_config.kmip.minor_versionyesMust be 4. Only KMIP 1.4 is supported.
kms_config.tls.keyyesPEM-encoded TLS client private key. The \n characters must be literal newlines in the JSON string.
kms_config.tls.certyesPEM-encoded TLS client certificate. Same newline requirement.
kms_config.tls.cipher_suitesnoList of allowed TLS cipher suites as uint16 values matching Go's crypto/tls constants. If omitted, Go uses a safe default list.
kms_config.tls.insecure_skip_verifynoSet to true only for testing — skips server certificate chain verification. Never use in production.

Step 3: Apply the Gateway Custom Resource​

The gateway operator mounts the ConfigMap and Secret into the DS3 Gateway pod automatically. You only need to point to them (or accept the defaults):

# gateway.yaml
apiVersion: gateway.cubbit.io/v1alpha1
kind: Gateway
metadata:
  name: my-gateway
  namespace: <namespace>
spec:
  gods3:
    enabled: true
    # These are the defaults — omit them if you use the standard names:
    applicationConfigMapName: gods3-config
    applicationSecretName: gods3-service

Apply it:

kubectl apply -f gateway.yaml

Note: The deprecated spec.gods3.keysSeedConfiguration field is ignored in current versions. All crypto configuration flows through the ConfigMap and Secret above. The gateway operator logs a deprecation warning if keysSeedConfiguration is populated, but no error is raised.

Step 4: Prepare the KMS (KMIP)​

Your KMS must expose per-tenant keys as KMIP Managed Objects with the following attributes:

KMIP AttributeValue
Object TypeSecret Data
Name (NameValue)The tenant UUID exactly as the Coordinator returns it (e.g., "67f1adf3-08e5-4eea-94e3-e611f263c87b")
Name TypeUninterpreted Text String
Key ValueBase64-encoded seed value (will be used as input to SHA-256 for X25519 key derivation)

Important rules:

  • Each tenant UUID must map to at most one Managed Object. Duplicates cause undefined behaviour.
  • Only tenants with zero_knowledge: true in the Coordinator will trigger a KMS lookup.
  • Tenants with zero_knowledge: false carry their key in the Coordinator's response — the KMS is not called for them.
  • If a zero_knowledge: true tenant has a key in the local Secret file (tenant_keys section), the file takes precedence over the KMS.

Testing with PyKMIP (development)​

For testing, you can run a PyKMIP server. Cubbit maintains a forked version pre-configured for the DS3 Gateway. The server listens on port 5696 and can be seeded with test tenant keys via a startup script. Refer to the Cubbit development documentation for the exact deployment manifests.

Step 5: Verify​

Check the DS3 Gateway pod logs​

kubectl logs -n <namespace> -l app=<gateway-name>-gods3 -c gods3 | grep -E 'keys|kms|kmip'

A healthy startup shows:

"Starting Keys Client update routine"

Check the gateway operator logs​

kubectl logs -n <namespace> -l app=gateway-operator -c manager | grep gods3

The ConfigMap and Secret volumes should be mounted correctly — you'll see the ConfigMap and Secret being wired into the deployment.

Simulate an upload (optional)​

Upload an object through the S3 API. If keys are resolved correctly, the payload is encrypted and uploaded to the Swarm.

If the KMS is unreachable during the periodic key refresh, you will see this in the logs:

"periodical update: failed to get key for tenant ..."

This does not affect in-flight operations — the Gateway keeps the last successfully loaded keys. If no key was ever loaded for a tenant (first start, KMS never reachable), uploads for that tenant return HTTP 500.

Troubleshooting​

All symptoms below appear in the DS3 Gateway pod logs (kubectl logs -n <namespace> -l app=<gateway-name>-gods3 -c gods3).

Symptom (from pod logs)Likely causeFix
kms config is requiredkms_config is absent or empty in the SecretAdd kms_config with the required fields
missing required configurationOne of kmip.address, tls.cert, or tls.key is emptyFill in all required fields
key not found for private tenantID ...No KMIP Managed Object with that tenant nameAdd the tenant key to the KMS
error writing kms request / error reading kms responseNetwork issue, firewall, TLS handshake failureCheck network connectivity, TLS cert validity, KMIP server logs
error decoding kms response / error decoding kms response payloadKMS returned an unexpected message formatVerify the KMS is KMIP 1.4 compliant and the response structure matches what the DS3 Gateway expects
invalid crypto.mode valueTypo in the mode fieldUse one of: single-key, local-multi-keys, remote-multi-keys
unsupported KMIP version, use 1.4major_version/minor_version not set to 1 and 4Correct the values in the Secret

What you do NOT need to do​

  • ❌ Do not set TENANT_ENCRYPTION_KEY_MODE — it is no longer used.
  • ❌ Do not populate spec.gods3.keysSeedConfiguration on the Gateway CR — it is deprecated.
  • ❌ Do not restart the DS3 Gateway after updating keys in the KMS — keys are hot-reloaded every update_period.

Reference: key derivation​

The DS3 Gateway uses X25519 (Curve25519) + AES-256-CTR hybrid encryption (ECIES). The key derivation chain is:

KMIP key value (base64)
  ↓ base64 decode
raw seed (32 bytes)
  ↓ SHA-256
private key (32 bytes)
  ↓ X25519 ScalarBaseMult
public key (32 bytes)  ← kept in the DS3 Gateway's in-memory key store, scoped per tenant

During encryption, an ephemeral keypair is generated per payload, an ECDH shared secret is derived, and the actual AES key is obtained by hashing that shared secret. This ensures per-payload forward secrecy.

Key format requirements​

PropertyValue
AlgorithmRaw symmetric seed (any cryptographically secure random)
Min decoded length32 bytes (256 bits)
Recommended length48 bytes (384 bits) — extra entropy for future-proofing
Encoding in KMIPBase64
KMIP Object TypeSecret Data
Pre-generatedNo — the Gateway derives the X25519 keypair from the seed

To generate a compliant seed:

openssl rand -base64 48

If the KMS becomes unavailable​

The Gateway caches resolved keys in memory. If the KMS goes down:

  • In-flight uploads/downloads → continue uninterrupted — keys already loaded at startup or from the last successful refresh are still valid.
  • New tenant requests for tenants whose keys have never been loaded → fail with HTTP 500 until the KMS is back.
  • Gateway logs → show "periodical update: failed to get key for tenant ..." at every update_period.
  • Automatic recovery → the Gateway retries on every refresh cycle; no manual intervention needed once the KMS is restored.

Backup and disaster recovery​

  • KMIP key seeds are the single source of truth for private tenants. Ensure your KMS has its own backup/DR strategy (replication, snapshots, cross-region).
  • Cached payloads that were encrypted with a key that is no longer available become unreadable. Objects stored in the Swarm are unaffected.

Production checklist​

  • KMIP server is reachable from the DS3 Gateway pod (check network policies, firewall rules)
  • TLS client cert is valid and signed by a CA the KMS trusts
  • insecure_skip_verify is false in production
  • update_period is set to a reasonable interval (1h recommended)
  • Tenant zero_knowledge flags are correctly set in the Coordinator
  • Each tenant that should use the KMS has a KMIP Managed Object with the matching UUID as the name
  • The Secret is stored in a secure location (Vault, ExternalSecret, etc.) — never committed to git
Last updated on