Install on a cloud cluster¶

A cloud installation is the same set of components as the local platform, applied with a production kustomize overlay that changes only what must differ from a laptop. The ankka repository carries an example production overlay in kustomization/overlays/cloud; copy it, set every value marked SET for your installation, and apply it by hand after the three controllers whose resources it uses.

deploy-local.sh is not used here. It refuses any kubectl context that is not the local kind cluster, on purpose: a script that deploys to whatever context happens to be current is a script that eventually deploys to the wrong one.

Prerequisites¶

Kubernetes 1.32 or later. Envoy Gateway's custom resource definitions use a validation rule an older API server rejects.
A network plugin that enforces NetworkPolicy. The platform restricts who may connect to every workload with network policies, and a network that stores them without enforcing them refuses nothing. GKE Dataplane V2, Calico, Cilium and k3s's built-in controller enforce them. See Check that the network enforces policy.
A domain you control, with DNS at a provider cert-manager can write to. The platform serves every exposed service under one wildcard certificate, *.<base domain>, and a wildcard can only be issued over the ACME DNS-01 challenge, which writes a TXT record into your zone. HTTP-01 cannot issue wildcards. cert-manager has built-in DNS-01 solvers for Cloudflare, Route53, Google Cloud DNS, Azure DNS, DigitalOcean, Akamai, ACME-DNS and RFC-2136 servers; other providers need a webhook solver.
A load balancer your cluster can provision for a Service of type LoadBalancer, and a DNS record pointing *.<base domain> at its address.
A container registry the cluster can pull from, holding the platform's images: the operator, the control plane, the sidecar, the console and, if you want the sample, the shopping cart. Every image is published for linux/amd64 and linux/arm64 under one tag, so the nodes may be of either architecture, or both.

What a production overlay changes¶

The overlay lists the same components as the local one, and patches or replaces five things:

	Local overlay	Production overlay
Gateway exposure	node ports 30080/30443, published by kind on 8080/8443	a `LoadBalancer` service, optionally with a pinned address
Wildcard certificate	a certificate authority created in the cluster, as a namespaced `Issuer`	an ACME issuer, such as Let's Encrypt, over DNS-01
Base domain and HTTPS port	`127.0.0.1.sslip.io`, port 8443	your domain, port 443
Identity provider admin	a development secret, `admin`/`admin`	the secret deleted; you create one out of band
Images	unqualified names loaded into the node	your registry, at a pinned release tag
The broker's size	one node, 2Gi, a 512MB heap	the values you set in `broker-size.yaml`, and three nodes with every topic copied three times when you list the `broker-three-nodes` component
The model server	not listed, unless deployed with `ANKKA_LOCAL_MODELS=on`	listed when the overlay names the `models` component, at the size, placement and storage you set in `models-size.yaml`
Topic defaults and bounds	seven days, one copy, any retention allowed	the `ANKKA_TOPIC_*` variables you set on the control plane before the first topic is declared

Both overlays install the same two authorities the platform issues workload certificates from — one for traffic between a service's own instances, one for HTTP between services and from the gateway — as self-signed roots in cert-manager's namespace, and trust-manager to share the second one's root with every ankka namespace. They are private on purpose: no public authority issues an ankka:// identity or a cluster-internal name, and nothing outside the cluster is asked to trust them. An installation that wants its authorities to come from elsewhere — a corporate PKI, a cloud certificate service — replaces the two ClusterIssuers ankka-cluster and ankka-service in kustomization/components/pki and nothing else; the operator asks for them by name. The root must not be an intermediate, because cluster remoting refuses a certificate issued by one.

The base domain¶

The base domain and HTTPS port are written once, in the overlay's ankka-platform ConfigMap, and kustomize replacements copy them into every place that must agree: the wildcard certificate, the gateway's listener, the control plane's and identity provider's routes, Keycloak's own hostname, and the ANKKA_BASE_DOMAIN of the operator and the control plane.

apiVersion: v1
kind: ConfigMap
metadata:
  name: ankka-platform
  namespace: ankka-gateway
data:
  baseDomain: example.com
  httpsPort: "443"

The control plane answers at https://api.example.com, the identity provider at https://auth.example.com, and exposed services at https://<service>-<project>.example.com.

The certificate¶

The gateway's HTTPS listener serves a Secret named ankka-wildcard-tls in the ankka-gateway namespace, holding a certificate for *.<base domain>. That Secret is the whole contract. The example overlay produces it with a cert-manager Certificate from an ACME ClusterIssuer; any issuer that produces it will do, and so will a bought certificate placed in that Secret.

Two details decide whether issuance works. Use the ACME provider's staging endpoint until issuance succeeds, because the production endpoint rate-limits failures. And mind where cert-manager looks for secrets: a ClusterIssuer reads the secrets it references from cert-manager's own namespace, and a namespaced Issuer from its own namespace. Put the DNS provider's credential where the issuer you choose will look for it. A webhook solver may add its own rule: the example overlay's webhook reads its token with its own permissions in cert-manager's namespace, which is why it uses a ClusterIssuer.

Custom hostnames¶

A service can answer at a name under a domain its owner brings (see Serve a service at your own domain). Each such name gets a certificate of its own, since the wildcard covers only the base domain. The example overlay names a second ClusterIssuer, letsencrypt-hostnames, in hostnameIssuer on the ankka-platform ConfigMap: ACME over HTTP-01, answered through the gateway's port-80 listener, because the installation cannot write to another domain's zone. The load balancer must therefore pass port 80 as well as 443. The components already carry what that needs: cert-manager's controller runs with --enable-gateway-api, and the Gateway admits listener sets and challenge routes from project namespaces. Leave hostnameIssuer empty to refuse every custom hostname.

Set gatewayAddress on the same ConfigMap to the load balancer's address so the control plane can tell a member the A record for an apex name, which cannot be a CNAME. Left empty, an apex is told that this installation has published no address.

The identity provider's administrator¶

The local platform's Keycloak administrator is admin/admin, in a Secret that is public in the ankka repository. A production overlay deletes that Secret rather than overriding it, so the real password never enters git, and Keycloak cannot start until you create the Secret yourself:

kubectl -n ankka-auth create secret generic ankka-keycloak-admin \
  --from-literal=username=admin --from-literal=password="$(openssl rand -base64 32)"

An identity provider that refuses to start is a loud and immediate failure. One on the internet with a published administrator password is neither. The realm contains no users at all; add them in the console. See Identity and machine accounts.

The object store's secrets¶

The example overlay deletes the object store's two development Secrets as it deletes the identity provider's, and the store and the operator cannot start until you create them. The administrator token is written twice, once for the store and once for the operator:

TOKEN="$(openssl rand -base64 32)"
kubectl -n garage-system create secret generic garage-secrets \
  --from-literal=rpc-secret="$(openssl rand -hex 32)" --from-literal=admin-token="$TOKEN"
kubectl -n ankka-operator create secret generic ankka-object-store-admin --from-literal=token="$TOKEN"

See Object storage.

The console¶

The overlay's ankka-platform ConfigMap names the console's address as consoleAuthority: console.<base domain>, with :<port> when HTTPS is not on 443. The overlay deletes the console's development Secret and removes the development secret from its realm client, so Keycloak generates one. Once Keycloak is up, create the Secret from that client secret and a session secret of your own; see Install and configure the console. Until the Secret exists the console does not start, and nothing else waits on it. An installation that does not want a console leaves the component out; see Install and configure the console.

Images¶

Every platform manifest names an unqualified image, such as ankka-controlplane:latest, with imagePullPolicy: IfNotPresent. That suits a node the image was loaded onto and is useless for a cluster that pulls. Every release publishes the images publicly on GitHub Container Registry, tagged with the release's version: ghcr.io/thinkmorestupidless/ankka-operator, ankka-controlplane, ankka-sidecar, ankka-console and sample-shopping-cart. A cluster can pull those directly, through a pull-through cache of ghcr.io in its own cloud, or from a registry of your own that you build them into by setting DOCKER_REPOSITORY:

DOCKER_REPOSITORY=registry.example.com/ankka sbt docker:publish

A cache in the cluster's own region keeps pulls local and keeps serving a version it has already fetched when ghcr.io is unavailable. Whichever you choose, rename the images in the overlay with an images: block, at the release you run:

images:
  - name: ankka-controlplane
    newName: registry.example.com/ankka/ankka-controlplane
    newTag: "0.16.1"
  - name: ankka-operator
    newName: registry.example.com/ankka/ankka-operator
    newTag: "0.16.1"

The sidecar needs one more step. It appears in no manifest, because the operator injects it beside every Python service, so the images: block never sees it. The operator learns its name from the ANKKA_SIDECAR_IMAGE variable on its own Deployment; patch that to the same registry and tag. Unset, a Python service fails with operator has no sidecar image rather than run a sidecar of the wrong version.

The proxy the operator runs beside every web-hosted service is the same: no manifest names it, and the operator reads it from ANKKA_PROXY_IMAGE on its own Deployment, which the example overlay patches to the registry and tag. Unset, a web-hosted service fails with operator has no proxy image. The operator also reads ANKKA_HTTPS_PORT, the port the gateway is reached on, which the overlays set from the same value as the control plane's.

The model server's keeper is ankka-model-keeper, named by the models component's manifests, so an overlay that lists the component renames it in the same images: block. The component's other image, Ollama's, is already pinned to a version and pulled from Docker Hub.

The keyring is ankka-keyring, named by the keyring component's manifests and published with every release beside the others; an overlay that lists the component renames it in the same images: block.

Pin a release tag rather than latest. A cluster should run a version that was built, tested and published as one.

The model server¶

The installation's model server is the models component: Ollama and a keeper that serves the models its administrator declares, to the installation's services only (see Models served by the installation). An overlay that wants one lists the component after the control plane and patches it with models-size.yaml, where the node it runs on, its toleration, its memory, CPU and accelerator, the number of calls it answers at once and its storage class and size are marked SET. The component's own values suit a laptop: one small model on a processor. Size the storage for every declared model together, since each is kept on the volume. An installation that wants no model server leaves the component out, and its services name a model provider with an address of its own.

The broker¶

The installation's broker is the broker component: Strimzi and one Kafka, which every project's services use (see The installation's broker). The example overlay patches its node pool with broker-size.yaml, where the node count, each node's storage, the heap and the memory and CPU are marked SET; the component's own values suit a laptop. Strimzi's images come from quay.io, so a cluster that pulls only through a cache of its own needs that cache to mirror quay.io/strimzi too. An installation that keeps a Kafka of its own leaves the component out, and its services name that Kafka in their descriptors.

Apply it, in order¶

kubectl config current-context       # confirm it is the cluster you mean
kubectl apply -k kustomization/components/cnpg --server-side --force-conflicts
kubectl apply -k kustomization/components/certmanager --server-side --force-conflicts
kubectl apply -k kustomization/components/envoy-gateway --server-side --force-conflicts
kubectl apply -f kustomization/components/broker/namespace.yaml --server-side
kubectl apply -k kustomization/components/broker/strimzi --server-side --force-conflicts
# wait for the four controllers to be ready; install a DNS-01 webhook solver here if yours needs one
kubectl apply -k kustomization/components/trust-manager --server-side --force-conflicts
kubectl -n cert-manager rollout status deployment/trust-manager
kubectl apply -k kustomization/components/keycloak-operator --server-side --force-conflicts
kubectl create namespace ankka-controlplane
kubectl create namespace ankka-auth
kubectl -n ankka-auth create secret generic ankka-keycloak-admin ...   # see above
kubectl apply -k kustomization/overlays/<your-overlay> --server-side --force-conflicts

The overlay carries the cloud-provider component and the installation's cloud settings in its ankka-platform ConfigMap, with no provider named. To name one, set cloudProvider, cloudAccount and cloudLocation there, and install the provider's own Deployment into ankka-cloud-provider after the overlay; see The cloud provider.

An installation whose buckets are in Google Cloud Storage names its object store there too (objectStoreBackend, objectStorePrefix, objectStoreSoftDeleteDays); see Install on GKE.

An overlay that lists the models component needs nothing applied before it: the component brings no custom resource definitions, and its certificate is issued by an authority the overlay already carries. Every service already running restarts once when it is added.

The controllers go first because each brings custom resource definitions that the overlay's resources are instances of, and a single kubectl apply -k gives no ordering guarantee between a definition and a resource of that kind. Apply with --server-side: CloudNativePG's definitions are too large for client-side apply's annotation.

The overlay carries everything else the platform needs, so the same apply works by hand and from a GitOps tool such as Flux. The control plane's database schema is a ConfigMap the overlay generates from the schema's single copy, created in the same apply as the Postgres cluster that runs it once at bootstrap. The realm is a KeycloakRealmImport in the overlay, which the Keycloak operator imports once Keycloak is ready.

kubectl apply of many documents applies everything it can and exits non-zero if anything failed. Check the exit code, not the count of lines that say applied.

The realm import is one-shot: it creates the ankka realm if it does not exist and never updates it, so a later change to kustomization/components/keycloak/realm-import.json is made in Keycloak's console on an installation that already has the realm. Create the first users there too; the realm holds none.

A cluster of your own machines¶

Nothing above assumes a cloud. A cluster you built yourself — k3s on a few machines, say — runs the platform the same way once it provides three things a cloud would otherwise hand you:

An address for the gateway. The gateway is a Service of type LoadBalancer, and something must give it an address: k3s's built-in ServiceLB gives it a node's, which is enough for one network; MetalLB gives it one from a pool you reserve. Point *.<base domain>, api.<base domain> and auth.<base domain> at that address. A private address behind a public name is ordinary: DNS-01 issues the wildcard certificate by writing a record in your zone and never connects to the cluster, so nothing need be reachable from the internet.
A default storage class. Every database the platform provisions asks the cluster for a volume of its default class and names none. k3s ships local-path, which keeps a volume on the node that first mounted it; that is fine for a lab and means a database follows its node.
Nodes of either architecture. amd64 or arm64, or a mix: every image of the platform's is published for both, and nothing in the platform pins a workload to one. Your own services' images are yours to build for the nodes you have; see Build an image.

Memory is the constraint such a cluster usually meets first. The control plane, the operator, the identity provider, the gateway, the certificate manager and the database operator sit beside every service you deploy, and each service is a JVM. Leave out the broker, garage and telemetry-store components until something needs them.

Check that the network enforces policy¶

Prove it once per cluster, with a pod that a deny-all policy should cut off:

kubectl create namespace netpol-check
kubectl -n netpol-check run server --image=busybox:1.36 --labels=probe=server --command -- httpd -f -p 8080
kubectl -n netpol-check run client --image=busybox:1.36 --command -- sleep 600
kubectl -n netpol-check wait --for=condition=Ready pod/server pod/client
IP=$(kubectl -n netpol-check get pod server -o jsonpath='{.status.podIP}')
kubectl -n netpol-check exec client -- sh -c "echo | nc -w 3 $IP 8080" && echo reachable
kubectl -n netpol-check apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: deny-all }
spec: { podSelector: { matchLabels: { probe: server } }, policyTypes: [Ingress] }
EOF
kubectl -n netpol-check exec client -- sh -c "echo | nc -w 3 $IP 8080" && echo "NOT ENFORCED"
kubectl delete namespace netpol-check

The first connection must succeed and the second must fail. On a cluster that prints NOT ENFORCED, every connection is still mutual TLS and every caller still named, but nothing is refused before the handshake: a workload of another project can reach a service's cluster ports and its database's port, and is stopped only by lacking a certificate.

After installing¶

ankka config set url https://api.example.com
ankka login

The console is at https://console.example.com. A publicly trusted certificate needs no config set ca. Grant the first administrator the platform-admin realm role in Keycloak's console, then create organizations and projects. See Organizations, projects and members.

Upgrading the installation is the same apply with new image tags. The operator, control plane, sidecar, console and CLI are released together as one version. Let the control plane's rollout finish before any service is deployed again: a service's runtime records each read of its secrets with the control plane, and refuses the read when the control plane does not yet know how.

Where secrets are kept is secretBackend on the ankka-platform ConfigMap, with cloudProvider, cloudAccount and cloudLocation for Google Secret Manager; the record of reads is kept in a database of its own, ankka-secret-reads-db, which every overlay installs beside the control plane's. See Secrets on the platform for each setting, what Google Cloud needs before an installation uses Secret Manager, and how an installation moves to it.

A spoke installation¶

Several installations can share one set of users. One of them, the hub, runs Keycloak as described on this page. The others, the spokes, run the control plane and the operator with no identity provider of their own, and trust the hub's realm. A person registered once at the hub logs in to every spoke with the same account.

A spoke's overlay is a copy of the production overlay with these differences:

It leaves out the keycloak-operator and keycloak components, and the patch that deletes the Keycloak administrator Secret, since there is no Keycloak to administer.
It sets two variables on the control plane Deployment, both naming the hub's realm by its public address:

- name: ANKKA_AUTH_ISSUER
  value: https://auth.example.com/realms/ankka
- name: ANKKA_AUTH_JWKS_URL
  value: https://auth.example.com/realms/ankka/protocol/openid-connect/certs

It empties ANKKA_AUTH_JWKS_CA on the same Deployment. The controlplane component sets it to the cluster's own service authority, which verifies the in-cluster Keycloak of a hub. The hub's public address has a certificate from a public issuer, so a spoke that keeps that setting refuses every token. Empty means the JVM's trust store:

- name: ANKKA_AUTH_JWKS_CA
  value: ""

It keeps its own base domain, for example caladan.example.com, so its control plane answers at api.caladan.example.com and its exposed services have hostnames under that domain.
Its console, at console.caladan.example.com, signs in at the hub. It reads the issuer from its own control plane, which names the hub's realm. Empty the two settings that point the console at an in-cluster Keycloak, so it calls the hub at its public address and verifies it by the system's trust store:

- name: ANKKA_CONSOLE_AUTH_BACKCHANNEL_URL
  value: ""
- name: ANKKA_CONSOLE_AUTH_CA
  value: ""

In the hub's realm, add https://console.caladan.example.com/* to the ankka-console client's valid redirect URIs. Give the spoke's ankka-console-secrets Secret that client's secret and a session secret of the spoke's own; see Install and configure the console.

The spoke accepts only tokens whose issuer is the hub's; a token issued anywhere else is refused with 401. Its login discovery advertises the hub's issuer, so ankka login against the spoke's address opens the hub's sign-in page. A platform-admin in the hub's realm is a platform administrator on every installation that trusts it. See Identity and machine accounts.