# Kubernetes operator The operator publishes cluster Services through Tuna tunnels. You describe a tunnel, a domain or a TCP port as a Kubernetes resource, and the operator creates it in your Tuna account and keeps the connection up. The cluster connects to Tuna on its own, over outbound connections, so you don't need a LoadBalancer, a public IP or open inbound ports. > **Not the same as tuna kubernetes.** The [`tuna kubernetes`](https://tuna.am/en/docs/tunnels/kubernetes.md) command gives temporary access to the cluster's Kubernetes API. The operator does something else: it publishes applications running in the cluster. ## Resources | Resource | What it does | When you need it | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | | [Tunnel](#tunnel) | an HTTP or TCP tunnel to a single Service | to publish one application or database | | [Ingress](#ingress) with `ingressClassName: tuna` | path routing to several Services behind one domain | `/api` goes to one Service, `/` to another | | [Domain](#domain) | a permanent subdomain or a [custom domain](https://tuna.am/en/docs/tunnels/guides/connect-self-domain.md) in your account | the address must not change; a host for an Ingress | | [TCPEndpoint](#tcp) | a reserved [TCP port](https://tuna.am/en/docs/tunnels/ports.md) with an alias | a permanent port for a TCP tunnel | Tunnel, Domain and TCPEndpoint are operator resources in the `k8s.tuna.am/v1alpha1` group. Ingress is a standard Kubernetes resource, and the operator serves it as an Ingress controller. ## What you need * Kubernetes 1.32 or newer; * [Nelm](https://github.com/werf/nelm) (recommended) or Helm 4; * an API key with write access to "Endpoints and domains" — see below for how to create it. ## API key for the operator The operator uses the key to reserve domains and TCP ports and to issue temporary tokens for tunnels. All of this belongs to the "Endpoints and domains" product, so write access to it (the `tunnels:write` scope) is all the key needs. 1. Open [API keys](https://my.tuna.am/api_keys?lang=en) in the dashboard and create a new key. 2. Under "What is the key for", choose "For API". 3. In "Description", name the cluster, for example `k8s-operator prod`. Create a separate key for each cluster: then you can revoke one cluster's access without affecting the others. 4. In "Expiration", choose "Unlimited". The "For API" preset sets 30 days by default, and once the key expires, the operator loses access to the API. If your security policy requires a limited lifetime, keep it, but plan to rotate the key. 5. Under "API access", choose "Selective" and set "Endpoints and domains" to "Write". Leave the other products at "None". 6. Create the key and copy it. You don't need to turn on "Run tunnels". The operator doesn't connect tunnels with its own key: it issues a separate temporary token for each tunnel. A key with full API access works too, but it gives the operator access to monitors, secrets and team settings it doesn't need. ## Installation The [`tuna-k8s-operator`](https://code.tuna.am/tuna/-/packages/helm/tuna-k8s-operator) chart is in the `code.tuna.am` package registry. The examples install the operator into the `tuna-system` namespace and pass the key in the `credentials.apiKey` parameter. Other ways to pass the key are in [Where to keep the key](#api-key-storage). **Nelm** Nelm shows the change plan first and then applies it: ```shell export NELM_FEAT_REMOTE_CHARTS=true nelm repo add tuna https://code.tuna.am/api/packages/tuna/helm nelm release plan install -n tuna-system -r tuna-operator tuna/tuna-k8s-operator \ --set credentials.apiKey= nelm release install -n tuna-system -r tuna-operator tuna/tuna-k8s-operator \ --set credentials.apiKey= ``` **Helm** ```shell helm repo add tuna https://code.tuna.am/api/packages/tuna/helm helm repo update helm install tuna-operator tuna/tuna-k8s-operator \ --namespace tuna-system --create-namespace \ --set credentials.apiKey= ``` Check that the operator is running: ```shell kubectl get pods -n tuna-system ``` The namespace runs two pods: `…-agent-manager` handles Tunnel and Ingress, `…-api-manager` handles Domain and TCPEndpoint. ### Where to keep the key | Parameter | How it works | When to choose it | | ---------------------------- | ---------------------------------------------------- | ----------------------------------------------------------------------------------------- | | `credentials.apiKey` | the chart creates a Secret with the key | a trial run, a test cluster | | `credentials.existingSecret` | the name of your own Secret with the `api-key` field | the key comes from External Secrets Operator or Sealed Secrets, or you created it by hand | | `extraEnv` | the `TUNA_API_KEY` variable from a secret injector | no Secret in Kubernetes: the key comes straight from the vault | An `extraEnv` example for [Bank-Vaults](https://bank-vaults.dev): ```yaml podAnnotations: vault.security.banzaicloud.io/vault-addr: https://vault.vault.svc:8200 vault.security.banzaicloud.io/vault-role: tuna-operator extraEnv: - name: TUNA_API_KEY value: vault:secret/data/tuna#TUNA_API_KEY ``` ### Chart parameters `helm show values tuna/tuna-k8s-operator` lists all parameters with descriptions. These are the ones you'll need most often: | Parameter | What it sets | | ------------------------------------- | -------------------------------------------------------------------------------------- | | `watchNamespace` | watch a single namespace; by default the operator watches all of them | | `ingressClass.default` | make `tuna` the default class for Ingresses without `ingressClassName` | | `agentManager.allowExternalUpstreams` | allow publishing addresses outside the resource's namespace, see [Security](#security) | | `apiManager.replicas` | the number of api-manager replicas; one is active, the rest are on standby | ## Tunnel to a Service ```yaml apiVersion: k8s.tuna.am/v1alpha1 kind: Tunnel metadata: name: web spec: protocol: http subdomain: my-app upstream: url: http://web:8080 ``` * `protocol` — `http` or `tcp`; * `subdomain` — a subdomain on the Tuna base domain. Instead, you can set `domain` — a domain reserved in your account; * `upstream.url` — a Service in the same namespace and its port. The public address appears in the `URL` column: ```shell kubectl get tunnel web ``` ### A separate key for a tunnel By default, a tunnel connects with a temporary token that the operator issues with its own key. If the tunnel should run under a different key, for example a team key with the team's limits, put that key in a Secret in the same namespace. The key only needs permission to run tunnels (`tunnels:connect`): choose the "For tunnels" preset when you create it. ```shell kubectl create secret generic team-tuna-api-key --from-literal=api-key= ``` ```yaml spec: apiKeySecretRef: name: team-tuna-api-key ``` ## Ingress with path routing An Ingress host can only be a reserved domain: a Domain resource or a domain from the [dashboard](https://my.tuna.am/domains?lang=en). `kubectl get domain my-app` shows the full domain name. ```yaml apiVersion: k8s.tuna.am/v1alpha1 kind: Domain metadata: name: my-app spec: subdomain: my-app --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: my-app spec: ingressClassName: tuna rules: - host: my-app.a.2na.am http: paths: - path: /api pathType: Prefix backend: service: name: api port: number: 8080 - path: / pathType: Prefix backend: service: name: web port: name: http ``` How it works: * each host gets one tunnel; * changes to paths and backends apply without reconnecting; * prefixes match by path segment: `/api` matches `/api/users` but not `/apix`; * the backend receives the public `Host` header along with `X-Forwarded-Host` and `X-Forwarded-Proto`. Limitations: * a backend can only be a Service; * rules without a host and wildcard hosts are skipped; * several Ingresses for the same host are not reconciled with each other. ## Custom domain ```yaml apiVersion: k8s.tuna.am/v1alpha1 kind: Domain metadata: name: api-example-com spec: domain: api.example.com reclaimPolicy: Retain ``` The DNS records to create at your DNS provider appear in `status.dnsRecords`: ```shell kubectl describe domain api-example-com ``` If the DNS check fails, the operator retries it every 5 minutes. You can't switch an existing resource between `subdomain` and `domain` — create a new one. What `reclaimPolicy` does is described in [Existing reservations](#reservations). ## TCP port A TCPEndpoint reserves the port, and a Tunnel with `protocol: tcp` publishes a Service through it: ```yaml apiVersion: k8s.tuna.am/v1alpha1 kind: TCPEndpoint metadata: name: postgres spec: alias: postgres --- apiVersion: k8s.tuna.am/v1alpha1 kind: Tunnel metadata: name: postgres spec: protocol: tcp portOrAlias: postgres upstream: url: tcp://postgres:5432 ``` `kubectl get tcpendpoint postgres` shows the host and the port number. For the full field reference, run `kubectl explain tunnel.spec`, `kubectl explain domain.spec` and `kubectl explain tcpendpoint.spec`. More examples are in the chart's `examples/` directory. ## Existing reservations If a domain or an alias already exists in your account — created in the dashboard or by another cluster — Domain and TCPEndpoint adopt that reservation instead of failing. Such a resource has `status.adopted: true`, and the operator never deletes its reservation. For all other resources, `reclaimPolicy` applies: * `Delete` (default) — release the reservation when the resource is deleted; * `Retain` — keep the reservation in your account. If the reservation is deleted in the dashboard, the operator creates it again. ## Status and errors Each resource reports its state in the `Ready` condition and in Kubernetes events: ```shell kubectl describe tunnel web ``` An Ingress has no conditions, so check its events: `TunnelConnected`, `TunnelStopped`, `RoutesUpdated`, `RuleSkipped`, `BackendSkipped`. ## Security * **Own namespace only.** Tunnel and Ingress publish only Services of their own namespace. IP addresses, external hosts, Services in other namespaces and `ExternalName` Services are rejected. The `agentManager.allowExternalUpstreams=true` parameter lifts the restriction, but then permission to create a Tunnel or an Ingress in any namespace means permission to publish any address reachable from the operator pod. * **A token per tunnel.** Each tunnel has its own temporary token. It can only run tunnels, has no API access, works with a single protocol and holds at most one tunnel at a time. The token is revoked when the resource is deleted. Tokens of resources deleted while the operator was down are revoked automatically. * **Minimal permissions in the cluster.** The pods run as a non-root user, with a read-only root filesystem and no capabilities. Only agent-manager reads Secrets and Services, and only with `get`. * **Traffic goes through the pod.** Tunnels run inside the agent-manager pod: traffic goes from Tuna through the pod to the Service. agent-manager has a single replica, so tunnels reconnect when you upgrade the operator. ## Upgrading **Nelm** Run the same `nelm release plan install` and `nelm release install` with a new `--chart-version`: CRDs are updated together with the release. **Helm** Helm doesn't upgrade CRDs, so apply them before upgrading the chart: ```shell helm pull tuna/tuna-k8s-operator --untar --version kubectl apply -f tuna-k8s-operator/crds/ helm upgrade tuna-operator tuna/tuna-k8s-operator --namespace tuna-system --reuse-values --version ``` ## Uninstalling Delete the resources first: Domain and TCPEndpoint have finalizers, and they need a running operator. ```shell kubectl delete tunnels,domains,tcpendpoints --all --all-namespaces nelm release uninstall -n tuna-system -r tuna-operator # or: helm uninstall tuna-operator -n tuna-system kubectl delete crd tunnels.k8s.tuna.am domains.k8s.tuna.am tcpendpoints.k8s.tuna.am ```