Curious Pandas

Who can do what in a cluster

An app gets "forbidden". The one-line fix makes the error vanish and hands its stolen token the whole cluster. Grant exactly enough with RBAC, then see what NetworkPolicy and Pod Security add.

18 min+45 XP for finishing

Every pod runs as a service account, and every request it makes to the Kubernetes API carries that account’s token. So the question “what can this app do?” is really “what could anyone holding this token do?” - because if the app is ever broken into, that’s exactly who will be holding it.

Out of the box the answer is close to nothing. The RBAC docs are explicit: the default policies give “no permissions to service accounts outside the kube-system namespace” beyond basic API discovery. Every permission an app has, somebody granted.

Four objects, one question

Object Says Scope
Role these verbs on these resources one namespace
ClusterRole the same, defined once not namespaced
RoleBinding this subject gets that role one namespace - even when the role is a ClusterRole
ClusterRoleBinding this subject gets that ClusterRole every namespace

Two rules make RBAC easy to reason about. Permissions are purely additive: there are no deny rules, so access is the union of every binding that applies. And a RoleBinding that points at a ClusterRole grants it only in the binding’s own namespace. That’s how you reuse one definition, such as the built-in view, without handing it out everywhere.

Visual guideTwo ways to make 'forbidden' go away
Two ways to make 'forbidden' go awayA service account is linked by a RoleBinding to a Role that allows getting one named ConfigMap. Alternatively it is linked by a ClusterRoleBinding to cluster-admin, which allows any verb on any resource in every namespace.ServiceAccountshop:reporterRoleBindingin shopRolesettings-readerget shop-settingsone name, one verbClusterRoleBindingcluster-widecluster-adminbuilt inany verb, anywhereincl. every SecretsubjectroleRefallowssubjectroleRefallowsTwo ways to make 'forbidden' go awayA service account is linked by a RoleBinding to a Role that allows getting one named ConfigMap. Alternatively it is linked by a ClusterRoleBinding to cluster-admin, which allows any verb on any resource in every namespace.ServiceAccountshop:reporterRoleBindingin shopRolesettings-readerget shop-settingsone name, one verbClusterRoleBindingcluster-widecluster-adminbuilt inany verb, anywhereincl. every SecretsubjectroleRefallowssubjectroleRefallows
  1. The app runs as a service account. On its own it may do almost nothing.
  2. A RoleBinding in the shop namespace names the account as its subject and points at a Role.
  3. The Role allows one verb on one ConfigMap, by name. That is all a stolen token from this pod is worth.
  4. The one-line alternative: a ClusterRoleBinding to cluster-admin. Same error gone - and the token now opens everything.

Just enough access

The reporter app logs a real-format error: it may not get the ConfigMap it needs. Find out what it’s allowed to do, make it work, then make it only work. The audit panel re-checks five questions after every command.

RBAC sandboxnamespace shop · app runs as shop:reporter

App log: configmaps "shop-settings" is forbidden: User "system:serviceaccount:shop:reporter" cannot get resource "configmaps" in API group "" in the namespace "shop"

Least-privilege audit · re-checked after every command

    The finished shape - one verb, one resource, one name, one namespace - isn’t a classroom ideal. It’s exactly the Role tilt’s own config server runs with:

    kind: Role
    metadata:
      name: tilt-config-manager
      namespace: tilt-system
    rules:
      - apiGroups: [""]
        resources: ["configmaps"]
        resourceNames: ["tilt-config"]
        verbs: ["get", "update", "patch"]

    The same platform binds Velero’s service account to cluster-admin. Velero backs up and restores whole namespaces, so it genuinely needs very broad access - which makes its token one of the most valuable in the cluster. (It’s off unless you turn it on in tilt-config.json.)

    Who may talk to whom

    RBAC governs the API: who may read and change Kubernetes objects. It says nothing about network traffic between pods. That’s NetworkPolicy, and its defaults surprise people: a pod that no policy selects is “non-isolated”, and all inbound and outbound connections are allowed.

    Visual guideDefault-open, then one rule
    Default-open, then one ruleWithout a NetworkPolicy every pod can reach the database, including a compromised pod. With a policy selecting the database that allows ingress only from the api pods, the compromised pod's connection is not allowed.webapp=webapiapp=apidbapp=dbcompromised podany labelsHTTPSQLallowed?Default-open, then one ruleWithout a NetworkPolicy every pod can reach the database, including a compromised pod. With a policy selecting the database that allows ingress only from the api pods, the compromised pod's connection is not allowed.webapp=webapiapp=apidbapp=dbcompromised podany labelsHTTPSQLallowed?
    1. web calls api, api calls the database. That is the traffic the app needs.
    2. With no policy selecting db, any pod in the cluster can connect to it - including one an attacker controls.
    3. A policy that selects app=db and allows ingress only from app=api makes db isolated. Every other connection to it is dropped.

    Policies are additive, like RBAC: the traffic allowed is the union of every policy that selects a pod. Selecting a pod at all is what isolates it; the rules then list what’s let back in.

    What pods may be

    The third control is about the pod itself: may it run privileged, mount the host’s disks, run as root? Pod Security admission applies one of three built-in levels - privileged, baseline or restricted - per namespace, with a label:

    metadata:
      labels:
        pod-security.kubernetes.io/enforce: baseline
        pod-security.kubernetes.io/warn: restricted

    Each level can be applied in three modes. enforce rejects the pod. audit records the violation in the audit log and lets it through. warn shows the user a warning and lets it through. It’s the same audit-versus-enforce trap as Kyverno in the policy and scanning lesson: a namespace labelled audit: restricted looks locked down and admits everything.

    Check yourself+10 XP

    An app needs to read one ConfigMap. Which grant is least privilege?

    Check yourself+10 XP

    A RoleBinding in namespace shop points at the ClusterRole edit. Can the subject read Secrets in kube-system?

    Check yourself+10 XP

    You apply a NetworkPolicy that should block traffic to the database, and kubectl shows it was created. What have you proved?

    Do it for real · your machine+50 XP bonus

    Everything above ran in your browser. Now do it on a real cluster withthe tilt platform. You need Docker, kind and Tilt, about 15 GB of free disk and 8 GB of spare RAM. Setup guide · Can my machine run it?

    With the platform up on profile gitops or above, ask the API server about a real service account. The config server’s Role allows exactly one ConfigMap:

    SA=system:serviceaccount:tilt-system:tilt-config-server
    kubectl auth can-i get configmaps/tilt-config -n tilt-system --as=$SA      # yes
    kubectl auth can-i get configmaps/kube-root-ca.crt -n tilt-system --as=$SA  # no: wrong name
    kubectl auth can-i get secrets -n tilt-system --as=$SA                    # no
    kubectl auth can-i --list -n tilt-system --as=$SA

    Then look for the broadest grants on your cluster, and for what’s actually enforcing network and pod rules:

    kubectl get clusterrolebindings -o wide | grep cluster-admin
    kubectl get networkpolicy -A
    kubectl get ns -L pod-security.kubernetes.io/enforce,pod-security.kubernetes.io/audit

    Every cluster-admin line is a token worth stealing. Ask of each one: does it need all of that?

    Self-reported. We can't see your machine, and we don't want to. This is a bonus, never required.

    Sources

    Finished reading?

    Missions and quizzes above earn their own XP. This marks the lesson done.