Challenge 5 ☆☆

Welcome to challenge Challenge 5. You need to guess the secret that is hidden in Java, Docker, Kubernetes, Vault, AWS or GCP.

K8s Configmap based secret

Hardcoding something in docker was not smart. But what about a config-map in the default namespace without RBAC?

If this challenge is enabled, you are running this from a K8s cluster. Can you use kubectl to find the secret? Or what about looking at the repository where the cluster is configured by? You might have spotted it as well if the configmap was committed to Git maybe.

Answer to solution :

You can solve this challenge by the following steps:

  1. Check the status of the configmap in Git

    • Can you see where in git we stored the secrets-config.yml? If not, just do a search.

    • Take a look at the Data field: what can you find there?

  2. Ask nicely using Kubectl:

    • Make sure you have Kubectl installed as defined in the README.MD & make sure kubectl is configured to send commands to the right cluster.

    • Do kubectl get configmap. Here you see all the configmaps active in the namespace you are in, which is for this app normally default (unless otherwise specified by your administrator/trainer).

    • Now do kubectl get configmap secrets-file -o Yaml. Can you see the secret?

  3. Exec into the pod and get the data:

    • Make sure you have Kubectl installed as defined in the README.MD & make sure kubectl is configured to send commands to the right cluster.

    • Do kubectl get pods. Here you see all the Pods active in the namespace you are in, which is for this app normally default (unless otherwise specified by your administrator/trainer).

    • Now for your instance of the WrongSecrets pod, do kubectl exec -it secret-challenge-<rest of the name of the pod from the prev.step> — /bin/sh.

    • Now do env | grep SPECIAL_K8S_SECRET and there is your secret.

BTW: with kubectl get <item> -A gives you an overview of all the items over all the namespaces you have access to. This shows how important it is not to give people access to every namespace in your cluster, as this might mean leaking important config/items to them.

Why using configmaps is not a good idea

Configmaps were never designed to host secrets. Instead, they were designed to host additional data for your Job or Pod and other K8s resources. Configmaps are often readable for everyone who needs to develop/operate the processes which require data from the configmap. This could still mean that you put restrictions on who can read the configmap using RBAC. Maybe it is a good idea in production to limit who can read the configmap: to start, how about "not everyone"? Next, configmaps are often stored in Git to speed up development, which means overall that the data is quite exposed.

Given that a configmap just works with strings, you cannot easily encrypt it, other than through "rolling your own" on top of a configmap. Of course, you can encrypt the storage of where the configmap is hosted in after deployment, but that means it is still readable in Git.

This makes a configmap not a very suitable item to place a secret in, especially without strict RBAC: many people within the organization that have access to it, can still read it.

So: try not use configmaps for secrets & make sure you apply RBAC properly!

Last but not least: we could easily exec into the container, to grep the ENV vars with the secret. This has to do with 3 things:

  • we are allowed to do so by means of RBAC, which should not be your normal case in PRD: otherwise everybody of your organization can poke around in the container.

  • we have executables within the container (sh/openssl/etc) which we can execute to setup a shell. Stripping your container from any non-necessary binary can help to reduce attack-surface and make it harder for any attacker that did an RCE at your container to jump to other places within the container to further gain execution.

  • we have exposed the configmap as an ENV. This means that anybody who got to the container runtime within the pod can now dump the secret. We brought the secret close to the consumer, but maybe not close enough yet (e.g. the app only).

Detailed explanation of why RBAC is important and example

  • Let’s assume we have a secure-middleware namespace and we have two deployments running inside redis and ngrok

  • Let’s assume we have 4 secrets in the secure-middleware namespace, and 1 of them is redis-master-secret which only should be accessed by redis deployment, and 2 others (defualt, xyz). Then we have the ngrok-api-key secret which only should be accessed by the ngrok service.

  • But due to poorly granular access the user was given following RBAC role for ServiceAccount

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: secure-middleware
  name: ngrok-api-key
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]
  • So if the attacker compromised ngrok service or someone who doesn’t have redis access within that team can still steal other stuff.

  • As the resources: ["secrets"] gives access to all the namespace level secrets which means they can read 4 secrets from this pod ServiceAccount.

export APISERVER=https://${KUBERNETES_SERVICE_HOST}
export SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount
export NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)
export TOKEN=$(cat ${SERVICEACCOUNT}/token)
export CACERT=${SERVICEACCOUNT}/ca.crt
curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api/v1/namespaces/${NAMESPACE}/secrets/redis-master-secret
  • So the RBAC policy ideally should as be below

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: secure-middleware
  name: ngrok-api-key
rules:
- apiGroups: [""]
  resourceNames: ["ngrok-api-key"]
  verbs: ["get", "watch", "list"]

0