Background
I recently worked on a project where I needed an infrequently accessed key-value store for my application. Since I was building this to run in Kubernetes, I decided that I could just use a ConfigMap because this was already available to me as a native resource. This would allow my application to remain stateless while keeping important pieces of configuration stored separately, but accessible via the Kubernetes API. All I had to do was figure out how to get my pods to authenticate with the Kubernetes API. I learned a lot in the process, so I figured I would share my findings!
Service Accounts
Service accounts are designed for automated processes and pods, and can be used for a variety of tasks. For this blog however, I’ll be focusing on how they can be used for authentication and RBAC with the Kubernetes API.
For further reading, here is some background information on service accounts:
- https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/
- https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
By default, when you run a pod in Kubernetes, it uses the default service account and mounts into the pod important information to authenticate to the Kubernetes API.
Example 1:
Setup:
Start by creating a minikube cluster (or any custer for that matter) and run the following:
kubectl run -it --rm ubuntu --image=ubuntu:latest
If you don't see a command prompt, try pressing enter.
root@ubuntu:/#
Now, we’ll look at the files that Kubernetes mounted into the pod for the default service account:
root@ubuntu:/# ls /var/run/secrets/kubernetes.io/serviceaccount/
ca.crt namespace token
To show that this is actually the default service account, we can cat the “token” file. If we take the output and paste it into jwt.io you can see the JWT and metadata showing this is a Kubernetes “default” service account.
cat var/run/secrets/kubernetes.io/serviceaccount/token
Making a Request:
Now that we have all the pieces we need, we can make a request to the Kubernetes API. We’ll just install a few tools into our Ubuntu pod so we can more easily make the request.
apt update && apt install -y dnsutils curl jq
First, we have to figure out where to make the request. Luckily, Kubernetes has a service endpoint for the API intuitively called kubernetes. To see this, we can do an nslookup and see that the cluster kube-dns will resolve this for us.
root@ubuntu:/# nslookup kubernetes
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: kubernetes.default.svc.cluster.local
Address: 10.96.0.1
You can also see this with kubectl:
kubectl get svc
Now that we know we have a DNS entry for the Kubernetes API, we can try making a request with curl (do this inside the Ubuntu pod).
TOKEN=$(cat /run/secrets/kubernetes.io/serviceaccount/token)
curl -s https://kubernetes -H "Authorization: Bearer $TOKEN" --cacert /run/secrets/kubernetes.io/serviceaccount/ca.crt | jq
The output should look something like this:
root@ubuntu:/# curl -s https://kubernetes -H "Authorization: Bearer $TOKEN" --cacert /run/secrets/kubernetes.io/serviceaccount/ca.crt | jq
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "forbidden: User \"system:serviceaccount:default:default\" cannot get path \"/\"",
"reason": "Forbidden",
"details": {},
"code": 403
}
This shows us that we authenticated to the Kubernetes API, but we got a message that we (the default service account) are not authorized. One way around this would be to give the default service account RBAC permissions. However, for security purposes, we should create an explicit service account. This is important because any pod that you haven’t specified a service account for, will use the default service account. That means if you give permissions to the default service account, you may be giving permissions to all your other workloads as well. In other words, be careful giving RBAC permissions to the default service account unless you want all your workloads to have the same permissions.
Example 2
For this example, we will follow almost the same process, but create a service account, role, and rolebinding. I won’t get into all the details of roles, and rolebindings, but this link gives an awesome overview if you want to go deeper on this topic.
Setup:
First, we’ll create the serviceaccount, role, and rolebinding that will allow us to get and list pods from the Kubernetes API. The following is a YAML manifest you can use to set this up. Save it as something like role.yml
#!yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: default
name: podgetter
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: podgetter
namespace: default
subjects:
- kind: ServiceAccount
name: podgetter
roleRef:
kind: Role
name: podgetter
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: podgetter
namespace: default
Then apply it to your cluster:
kubectl apply -f role.yml
Now we’ll run the same pod as before, but this time injecting our custom service account podgetter.
kubectl run -it --rm ubuntu --image=ubuntu:latest --serviceaccount=podgetter
Once you have a prompt in this new pod, we’ll do the same setup and try to list the pods in the Kubernetes API endpoint.
apt update && apt install -y curl jq
TOKEN=$(cat /run/secrets/kubernetes.io/serviceaccount/token)
curl -s https://kubernetes/api/v1/namespaces/default/pods -H "Authorization: Bearer $TOKEN" --cacert /run/secrets/kubernetes.io/serviceaccount/ca.crt | jq
The result should look something like this:
{
"kind": "PodList",
"apiVersion": "v1",
"metadata": {
"selfLink": "/api/v1/namespaces/default/pods",
"resourceVersion": "19905"
},
"items": [
{
"metadata": {
"name": "ubuntu",
"namespace": "default",
"selfLink": "/api/v1/namespaces/default/pods/ubuntu",
"uid": "ee1bcfba-6457-426c-85ff-5de1e6851cde",
"resourceVersion": "19804",
"creationTimestamp": "2020-03-29T22:54:31Z",
"labels": {
"run": "ubuntu"
}
},
"spec": {
"volumes": [
{
"name": "podgetter-token-tkq9s",
"secret": {
"secretName": "podgetter-token-tkq9s",
"defaultMode": 420
}
}
],
"containers": [
{
"name": "ubuntu",
"image": "ubuntu:latest",
...
Since we got a response back, the pod successfully authenticated, and was authorized to access the pod listing endpoint! In our role and rolebinding we only allowed “get” and “list” for pods, so if we tried to curl a different endpoint like ConfigMaps we would get an unauthorized response. For example:
root@ubuntu:/# curl -s https://kubernetes/api/v1/namespaces/default/configmaps -H "Authorization: Bearer $TOKEN" --cacert /run/secrets/kubernetes.io/serviceaccount/ca.crt | jq
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "configmaps is forbidden: User \"system:serviceaccount:default:podgetter\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"default\"",
"reason": "Forbidden",
"details": {
"kind": "configmaps"
},
"code": 403
}
Example 3
This final example may be less practical for a real workload, but can help if you just need to quickly test and easily parse the output from the Kubernetes API by using kubectl. Kubectl will natively look for pod credentials in the /run/secrets/kubernetes.io/serviceaccount/ directory. We can use our same serviceaccount, role, and rolebinding but this time run kubectl:
kubectl run -it --rm kubectl --image=bitnami/kubectl --command --serviceaccount=podgetter -- bash
Examples:
If you don't see a command prompt, try pressing enter.
I have no name!@kubectl:/$ kubectl get pods
NAME READY STATUS RESTARTS AGE
kubectl 1/1 Running 0 43s
I have no name!@kubectl:/$
I have no name!@kubectl:/$ kubectl get configmaps
Error from server (Forbidden): configmaps is forbidden: User "system:serviceaccount:default:podgetter" cannot list resource "configmaps" in API group "" in the namespace "default"
I have no name!@kubectl:/$
Conclusion
In reality you will probably be using a programming language to make requests to the Kubernetes API, and there is a ton of support for this with various client libraries. You can still make these calls to the Kubernetes REST API manually with any HTTP library, but the client libraries will help with the authentication pieces mounted into the pod.
Hopefully some of these examples help your understanding on how the Kubernetes API is just a flexible and well implemented REST API. With the appropriate RBAC and service accounts, you can build on top of the API by making requests from pods.
If there’s anything I can improve on with this blog please feel free to let me know in the comments!