Kubernetes is a very complex product where creating and managing clusters requires a great deal of knowledge on a wide range of topics. The introduction of managed clusters brought simplicity to the process allowing users to focus on extracting the most out of the system.
One of the areas of most interest and different configurations is authentication and authorization. In authentication, the main objective, and most critical of all, is to ensure the identity and validity of users and machines. How does Kubernetes validate, for example, X.509 certificates or service account tokens? How can the cluster ensure that a node joining has been authorized and is not an imposter?
Kubernetes has a wide range of authentication methods, and we will detail each of them and their uses throughout this article.
When users manage clusters with different nodes, they need a simple way to ensure the authenticity of each of these nodes during the cluster join process. Bootstrap tokens are a type of secret that allows nodes to authenticate themselves for the first time and request Kubernetes to sign certificates so in the future authentication occurs using X.509 certificates.
A Bootstrap token is a 23-character bearer token whose composition contains two distinct parts separated by a dot: {token-id}.{token-secret}. The first part, with the size of six characters, is considered the public part of the key that identifies the token. The second component, {token-secret}, acts as the secret and with the identifier guarantees access to the API. Kubernetes stores Bootstrap tokens using Kubernetes Secret objects whose name follows the following format bootstrap-token-{token-id}. The secret’s content includes the token-secret value. The token’s expiration date is also stored in the secret content and allows defining date after which the token becomes invalid.
In addition to the default node joining policies, it is possible to grant other permissions — e.g., ingress permissions — to members using these tokens. However, these groups must have a particular name starting with system:bootstrappers:.
When a node uses the Bootstrap token for authentication, it sends the token in the Authorization header. The API checks if the token matches a Bootstrap token format and confirms the existence of the secret bootstrap-token-{token-id}. If it exists and the {token-secret} matches the expected value, the token is validated, and the node is allowed to join.
Static tokens are a type of authentication where administrators define a set of API access tokens without validity configured. These credentials have metadata associated that allow picking the user and RBAC groups to be used during the authorization process.
Static token configuration is achieved by setting up a CSV file on the API server with the following format.
After configuration, users can use the tokens as Bearer tokens in API requests.
Since no revocation method exists and any modification to the csv file requires the kube-apiserver to restart, this authentication method is highly discouraged and exists only for the simplicity of configuration.
The Kubernetes API has an authorization mode that fully delegates the authentication process to a proxy server. In this mode, the API receives requests from an intermediate proxy that manages authentication and sends the details associated with the user — including the username or RBAC group — via HTTP headers.
Since the Kubernetes API server has no way to validate the content of the headers, the only way to secure the platform against header spoofing attacks is to ensure their provenance. To prevent header spoofing attacks, the authenticating proxy must present a valid X.509 client certificate to the API server for validation against the configured CA.
Once the API server authenticates the proxy server, the request is authorized based on the content received in the headers.
This authentication method is suitable when you leverage an existing authentication infrastructure that contains Kubernetes authorization principals, but it requires access to the kube-apiserver settings and you need to manage a different certificate authority.
Continuing with the integration of external authentication management services, Webhook Token Authentication allows you to configure authorization with third-party generated tokens.
In this authentication mode, API requests contain a token generated by external services. Since the token does not come from Kubernetes and Kubernetes cannot validate them with OIDC mechanisms, the cluster must call a remote service to check their trustworthiness. Therefore, the API server will validate each token received by calling a third-party service validation endpoint. This service is responsible for returning the user details such as username, user UID and user groups for authorization with Kubernetes RBAC when the token is valid.
To ensure the security of the token validation process, the API server can be configured to perform authentication to the third-party service, including mutual TLS (mTLS).
This authentication method is convenient if you need to integrate Kubernetes access to services that already use third-party tokens for different workloads.
OpenID Connect (OIDC) is an extension of the OAuth 2.0 authentication standard. The main change in the protocol compared to OAuth2 is an additional field returned with the access token called the ID Token. This token contains various information about the user; namely the email, user ID, groups, etc. The ID Token is encoded as a JSON Web Token (JWT) and signed by a third party with their private key. While the private key is private, the public key is available at a well-known URI — /.well-known/openid-configuration — and is used to validate the token.
The introduction of support for this authentication method in Kubernetes was intended to solve a problem with the local authentication of remote users. This means that the OIDC authentication layer can validate users if they present an unexpired and valid JWT token without invoking a third-party service such as GitHub or other auth providers.
When using OIDC, the external entity returns a signed token, usually JSON Web Tokens (JWT). In its compact form, a JWT contains three parts separated by periods (.), which are:
The header identifies the token type and the algorithm used to sign the data. The payload includes the user claims we want to carry, including the RBAC user and RBAC group claims for Kubernetes authorization. Finally, the signature is the cryptographic signature of the header and payload data. This signature is generated using the private key and then verified using the public key in the Kubernetes API. It ensures that the information in the token originates from the authentication service and is not subsequently tampered with by malicious users.
With an OIDC integration, you can use your identity provider infrastructure to access your Kubernetes cluster, although it is fairly complex to grant different levels of access per cluster. It is also impossible to revoke tokens, and once issued, they are valid until expiration date is reached.
Service accounts are the basic tool for controlling pod access to the Kubernetes API. When you create a pod in a cluster, a service account is created for it, allowing the Kubernetes control plane to identify the pod, and thus be able to control what each one is allowed to do.
When a pod starts, Kubernetes mounts a Service Account token at a predefined location /var/run/secrets/kubernetes.io/serviceaccount/token. This JWT token was signed by Kubernetes Certificate Authority and contains information about the service account such as Kubernetes Namespace, Service Account Name, and Service Account ID. When your application uses a Kubernetes client, the client will load the token present at the well-known location and will use it to authenticate to the Kubernetes API by sending it as an Authorization header.
Below you can find an example of an SA token in its compact format.
When decoded, the SA token contains the following content.
Once the Kubernetes API receives a Bearer token with a Service Account payload inside, it validates the existence of the service account and checks if the system has any RBAC rules attached that allow/deny the current action.
As of Kubernetes 1.21, there is no longer an automatic creation of SA secrets, and tokens are linked to the Service account and to the pod that owns them. This means that each time a pod is removed, its Service Account token will no longer be valid as long as access from other pods remains untouched.
X.509 certificates are the most widely used PKI ecosystem certificates and are the basis of secure communications on the Internet. They allow not only to keep the content of the communication private but also to guarantee the identity of the actors.
In Kubernetes, certificates are used to authenticate users and machines. For example, they are used to ensure that the different components that are part of a cluster such as kubelet, kube-controller, and kube-scheduler can authenticate themselves to the API.
To use X.509 for authentication, the configuration of the API server requires a Certificate Authority(CA) configured. This CA is used to sign new certificates and to validate clients when they present their certificates.
When a client establishes a connection to the API, it submits its certificate to be validated by the API server (mTLS). When the server validates that the presented certificate was signed by a trusted CA, it extracts the username from the “Common Name” field and the RBAC groups from the Organization fields. Then, the authorization layer validates access to the desired resource based on the details extracted from the certificate.
An example of a certificate for the user foo. This certificate is valid for 24h and grants the permissions of the bar group.
Although X.509 is the most secure way to access clusters, at the time of this writing, it is still not possible to revoke certificates issued by the cluster (Kubernetes Issue #18982). This means that if someone stole a certificate, it will remain valid until its expiration unless the administrator replaces the entire CA chain in the cluster. It is a highly complex process and requires replacing all certificates in the cluster, including for Kubernetes components.
One method to reduce the risk is to use certificates with a short expiration date, however Kubernetes does not support extension of previously signed certificates. Kubernetes API requires a new signing request to be made and approved, which makes adoption very complicated.
Impersonation is a capability that allows machines or users to behave with rules associated with third-party groups or users.
When a user has access to impersonation mode and sends the correct headers, access to the specified resource is not evaluated by their access rules; instead, it is evaluated based on the users or groups that are impersonated.
The behavior of a request with impersonation is as follows:
This method is widely used, including by Remoteler, in configurations where cluster access is managed through a proxy but where it is not possible to change the API settings.
As we have seen throughout this article, user authentication on Kubernetes clusters is a complex topic that has many challenges and problems. The simplest authentication methods are highly insecure while the most secure are highly difficult to manage.
There is widespread interest in enterprise adoption of Kubernetes. However, due to the complexity, many clusters are configured insecurely. A recent study by Cyble found more than 900,000 Kubernetes clusters exposed and accessible over the Internet. Some of them with weak authentication configurations.
Remoteler Kubernetes Access aims to solve all the above problems as it secures the cluster using kubectl, ensures user identity, manages role-based access control in a centralized way while allowing you to control and review each user action through Audit logging.