fidzu : kubernets

The Kubernetes Container Runtime Interface (CRI) acts as the main connection between the kubelet and the Container Runtime. Those runtimes have to provide a gRPC server which has to fulfill a Kubernetes defined Protocol Buffer interface. This API definition evolves over time, for example when contributors add new features or fields are going to become deprecated.

In this blog post, I'd like to dive into the functionality and history of three extraordinary Remote Procedure Calls (RPCs), which are truly outstanding in terms of how they work: Exec, Attach and PortForward.

Exec can be used to run dedicated commands within the container and stream the output to a client like kubectl or crictl. It also allows interaction with that process using standard input (stdin), for example if users want to run a new shell instance within an existing workload.

Attach streams the output of the currently running process via standard I/O from the container to the client and also allows interaction with them. This is particularly useful if users want to see what is going on in the container and be able to interact with the process.

PortForward can be utilized to forward a port from the host to the container to be able to interact with it using third party network tools. This allows it to bypass Kubernetes services for a certain workload and interact with its network interface.

What is so special about them?

All RPCs of the CRI either use the gRPC unary calls for communication or the server side streaming feature (only GetContainerEvents right now). This means that mainly all RPCs retrieve a single client request and have to return a single server response. The same applies to Exec, Attach, and PortForward, where their protocol definition looks like this:

// Exec prepares a streaming endpoint to execute a command in the container.
rpc Exec(ExecRequest) returns (ExecResponse) {}

// Attach prepares a streaming endpoint to attach to a running container.
rpc Attach(AttachRequest) returns (AttachResponse) {}

// PortForward prepares a streaming endpoint to forward ports from a PodSandbox.
rpc PortForward(PortForwardRequest) returns (PortForwardResponse) {}

The requests carry everything required to allow the server to do the work, for example, the ContainerId or command (Cmd) to be run in case of Exec. More interestingly, all of their responses only contain a url:

message ExecResponse {
 // Fully qualified URL of the exec streaming server.
 string url = 1;
}

message AttachResponse {
 // Fully qualified URL of the attach streaming server.
 string url = 1;
}

message PortForwardResponse {
 // Fully qualified URL of the port-forward streaming server.
 string url = 1;
}

Why is it implemented like that? Well, the original design document for those RPCs even predates Kubernetes Enhancements Proposals (KEPs) and was originally outlined back in 2016. The kubelet had a native implementation for Exec, Attach, and PortForward before the initiative to bring the functionality to the CRI started. Before that, everything was bound to Docker or the later abandoned container runtime rkt.

The CRI related design document also elaborates on the option to use native RPC streaming for exec, attach, and port forward. The downsides outweighed this approach: the kubelet would still create a network bottleneck and future runtimes would not be free in choosing the server implementation details. Also, another option that the Kubelet implements a portable, runtime-agnostic solution has been abandoned over the final one, because this would mean another project to maintain which nevertheless would be runtime dependent.

This means, that the basic flow for Exec, Attach and PortForward was proposed to look like this:

Clients like crictl or the kubelet (via kubectl) request a new exec, attach or port forward session from the runtime using the gRPC interface. The runtime implements a streaming server that also manages the active sessions. This streaming server provides an HTTP endpoint for the client to connect to. The client upgrades the connection to use the SPDY streaming protocol or (in the future) to a WebSocket connection and starts to stream the data back and forth.

This implementation allows runtimes to have the flexibility to implement Exec, Attach and PortForward the way they want, and also allows a simple test path. Runtimes can change the underlying implementation to support any kind of feature without having a need to modify the CRI at all.

Many smaller enhancements to this overall approach have been merged into Kubernetes in the past years, but the general pattern has always stayed the same. The kubelet source code transformed into a reusable library, which is nowadays usable from container runtimes to implement the basic streaming capability.

How does the streaming actually work?

At a first glance, it looks like all three RPCs work the same way, but that's not the case. It's possible to group the functionality of Exec and Attach, while PortForward follows a distinct internal protocol definition.

Exec and Attach

Kubernetes defines Exec and Attach as remote commands, where its protocol definition exists in five different versions:

On top of that, there is an overall effort to replace the SPDY transport protocol using WebSockets as part KEP #4006. Runtimes have to satisfy those protocols over their life cycle to stay up to date with the Kubernetes implementation.

Let's assume that a client uses the latest (v5) version of the protocol as well as communicating over WebSockets. In that case, the general flow would be:

1. The client requests an URL endpoint for Exec or Attach using the CRI.

   • The server (runtime) validates the request, inserts it into a connection tracking cache, and provides the HTTP endpoint URL for that request.

2. The client connects to that URL, upgrades the connection to establish a WebSocket, and starts to stream data.

   • In the case of Attach, the server has to stream the main container process data to the client.
   • In the case of Exec, the server has to create the subprocess command within the container and then streams the output to the client.

   If stdin is required, then the server needs to listen for that as well and redirect it to the corresponding process.

Interpreting data for the defined protocol is fairly simple: The first byte of every input and output packet defines the actual stream:

How should runtimes now implement the streaming server methods for Exec and Attach by using the provided kubelet library? The key is that the streaming server implementation in the kubelet outlines an interface called Runtime which has to be fulfilled by the actual container runtime if it wants to use that library:

// Runtime is the interface to execute the commands and provide the streams.
type Runtime interface {
 Exec(ctx context.Context, containerID string, cmd []string, in io.Reader, out, err io.WriteCloser, tty bool, resize <-chan remotecommand.TerminalSize) error
 Attach(ctx context.Context, containerID string, in io.Reader, out, err io.WriteCloser, tty bool, resize <-chan remotecommand.TerminalSize) error
 PortForward(ctx context.Context, podSandboxID string, port int32, stream io.ReadWriteCloser) error
}

Everything related to the protocol interpretation is already in place and runtimes only have to implement the actual Exec and Attach logic. For example, the container runtime CRI-O does it like this pseudo code:

func (s StreamService) Exec(
 ctx context.Context,
 containerID string,
 cmd []string,
 stdin io.Reader, stdout, stderr io.WriteCloser,
 tty bool,
 resizeChan <-chan remotecommand.TerminalSize,
) error {
 // Retrieve the container by the provided containerID
 // …

 // Update the container status and verify that the workload is running
 // …

 // Execute the command and stream the data
 return s.runtimeServer.Runtime().ExecContainer(
 s.ctx, c, cmd, stdin, stdout, stderr, tty, resizeChan,
 )
}

PortForward

Forwarding ports to a container works a bit differently when comparing it to streaming IO data from a workload. The server still has to provide a URL endpoint for the client to connect to, but then the container runtime has to enter the network namespace of the container, allocate the port as well as stream the data back and forth. There is no simple protocol definition available like for Exec or Attach. This means that the client will stream the plain SPDY frames (with or without an additional WebSocket connection) which can be interpreted using libraries like moby/spdystream.

Luckily, the kubelet library already provides the PortForward interface method which has to be implemented by the runtime. CRI-O does that by (simplified):

func (s StreamService) PortForward(
 ctx context.Context,
 podSandboxID string,
 port int32,
 stream io.ReadWriteCloser,
) error {
 // Retrieve the pod sandbox by the provided podSandboxID
 sandboxID, err := s.runtimeServer.PodIDIndex().Get(podSandboxID)
 sb := s.runtimeServer.GetSandbox(sandboxID)
 // …

 // Get the network namespace path on disk for that sandbox
 netNsPath := sb.NetNsPath()
 // …

 // Enter the network namespace and stream the data
 return s.runtimeServer.Runtime().PortForwardContainer(
 ctx, sb.InfraContainer(), netNsPath, port, stream,
 )
}

Future work

The flexibility Kubernetes provides for the RPCs Exec, Attach and PortForward is truly outstanding compared to other methods. Nevertheless, container runtimes have to keep up with the latest and greatest implementations to support those features in a meaningful way. The general effort to support WebSockets is not only a plain Kubernetes thing, it also has to be supported by container runtimes as well as clients like crictl.

For example, crictl v1.30 features a new --transport flag for the subcommands exec, attach and port-forward (#1383, #1385) to allow choosing between websocket and spdy.

CRI-O is going an experimental path by moving the streaming server implementation into conmon-rs (a substitute for the container monitor conmon). conmon-rs is a Rust implementation of the original container monitor and allows streaming WebSockets directly using supported libraries (#2070). The major benefit of this approach is that CRI-O does not even have to be running while conmon-rs can keep active Exec, Attach and PortForward sessions open. The simplified flow when using crictl directly will then look like this:

All of those enhancements require iterative design decisions, while the original well-conceived implementation acts as the foundation for those. I really hope you've enjoyed this compact journey through the history of CRI RPCs. Feel free to reach out to me anytime for suggestions or feedback using the official Kubernetes Slack.

01 May 2024 12:00am GMT

With the release of Kubernetes 1.30, the feature to prevent the modification of the volume mode of a PersistentVolumeClaim that was created from an existing VolumeSnapshot in a Kubernetes cluster, has moved to GA!

The problem

The Volume Mode of a PersistentVolumeClaim refers to whether the underlying volume on the storage device is formatted into a filesystem or presented as a raw block device to the Pod that uses it.

Users can leverage the VolumeSnapshot feature, which has been stable since Kubernetes v1.20, to create a PersistentVolumeClaim (shortened as PVC) from an existing VolumeSnapshot in the Kubernetes cluster. The PVC spec includes a dataSource field, which can point to an existing VolumeSnapshot instance. Visit Create a PersistentVolumeClaim from a Volume Snapshot for more details on how to create a PVC from an existing VolumeSnapshot in a Kubernetes cluster.

When leveraging the above capability, there is no logic that validates whether the mode of the original volume, whose snapshot was taken, matches the mode of the newly created volume.

This presents a security gap that allows malicious users to potentially exploit an as-yet-unknown vulnerability in the host operating system.

There is a valid use case to allow some users to perform such conversions. Typically, storage backup vendors convert the volume mode during the course of a backup operation, to retrieve changed blocks for greater efficiency of operations. This prevents Kubernetes from blocking the operation completely and presents a challenge in distinguishing trusted users from malicious ones.

Preventing unauthorized users from converting the volume mode

In this context, an authorized user is one who has access rights to perform update or patch operations on VolumeSnapshotContents, which is a cluster-level resource.
It is up to the cluster administrator to provide these rights only to trusted users or applications, like backup vendors. Users apart from such authorized ones will never be allowed to modify the volume mode of a PVC when it is being created from a VolumeSnapshot.

To convert the volume mode, an authorized user must do the following:

1. Identify the VolumeSnapshot that is to be used as the data source for a newly created PVC in the given namespace.
2. Identify the VolumeSnapshotContent bound to the above VolumeSnapshot.

kubectl describe volumesnapshot -n <namespace> <name>

3. Add the annotation snapshot.storage.kubernetes.io/allow-volume-mode-change: "true" to the above VolumeSnapshotContent. The VolumeSnapshotContent annotations must include one similar to the following manifest fragment:

kind: VolumeSnapshotContent
metadata:
 annotations:
 - snapshot.storage.kubernetes.io/allow-volume-mode-change: "true"
...

Note: For pre-provisioned VolumeSnapshotContents, you must take an extra step of setting spec.sourceVolumeMode field to either Filesystem or Block, depending on the mode of the volume from which this snapshot was taken.

An example is shown below:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotContent
metadata:
 annotations:
 - snapshot.storage.kubernetes.io/allow-volume-mode-change: "true"
 name: <volume-snapshot-content-name>
spec:
 deletionPolicy: Delete
 driver: hostpath.csi.k8s.io
 source:
 snapshotHandle: <snapshot-handle>
 sourceVolumeMode: Filesystem
 volumeSnapshotRef:
 name: <volume-snapshot-name>
 namespace: <namespace>

Repeat steps 1 to 3 for all VolumeSnapshotContents whose volume mode needs to be converted during a backup or restore operation. This can be done either via software with credentials of an authorized user or manually by the authorized user(s).

If the annotation shown above is present on a VolumeSnapshotContent object, Kubernetes will not prevent the volume mode from being converted. Users should keep this in mind before they attempt to add the annotation to any VolumeSnapshotContent.

Action required

The prevent-volume-mode-conversion feature flag is enabled by default in the external-provisioner v4.0.0 and external-snapshotter v7.0.0. Volume mode change will be rejected when creating a PVC from a VolumeSnapshot unless the steps described above have been performed.

What's next

To determine which CSI external sidecar versions support this feature, please head over to the CSI docs page. For any queries or issues, join Kubernetes on Slack and create a thread in the #csi or #sig-storage channel. Alternately, create an issue in the CSI external-snapshotter repository.

30 Apr 2024 12:00am GMT

With Kubernetes 1.30, we (SIG Auth) are moving Structured Authorization Configuration to beta.

Today's article is about authorization: deciding what someone can and cannot access. Check a previous article from yesterday to find about what's new in Kubernetes v1.30 around authentication (finding out who's performing a task, and checking that they are who they say they are).

Introduction

Kubernetes continues to evolve to meet the intricate requirements of system administrators and developers alike. A critical aspect of Kubernetes that ensures the security and integrity of the cluster is the API server authorization. Until recently, the configuration of the authorization chain in kube-apiserver was somewhat rigid, limited to a set of command-line flags and allowing only a single webhook in the authorization chain. This approach, while functional, restricted the flexibility needed by cluster administrators to define complex, fine-grained authorization policies. The latest Structured Authorization Configuration feature (KEP-3221) aims to revolutionize this aspect by introducing a more structured and versatile way to configure the authorization chain, focusing on enabling multiple webhooks and providing explicit control mechanisms.

The Need for Improvement

Cluster administrators have long sought the ability to specify multiple authorization webhooks within the API Server handler chain and have control over detailed behavior like timeout and failure policy for each webhook. This need arises from the desire to create layered security policies, where requests can be validated against multiple criteria or sets of rules in a specific order. The previous limitations also made it difficult to dynamically configure the authorizer chain, leaving no room to manage complex authorization scenarios efficiently.

The Structured Authorization Configuration feature addresses these limitations by introducing a configuration file format to configure the Kubernetes API Server Authorization chain. This format allows specifying multiple webhooks in the authorization chain (all other authorization types are specified no more than once). Each webhook authorizer has well-defined parameters, including timeout settings, failure policies, and conditions for invocation with CEL rules to pre-filter requests before they are dispatched to webhooks, helping you prevent unnecessary invocations. The configuration also supports automatic reloading, ensuring changes can be applied dynamically without restarting the kube-apiserver. This feature addresses current limitations and opens up new possibilities for securing and managing Kubernetes clusters more effectively.

Sample Configurations

Here is a sample structured authorization configuration along with descriptions for all fields, their defaults, and possible values.

apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthorizationConfiguration
authorizers:
 - type: Webhook
 # Name used to describe the authorizer
 # This is explicitly used in monitoring machinery for metrics
 # Note:
 # - Validation for this field is similar to how K8s labels are validated today.
 # Required, with no default
 name: webhook
 webhook:
 # The duration to cache 'authorized' responses from the webhook
 # authorizer.
 # Same as setting `--authorization-webhook-cache-authorized-ttl` flag
 # Default: 5m0s
 authorizedTTL: 30s
 # The duration to cache 'unauthorized' responses from the webhook
 # authorizer.
 # Same as setting `--authorization-webhook-cache-unauthorized-ttl` flag
 # Default: 30s
 unauthorizedTTL: 30s
 # Timeout for the webhook request
 # Maximum allowed is 30s.
 # Required, with no default.
 timeout: 3s
 # The API version of the authorization.k8s.io SubjectAccessReview to
 # send to and expect from the webhook.
 # Same as setting `--authorization-webhook-version` flag
 # Required, with no default
 # Valid values: v1beta1, v1
 subjectAccessReviewVersion: v1
 # MatchConditionSubjectAccessReviewVersion specifies the SubjectAccessReview
 # version the CEL expressions are evaluated against
 # Valid values: v1
 # Required, no default value
 matchConditionSubjectAccessReviewVersion: v1
 # Controls the authorization decision when a webhook request fails to
 # complete or returns a malformed response or errors evaluating
 # matchConditions.
 # Valid values:
 # - NoOpinion: continue to subsequent authorizers to see if one of
 # them allows the request
 # - Deny: reject the request without consulting subsequent authorizers
 # Required, with no default.
 failurePolicy: Deny
 connectionInfo:
 # Controls how the webhook should communicate with the server.
 # Valid values:
 # - KubeConfig: use the file specified in kubeConfigFile to locate the
 # server.
 # - InClusterConfig: use the in-cluster configuration to call the
 # SubjectAccessReview API hosted by kube-apiserver. This mode is not
 # allowed for kube-apiserver.
 type: KubeConfig
 # Path to KubeConfigFile for connection info
 # Required, if connectionInfo.Type is KubeConfig
 kubeConfigFile: /kube-system-authz-webhook.yaml
 # matchConditions is a list of conditions that must be met for a request to be sent to this
 # webhook. An empty list of matchConditions matches all requests.
 # There are a maximum of 64 match conditions allowed.
 #
 # The exact matching logic is (in order):
 # 1. If at least one matchCondition evaluates to FALSE, then the webhook is skipped.
 # 2. If ALL matchConditions evaluate to TRUE, then the webhook is called.
 # 3. If at least one matchCondition evaluates to an error (but none are FALSE):
 # - If failurePolicy=Deny, then the webhook rejects the request
 # - If failurePolicy=NoOpinion, then the error is ignored and the webhook is skipped
 matchConditions:
 # expression represents the expression which will be evaluated by CEL. Must evaluate to bool.
 # CEL expressions have access to the contents of the SubjectAccessReview in v1 version.
 # If version specified by subjectAccessReviewVersion in the request variable is v1beta1,
 # the contents would be converted to the v1 version before evaluating the CEL expression.
 #
 # Documentation on CEL: https://kubernetes.io/docs/reference/using-api/cel/
 #
 # only send resource requests to the webhook
 - expression: has(request.resourceAttributes)
 # only intercept requests to kube-system
 - expression: request.resourceAttributes.namespace == 'kube-system'
 # don't intercept requests from kube-system service accounts
 - expression: !('system:serviceaccounts:kube-system' in request.user.groups)
 - type: Node
 name: node
 - type: RBAC
 name: rbac
 - type: Webhook
 name: in-cluster-authorizer
 webhook:
 authorizedTTL: 5m
 unauthorizedTTL: 30s
 timeout: 3s
 subjectAccessReviewVersion: v1
 failurePolicy: NoOpinion
 connectionInfo:
 type: InClusterConfig

The following configuration examples illustrate real-world scenarios that need the ability to specify multiple webhooks with distinct settings, precedence order, and failure modes.

Protecting Installed CRDs

Ensuring of Custom Resource Definitions (CRDs) availability at cluster startup has been a key demand. One of the blockers of having a controller reconcile those CRDs is having a protection mechanism for them, which can be achieved through multiple authorization webhooks. This was not possible before as specifying multiple authorization webhooks in the Kubernetes API Server authorization chain was simply not possible. Now, with the Structured Authorization Configuration feature, administrators can specify multiple webhooks, offering a solution where RBAC falls short, especially when denying permissions to 'non-system' users for certain CRDs.

Assuming the following for this scenario:

• The "protected" CRDs are installed.
• They can only be modified by users in the group admin.

apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthorizationConfiguration
authorizers:
 - type: Webhook
 name: system-crd-protector
 webhook:
 unauthorizedTTL: 30s
 timeout: 3s
 subjectAccessReviewVersion: v1
 matchConditionSubjectAccessReviewVersion: v1
 failurePolicy: Deny
 connectionInfo:
 type: KubeConfig
 kubeConfigFile: /files/kube-system-authz-webhook.yaml
 matchConditions:
 # only send resource requests to the webhook
 - expression: has(request.resourceAttributes)
 # only intercept requests for CRDs
 - expression: request.resourceAttributes.resource.resource = "customresourcedefinitions"
 - expression: request.resourceAttributes.resource.group = ""
 # only intercept update, patch, delete, or deletecollection requests
 - expression: request.resourceAttributes.verb in ['update', 'patch', 'delete','deletecollection']
 - type: Node
 - type: RBAC

Preventing unnecessarily nested webhooks

A system administrator wants to apply specific validations to requests before handing them off to webhooks using frameworks like Open Policy Agent. In the past, this would require running nested webhooks within the one added to the authorization chain to achieve the desired result. The Structured Authorization Configuration feature simplifies this process, offering a structured API to selectively trigger additional webhooks when needed. It also enables administrators to set distinct failure policies for each webhook, ensuring more consistent and predictable responses.

apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthorizationConfiguration
authorizers:
 - type: Webhook
 name: system-crd-protector
 webhook:
 unauthorizedTTL: 30s
 timeout: 3s
 subjectAccessReviewVersion: v1
 matchConditionSubjectAccessReviewVersion: v1
 failurePolicy: Deny
 connectionInfo:
 type: KubeConfig
 kubeConfigFile: /files/kube-system-authz-webhook.yaml
 matchConditions:
 # only send resource requests to the webhook
 - expression: has(request.resourceAttributes)
 # only intercept requests for CRDs
 - expression: request.resourceAttributes.resource.resource = "customresourcedefinitions"
 - expression: request.resourceAttributes.resource.group = ""
 # only intercept update, patch, delete, or deletecollection requests
 - expression: request.resourceAttributes.verb in ['update', 'patch', 'delete','deletecollection']
 - type: Node
 - type: RBAC
 - name: opa
 type: Webhook
 webhook:
 unauthorizedTTL: 30s
 timeout: 3s
 subjectAccessReviewVersion: v1
 matchConditionSubjectAccessReviewVersion: v1
 failurePolicy: Deny
 connectionInfo:
 type: KubeConfig
 kubeConfigFile: /files/opa-default-authz-webhook.yaml
 matchConditions:
 # only send resource requests to the webhook
 - expression: has(request.resourceAttributes)
 # only intercept requests to default namespace
 - expression: request.resourceAttributes.namespace == 'default'
 # don't intercept requests from default service accounts
 - expression: !('system:serviceaccounts:default' in request.user.groups)

What's next?

From Kubernetes 1.30, the feature is in beta and enabled by default. For Kubernetes v1.31, we expect the feature to stay in beta while we get more feedback from users. Once it is ready for GA, the feature flag will be removed, and the configuration file version will be promoted to v1.

Learn more about this feature on the structured authorization configuration Kubernetes doc website. You can also follow along with KEP-3221 to track progress in coming Kubernetes releases.

Call to action

In this post, we have covered the benefits of the Structured Authorization Configuration feature in Kubernetes v1.30 and a few sample configurations for real-world scenarios. To use this feature, you must specify the path to the authorization configuration using the --authorization-config command line argument. From Kubernetes 1.30, the feature is in beta and enabled by default. If you want to keep using command line flags instead of a configuration file, those will continue to work as-is. Specifying both --authorization-config and --authorization-modes/--authorization-webhook-* won't work. You need to drop the older flags from your kube-apiserver command.

The following kind Cluster configuration sets that command argument on the APIserver to load an AuthorizationConfiguration from a file (authorization_config.yaml) in the files folder. Any needed kubeconfig and certificate files can also be put in the files directory.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
 StructuredAuthorizationConfiguration: true # enabled by default in v1.30
kubeadmConfigPatches:
 - |
 kind: ClusterConfiguration
 metadata:
 name: config
 apiServer:
 extraArgs:
 authorization-config: "/files/authorization_config.yaml"
 extraVolumes:
 - name: files
 hostPath: "/files"
 mountPath: "/files"
 readOnly: true
nodes:
- role: control-plane
 extraMounts:
 - hostPath: files
 containerPath: /files

We would love to hear your feedback on this feature. In particular, we would like feedback from Kubernetes cluster administrators and authorization webhook implementors as they build their integrations with this new API. Please reach out to us on the #sig-auth-authorizers-dev channel on Kubernetes Slack.

How to get involved

If you are interested in helping develop this feature, sharing feedback, or participating in any other ongoing SIG Auth projects, please reach out on the #sig-auth channel on Kubernetes Slack.

You are also welcome to join the bi-weekly SIG Auth meetings held every other Wednesday.

Acknowledgments

This feature was driven by contributors from several different companies. We would like to extend a huge thank you to everyone who contributed their time and effort to make this possible.

26 Apr 2024 12:00am GMT