Chapter 2: Understanding the Brain of the Cluster
In the field of cloud-native orchestration and scheduling, Kubernetes has become the de facto standard.
When we try to understand how Kubernetes works, the controller is definitely a difficult point. Several concepts we often hear, such as Declarative API, Operator, Desired State Design, etc., are actually all about the same thing: the Controller Pattern.
The Kubernetes controller pattern is not easy to understand for two reasons: first, there are many controllers, and their specific implementations vary widely, making it impractical to understand each one individually; second, the implementation of controllers uses some rather obscure mechanisms that are difficult to master. However, we cannot avoid the controller pattern because it is the brain of the cluster.
In this chapter, we deeply analyze the design process of a simple refrigerator to understand the origin, evolution, and underlying logic of Kubernetes controllers.
2.1 Viewing the Cluster from the Controller’s Perspective
As shown in Figure 2-1, we can see the simplified architecture of Kubernetes. The architecture diagram includes the cluster’s core database Etcd, the Scheduler, the cluster’s entry point API Server, the Controller, the service proxy Proxy, and the node agent Kubelet that directly manages containers.
From a logical perspective, these components can be divided into three parts: the core component Etcd, the entry component API Server that directly operates on Etcd, and the other components. The reason the other components can be classified together is that they can all be seen as controllers of the cluster.
Figure
Figure 2-1: Simplified architecture of Kubernetes
(Image description: The diagram shows Etcd at the center, API Server above it, and arrows pointing to Controller, Scheduler, Proxy, and Kubelet. Kubelet connects to containers.)
2.2 Origin and Evolution of the Controller
Although the controller is a relatively complex component in Kubernetes, the concept of a controller itself is not unfamiliar to us. The washing machines, refrigerators, air conditioners, etc., we use in daily life all require controllers to function properly.
In this section, we think through the design process of a simple refrigerator to understand the principles and implementation of the Kubernetes cluster controller.
2.2.1 Designing a Refrigerator
Figure 2-2 shows a simple refrigerator. The refrigerator consists of five components: the cabinet, the cooling system, the lighting system, the thermostat, and the door. The refrigerator has two typical usage scenarios: when someone opens the refrigerator door, the light inside automatically turns on; when someone adjusts the thermostat, the cooling system adjusts the temperature inside the refrigerator according to the temperature setting.
Figure
Figure 2-2: A simple refrigerator (only some components are drawn)
(Image description: A simple box-shaped refrigerator with a door, a light bulb inside, a cooling element, and a thermostat knob. Labeled components: cabinet, door, lighting system, cooling system, thermostat.)
2.2.2 Unified Operation Entry
In fact, we can simply abstract the refrigerator into two parts as shown in Figure 2-3: a unified operation entry and the other components. Users can only operate the refrigerator through the operation entry, which provides two interfaces: opening/closing the door and adjusting the thermostat. When the user invokes these two interfaces, the entry logic adjusts the state of the door or the thermostat.
Figure
Figure 2-3: Refrigerator system after adding a unified entry
(Image description: A box labeled “Unified Operation Entry” has two arrows going out: one to “Door” and one to “Thermostat”. There is no connection to the lighting or cooling system.)
However, there is a problem: through these two interfaces, the user can neither turn on the light inside the refrigerator nor adjust the temperature inside the refrigerator, because these two interfaces have no direct relationship with the lighting or cooling systems.
2.2.3 Introducing the Controller
As shown in Figure 2-4, the controller is born to solve the above problem. The controller acts as a bridge between user operations and the states of the various refrigerator components. When the user opens the door, the controller “observes” the change in the door and helps the user turn on the light inside the refrigerator; when the user adjusts the thermostat, the controller “observes” the temperature set by the user and manages the cooling system for the user to adjust the temperature inside the refrigerator.
Figure
Figure 2-4: Refrigerator system with added controller
(Image description: The “Unified Operation Entry” connects to “Door” and “Thermostat”. A new box labeled “Controller” sits between the entry and the components. The controller observes the door and thermostat states, and controls the lighting and cooling systems.)
2.2.4 Unified Management of Controllers
Because the refrigerator has a lighting system and a cooling system, it is more reasonable to set up a separate controller for each component rather than having one controller manage two components. At the same time, for ease of management, a controller manager can be set up to uniformly maintain all these controllers and ensure they are working normally, as shown in Figure 2-5.
Figure
Figure 2-5: Refrigerator system after adding a controller manager
(Image description: A box labeled “Controller Manager” oversees two sub-boxes: “Lighting Controller” and “Cooling Controller”. The lighting controller connects to the lighting system; the cooling controller connects to the cooling system. The unified entry still connects to door and thermostat, and the controllers observe those states.)
2.2.5 Shared Informer
With the controller and controller manager in place, the refrigerator design seems quite good. However, as the refrigerator’s functionality increases, new controllers will inevitably be added continuously. These controllers all need to continuously monitor the state changes of the components they “care about” through the refrigerator entry, just like the lighting system controller constantly monitors the refrigerator door state. When a large number of controllers continuously communicate with the operation entry, it increases the pressure on the operation entry.
As shown in Figure 2-6, we can delegate the task of monitoring the state changes of refrigerator components to a new module called the Shared Informer. The Shared Informer acts as an agent for the controllers, monitoring the state changes of refrigerator components on their behalf and notifying the corresponding controllers of the state changes of different components according to the controllers’ “preferences”.
The addition of the Shared Informer module can greatly alleviate the pressure on the refrigerator operation entry.
Figure
Figure 2-6: Refrigerator system after adding the Shared Informer module
(Image description: A box labeled “Shared Informer” sits between the “Unified Operation Entry” and the controllers (Lighting Controller, Cooling Controller). The shared informer watches the entry for state changes and pushes updates to the appropriate controllers. The controllers still connect to their respective systems.)
2.2.6 List Watcher
The Shared Informer makes it convenient for controllers to monitor refrigerator components, and the core functionality of this mechanism is, of course, actively obtaining component states and passively receiving notifications of state changes. These two functions together form the List Watcher mechanism, as shown in Figure 2-7.
Assume that the Shared Informer and the refrigerator entry communicate via the HTTP protocol, then HTTP chunked transfer encoding is a good choice for implementing the List Watcher.
The controller sends a query request to the refrigerator entry through the List Watcher and then waits. When there is a change in a refrigerator component, the entry notifies the controller through a chunked HTTP response. The controller, upon “seeing” the Chunked response, assumes that the response data has not been fully sent yet, so it continues waiting, as shown in Figure 2-8.
Figure
Figure 2-7: Refrigerator system after adding the List Watcher
(Image description: A box labeled “List Watcher” is embedded within the “Shared Informer”. It shows two arrows: one for “List” (active get) and one for “Watch” (passive notification).)
Figure
Figure 2-8: Implementing List Watcher using HTTP chunked transfer encoding
(Image description: A sequence diagram. Controller sends HTTP GET request with “Watch” header to Refrigerator Entry. Entry sends back HTTP 200 OK with Transfer-Encoding: chunked. Then when a state change occurs, Entry sends a chunk with the new state data. Controller keeps connection open.)
2.3 Controller Examples
Above, we learned about the meaning, role, and implementation method of the controller through the evolution of a simple refrigerator. Now we return to the Kubernetes cluster.
The Kubernetes cluster allows a large number of controllers to be implemented, and in the foreseeable future, controllers with new functions will continue to appear, while some old controllers will be gradually phased out.
As of now, the more commonly used controllers include the Pod Controller, Deployment Controller, Service Controller, Route Controller, ReplicaSet Controller, etc. Some of these controllers are implemented and managed by the Kube Controller Manager, while others are implemented by the Cloud Controller Manager, such as the Service controller and the Route controller.
The reason the Cloud Controller Manager exists is that in different cloud environments, due to differences in cloud vendors and environments, some controller implementations vary significantly. This type of controller is separated out and implemented independently by cloud vendors based on the Cloud Controller Manager framework.
We take the Service controller and Route controller implemented by the Alibaba Cloud Kubernetes cluster Cloud Controller Manager as examples to briefly explain the working principle of Kubernetes controllers.
2.3.1 Service Controller
First, a user requests the API Server to create a Load Balancer type service. The API Server receives the request and writes the detailed information of this service into the Etcd database.
Second, this change is “observed” by the Service controller.
Finally, the Service controller understands the Load Balancer type service. Because this type of service, in addition to the service resource stored in Etcd, also requires an SLB (Server Load Balancer) as an entry point and several Endpoints as backends. Therefore, the Service controller requests the cloud API and the API Server respectively to create the SLB instance and Endpoint resources, as shown in Figure 2-9.
Figure
Figure 2-9: Service controller system
(Image description: User → API Server → Etcd. Service controller watches Etcd changes. Then it calls cloud API to create SLB instance, and calls API Server to create Endpoint resources. Arrows show the flow.)
2.3.2 Route Controller
As shown in Figure 2-10, when a new node joins the cluster, the cluster needs to add a route entry in the VPC (Virtual Private Cloud) route table to build the Pod network backbone for this newly added node.
The task of adding the route entry is performed by the Route controller. The process by which the Route controller accomplishes this task is very similar to the processing flow of the Service controller above, so we will not repeat it here.
Figure
Figure 2-10: Route controller system
(Image description: New node → API Server → Etcd. Route controller watches for new node events. It then calls the cloud API to add a route in the VPC route table. The new node gets connected to the Pod network.)
2.4 Summary
Overall, the controllers in a Kubernetes cluster play the role of the cluster’s brain. With controllers, the Kubernetes cluster breaks free from being mechanical and passive, and becomes an automatic, intelligent, and highly useful system.