Kubernetes Worker Nodes are the worker machines in the cluster that run containerized applications. They can be physical machines or virtual machines. Each worker node runs the necessary components to manage and run Pods.
Worker Node Components
1. kubelet
kubelet is the primary agent on the worker node, responsible for communicating with the control plane and managing Pods.
Main Responsibilities:
- Listens to the API Server to get the desired state of Pods
- Ensures that containers run according to Pod specifications
- Periodically reports node and Pod status to the control plane
- Handles container lifecycle (creation, startup, stop, deletion)
- Mounts and unmounts storage volumes
- Runs health checks (livenessProbe, readinessProbe, startupProbe)
How It Works:
- kubelet gets Pod specifications assigned to the node from the API Server
- Interacts with the container runtime through the Container Runtime Interface (CRI)
- Uses the container runtime to create and manage containers
- Periodically reports node and Pod status to the API Server
Configuration File: /var/lib/kubelet/config.yaml
Default Ports: 10250 (HTTPS), 10248 (health check), 10255 (read-only HTTP)
2. kube-proxy
kube-proxy is responsible for maintaining network rules on the node, implementing Service load balancing and service discovery.
Main Responsibilities:
- Monitors Service and Endpoint changes in the API Server
- Maintains network rules (iptables or IPVS)
- Forwards traffic to backend Pods
- Implements Service load balancing
Operating Modes:
-
iptables mode (default):
- Uses iptables rules to implement traffic forwarding
- Good performance, but rule updates have latency
- Does not support complex load balancing algorithms
-
ipvs mode:
- Uses Linux IPVS to implement load balancing
- Supports multiple load balancing algorithms (round-robin, least connections, source hash, etc.)
- Higher performance, suitable for large-scale clusters
- Requires loading the ipvs kernel module
-
userspace mode (deprecated):
- Implements load balancing in user space
- Poor performance, no longer recommended
Configuration Example:
yamlapiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration mode: "ipvs"
3. Container Runtime
The container runtime is the core component responsible for running containers. Kubernetes uses the Container Runtime Interface (CRI) to interact with the container runtime.
Supported Container Runtimes:
-
containerd (recommended):
- CNCF graduated project
- Lightweight and high-performance
- Docker's underlying runtime
-
CRI-O:
- Designed specifically for Kubernetes
- Lightweight and secure
- OCI compatible
-
Docker Engine (via dockershim, deprecated):
- Kubernetes 1.24+ has removed dockershim
- No longer recommended
CRI Workflow:
- kubelet calls the container runtime through the CRI interface
- The container runtime creates and manages containers
- The container runtime returns container status information
Node Status
Node Conditions
Node conditions describe the health status of the node:
-
Ready: Whether the node is healthy and can run Pods
- True: Node is healthy
- False: Node is unhealthy
- Unknown: Node controller cannot get node status
-
MemoryPressure: Node memory pressure
- True: Node has insufficient memory
-
DiskPressure: Node disk pressure
- True: Node has insufficient disk space
-
PIDPressure: Node process pressure
- True: Node has too many processes
-
NetworkUnavailable: Node network unavailable
- True: Node network configuration has issues
Node Capacity and Allocatable Resources
- Capacity: Total resources of the node (CPU, memory, storage)
- Allocatable: Resources that can be allocated to Pods (minus system reservation)
Node Information
- OS Image: Operating system image
- Kernel Version: Kernel version
- Kubelet Version: kubelet version
- Container Runtime Version: Container runtime version
Node Management
Node Maintenance
- Safely drain Pods:
bashkubectl drain <node-name> --ignore-daemonsets
- Mark node as unschedulable:
bashkubectl cordon <node-name>
- Unmark node as unschedulable:
bashkubectl uncordon <node-name>
Node Taints
Taints are used to prevent Pods from being scheduled to specific nodes:
bash# Add taint kubectl taint nodes <node-name> key=value:NoSchedule # View taints kubectl describe node <node-name> | grep Taint # Remove taint kubectl taint nodes <node-name> key:NoSchedule-
Taint types:
- NoSchedule: Do not schedule new Pods, existing Pods are not affected
- PreferNoSchedule: Try not to schedule, but not mandatory
- NoExecute: Do not schedule new Pods, existing Pods will be evicted (unless they have tolerations)
Node Labels
Labels are used to organize and select nodes:
bash# Add label kubectl label nodes <node-name> disktype=ssd # View labels kubectl get nodes --show-labels # Remove label kubectl label nodes <node-name> disktype-
Node Health Checks
kubelet periodically performs health checks:
- Node status check: Check if node resources are sufficient
- Container health check: Execute livenessProbe and readinessProbe
- Image pull check: Check if images are available
Best Practices
-
Resource Reservation: Reserve sufficient CPU and memory for system processes and kubelet
-
Monitor Nodes: Monitor CPU, memory, disk, and network usage of nodes
-
Log Collection: Collect logs from kubelet and container runtime
-
Security Hardening:
- Use TLS for encrypted communication
- Limit kubelet API access
- Regularly update component versions
-
Node Maintenance: Use the drain command to safely maintain nodes
-
Auto-scaling: Use Cluster Autoscaler to automatically adjust the number of nodes
-
Taints and Tolerations: Reasonably use taints and tolerations to control Pod scheduling
-
Node Affinity: Use node affinity to optimize Pod allocation
Troubleshooting
- View node status:
bashkubectl describe node <node-name>
- View kubelet logs:
bashjournalctl -u kubelet
- View container runtime logs:
bashjournalctl -u containerd
- Check node resources:
bashkubectl top nodes