🏷️ Protect node metadata and endpoints#
Kubelet#
What is Kubelet?#
Node agent that runs on each node in a Kubernetes cluster
Role in Kubernetes architecture:
Communicates with control plane components like API server
Manages pods and containers on its node
Ensures pods are healthy and runnig as specified
Pod Lifecycle Management#
Creating and deleting pods
Receives pod specs from API server
Pulls container images and runs containers
Terminates pods when deleted via API server
Maintaining pod state
Restarts containers if they fail
Mount and umounts volumes for persistent storage
Reporting pod status
Updates API server with pod phase (Pending, Running, Succeeded, Failed, Unknown)
Enables control plane to track pod state
Resource Monitoring and Allocation#
Monitoring node and pod resource usage
Collects metrics on CPU, memory, disk usage
Enforcing resource limits
Ensures containers do not exceed CPU and memory limits defined in pod specs
Terminates or throttles containers exceeding limits
Allocating resources to pods
Assigns available node CPU and memory to pods based on requests and limits
Node Health Monitoring#
Checking node health
Runs node problem detector to identify issues like kernel deadlocks, corrupted filesystem, …
Reports node conditions (Ready, MemoryPressure, DiskPressure, PIDPressure,NetworkUnavailable) to API server
Handling node failures
Continues running existing pods if node is partitioned from master
Marks node as unreachable if kubelet stops posting status
Enables rescheduling of pods to healthy nodes
Participating in cluster autoscaling
Annotates nodes to influence scale up/down behavior
Drains pods from node before scale down
Security and Logging#
Authenticating to API server
Uses TLS certificates to verify identity
Rotates certificates automatically to prevent expiration
Restricting access to sensitive APIs
Limits which pods can access node endpoints like logs, metrics, exec
Managing container logs
Collects stdout/stderr logs from containers
Writes logs to files (
/var/log/pods,/var/log/containers) for access viakubectl logsRotates and compresses logs to save disk space
Providing audit trail
Records events for pod lifecycle, resource usage, node conditions
Enables debugging and tracing of cluster activity
Kubelet API
The Kubelet API is an internal, undocumented REST API exposed by the Kubelet, the primary agent running on each Kubernetes node. It enables communication between the Kubernetes control plane and nodes, managing pod lifecycle, container runtime, and node health. By default, it listens on port 10250 (secure port) and port 10255 (read-only port), with the latter allowing unauthenticated access to limited endpoints.
Key Endpoints
Read-Only Port (10255):
/pods– Lists pods on the node./stats/summary– Provides node and pod resource statistics./metrics– Exposes Kubelet and container metrics (e.g., CPU, memory)./metrics/cadvisor– Detailed metrics from cAdvisor (container resource usage)./spec/– Returns node specifications (CPU, memory, disk, network).
Secure Port (10250):
/run/{namespace}/{podName}/{containerName}– Executes commands in a container./exec/{namespace}/{podName}/{containerName}– Runs commands interactively./portForward/{namespace}/{podName}– Forwards ports from the pod to the local machine./containerLogs/{namespace}/{podName}/{containerName}– Retrieves container logs./configz– Exposes Kubelet configuration./cri/– Access to container runtime interface (CRI).
Security and Access
Default Configuration: By default, Kubelet allows anonymous access to the read-only port (10255), which can be exploited if exposed externally.
Authentication & Authorization:
Certificate-based: Recommended for production.
Token-based: Uses service account tokens.
Anonymous Access: Disabled by setting
--anonymous-auth=false.
Authorization Modes:
AlwaysAllow– Allows all requests (dangerous).Webhook– Uses external authorization system.RBAC– Most secure and recommended.
Risks and Best Practices
Security Risk: Unauthenticated access to the Kubelet API can lead to remote code execution, privilege escalation, and cluster takeover.
Best Practices:
Disable anonymous access (
--anonymous-auth=false).Use certificate-based authentication.
Restrict network access to the Kubelet API (only allow API server and trusted components).
Regularly audit Kubelet configuration and use tools like
kubeletctlfor discovery and testing.
Note: The Kubelet API is not officially documented and may change between versions. Always refer to the source code or trusted tools like
kubeletctlfor up-to-date endpoints.
Kubernetes Network Ports#
Control Plane
Protocol |
Direction |
Port Range |
Purpose |
Used By |
|---|---|---|---|---|
TCP |
Inbound |
6443 |
Kubernetes API server |
All |
TCP |
Inbound |
2379-2380 |
etcd server client API |
kube-apiserver, etcd |
TCP |
Inbound |
10250 |
Kubelet API |
Self, Control plane |
TCP |
Inbound |
10259 |
kube-scheduler |
Self |
TCP |
Inbound |
10257 |
kube-controller-manager |
Self |
Worker
Protocol |
Direction |
Port Range |
Purpose |
Used By |
|---|---|---|---|---|
TCP |
Inbound |
10250 |
Kubelet API |
Self, Control plane |
TCP |
Inbound |
10256 |
kube-proxy |
Self, Load balancers |
TCP |
Inbound |
30000-32767 |
NodePort Services |
All |
UDP |
Inbound |
30000-32767 |
NodePort Services |
All |
Configure Network Firewall Rules#
Restrict access to exposed ports
Kubelet API (10250): Allow only from control plane IPs
iptables -A INPUT -p tcp -s <cp-ip-range> --dport 10250 -j ACCEPT
API server: Limit acces to authorized users and control plane
etcd (2379-2380): Allow only from control plane, encrypt traffic
NodePort services (30000-32767): Open slectively based on exposed services
Configure cloud provider firewall rules
GKE: Use master authorized networks to restrict control plane and node access
AWS: Leverage security groups to limit API server access and node-to-node access
On-Premise: Set up perimeter firewalls to filter traffic to node ports by source IP
Monitor Network Traffic#
Use network monitoring tools to track flows between pods and nodes
Inspect logs from firewalls and Kubernetes components for unauthorized access attempts
Set up alerts for any traffic that violates expected ingress/egress rules
Check opened ports on control plane node (kubeadm installed)#
sudo ss -ltpn | awk 'NR==1||$6~/etcd|kube/{printf "%-10s %-20s %s\n", $1, $4, $6}'
State Local Peer
LISTEN 172.16.0.73:2379 users:(("etcd",pid=6937,fd=7))
LISTEN 172.16.0.73:2380 users:(("etcd",pid=6937,fd=3))
LISTEN 127.0.0.1:2379 users:(("etcd",pid=6937,fd=6))
LISTEN 127.0.0.1:2381 users:(("etcd",pid=6937,fd=12))
LISTEN 127.0.0.1:10259 users:(("kube-scheduler",pid=7671,fd=4))
LISTEN 127.0.0.1:10257 users:(("kube-controller",pid=7523,fd=4))
LISTEN 127.0.0.1:10248 users:(("kubelet",pid=8595,fd=17))
LISTEN 127.0.0.1:10249 users:(("kube-proxy",pid=7859,fd=25))
LISTEN *:6443 users:(("kube-apiserver",pid=7185,fd=4))
LISTEN *:10256 users:(("kube-proxy",pid=7859,fd=24))
LISTEN *:10250 users:(("kubelet",pid=8595,fd=18))
curl -sk https://localhost:10250/pods
Unauthorized
sudo curl --cacert /etc/kubernetes/pki/ca.crt \
--cert /etc/kubernetes/pki/apiserver-kubelet-client.crt \
--key /etc/kubernetes/pki/apiserver-kubelet-client.key \
-sk https://localhost:10250/pods | jq '.items[].metadata.name'
"kube-controller-manager-controlplane-01"
"notes-58cd4f7564-cmjtw"
"kube-scheduler-controlplane-01"
"metrics-server-799f7ccf68-4fvpp"
"ngf-nginx-gateway-fabric-5bff9d865-c24sx"
"cilium-envoy-c2qs5"
"hubble-ui-7bcb645fcd-wzvcr"
"app2-6c66f57cdf-p96vf"
"ingress-nginx-controller-579c674b5d-jstfh"
"notes-nginx-76bdc8897b-h7ws9"
"etcd-controlplane-01"
"kube-proxy-6rhzf"
"hubble-relay-7d95fb4b67-h447k"
"cilium-lrf9z"
"kube-apiserver-controlplane-01"
"app1-67745fc6b4-89rqh"
sudo etcdctl endpoint status -w table \
--cacert="/etc/kubernetes/pki/etcd/ca.crt" \
--cert="/etc/kubernetes/pki/etcd/server.crt" \
--key="/etc/kubernetes/pki/etcd/server.key"
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 127.0.0.1:2379 | a281042b761775aa | 3.6.6 | 23 MB | true | false | 49 | 9121193 | 9121193 | |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
Kubelet authentication#
Ensure anonymous authentication is disabled.
sudo systemctl cat kubelet.service
awk 'NR>1&&NR<8' /var/lib/kubelet/config.yaml
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 0s
enabled: true
Ensure apiserver has kubelet cert and key.
sudo grep kubelet /etc/kubernetes/manifests/kube-apiserver.yaml
- --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
- --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
Headlamp#
Headlamp is an open-source, extensible Kubernetes web UI designed to provide a modern, user-friendly interface for managing and visualizing Kubernetes clusters.
helm repo add headlamp https://kubernetes-sigs.github.io/headlamp/
helm template -n kube-system headlamp headlamp/headlamp --set replicaCount=2
helm install -n kube-system headlamp headlamp/headlamp --set replicaCount=2
Install Headlamp with Helm and TLS
To install Headlamp using Helm with Ingress and TLS provided by cert-manager, follow these steps:
Install cert-manager (if not already installed):
# Add the Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io
helm repo update
# Install cert-manager with CRDs
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--set crds.enabled=true
Configure Headlamp Helm chart with Ingress and TLS:
Use a values.yaml file with the following settings:
# Enable Ingress and configure TLS
ingress:
enabled: true
hosts:
- your-domain.com
annotations:
kubernetes.io/ingress.class: "nginx" # or your ingress controller class
cert-manager.io/cluster-issuer: "letsencrypt-prod" # Replace with your cluster issuer
tls:
- hosts:
- your-domain.com
secretName: headlamp-tls # This secret will be managed by cert-manager
# Enable plugins (optional, e.g., cert-manager plugin)
pluginsManager:
enabled: true
configContent: |
plugins:
- name: cert-manager
source: https://artifacthub.io/packages/headlamp/headlamp-plugins/headlamp_cert-manager
version: 0.1.0
Install Headlamp:
helm install headlamp headlamp/headlamp \
--namespace kube-system \
-f values.yaml
Verify TLS Certificate:
After installation, cert-manager will automatically create the headlamp-tls secret using the specified cluster issuer. Monitor the certificate status:
kubectl get certificate -n kube-system
kubectl describe certificate headlamp-tls -n kube-system
Access Headlamp:
Obtain the Ingress IP or hostname.
Access the UI via
https://your-domain.com.Authenticate using a token (if OIDC is not configured):
kubectl create token my-headlamp -n kube-system
Note: Ensure your ingress controller is configured to handle TLS termination and that the
cert-manager.io/cluster-issuerannotation matches a valid issuer in your cluster.