Troubleshooting Kubernetes Networking — Part1
Troubleshooting Kubernetes Networking: As we are seeing one by one issues on Kubernetes and how to fix it, part of that today we are going to see, another important topic, which is nothing but networking. As this is vest topic to cover, we will be separate one by one and provide the solutions. In this topic, we will be taking some example scenarios and how to fix it.
In this series we should get understand the basics and few related topics, then will see very basic issue and how to fix in this, in coming post will be seeing another extended issue and how to fix it.
Before we proceed, let’s check about Kubernetes service, A Kubernetes service is a logical abstraction for a deployed group of pods in a cluster (which all perform the same function). Since pods are ephemeral, a service enables a group of pods, which provide specific functions (web services, image processing, etc.) to be assigned a name and unique IP address (clusterIP). As long as the service is running that IP address, it will not change. Services also define policies for their access.
Types of Kubernetes services?
- ClusterIP. Exposes a service which is only accessible from within the cluster.
- NodePort. Exposes a service via a static port on each node’s IP.
- LoadBalancer. Exposes the service via the cloud provider’s load balancer.
- ExternalName. Maps a service to a predefined externalName field by returning a value for the CNAME record.
Container Network Interface
As we know, Every Pod in a cluster gets its own unique cluster-wide IP address. This means you do not need to explicitly create links between Pods and you almost never need to deal with mapping container ports to host ports. This creates a clean, backwards-compatible model where Pods can be treated much like VMs or physical hosts from the perspectives of port allocation, naming, service discovery, load balancing, application configuration, and migration.
Kubernetes imposes the following fundamental requirements on any networking implementation (barring any intentional network segmentation policies): pods can communicate with all other pods on any other node without NAT agents on a node (e.g. system daemons, kubelet) can communicate with all pods on that node.
Kubernetes networking addresses four concerns:
- Containers within a Pod use networking to communicate via loopback.
- Cluster networking provides communication between different Pods.
- The Service resource lets you expose an application running in Pods to be reachable from outside your cluster.
- You can also use Services to publish services only for consumption inside your cluster.
Here are some familiar or most used CNI plugins,
How to choose a CNI Provider?
There is no single CNI provider that meets all our project needs, here some details about each provider. For easy setup and configuration, Flannel and Weavenet provide great capabilities. Calico is better for performance since it uses an underlay network through BGP. Cilium utilizes a completely different application-layer filtering model through BPF and is more geared towards enterprise security.
As we have seen, Kubernetes supports a variety of networking plugins as seen above and each fails in its own way. To troubleshoot the issues in networking we should understand the core. Kubernetes relies on the Netfilter kernel module to set up low level cluster IP load balancing. This requires two critical modules, IP forwarding and bridging.
Kernel IP forwarding
IP forwarding is a kernel setting that allows forwarding of the traffic coming from one interface to be routed to another interface. This setting is necessary for Linux kernel to route traffic from containers to the outside world.
What it causes?
Sometimes this setting could be reset by a security team running while security scans/enforcements or some system changes, or have not been configured to survive a reboot. When this happens networking starts failing.
Pod to service connection times out:
* connect to 10.0.21.231 port 3000 failed: Connection timed out
* Failed to connect to 10.0.21.231 port 3000: Connection timed out
* Closing connection 0
curl: (7) Failed to connect to 10.0.21.231 port 3000: Connection timed out
Tcpdump could show that lots of repeated SYN packets are sent, but no ACK is received.
How to diagnose
Check that ipv4 forwarding is enabled
Continue reading it on https://foxutech.com/troubleshooting-kubernetes-networking-part1/