As we have seen some Kubernetes troubleshooting articles before, today will see another Kubernetes issue and will see how to troubleshoot it. Today will see what is Exit Code 137 error and what is cause and why this happens.
Exit Code 137 errors happen when a container or pod was terminated due to high memory usage. Your container or Kubernetes pod will be stopped to prevent the excessive memory consumption from affecting your host’s reliability.
You should identify the processes that ended with exit code 137 and should investigate in detail. One way we can simply conclude the system may reached the limit we defined in the manifest, but only this cannot be reason always. However, there might also be a memory leak or sub-optimal programming inside your application that’s causing resources to be consumed excessively, there are lot of factor to check before we conclude anything.
In this article, you’ll learn how to identify, and debug exit code 137 so your containers runs reliably. This will reduce your maintenance overhead and help stop inconsistencies caused by services stopping unexpectedly. Although some causes of exit code 137 can be highly specific to your environment, most problems can be solved with a simple troubleshooting sequence. Always note that, any problem leading to distruption is business loss. We should be too causes and vigilance on the issue and should fix it permanently.
What Is Exit Code 137?
As we know, All the processes emit an exit code when they terminate/killed unexpectly. Exit codes provide a mechanism for informing the user, operating system, and other applications why the process stopped. Each code is a number between 0 and 255. The meaning of codes below 125 is application-dependent, while higher values have special meanings.
A 137 code is issued when a process is terminated externally because of its memory consumption. The operating system’s out of memory manager (OOM) intervenes to stop the program before it destabilizes the host.
Pods running in Kubernetes will show a status of OOMKilled when they encounter a 137 exit code. Although this looks like any other Kubernetes status, it’s caused by the operating system’s OOM killer terminating the pod’s process. You can check the pod and understand the error.
# kubectl get podsCauses of Container Memory Issues
Understanding the situations that lead to memory-related container terminations is the first step towards debugging exit code 137. Here are some of the most common issues that you might experience.
Container Memory Limit Exceeded
Kubernetes pods will be terminated when they try to use more memory than their configured limit allows.
Fix: You might be able to resolve this situation by increasing the limit if your cluster has additional capacity available. Lets see in-detail below.
Application Memory Leak
Poorly optimized code can create memory leaks. A memory leak occurs when an application uses memory but doesn’t release it when the operations complete. This causes the memory to gradually fill up and will eventually consume all the available capacity.
Fix: There is no direct solution for this issue, to fix this, you should understand you changes and should see where there is more memory consuming or what causing to utilize the memory.
Increases in the Load
Sometimes we may experience this issue when there is unexpected load we receive on our tool. We could plan our system resource for some limited number of users. But due to some demand, if there are high number of users, memory will be get consumed more (if the application is memory intensive).
Fix: You should have extensive monitoring to get alerted when there is high observed, or you can enable supported auto-scaling to fix this issue or check your architect and implement suitable solution.
Continue reading on https://foxutech.com/how-to-fix-exit-code-137-kubernetes-memory-issues/
