What are Kubernetes & Containers Exit Codes — How to Fix it

When we work on the Linux system earlier our carrier, we may experienced some of our shell scripts get failed or some services get fails, best way to troubleshoot those issues are checking the exit code or how the service or script got stopped. This legacy continues, and even in modern container or microservices world, this continues. Lets check more about container exit codes and will understand more about the each exit codes. This should help to troubleshoot the issues better way.

Container Exit Codes

In case if the container got exited/terminated unexpectedly, container engines reports why the container got terminated with exit codes. The exit codes help to understand the issue and fix it. You may experience the same when you worked on Kubernetes, as container failures are one of the most commons caused on pod exceptions.

Related Article: K8s Troubleshooting Archives — FoxuTech

To troubleshoot the issues, we may need to check the logs and should try to get the exit codes, so it will help to fix the issue sooner sometimes.

The most common exit codes used by containers are:

CODE #ReasondetailsExit Code 0Purposely stoppedUsed by developers/devops/sre to indicate that the container was automatically stoppedExit Code 1Application errorContainer was stopped due to application error or incorrect reference in the image/configuration specificationExit Code 125Container failed to run errorThe docker run command did not execute successfullyExit Code 126Command invoke errorA command specified in the image specification could not be invokedExit Code 127File or directory not foundFile or directory specified was not foundExit Code 128Invalid argument used on exitExit was triggered with an invalid exit codeExit Code 134Abnormal termination (SIGABRT)The container aborted itself using the abort () function.Exit Code 137Immediate termination (SIGKILL)Container was immediately terminated by the operating system via SIGKILL signalExit Code 139Segmentation fault (SIGSEGV)Container attempted to access memory that was not assigned to it and was terminatedExit Code 143Graceful termination (SIGTERM)Container received warning that it was about to be terminated, then terminatedExit Code 255Exit Status Out of RangeContainer exited, returning an exit code outside the acceptable range, meaning the cause of the error is not known

Let’s check in-details each exit code and how to troubleshoot each, to check the failed containers.

Container Lifecycle

To better understand the causes of container failure, let’s discuss the lifecycle of a container first. Taking Docker as an example — at any given time, a Docker container can be in one of several states:

• Created — the Docker container is created but not started yet (this is the status after running docker create, but before actually running the container)
• Up — the Docker container is currently running. This means the operating system process managed by the container is running. This happens when you use the commands docker start or docker run can happen using docker start or docker run.
• Paused — the container process was running, but Docker purposely paused the container. Typically, this happens when you run the Docker pause command.
• Exited — the Docker container has been terminated, usually because the container’s process was killed.

When a container reaches the Exited status, Docker will report an exit code in the logs, to inform you what happened to the container that caused it to shut down.

Understanding Container Exit Codes

Below we cover each of the exit codes in more detail.

Exit Code 0: Purposely Stopped

Exit Code 0 is triggered by developers/SRE/DevOps team when they purposely stop their container after a task completes. Technically, Exit Code 0 means that the foreground process is not attached to a specific container. This one as we are aware, we no need to worry much about it.

Things to while Exit Code 0?

1. Check the container logs to identify what caused the issue and why the container got exit.
2. Review the code and identify why it triggered Exit Code 0, and whether it is functioning correctly.

Exit Code 1: Application Error

Exit Code 1 indicates that the container was stopped due to one of the following:

• An application error — this could be a programming error in code run on the container, such as “divide by zero”, or advanced errors related to the runtime environment.
• An invalid reference — this means the image spec refers to a file that does not exist in the container image.

Things to while Exit Code 1?

1. Check the container log to see if one of the files listed in the image spec could not be found. If this is the issue, correct the image spec to point to the correct path and filename.
2. If you cannot find an incorrect file reference, check the container logs for an application error, and debug the library that caused the error.

Exit Code 125: Container Failed to Run

Exit Code 125 means that the command is used to run the container. For example, docker run was invoked in the system shell but did not execute successfully. Here are common reasons this might happen:

• An undefined flag was used in the command, for example docker run --abcd
• The user-defined in the image spec does not have sufficient permissions on the machine.
• Incompatibility between the container engine and the host operating system or hardware

Things to while Exit Code 125?

1. Check if the command used to run the container uses the proper syntax.
2. Check if the user running the container, or the context in which the command is executed in the image spec, has sufficient permissions to create containers on the host.
3. If your container engine provides other options for running a container, try them. For example, in Docker, try docker start instead of docker run
4. Test if you can run other containers on the host using the same username or context. If not, reinstall the container engine, or resolve the underlying compatibility issue between the container engine and the host setup.

Exit Code 126: Command Invoke Error

Exit Code 126 means that a command used in the container spec could not be invoked. This is often the cause of a missing dependency or an error in a continuous integration script used to run the container.

Things to while Exit Code 126?

1. Check the container logs to see which command could not be invoked.
2. Try running the container spec without the command to ensure you isolate the problem.
3. Troubleshoot the command to ensure you are using the correct syntax and all dependencies are available.
4. Correct the container spec and retry running the container.

Exit Code 127: File or Directory Not Found

Exit Code 127 means a command specified in the container specification refers to a non-existent file or directory.

Things to while Exit Code 127?

Same as Exit Code 126, identify the failing command and make sure you reference a valid filename and file path available within the container image.

Exit Code 128: Invalid Argument Used on Exit

Exit Code 128 means that code within the container triggered an exit command but did not provide a valid exit code. The Linux exit command only allows integers between 0–255, so if the process was exited with, for example, exit code 3.5, the logs will report Exit Code 128.

Things to while Exit Code 128?

1. Check the container logs to identify which library caused the container to exit.
2. Identify where the offending library uses the exit command and correct it to provide a valid exit code.

Exit Code 134: Abnormal Termination (SIGABRT)

Exit Code 134 means that the container abnormally terminated itself, closed the process and flushed open streams. This operation is irreversible, like SIGKILL (see Exit Code 137 below). A process can trigger SIGABRT by doing one of the following:

• Calling the abort() function in the libc library
• Calling the assert() macro, used for debugging. The process is then aborted if the assertion is false.

Things to while Exit Code 134?

1. Check container logs to see which library triggered the SIGABRT signal.
2. Check if process abortion was planned (for example because the library was in debug mode), and if not, troubleshoot the library and modify it to avoid aborting the container.

Exit Code 137: Immediate Termination (SIGKILL)

Exit Code 137 means that the container has received a SIGKILL signal from the host operating system. This signal instructs a process to terminate immediately, with no grace period. This can be either:

• Triggered when a container is killed via the container engine, for example when using the docker kill command.
• Triggered by a Linux user sending a kill -9 command to the process.
• Triggered by Kubernetes after attempting to terminate a container and waiting for a grace period of 30 seconds (by default)
• Triggered automatically by the host, usually due to running out of memory. In this case, the docker inspect command will indicate an OOMKilled error.

Things to while Exit Code 137?

1. Check logs on the host to see what happened prior to the container terminating, and whether it previously received a SIGTERM signal (graceful termination) before receiving SIGKILL.
2. If there was a prior SIGTERM signal, check if your container process handles SIGTERM and can gracefully terminate.
3. If there was no SIGTERM and the container reported an OOMKilled error, troubleshoot memory issues on the host.

Exit Code 139: Segmentation Fault (SIGSEGV)

Exit Code 139 means that the container received a SIGSEGV signal from the operating system. This indicates a segmentation error — a memory violation, caused by a container trying to access a memory location to which it does not have access. There are three common causes of SIGSEGV errors:

Continue Reading remaining sections on What are Kubernetes & Containers Exit Codes — How to Fix it (foxutech.com)