Troubleshooting SIGSEGV: Segmentation Fault on Linux Containers (Exit Code 139) (2023)

The SIGSEGV Linux signal indicates a segmentation violation within a running process. Segmentation errors occur when a program tries to access unallocated memory. This can be due to accidentally buggy code or intentional malicious activity.

SIGSEGV signals originate at the operating system level, but you can also find them associated with containerization technologies such as Docker andKubernetes. If a container with status code 139 disembarks, it is because it received a SIGSEGV signal. The operating system terminated the container process to protect itself from a memory integrity violation.

It's important to investigate what's causing the segmentation errors when your containers end with code 139. This often indicates a programming error in the languages ​​that give you direct memory access. If the error occurs in containers running a third-party image, there may be a bug in that software or an incompatibility with your environment.

In this article, we explain what SIGSEGV signals are, how they affect your Linux containers in Kubernetes, and how to troubleshoot and troubleshoot segmentation errors in your application.

What is a segmentation error?

Asegmentation errormay seem like a rather obscure concept. The meaning is quite simple: a process receiving a SIGSEGV signal attempted to read or write to memory that it is not allowed to access. The kernel normally kills the process to prevent memory corruption. This behavior can be modified by explicitly handling the signal in program code.

Segmentation errors are named to reflect the way memory is partitioned by purpose. Data segments store values ​​that can be determined at compile time, text segments contain program statements, and heap segments encapsulate dynamically allocated variables created at run time.

Most real-world segmentation errors fall into the latter category. operations such as Out-of-bounds array accesses, such as bad pointer definitions, read-only memory writes, and out-of-bounds array accesses, attempt to access memory that is out of the heap.

Here is a trivial example of a C program showing a segmentation error:

int main () { char * buffer; buffer[0] = 0; returns 0 ;}

Save the program as <terminal inline>hello-world.c<terminal inline> and compile it with <terminal inline>make<terminal inline>:

$ hello world

Now run the compiled binary:

$ ./hello-worldSegfault (Core Dump)

The program is terminated immediately and a segmentation error is reported. If you check the exit code, you will see that it is 139, which corresponds to a segmentation error:

$echo $?139

Why is that happend? The program created a variable named <terminal inline>buffer<terminal inline> but did not allocate any memory to it. As a result, mapping <terminal inline>buffer[0] = 0<terminal inline> resulted in writing to unallocated memory. You can fix the program by making sure that <terminal inline>buffer<terminal inline> is big enough to cover the data to be saved:

int main() { char *buffer[1]; buffer[0] = 0; returns 0 ;}

Allocating <inline terminal>buffer<inline terminal> one byte of memory is enough to process the allocated value. This program runs successfully and exits with status code 0.

Segmentation errors in containers

Now let's see what happens when a segmentation error occurs in a container. Here is a simple Dockerfile for the crashed app described above:

DESDE alpine:latestRUN apk install --upgrade build-baseCOPY hello-world.c .RUN make hello-world && mv hello-world /usr/bin/hello-worldCMD ["hello-world"]

Build your container image with the following command:

$ docker build -t segfault:last .

Now start a container:

$ docker run segfault: last

The container starts, runs the command, and immediately exits. Use <terminal inline>docker ps<terminal inline> with the <terminal inline>-a<terminal inline> flag to get the details of the stopped container:

The exit code 139 is reported due to the segmentation error in the application.

Debugging Kubernetes Segfaults

You can also troubleshoot segmentation errors in Kubernetes containers. Use a project likeMicroK8sÖK3sto start a restaurantKubernetes-Clusteron your machine. Next, create a pod manifest that launches a container with your image:

apiVersion: v1kind: Podmetadata: name: segfaultspec: container: - name: segfault image: segfault:latest

UsekubectlTo add the pod to your cluster:

$ kubectl apply -f pod.yaml

Now get the pod details:

The pod is stuck and stuck in a reboot loop. Use the <terminal inline>describe<terminal inline> command to find out the cause:

$ kubectl describe pod/segfaultName: segfaultNamespace: default...Containers: segfault: ... Last State: Terminated Reason: Error Exit Code: 139

The exit code is reported as 139, indicating that a segmentation error caused the application in the container to crash.

Solution for segmentation errors

Once you've identified segmentation errors as the cause of your container terminations, you can proceed to mitigate them and prevent future recurrences.

If the error occurs in a third-party container image, your options are limited. You should report an issue with the developer to investigate the cause of the unexpected memory access attempts. If the problem is in your own software, you can take more specific troubleshooting steps to figure out what's wrong.

Error code identification

First, look for obvious areas of your code that might be affected by segmentation issues. You may be able to use your container's logs to calculate the sequence of events that led to the error:

$ docker registry mi container $ kubectl registry pod/my-pod

Use the container activity to find where in the source the error originated. If there is an array access, pointer reference, or unprotected memory write in the area, this may be the cause of the problem.

environmental intolerance

Another common cause of these errors is when a shared library update introduces incompatibilities with existing binaries. This can cause memory access violations if the versions loaded deviate from the supported range.

Try reverting any recent dependency changes in your containers. This can help eliminate problems caused by third-party library updates.

In rare cases, persistent segmentation errors with no obvious explanation can be caused by incompatibilities with the computer's physical hardware. They could even be symptoms of a memory defect. This type of problem is less likely in the context of a typical Kubernetes cluster running on onePublic cloud provider K8s🇧🇷 RunningStorageServicing your own hardware can help rule out physical problems.

directed debugging

You can use Linux tools to debug SIGSEGV signals with greater precision. Segmentation errors always generate kernel log messages. Because containers run as processes within their host's kernel, they are written even if the error occurred inside a container.

Check your system log by viewing the contents of <inline terminal>/var/log/syslog<inline terminal>:

$ sudo cola -f /var/log/syslog

This command continuously streams logs to your terminal until you cancel it with Ctrl+C. Now try to reproduce the event that caused the segmentation error. The SIGSEGV signal looks like this in the protocol:

hola-mundo[2631584]: Segmentfehler in 7f4624c6cfe0 ip 000055730c3621ed sp 00007ffce90e35f0 Fehler 7 in hola-mundo[55730c362000+1000]

The log can be interpreted as follows:

• <bold terminal online>at <address><bold terminal online>:The forbidden memory address that the code tried to access.
• <bold online terminal>ip <pointer><bold online terminal>:The memory address of the code that committed the violation.
• <bold inline terminal>sp <pointer><bold inline terminal>:The stack pointer for the operation, providing the address of the last program request on the stack.
• <terminal bold inline> Fehler <code><terminal bold inline>:The error code gives an indication of the type of operation attempted. Common codes include <line terminal>6<line terminal>, writing to an unallocated area; <inline terminal>7<inline terminal>, writing to a readable but not writable area; <line terminal>4<line terminal>, read from unused area; and <inline terminal>5<inline terminal>, reading from a write-only area.

Accessing the kernel log gives you a better understanding of what the code is doing at the time the error occurs. Although this log cannot be accessed directly from the containers, you should still be able to get the details of the segmentation errors if you have root access to the host machine.

Elegant handling of segmentation errors

Another way to fix segmentation errors is to properly handle them within your code. You can use libraries likeSegv captureto capture SIGSEGV signals and convert them into software exceptions. You can then treat them like any other exception, giving you the ability to log details to your error monitoring platform and recover without error.

While dealing with SIGSEGV is a good way to avoid fatal errors, it is worth thoroughly investigating and fixing each occurrence of this error. A segmentation error indicates that the program is doing something that the Linux kernel specifically forbids, indicating serious reliability or security vulnerabilities in your code. Simply capturing the signal and ignoring it could cause other problems in your program if you expect to have read or write memory that turns out to be out of bounds.

final thoughts

Segmentation errors occur when a program tries to use memory that it is not allowed to access. They also arise when data is written to read-only memory and vice versa. As you saw in this article, these errors are often the result of simple programming errors. You also discussed how to identify a segfault as the cause of container terminations and how to start troubleshooting the segfaults that occur in your programs.

Staying one step ahead of these errors will ensure your applications run with maximum reliability and uptime.ContainIQ's Kubernetes monitoring platformallows you to track container terminations and their causes in real-time, helping you identify when things aren't working as they should. You can use it to configure alerts for failed containers and then examine their exit codes to identify segmentation failures in your cluster. This gives you the ability to respond to segmentation failures as they occur and keep your workloads in a healthy state.