Troubleshooting SIGSEGV: Segmentation Fault on Linux Containers (Exit Code 139) (2023)

The SIGSEGV Linux signal indicates a segmentation violation within a running process. Segmentation errors occur when a program tries to access unallocated memory. This can be due to accidentally buggy code or intentional malicious activity.

SIGSEGV signals originate at the operating system level, but you can also find them associated with containerization technologies such as Docker andKubernetes. If a container with status code 139 disembarks, it is because it received a SIGSEGV signal. The operating system terminated the container process to protect itself from a memory integrity violation.

It's important to investigate what's causing the segmentation errors when your containers end with code 139. This often indicates a programming error in the languages ​​that give you direct memory access. If the error occurs in containers running a third-party image, there may be a bug in that software or an incompatibility with your environment.

In this article, we explain what SIGSEGV signals are, how they affect your Linux containers in Kubernetes, and how to troubleshoot and troubleshoot segmentation errors in your application.

What is a segmentation error?

Asegmentation errormay seem like a rather obscure concept. The meaning is quite simple: a process receiving a SIGSEGV signal attempted to read or write to memory that it is not allowed to access. The kernel normally kills the process to prevent memory corruption. This behavior can be modified by explicitly handling the signal in program code.

Segmentation errors are named to reflect the way memory is partitioned by purpose. Data segments store values ​​that can be determined at compile time, text segments contain program statements, and heap segments encapsulate dynamically allocated variables created at run time.

Most real-world segmentation errors fall into the latter category. operations such as Out-of-bounds array accesses, such as bad pointer definitions, read-only memory writes, and out-of-bounds array accesses, attempt to access memory that is out of the heap.

Here is a trivial example of a C program showing a segmentation error:

int main () { char * buffer; buffer[0] = 0; returns 0 ;}

Save the program as <terminal inline>hello-world.c<terminal inline> and compile it with <terminal inline>make<terminal inline>:

$ hello world

Now run the compiled binary:

$ ./hello-worldSegfault (Core Dump)
(Video) `Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)` after...

The program is terminated immediately and a segmentation error is reported. If you check the exit code, you will see that it is 139, which corresponds to a segmentation error:

$echo $?139

Why is that happend? The program created a variable named <terminal inline>buffer<terminal inline> but did not allocate any memory to it. As a result, mapping <terminal inline>buffer[0] = 0<terminal inline> resulted in writing to unallocated memory. You can fix the program by making sure that <terminal inline>buffer<terminal inline> is big enough to cover the data to be saved:

int main() { char *buffer[1]; buffer[0] = 0; returns 0 ;}

Allocating <inline terminal>buffer<inline terminal> one byte of memory is enough to process the allocated value. This program runs successfully and exits with status code 0.

Segmentation errors in containers

Now let's see what happens when a segmentation error occurs in a container. Here is a simple Dockerfile for the crashed app described above:

DESDE alpine:latestRUN apk install --upgrade build-baseCOPY hello-world.c .RUN make hello-world && mv hello-world /usr/bin/hello-worldCMD ["hello-world"]

Build your container image with the following command:

$ docker build -t segfault:last .

Now start a container:

$ docker run segfault: last

The container starts, runs the command, and immediately exits. Use <terminal inline>docker ps<terminal inline> with the <terminal inline>-a<terminal inline> flag to get the details of the stopped container:

$ docker ps -a

6e6944f7f339Segment error: latest"Hello World"17 seconds agoFinished (139) 16 seconds ago
(Video) C++ | How to fix exit code 139 (interrupted by signal 11) | Student Help

The exit code 139 is reported due to the segmentation error in the application.

Troubleshooting SIGSEGV: Segmentation Fault on Linux Containers (Exit Code 139) (1) Troubleshooting SIGSEGV: Segmentation Fault on Linux Containers (Exit Code 139) (2)

Monitor Kubernetes events in real time

Monitor the health of your cluster and resolve issues faster with pre-built dashboards that just work.

Learn more Book a demo

Troubleshooting SIGSEGV: Segmentation Fault on Linux Containers (Exit Code 139) (3)

Troubleshooting SIGSEGV: Segmentation Fault on Linux Containers (Exit Code 139) (4) Troubleshooting SIGSEGV: Segmentation Fault on Linux Containers (Exit Code 139) (5)

Debugging Kubernetes Segfaults

You can also troubleshoot segmentation errors in Kubernetes containers. Use a project likeMicroK8sÖK3sto start a restaurantKubernetes-Clusteron your machine. Next, create a pod manifest that launches a container with your image:

(Video) How to Solve Segmentation Fault Problem

apiVersion: v1kind: Podmetadata: name: segfaultspec: container: - name: segfault image: segfault:latest

UsekubectlTo add the pod to your cluster:

$ kubectl apply -f pod.yaml

Now get the pod details:

$kubectl gets pod/segfault

segment defective0/1CrashLoopBackOff1 (7s ago)19er

The pod is stuck and stuck in a reboot loop. Use the <terminal inline>describe<terminal inline> command to find out the cause:

$ kubectl describe pod/segfaultName: segfaultNamespace: default...Containers: segfault: ... Last State: Terminated Reason: Error Exit Code: 139

The exit code is reported as 139, indicating that a segmentation error caused the application in the container to crash.

Solution for segmentation errors

Once you've identified segmentation errors as the cause of your container terminations, you can proceed to mitigate them and prevent future recurrences.

If the error occurs in a third-party container image, your options are limited. You should report an issue with the developer to investigate the cause of the unexpected memory access attempts. If the problem is in your own software, you can take more specific troubleshooting steps to figure out what's wrong.

Error code identification

First, look for obvious areas of your code that might be affected by segmentation issues. You may be able to use your container's logs to calculate the sequence of events that led to the error:

(Video) What Is Segmentation Fault & How To Find That In C & C++?

$ docker registry mi container $ kubectl registry pod/my-pod

Use the container activity to find where in the source the error originated. If there is an array access, pointer reference, or unprotected memory write in the area, this may be the cause of the problem.

environmental intolerance

Another common cause of these errors is when a shared library update introduces incompatibilities with existing binaries. This can cause memory access violations if the versions loaded deviate from the supported range.

Try reverting any recent dependency changes in your containers. This can help eliminate problems caused by third-party library updates.

In rare cases, persistent segmentation errors with no obvious explanation can be caused by incompatibilities with the computer's physical hardware. They could even be symptoms of a memory defect. This type of problem is less likely in the context of a typical Kubernetes cluster running on onePublic cloud provider K8s🇧🇷 RunningStorageServicing your own hardware can help rule out physical problems.

directed debugging

You can use Linux tools to debug SIGSEGV signals with greater precision. Segmentation errors always generate kernel log messages. Because containers run as processes within their host's kernel, they are written even if the error occurred inside a container.

Check your system log by viewing the contents of <inline terminal>/var/log/syslog<inline terminal>:

$ sudo cola -f /var/log/syslog

This command continuously streams logs to your terminal until you cancel it with Ctrl+C. Now try to reproduce the event that caused the segmentation error. The SIGSEGV signal looks like this in the protocol:

hola-mundo[2631584]: Segmentfehler in 7f4624c6cfe0 ip 000055730c3621ed sp 00007ffce90e35f0 Fehler 7 in hola-mundo[55730c362000+1000]

The log can be interpreted as follows:

  • <bold terminal online>at <address><bold terminal online>:The forbidden memory address that the code tried to access.
  • <bold online terminal>ip <pointer><bold online terminal>:The memory address of the code that committed the violation.
  • <bold inline terminal>sp <pointer><bold inline terminal>:The stack pointer for the operation, providing the address of the last program request on the stack.
  • <terminal bold inline> Fehler <code><terminal bold inline>:The error code gives an indication of the type of operation attempted. Common codes include <line terminal>6<line terminal>, writing to an unallocated area; <inline terminal>7<inline terminal>, writing to a readable but not writable area; <line terminal>4<line terminal>, read from unused area; and <inline terminal>5<inline terminal>, reading from a write-only area.

Accessing the kernel log gives you a better understanding of what the code is doing at the time the error occurs. Although this log cannot be accessed directly from the containers, you should still be able to get the details of the segmentation errors if you have root access to the host machine.

Elegant handling of segmentation errors

Another way to fix segmentation errors is to properly handle them within your code. You can use libraries likeSegv captureto capture SIGSEGV signals and convert them into software exceptions. You can then treat them like any other exception, giving you the ability to log details to your error monitoring platform and recover without error.

(Video) Process Exited With Code 1 Command Prompt Error on Windows 10/11 FIX

While dealing with SIGSEGV is a good way to avoid fatal errors, it is worth thoroughly investigating and fixing each occurrence of this error. A segmentation error indicates that the program is doing something that the Linux kernel specifically forbids, indicating serious reliability or security vulnerabilities in your code. Simply capturing the signal and ignoring it could cause other problems in your program if you expect to have read or write memory that turns out to be out of bounds.

final thoughts

Segmentation errors occur when a program tries to use memory that it is not allowed to access. They also arise when data is written to read-only memory and vice versa. As you saw in this article, these errors are often the result of simple programming errors. You also discussed how to identify a segfault as the cause of container terminations and how to start troubleshooting the segfaults that occur in your programs.

Staying one step ahead of these errors will ensure your applications run with maximum reliability and uptime.ContainIQ's Kubernetes monitoring platformallows you to track container terminations and their causes in real-time, helping you identify when things aren't working as they should. You can use it to configure alerts for failed containers and then examine their exit codes to identify segmentation failures in your cluster. This gives you the ability to respond to segmentation failures as they occur and keep your workloads in a healthy state.


How do I fix exit code 139? ›

This is a very common mistake and can be fixed by simply updating the version number whenever you update a library and its binaries.

What is exit code 139 in container? ›

Exit Code 139 means that the container received a SIGSEGV signal from the operating system. This indicates a segmentation error – a memory violation, caused by a container trying to access a memory location to which it does not have access.

What is exit code 139 in Linux? ›

When a container exits with status code 139, it's because it received a SIGSEGV signal. The operating system terminated the container's process to guard against a memory integrity violation. It's important to investigate what's causing the segmentation errors if your containers are terminating with code 139.

How do I fix segmentation fault in Linux? ›

In Linux, the error “segmentation fault (core dumped)” occurs when the process requires additional memory that the operating system does not permit access. It can be resolved by removing the “lock” files through the “rm” command, clearing the cache repository, or killing the process via “process id”.

What does the exit status code 139 says something went wrong? ›

It means the program crashed before it exited. You need to debug the program. For example, you need to check whether the file is successfully opened after fopen .

How to fix SIGSEGV error in C? ›

SIGSEGV Error is caused by an invalid memory reference or segmentation fault. The most common causes are accessing an array element out of bounds, or using too much memory. Check this link for more clarity. This type of problem generally comes due to array out of index problem.

What causes SIGSEGV? ›

1) Segmentation Fault (also known as SIGSEGV and is usually signal 11) occur when the program tries to write/read outside the memory allocated for it or when writing memory which can only be read.In other words when the program tries to access the memory to which it doesn't have access to.

How do I get rid of a container that is not running? ›

It's possible to remove containers that aren't running? You can use the command docker container list --all to see all the container no matter the state, and then use the command docker rm containerName to delete the container.

How do you exit a container in Linux? ›

Just Stopping the Container

If you want to stop and exit the container, and are in an interactive, responsive shell - press ctrl+d to exit the session. You could as well type the exit command. TL;DR: press ctrl+c then ctrl+d - that means, keep the ctrl key pressed, type a c, and let go of ctrl.

How do you check the exit code in Linux? ›

To check the exit code we can simply print the $? special variable in bash. This variable will print the exit code of the last run command.

How do I exit Linux safely? ›

At the command prompt, type exit. Depending on the device configuration, you may be presented with another menu, for example: Access selection menu: a: Admin CLI s: Shell q: Quit Select access or quit [admin] : Type q or quit to exit.

How to avoid segmentation fault in Linux? ›

5. How to Prevent Segmentation Faults?
  1. Dynamic memory allocation.
  2. Indirect memory access through pointers.
  3. Array indexes with values higher than their current allocated sizes.
  4. Type consistency throughout the code and function parameters calling convention.
  5. String and buffer operations.
Jan 1, 2022

How to fix the segmentation fault in C on Linux? ›

  1. Step 1: Remove the lock files present at different locations. ...
  2. Step 2: Remove repository cache. ...
  3. Step 3: Update and upgrade your repository cache. ...
  4. Step 4: Now upgrade your distribution, it will update your packages. ...
  5. Step 5: Find the broken packages and delete them forcefully.
Apr 2, 2019

What does segmentation fault mean in Linux? ›

A segmentation fault occurs when a program attempts to access a memory location that it is not allowed to access, or attempts to access a memory location in a way that is not allowed (for example, attempting to write to a read-only location, or to overwrite part of the operating system).

What is exit code 139 interrupted by signal 11 SIGSEGV? ›

This indicates a bug in your program. In a Python program, this is either a bug in the interpreter or in an extension module being used (and the latter is the most common cause).

Can you block SIGSEGV? ›

SIGSEGV is an MMU (virtual memory hardware) exception. SIGSEGV does not necessarily imply that any UB occurred, it only means you tried to access memory that isn't mapped. The reason you can't block it is because after the signal is raised the program retries the same operation that caused the problem.

How do I clear all containers? ›

  1. Stop the container(s) using the following command: docker-compose down.
  2. Delete all containers using the following command: docker rm -f $(docker ps -a -q)
  3. Delete all volumes using the following command: docker volume rm $(docker volume ls -q)
  4. Restart the containers using the following command:

How do I find out what containers are running? ›

To check the container status and run IBM Workload Automation commands, you need to access the containers as described below:
  1. Obtain the container ID by running the following command: docker ps. ...
  2. Access the Docker container by running the following command: docker exec -it <container_id> /bin/bash.

How do I force remove all containers? ›

Remove all Containers: To remove all containers from the docker-machine, we need to get the ids of all the containers. We can simply get the ids of the containers with the command docker ps -aq, then by using the docker rm command, we can remove all the containers in the docker-machine.

How do I find the exit code for a container? ›

Checking Exit Codes in Kubernetes

If you are running containers as part of a Kubernetes cluster, you can find the exit code by gathering information about a pod. Run the kubectl describe pod [POD_NAME] command.

What is the command to exit from container? ›

You can leave the container with exit or ctrl-D .

What is the command to exit a container without killing the process? ›

Press Ctrl-P, followed by Ctrl-Q, to detach from your connection. You'll be dropped back into your shell but the previously attached process will remain alive, keeping your container running. You can check this by using docker ps to get a list of running containers.

How can the errors in segmentation fault be avoided? ›

You have to check that no_prod is < 1024 before writing to it, otherwise you'll write in unallocated memory, which is what gives you a segmentation fault. Once no_prod reached 1024 you have to abort the program (I assume you haven't worked with dynamic allocation yet).

How to find out what causes segmentation fault in C? ›

List of Common Reasons for Segmentation Faults in C/C++
  1. Accessing an array out of bounds.
  2. Dereferencing NULL pointers.
  3. Dereferencing freed memory.
  4. Dereferencing uninitialized pointers.
  5. Incorrect use of the "&" (address of) and "*" (dereferencing) operators.
  6. Improper formatting specifiers in printf and scanf statements.
Jan 27, 2020

What kind of error is this segmentation fault? ›

Segmentation Fault in C/C++

Segmentation faults in C/C++ occur when a program attempts to access a memory location it does not have permission to access. Generally, this occurs when memory access is violated and is a type of general protection fault.

What are Linux exit codes? ›

In Linux, an exit code indicates the response from the command or a script after execution. It ranges from 0 to 255. The exit codes help us determine whether a process ran: Successfully.

What is bash default exit code? ›

For the bash shell's purposes, a command which exits with a zero (0) exit status has succeeded. A non-zero (1-255) exit status indicates failure. If a command is not found, the child process created to execute it returns a status of 127. If a command is found but is not executable, the return status is 126.

What is the command to view source code in Linux? ›

There is an apt-get paramater that you can use to get packages source code: apt-get source . To get coreutils' source code, use sudo apt-get source coreutils .

How do I exit Linux manually? ›

Just press “q” key to quit out of a man page. Yes it is that easy, simply pressing “q” will exit out of the man command properly.

How do I boot into safe mode in Linux? ›

Boot to Recovery Mode in Ubuntu

If your computer boots too quickly, you're going to need to do this immediately after powering it on. Quickly press either the Shift or Escape key. On newer computers, it's probably Escape . The timing has to be near perfect on some computers, so you may have to press it repeatedly.

How to cause a segmentation fault Linux? ›

Segmentation faults can arise from similar conditions. A buffer overflow, such as trying to reach outside the bounds of an array, can cause a segfault, or trying to access memory that has not been allocated or has been deleted. Trying to write to memory that is read-only can also cause a memory error.

Is a segmentation fault a memory leak? ›

Most memory errors which aren't memory leaks end up resulting in a segmentation fault. A segmentation fault is raised when the operating system realizes that your program is trying to access memory that it shouldn't have access to.

What is exit code 139 in AWS? ›

139 – Occurs when a segmentation fault is experienced. Likely, the application tried to access a memory region which it not available, or there is an unset or invalid environment variable. 255 – Occurs when the ENTRYPOINT CMD command in your container failed due to an error.

What does the git process exited with the code 139? ›

139 is a standard exit code to indicate "the program exited because of a Segmetation fault" -- e.g: most porbably a null pointer bug. Does IntelliJ have a console somewhere, where you could see exactly what git commands are executed ? this error may be triggered by a hook rather than by git itself. Are you using WSL?

How do you avoid SIGSEGV errors? ›

Don't use pointers. And if you do always check for NULL pointers. Minimize the use of pointers, access any kind of arrays within its bounds, check whether pointer is NULL before using, make sure you allocate new memory before using a pointer which you have already freed (it may not be giving NULL ).

What triggers SIGSEGV? ›

SIGSEGV is triggered by the operating system, which detects that a process is carrying out a memory violation, and may terminate it as a result.

How do I reset my AWS environment? ›

Sign in to the AWS Management Console and open the Amazon Redshift console at .
  1. On the navigation menu, choose Clusters to display your list of clusters.
  2. Choose the examplecluster cluster. For Actions, choose Delete. ...
  3. Confirm the cluster to be deleted, then choose Delete cluster.

Why do I get Segmentation Fault in C? ›

In practice, segfaults are almost always due to trying to read or write a non-existent array element, not properly defining a pointer before using it, or (in C programs) accidentally using a variable's value as an address (see the scanf example below).

How do you check the exit code? ›

To display the exit code for the last command you ran on the command line, use the following command: $ echo $? The displayed response contains no pomp or circumstance. It's simply a number.

How do I fix process exited code? ›

How Can I Fix Process Exited With Code 1 Error in Windows 11/10?
  1. Try restarting your computer. ...
  2. Try opening the Task Manager and taking a look at your processes. ...
  3. Try running your program as an administrator.
  4. Make sure you have all the required dependencies installed.
  5. Check your system for viruses and malware.

How do I resolve a git issue? ›

Procedure to solve an issue on GIT repositories:
  1. Firstly, we must update the code in our local system.
  2. Review the issues on GIT.
  3. When we click on issue, that issue will expand, after reading the issue put the status to this issue as ”S2 – In analysis”.
  4. Assign that issue to yourself on git.

How do I reset git? ›

git reset --soft : Known as a soft reset, this updates the current branch tip to the specified commit and makes no other changes. git reset --hard : Known as a hard reset, this updates the current branch tip to the specified commit, unstages any changes, and also deletes any changes from the working directory.


1. How to solve apache2 error while starting | job for apache2.service failed
2. George Hotz | Programming | simple C backend for tinygrad (the tiny corp needs contracts) | buy c3
(george hotz archive)
3. HackTheBox - Ellingson
4. Accelerate the Adoption of "You Build it, You Own it" with Self-Service Operations
5. 2020.07.28 - Brown Bag: Creating a Snapshot-based, Feedback-guided Fuzzer
(GitLab Unfiltered)
6. Pair Programming with OhMyPosh's Jan De Dobbeleer - Together we code a Nightscout segment in Go!
(Scott Hanselman)
Top Articles
Latest Posts
Article information

Author: Maia Crooks Jr

Last Updated: 06/17/2023

Views: 6508

Rating: 4.2 / 5 (63 voted)

Reviews: 86% of readers found this page helpful

Author information

Name: Maia Crooks Jr

Birthday: 1997-09-21

Address: 93119 Joseph Street, Peggyfurt, NC 11582

Phone: +2983088926881

Job: Principal Design Liaison

Hobby: Web surfing, Skiing, role-playing games, Sketching, Polo, Sewing, Genealogy

Introduction: My name is Maia Crooks Jr, I am a homely, joyous, shiny, successful, hilarious, thoughtful, joyous person who loves writing and wants to share my knowledge and understanding with you.