Memory Corruption – Debugging Tools in Linux

By | 08/07/2013

In part I, we learnt about the memory corruption and the probable causes. Presently, there are plethora Linux tools available to combat the issues of memory corruption. Such Linux tools assist a great deal in detecting the memory corruption and resolving them. In this article we will cover 3 popular open source tools available for debugging memory corruption related problem on Linux.

NOTE – Information related to installation of debugging tools is Ubuntu specific.

 

Memory Corruption Debugging Tools

1. Electric Fence

Electric Fence is a memory debugger, or sometimes also called malloc debugger as it detects memory corruption related to memory allocated by malloc(). It excels in detecting two kinds of programming issues related to heap memory corruption

  1. The buffer overrun of a memory allocated by ‘malloc()’
  2. Access to memory that has been freed by ‘free()’. Well, electric fence will detect even a read access, along with the write.

The way it helps is, we run our executable in a debugger along with electric fence. and electric fence will make the program error at the point either where the buffer is going out of bounds of malloc-ed boundaries, or any access to a memory already freed. Hence, this way we come to know (with the error thrown by electric fence) about the statement attempting to corrupt a memory.

In all, the crash point is moved to the precise point of the first invalid memory write/read and hence helping us to determine where memory corruption is taking place. Well, the way things change is, with gdb we’ll see the crash wherever it happens, but with efence, the crash location changes to where the corruption happens.

To begin with, we will see how to set up electric fence.

The following command works on Ubuntu system to install the open source tool.

$ sudo apt-get install electric-fence

However, one can also install electric-fence through synaptic or aptitude.

Once installed, one can see that it is a library which contains overloaded definitions of malloc(), free(), calloc(), and other such memory related api’s which are generally available in libc. The way it works is, efence places an inaccessible page after each memory block allocated by ‘malloc()’. And for which it has to use the virtual memory hardware of the system. Once, we go beyond the allocated memory, it will come to know the invalid access and will trigger the error.

To use electric fence, just compile the sources with ‘-g’ debug option and link it to the efence library.

Lets take our old heap corruption example, which looks like,

#include <stdio.h>
#include <stdlib.h>
int main()
{
  int *pData = NULL;
  int num = 12;
  pData = (int*) malloc (num * sizeof (int));
  //...do stuff use the memory
  free(pData);

  pData[0] = -1;
  pData = (int*) malloc (num * sizeof (int));
  //...do stuff use the memory
  free(pData);
  return 0;
}

Here is how we compile the executable now using efence .

$ gcc -g -Wall  heapcorrupt.c -o heapcorrupt   -lefence

Its essential to understand in the above linking, that we are asking the final built executable to use symbols ‘malloc’ and ‘free’ from the libefence library rather than libc. For me, libefence library is place at

/usr/lib/

In certain cases, if gcc is still using libc definitions, try giving the path to the library in the compilation options.

Lets confirm, if our binary is using the required ‘malloc’ and ‘free’ symbols through ‘nm’ tool, which lists all the symbols.

$ nm -a heapcorrupt | grep malloc

        U malloc

The above nm output implies, the symbol ‘malloc’ is unknown (U), hence its externally defined in a library.

Now, we are sure the efence will be effective in error-ing out the program where the invalid memory access takes place. Lets run the built binary.

$ ./heapcorrupt

 Electric Fence 2.1 Copyright (C) 1987-1998 Bruce Perens.

Segmentation fault (core dumped)

We still get the segmentation fault, but the Electric Fence copyright information in the output indicates the seg fault is at the detected location of memory corruption. To find that, we shall again use gdb debugger,

$ gdb ./heapcorrupt

GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu1) 7.4-2012.04

Copyright (C) 2012 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "i686-linux-gnu".

For bug reporting instructions, please see:

<http://bugs.launchpad.net/gdb-linaro/>...

Reading symbols from /home/ubuntu/progs/memorycorruption/heapcorrupt...done.

(gdb) break 8

Breakpoint 1 at 0x80484ed: file heapcorrupt.c, line 8.

(gdb) run

Starting program: /home/ubuntu/progs/memorycorruption/heapcorrupt

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".

Breakpoint 1, main () at heapcorrupt.c:8

8       pData = (int*) malloc (num * sizeof (int));

(gdb) step

 Electric Fence 2.1 Copyright (C) 1987-1998 Bruce Perens.

10       free(pData);

(gdb) step

12       pData[0] = -1;

(gdb) step

Program received signal SIGSEGV, Segmentation fault.

0x08048510 in main () at heapcorrupt.c:12

12       pData[0] = -1;

In the above gdb debugging session, we have put a breakpoint at line 8, i.e. where the first memory allocation takes place. Then we step through the program until we get a seg fault at line 12, i.e.

    pData[0] = -1;

This is the precise location which is corrupting the memory by writing onto a memory location which has already been free-ed. Thanks to electric fence. :)

2. Valgrind

Valgrind is not just one tool, but a suite of tools for debugging and profiling memory issues. It has the potential to do a complete memory mismanagement check up of the software. Hence, valgrind is one of the most useful and popular tool for most of the memory related issues in a program.

The valgrind suite of tools includes a lot of tools that do profiling as well as checking. To know about what all these tools are along with their options, refer this page.

However, for this article, we shall focus just on the tools related to memory checking. The major one being “memcheck”.

Before we jump into beginning to use valgrind tool memcheck, we should be familiar with what kind of memory related problems it helps sorting and how does it work.

Following a like a list in brevity stating all the possible scenarios valgrind will be most useful in,

  1. Accessing overflowed or under-flowed buffer memory
  2. Issues related to uninitialized memory
  3. Accessing memory which has been free-ed.
  4. Accessing inappropriate areas of stack
  5. Free-ing an already free-ed memory
  6. Mismatched malloc/new and free/delete combinations.
  7. Memory Leaks
  8. Overlapping of memory areas during memcpy, etc

We are all set to start using valgrind memcheck tool and see how it works towards detecting and debugging memory errors.

Here is a program which, which has a couple of memory related issues. I won’t disclose them right now, though they are not so difficult to find out. This is just for an example, hence lets use valgrind memcheck tool to first detect and then debug the issues.

#include <stdio.h>
#include <stdlib.h>

#define MAX 7
#define MAGIC 65

int main()

{
    char *pChar = NULL;
    int i = 0;
    pChar = (char*) malloc (MAX *sizeof(char));

    if (!pChar)
    {
       printf("Mem Error!\n");
       return 0;
    }   

    for ( ; i < MAX; i++)
    {
       pChar[i] = MAGIC + i;
    }  

    pChar[i] = 0;
    printf("String is %s\n", pChar);
    free(pChar);

    return 0;
}

Valgrind memcheck tool works in a little different way, as it doesn’t need anything special to be done during compilation. The source is built normally as for any generic debugging with the –g debug option.

$ gcc –g –Wall memprog.c  -o  memprog

Once, we have the executable built we use valgrind virtual machine to run it and insert instrumentation information to detect memory issues and come up with related memory errors if there exist any. The way we run the valgrind memcheck tool is

$ valgrind –tool=memcheck ./memprog

So the syntax looks like

$ valgrind – tool= [toolname] [options] [command –to-run]

For our above valgrind command, I get following output

==2132== Memcheck, a memory error detector

==2132== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.

==2132== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info

==2132== Command: ./memprog

==2132==

==2132== Invalid write of size 1

==2132==    at 0x80484DC: main (memprog.c:22)

==2132==  Address 0x41ef02f is 0 bytes after a block of size 7 alloc'd

==2132==    at 0x402BE68: malloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)

==2132==    by 0x8048498: main (memprog.c:11)

==2132==

==2132== Invalid read of size 1

==2132==    at 0x4089E29: vfprintf (vfprintf.c:1630)

==2132==    by 0x4091EFE: printf (printf.c:35)

==2132==    by 0x405E4D2: (below main) (libc-start.c:226)

==2132==  Address 0x41ef02f is 0 bytes after a block of size 7 alloc'd

==2132==    at 0x402BE68: malloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)

==2132==    by 0x8048498: main (memprog.c:11)

==2132==

String is ABCDEFG

==2132==

==2132== HEAP SUMMARY:

==2132==     in use at exit: 0 bytes in 0 blocks

==2132==   total heap usage: 1 allocs, 1 frees, 7 bytes allocated

==2132==

==2132== All heap blocks were freed -- no leaks are possible

==2132==

==2132== For counts of detected and suppressed errors, rerun with: -v

==2132== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)

Wow… so without even have to stare at the code, it tells us the source contains two memory errorrs. That is,

==2132== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)

Not just the error information details, lets try to understand what each line of the obtained output means here,

The first few line just describes that we have used the memcheck tool from the valgrind suite along with the copyright details.

==2132== Memcheck, a memory error detector

==2132== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.

==2132== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info

==2132== Command: ./memprog

The next piece of information refers to the first memory error.

==2132== Invalid write of size 1

==2132==    at 0x80484DC: main (memprog.c:22)

==2132==  Address 0x41ef02f is 0 bytes after a block of size 7 alloc'd

==2132==    at 0x402BE68: malloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)

==2132==    by 0x8048498: main (memprog.c:11)

It says an invalid write of size one. Seems like, there has been a 1 byte write to an overflowed or invalid memory. The second line provides us with the exact instruction address along with the line number in our source code – memprog.c : 22, i.e. line number 22. The line 22 is,

    pChar[i] = 0;

It is a statement which is writing to a buffer. Lets check out the size of the allocated buffer and index value ‘I’ at line 22. We can do it by using gdb or a simple printf() statement.

$ gdb memprog

GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu1) 7.4-2012.04

Copyright (C) 2012 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "i686-linux-gnu".

For bug reporting instructions, please see:

<http://bugs.launchpad.net/gdb-linaro/>...

Reading symbols from /home/ubuntu/progs/memorycorruption/memprog...done.

(gdb) break 22

Breakpoint 1 at 0x80484d4: file memprog.c, line 22.

(gdb) run

Starting program: /home/ubuntu/progs/memorycorruption/memprog

Breakpoint 1, main () at memprog.c:22

22        pChar[i] = 0;

(gdb) print i

$1 = 7

The value of index ‘i’ comes out to be 7. Yes, this write is resulting to a buffer overflow as the memory block is till index 6. Hence, leading to a memory overflow. Thanks to valgrind memcheck tool for saving the effort of zeroing down to this line of code directly.

Make the needed changes to rectify this error, and be back on to the next piece of information in the output by the tool

==2132== Invalid read of size 1

==2132==    at 0x4089E29: vfprintf (vfprintf.c:1630)

==2132==    by 0x4091EFE: printf (printf.c:35)

==2132==    by 0x405E4D2: (below main) (libc-start.c:226)

==2132==  Address 0x41ef02f is 0 bytes after a block of size 7 alloc'd

==2132==    at 0x402BE68: malloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)

==2132==    by 0x8048498: main (memprog.c:11)

Now its an invalid read i.e. read access to an inappropriate memory and that too during a printf() statement. Lets take a thoughtful look at the printf() statement.

    printf("String is %s\n", pChar);

Oh yes, it wants to print a string which does not has a terminating null character. Hence, it gets to read memory location beyond the string buffer limits. In all, we are missing a terminating null character here. There are two programming rectifications to it – either increase the buffer limit to 8 or store only 6 characters to accommodate the null character. The solution is totally at the programmer’s discretion depending upon the design needs.

Though, we are done with debugging the two detected memory errors, lets see what memcheck has to say more.

String is ABCDEFG

==2132==

==2132== HEAP SUMMARY:

==2132==     in use at exit: 0 bytes in 0 blocks

==2132==   total heap usage: 1 allocs, 1 frees, 7 bytes allocated

==2132==

==2132== All heap blocks were freed -- no leaks are possible

==2132==

==2132== For counts of detected and suppressed errors, rerun with: -v

==2132== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)

It prints the statement and gives a summary of heap stats in terms of any heap memory leaks and a final summary of the memory errors.

We are done, and our example source code should be memory error free now. J

So, now we know there is this awesome tool for all the developers who work pretty much within the memories as it helps in making the code memory error proof.

3. Libsafe

This is a Linux library that works with gcc and makes sure the return address of a buffer on the stack is not clobbered. Hence, this also protects the binary from all the attacks which are possible due to buffer overruns in stack corruption.

Libsafe is a dynamically loadable library which comes with its own definitions of

strcpy(char *dest, const char *src)

strpcpy(char *dest, const char *src)

wcscpy(wchar_t *dest, const wchar_t *src)

wcpcpy(wchar_t *dest, const wchar_t *src)

strcat(char *dest, const char *src)

wcscat(wchar_t *dest, const wchar_t *src)

getwd(char *buf)

gets(char *s)

scanf(const char *format, ...)

realpath(char *path, char resolved_path[])

sprintf(char *str, const char *format, ...)

All the above api’s are unsafe to use normally as they don’t explicitly have a check/buffer-limit for the buffer overflows. If a function return address or a variable which resides in the stack is susceptible to buffer overflows, this raises a vulnerability to attack and gain unethical access to the user space of the system. One can also use this vulnerability to run an infectious program.

Libsafe re-defines these unsafe functions and handles the buffers securely. How it avoids the buffer overflows is by setting a realistic limits to the size of the buffers, accesses to beyond which will not be allowed.

To setup libsafe, one may download the complete package from synaptic or attribute, and then make and install the same. More details on libsafe and how to setup libsafe is here

Stack Smashing Protection

Stack Smashing Protection, also abbreviated as SSP is kind of an extension to GCC to provide security towards the stack overflows and its vulnerabilities. Basically it validates that the return address of a function which is stored in stack has not be clobbered by anyone. Besides, it also protects the local variables/registers saved in the stack from being altered. It may also sort the array variables wherever needed to safeguard from the buffer overruns.

Since it is an extension to the GCC, with the latest versions it has been included with the standard GCC. In case, one uses an older version or SSP disabled GCC, one can enable it simply by using following option

-fstack-protector-all

Therefore, taking a simple example source bugged with a buffer overflow

#include <stdio.h>
#include <string.h>

#define LEN 6

void cpyPrint(char *str)
{
   char aBuf[LEN];
   strcpy(aBuf, str);
   printf("String is %s\n", aBuf);
}

int main()
{
   char *aStr = "MyLinux";
   cpyPrint(aStr);

   return 0;
}

For me, it is included in the standard GCC, so I don’t need to use any option.

$ gcc -Wall bufferoverflow.c -o bufferoverflow

Running the code, gives us a descriptive error to alert us regarding the code being unsafe for

stack corruption.

$ ./bufferoverflow

String is MyLinux

*** stack smashing detected ***: ./bufferoverflow terminated

======= Backtrace: =========

/lib/i386-linux-gnu/libc.so.6(__fortify_fail+0x45)[0xb76e30e5]

/lib/i386-linux-gnu/libc.so.6(+0x10409a)[0xb76e309a]

./bufferoverflow[0x80484b2]

./bufferoverflow[0x80484d1]

/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75f84d3]

./bufferoverflow[0x80483d1]

======= Memory map: ========

08048000-08049000 r-xp 00000000 08:01 393821     /home/ubuntu/progs/memorycorruption/bufferoverflow

08049000-0804a000 r--p 00000000 08:01 393821     /home/ubuntu/progs/memorycorruption/bufferoverflow

0804a000-0804b000 rw-p 00001000 08:01 393821     /home/ubuntu/progs/memorycorruption/bufferoverflow

08cdc000-08cfd000 rw-p 00000000 00:00 0          [heap]

b75b0000-b75cc000 r-xp 00000000 08:01 658895     /lib/i386-linux-gnu/libgcc_s.so.1

b75cc000-b75cd000 r--p 0001b000 08:01 658895     /lib/i386-linux-gnu/libgcc_s.so.1

b75cd000-b75ce000 rw-p 0001c000 08:01 658895     /lib/i386-linux-gnu/libgcc_s.so.1

b75de000-b75df000 rw-p 00000000 00:00 0

b75df000-b7782000 r-xp 00000000 08:01 658295     /lib/i386-linux-gnu/libc-2.15.so

b7782000-b7784000 r--p 001a3000 08:01 658295     /lib/i386-linux-gnu/libc-2.15.so

b7784000-b7785000 rw-p 001a5000 08:01 658295     /lib/i386-linux-gnu/libc-2.15.so

b7785000-b7788000 rw-p 00000000 00:00 0

b7796000-b779a000 rw-p 00000000 00:00 0

b779a000-b779b000 r-xp 00000000 00:00 0          [vdso]

b779b000-b77bb000 r-xp 00000000 08:01 658307     /lib/i386-linux-gnu/ld-2.15.so

b77bb000-b77bc000 r--p 0001f000 08:01 658307     /lib/i386-linux-gnu/ld-2.15.so

b77bc000-b77bd000 rw-p 00020000 08:01 658307     /lib/i386-linux-gnu/ld-2.15.so

bfea9000-bfeca000 rw-p 00000000 00:00 0          [stack]

Aborted (core dumped)

Otherwise, due to some personal requirement, one wants to disable SSP, it can be done using option ‘-fno-stack-protector’.

$ gcc -Wall -fno-stack-protector bufferoverflow.c -o bufferoverflow

After disabling, the code runs all fine silent about the buffer overflow and leaving loopholes for the attackers.

$ ./bufferoverflow

String is MyLinux

In the End

We learnt about quite a lot about tools which can be use to debug and resolve different types and causes of memory corruptions and errors. Detecting and debugging memory corruption issues have always been a challenge and have taken a lot of the time and resources of the developers. Hence, an abundance of tools are available to help in fixing such kinds of errors/bugs which can lead to the scary behaviours and results. There are tons of more such tools but it may not be needed to know them all. However, we would be happy to hear more of any tool available which helps in some disparate way.

Finally, its is a considered as a good practice to verify one’s software with memory checking tools, irrespective of any unexpected behaviour/error. As we have seen, many of the bugs may pass through undetected silently unless they are being exploited or leading to a propagated bug. Hence, every developer should make sure the code is memory errors free at the end of the day. Therefore, do use memory debuggers assuring the software to be using memory properly and appropriately.

One thought on “Memory Corruption – Debugging Tools in Linux

  1. sumit kumar

    Nice article.
    The glibc also includes some memory-checking tools. One of them is mcheck() and MALLOC_CHECK_ (environment variable), enforce heap data structure consistency checking and other is mtrace(), traces memory allocation and deallocation to detect memory leaks issues.
    http://www.gnu.org/software/libc/manual/html_node/Heap-Consistency-Checking.html
    http://www.gnu.org/software/libc/manual/html_node/Allocation-Debugging.html
    Also Its better, if we first run any static analysis tool like cppcheck to detect buffer overflow, memory/resource leaks and un-initialized variables.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *