In the last article, we fuzzed annocheck with radamsa. This found several crashes. But annobin-9.94 has everything cleaned up that radamsa can find. We're not done fuzzing yet. We need to try a guided coverage fuzzer like AFL++ to see if there are more problems.
The strategy for using AFL++ is to build annocheck as a static application. We will start fuzzing it and make some adjustments based on how the initial runs go. Then we will let it run for several hours to see what it finds.
The first step is to download a copy of annobin-9.94 from here. Next install and build the source rpm. Then cd into the annobin-9.94 source directory. If you need to setup an rpm build environment, there are steps here.
Next, we build annocheck as follows:
export AFL_PATH=/home/builder/working/BUILD/repos/git-repos/AFLplusplus
CC=afl-clang-lto CXX=afl-clang-lto++ RANLIB=llvm-ranlib AR=llvm-ar LD=afl-clang-lto++ NM="llvm-nm -g" ./configure --without-gcc_plugin --without-tests --without-docs --enable-static --with-gcc-plugin-dir=`gcc -print-file-name=plugin`
The configure script errors out like this:
config.lt: creating libtool
checking for GCC plugin headers... no
configure: error: GCC plugin headers not found; consider installing GCC plugin development package
After much digging around, I decided that since we are making a static app and not worried about the gcc plugins, we'll just fix the tests to think everything is OK. Apply the following patch:
--- annobin-9.94.orig/config/gcc-plugin.m4 2021-08-31 10:02:12.000000000 -0400
+++ annobin-9.94/config/gcc-plugin.m4 2021-09-15 16:31:49.946843182 -0400
@@ -129,7 +129,7 @@ int main () {}
[gcc_plugin_headers=yes],
[gcc_plugin_headers=no])
-if test x"$gcc_plugin_headers" = xyes; then
+if test x"$gcc_plugin_headers" = xyes -o x"$static_plugin" = xyes; then
AC_MSG_RESULT([yes])
else
AC_MSG_RESULT([no])
Since we modified the m4 scripts, we need to regenerate the configure script. I used the autogen.sh file from the audit-userspace project as a convenience. Download it and do the following:
CC=afl-clang-lto CXX=afl-clang-lto++ RANLIB=llvm-ranlib AR=llvm-ar LD=afl-clang-lto++ NM="llvm-nm -g" ./configure --without-gcc_plugin --without-tests --without-docs --enable-static --with-gcc-plugin-dir=`gcc -print-file-name=plugin`
Now configure finishes correctly. We are now ready to build it. The first step is to export the instrument variable and the run make:
make
This fails as follows:
error: unable to load plugin 'annobin': 'annobin: cannot open shared object file: No such file or directory'
make[1]: *** [Makefile:442: annocheck-annocheck.o] Error 1
After much digging around, I found that the cause of this failure is a plugin option in the annocheck_CFLAGS. Open annocheck/Makefile and go down to line 338 and remove -fplugin=annocheck. Now run make again. Now it compiles annocheck.
Time to setup for fuzzing. We will use the same program that we created to fuzz using radamsa. Go check the last article if you need the recipe. Do the following:
mkdir in
cp ~/test/hello in
mkdir -p /tmp/proj/out
ln -s /tmp/proj/out out
# Then we switch to root and do some house keeping to let this
# run as fast as possible
su - root
service auditd stop
service systemd-journald stop
service abrtd stop
auditctl -D
auditctl -e 0
echo core >/proc/sys/kernel/core_pattern
cd /sys/devices/system/cpu
echo performance | tee cpu*/cpufreq/scaling_governor
exit
Now it's time to fuzz!
And we get the following:
There are plenty of guides that tell you what each of these items mean. Please search the internet and find one if you are curious. My main goal in this is to show you how to overcome many obstacles to get to the prize.
What I'd like to point out is 2 things. We are getting about 248 executions per second. That is kind of low. And we hit 2 crashes in 25 seconds. I hit Ctl-C to exit so that we can take an initial look at the crashes. The directory structure of the out directory looks like this:
out
└── default
├── cmdline
├── crashes
│ ├── id:000000,sig:06,src:000000,time:17826,op:havoc,rep:4
│ ├── id:000001,sig:11,src:000000,time:23980,op:havoc,rep:4
│ └── README.txt
├── fuzz_bitmap
├── fuzzer_setup
├── fuzzer_stats
The actual test cases are in the crashes directory. So, what do we actually do with these? The answer is that we build another copy of annocheck using the address sanitizer and pass these to the sanitized annocheck to see what happens.
The last article explains how to build a sanitized copy of annocheck. Just open the tarball in a different location so that you don't overwrite the AFL++ build of annocheck and compile it. I built it in a dirrectory called annobin-9.94.test. You can do it anywhere, just correct the location.
Next, from the sanitized annocheck dir, we run:
annocheck: Version 9.94.
Hardened: id:000000,sig:06,src:000000,time:17826,op:havoc,rep:4: FAIL: pie test because not built with '-Wl,-pie' (gcc/clang) or '-buildmode pie' (go)
Hardened: id:000000,sig:06,src:000000,time:17826,op:havoc,rep:4: FAIL: bind-now test because not linked with -Wl,-z,now
annocheck: annocheck.c:1250: find_symbol_in: Assertion `symndx == sym_hdr->sh_size / sym_hdr->sh_entsize' failed.
Aborted
Aborted? That stinks. Programs that call abort trick AFL++ into thinking this is a valid crash. This is because abort creates a core dump and that's exactly what AFL++ is looking for, The solution to this is that we need to override the abort call. The program depends on abort stopping execution. Removing it altogether means it will be running in code that the developer never intended. Since it's a false positive, we'll replace the call to abort with exit. So, let's go find it and recompile.
Grep'ing for abort doesn't find anything. But looking again at the error message says this occurs on line 1250 in annocheck.c. Opening that file finds an assert macro at that line. That means we need to override the assert macro with one that calls exit.
In annocheck/annocheck.h we find the include of assert.h around line 21. Comment that out using old style /* */ syntax. Next, a little farther down after all the includes, put the new macro:
#define assert(e) ((void) ((e) ? 0 : __assert (#e, __FILE__, __LINE__)))
#define __assert(e, file, line) ((void)printf ("%s:%u: failed assertion `%s'\n", file, line, e), exit(1), 0)
Now, save, run make clean, and make. One other thing to note, annocheck makes temp directories to work with. When it aborts or crashes, it cannot clean those up. For now
Back to fuzzing. We really do not need the previous runs. You can clear them out by
Actually, let's take a quick detour before going back to fuzzing. If you recall from above, I mentioned that the number of executions per second is kind of low. Let's fix that before we start fuzzing.
AFL++ has a mode of operation called deferred instrumentation mode. What this does is instead of starting an application from scratch, letting the runtime linker resolve all symbols, open and parse configuration files, and do general initialization...you can insert code that marks a spot where all of that has completed. AFL++ will fork new copies from that spot. The placement matters. You need to do it after general startup, and before it reads the file with the fuzzing data in it.
So, if we open annocheck.c and locate the main function at line 1943, let's scan down looking for the place to put it. It checks some arguments, checks elf version, loads rpm's configuration (that takes some time), and then processes the command line arguments. This looks like the spot. Copy and paste this at line 1967 just above processing command line args:
__AFL_INIT();
#endif
Let's make one more change. Annocheck creates a temporary directory while processing a file. Let's put that over in /tmp so as not to wear out a real hard drive. Look for a function named create_tmpdir. At line 1296 it copies a file name template into a buffer. Let's prepend "/tmp/" to that string. If you look a little further down, it calls concat and rewrites the tempdir variable with the current working directory as the path root. We don't want that. So, comment it out and the following line which outputs where the tempdir is. You have to use /* */ commenting on this program.
Save and exit. Rerun make. In another terminal, run the following script. It will periodically clear out the annocheck.data temp directories that aren't removed on a crash.
And in the other terminal, from the annocheck directory, run this:
Now let's look at the executions per second:
Now we are getting about 4.5 times the speed. Using deferred instrumentation mode makes a huge improvement. The reason we want it to run faster is that the more executions per second, the more test cases it can try in the same amount of time. On very mature programs, it can take days of fuzzing to find even one bug.
We'll stop here. You can let it run for a couple hours. You will probably have over a hundred "unique" crashes to choose from. In the next article, I'll go over how to sort through all that efficiently. If you do get a collection, remember to copy them to persistent disk storage. The /tmp dir goes away when you reboot the computer.
