Tuesday, April 25, 2017

Aide made easy

The aide program is a good program to use to see if anything important has changed on your system. It works by creating a baseline which at some future point you use to compare with the current system to see what changed. It can track added files, deleted files, and changed files. The changed files it can tell you which attribute changed such as owner, group, other permissions, size, time, extended attributes, or if the file contents changed yielding a new SHA256 hash of the file.

The only wrinkle is that it requires you to actually move the database after you create it. From the command line that is a bit cumbersome because you need to figure out where things go.

I have a couple of little scripts that simplifies using the aide program: aide-init and aide-check. You can install these into /root/bin for ease of use. This is aide-init:

#!/bin/sh
echo "Creating temp copy of aide database"
rm -f /var/lib/aide/*
aide -i
echo "Overwriting stored aide database."
mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz
echo "Aide is ready for use."


and aide-check:

#!/bin/sh
aide -C


The only reason I have aide-check is just for symmetry. If have an init, you shoulf have a check.

So, the way to use these is to run aide-init to establish a baseline. Then next time you login, run aide-check to see if anything has changed. If so, investigate. If you are satisfied all changes are explained run aide-init again. Also if you see an update that needs to be installed, immediately do an aide-init after the update so that you have all changes rolled up to the database.

I have one interesting side note on this. I am leasing a VPS system for security research. A VPS system is a Virtual Private Server which is based on container technology. The interesting thing is that I can see when the host updates under me (since I share the base image with other servers).

Anyways...just thought I pass this along in case anyone finds this useful.

Monday, April 24, 2017

Sending email when audisp program sees event

I've been working our way up to writing an audispd plugin that will send an email whenever a specific event occurs. To do this, we will need a plan of action. The way it should work is that we need a way of identifying the event that we are interested in. We will use this to trigger the sending of email.

Needle in a Haystack
The first issue is we need to find the event. The easiest way to do this is to add a key to the audit rule. Then what we can do is write an audispd plugin that will have a callback function that uses the auparse_normalize API to normalize the event. Then we have the subject readily available. The normalizer API also hands us the key without needing to search for it. But that leads us to the question of what are we looking for?

Remember that audisp plugins can take a command line argument. So instead of opening a config file and parsing it, we''ll just ask that whatever the key is be passed as an argument on the command line. And just in case you want to match on multiple but related keys, we'll do substring matching using the strstr function. A more professional program might have a config file with many things or pairs of things to match against before alerting. But I'm going to keep it simple while illustrating the point.

So, the initial code looks like this:

#define _GNU_SOURCE
#include <stdio.h>
#include <sys/select.h>
#include <string.h>
#include <errno.h>
#include <stdlib.h>
#include <unistd.h>
#include <libaudit.h>
#include <auparse.h>

const char *needle = NULL;

// send_alert goes here

static void handle_event(auparse_state_t *au,
        auparse_cb_event_t cb_event_type, void *user_data)
{
    char msg[256], *name = NULL, *key = NULL;
    if (cb_event_type != AUPARSE_CB_EVENT_READY)
        return;

    /* create a message */
    if (!auparse_normalize(au, NORM_OPT_NO_ATTRS)) {
        if (auparse_normalize_key(au) == 1)
            key = auparse_interpret_field(au);
            if (key && strstr(needle, key)) {
                if (auparse_normalize_subject_primary(au) == 1)
                    name = strdup(auparse_interpret_field(au));
                /* send a message */
                printf("Alert, %s triggered our rule\n", name);
                //send_alert(name);
                free(name);
        }
    }
}

int main(int argc, char *argv[])
{
    auparse_state_t *au = NULL;
    char tmp[MAX_AUDIT_MESSAGE_LENGTH+1], bus[32];

    if (argc != 2) {
        fprintf(stderr, "Missing key to look for\n");
        return 1;
    }
    needle = argv[1];

    /* Initialize the auparse library */
    au = auparse_init(AUSOURCE_FEED, 0);
    auparse_add_callback(au, handle_event, NULL, NULL);

    do {
        int retval;
        fd_set read_mask;

        FD_ZERO(&read_mask);
        FD_SET(0, &read_mask);

        do {
            retval = select(1, &read_mask, NULL, NULL, NULL);
        } while (retval == -1 && errno == EINTR);

        /* Now the event loop */
         if (retval > 0) {
            if (fgets_unlocked(tmp, MAX_AUDIT_MESSAGE_LENGTH,
                stdin)) {
                auparse_feed(au, tmp,
                    strnlen(tmp, MAX_AUDIT_MESSAGE_LENGTH));
            }
        } else if (retval == 0)
            auparse_flush_feed(au);
        if (feof(stdin))
            break;
    } while (1);

    /* Flush any accumulated events from queue */
    auparse_flush_feed(au);
    auparse_destroy(au);

    return 0;
}


We can compile it like this:

gcc -o audisp-example4 audisp-example4.c -lauparse -laudit


And let's test it by triggering any time the "w" command is used. Let's add the following audit rule:

auditctl -a always,exit -F path=/usr/bin/w -F perm=x -F key=alert
Then run "w" and let's collect the audit log for testing.

$ w
 16:15:00 up 16 min,  1 user,  load average: 0.34, 0.40, 0.37
USER     TTY        LOGIN@   IDLE   JCPU   PCPU WHAT
sgrubb   tty2      16:04   16:04   2:10  49.27s /usr/lib64/firefox/firefox -con

$ ausearch --start recent --raw > test.log


Now we have some data to test the plugin.

$ cat test.log | ./audisp-example4 alert
Alert, sgrubb triggered our rule


Sendmail
The next trick is that we need to send an email notifying that something we're interested in happened. It turns out that there is a program called sendmail which is not exactly the MTA daemon sendmail, but a command line helper program. Its found at /usr/lib/sendmail. Which seems a bit odd but this is the historic place where it was located since the 1990's. I also think all MTA's provide their own stub for compatibility with this ancient standard. Fedora uses postfix, so let's just verify that we have a sendmail stub.

$ file /usr/lib/sendmail
/usr/lib/sendmail: symbolic link to /etc/alternatives/mta-sendmail
$ file /etc/alternatives/mta-sendmail
/etc/alternatives/mta-sendmail: symbolic link to /usr/lib/sendmail.postfix
$ file /usr/lib/sendmail.postfix
/usr/lib/sendmail.postfix: symbolic link to ../sbin/sendmail.postfix
$ file ../sbin/sendmail.postfix
../sbin/sendmail.postfix: cannot open `../sbin/sendmail.postfix' (No such file or directory)
$ file /sbin/sendmail.postfix
/sbin/sendmail.postfix: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=d82eedab7c1ad5c7b01a2a82a3b07919fc1b9089, stripped

OK. There it is. After 4 symlinks we land on the real program.

The way that you use it is to send via stdin an email composed line by line like you would a normal email. You have to have a To, From, Subject, and then the body of the email. It knows the email is finished when it sees a period followed by 2 consecutive newlines. So we might want an email like this:

To: security-team@company
From: root
Subject: Audit Alert
Just wanted to let you know that %s triggered the rule

.



This is pretty much what would show up in your inbox. OK, so let's add this to our program. The trick here is that we open sendmail with popen() writable and then pass our pre-formatted email to it and then close. It will take care of the rest assuming that you substitute a valid email address that is routable for our stand-in on the "To" line.

The code for send alert looks like this:

static void send_alert(const char *name)
{
    FILE *mail;
    mail = popen("/usr/lib/sendmail", "w");
    if (mail) {
        fprintf(mail, "To: %s\n", mail_acct);
        fprintf(mail, "From: root\n");
        fprintf(mail, "Subject: Audit Alert\n\n");
        fprintf(mail, "Just wanted to let you know %s triggered the audit rule\n",
             name);
        fprintf(mail, ".\n\n");         // Close it up...
        fclose(mail);
    }
}


So, copy and paste this into the program above the event_handler. Delete printf in event_handler and uncomment the call to send_alert. You should still be able to test this by cat'ing the test.log file into stdin of the program.

If you are happy with this, then you need to install the program by copying it to /sbin and then setup the configuration file so that audispd starts and passes the right key to the program to look for. The conf file is almost identical to the one from the audispd plugin blog post.

active = yes
direction = out
path = /sbin/audisp-example4
type = always
args = alert
format = string


Copy this to /etc/audisp/plugins.d/audisp-example4.conf and restart the audit daemon. You may need to disable selinux to get this to run. Selinux wraps the audit daemon and anything that is a child of it gets the same type as audit. So, the child is bound by auditd's selinux policy. Auditd can send emails, so there is hope. Ideally you would put this program in either a permissive domain or write a simple policy for it if you used this long term.

To test the program run the "w" program again and see if you get email. A more professional program would also want to build some hysteresis or rate limiting into the program so it cannot flood the recipient of the alert.


Conclusion
Once you have a basic audispd plugin, you can alert in many ways. You might even prefer to create a snmp trap. The beauty of this is that the logs can be tested in realtime as something occurs so that you can react to it right away. The technique that we used in this post was to give the event we are interested in a special key, we filter for that key. When we see our event, we send an email. This is one of the tricks you can use to spot things for alerting.

Tuesday, April 18, 2017

Writing a program using the auparse_normalize functions

In the last post we took a look at how an auparse program should be changed to run under the audit event dispatcher. While writing it, we saw a problem where we needed to look at several fields to figure out which field we needed to use. When this changes based on the event type and success vs fail, it can become confusing. There is a solution for that and we will look at what that is.

Auparse Normalizer
The auparse library has gained a new API starting around audit-2.7. It wasn't really ready for people to use until around 2.7.2 and still under active development. Its close enough to done that we can start using it in other programs.

In essence, all that is different in using the normalizer is that you call auparse_normalize() and then use a different field accessor function. You do not need to use auparse_find_field().

The field accessor functions come in 2 types. There are some that return a character string, and there are those that return an integer. The ones that return a string are returning metadata about the event and can be used directly. The ones that return an integer only move the auparse internal cursor to the correct field. Then you use the field just as we have in earlier programs by calling auparse_interpret_field(), or auparse_get_field_str(), or auparse_get_field_type(), etc.

The normalizer API will locate and save the field locations for the event's

  • session
  • subject's primary identity
  • subject's secondary identity
  • object's primary identity
  • object's secondary identity
  • results
  • key

It also provides metadata about the event's

  • kind
  • action being performed by subject
  • how the event was being performed


To illustrate how to use the API, consider the following program. Notice that we loop at the event level and do not need to iterate across records or fields.


#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <auparse.h>

int main(int argc, char *argv[])
{
        auparse_state_t *au = auparse_init(AUSOURCE_DESCRIPTOR, 0);
        if (au == NULL) {
                printf("Error initializing event source\n");
                return 1;
        }

        while (auparse_next_event(au) > 0) {
                // Event level
                const char *item, *evkind, *action, *str, *how, *field;
                printf("---\n");

                // Type:
                printf("event: %s\n", auparse_get_type_name(au));

                // Do normalization
                if (auparse_normalize(au, NORM_OPT_NO_ATTRS) == 1) {
                        printf("error normalizing - skipping\n");
                        continue;
                }

                // Event kind
                evkind = auparse_normalize_get_event_kind(au);
                if (evkind)
                        printf("  event-kind: %s\n", evkind);

                // Session
                if (auparse_normalize_session(au) == 1) {
                        const char *ses = auparse_interpret_field(au);
                        printf("  session: %s\n", ses);
                }

                // Subject
                if (auparse_normalize_subject_primary(au) == 1) {
                        const char *subj = auparse_interpret_field(au);
                        field = auparse_get_field_name(au);
                        if (strcmp(subj, "unset") == 0)
                                subj = "system";
                        printf("  subject.primary: %s=%s\n", field, subj);
                }

                if (auparse_normalize_subject_secondary(au) == 1) {
                        const char *subj = auparse_interpret_field(au);
                        field = auparse_get_field_name(au);
                        printf("  subject.secondary: %s=%s\n", field, subj);
                }

                // Action
                action = auparse_normalize_get_action(au);
                if (action) {
                        printf("  action: %s\n", action);
                }

                // Object
                if (auparse_normalize_object_primary(au) == 1) {
                        field = auparse_get_field_name(au);
                        printf("  object.primary: %s=%s\n", field,
                                        auparse_interpret_field(au));
                }

                if (auparse_normalize_object_secondary(au) == 1 ) {
                        field = auparse_get_field_name(au);
                        printf("  object.secondary:%s=%s\n", field,
                                        auparse_interpret_field(au));
                }

                // Object kind
                str = auparse_normalize_object_kind(au);
                if (strcmp(str, "unknown"))
                        printf("  object-kind: %s\n", str);

                // How action was performed
                how = auparse_normalize_how(au);
                if (how)
                        printf("  how: %s\n", how ? how : "missing-how");

                if (auparse_normalize_key(au) == 1) {
                        const char *key = auparse_interpret_field(au);
                        printf("  key: %s\n", key);
                }

                // Results
                if (auparse_normalize_get_results(au) == 1) {
                        int i = 0;
                        const char *res[] = { "fail", "success" };
                        item = auparse_interpret_field(au);
                        if (strcmp(item, "yes") == 0)
                                i = 1;
                        else if (strncmp(item, "suc", 3) == 0)
                                i = 1;
                        printf("  results: %s\n", res[i]);
                }
        }
        printf("---\n");
        auparse_destroy(au);
        return 0;
}


Compile this program using:

# gcc -o normalize  normalize.c -lauparse


To run the program pass log data to it from stdout

ausearch --start today --raw | ./normalize


You should see some output similar to this:

event: DAEMON_START
  event-kind: audit-daemon
  session: unset
  subject.primary: auid=system
  subject.secondary: uid=root
  action: started-audit
  object-kind: service
  results: success


You can use this program to look over normalized log data to aid in understanding where each of the parts come from.

OK, so how does this fix up the problem we saw last time? Now all we need to do is call auparse_normalize() and then auparse_normalize_subject_primary(). That's it. No conditional. Rather than present the whole program again, I will just give you the changed code that goes in the handle_event function.  Update: I am going to publish the whole function to emphasize that there is no longer a loop across records when using the auparse_normalize API.

static void handle_event(auparse_state_t *au,
        auparse_cb_event_t cb_event_type, void *user_data)
{
    if (cb_event_type != AUPARSE_CB_EVENT_READY)
        return;

    auparse_first_record(au);
    int type = auparse_get_type(au);
    if (type == AUDIT_USER_LOGIN) {
        char msg[256],  *name = NULL;
        const char *res = NULL;

        /* create a message */
        if (!auparse_normalize(au, NORM_OPT_NO_ATTRS)) {
            if (auparse_normalize_subject_primary(au) == 1)
                name = strdup(auparse_interpret_field(au));
            if (auparse_normalize_get_results(au) == 1)
                res = auparse_get_field_str(au);
        }
        snprintf(msg, sizeof(msg), "%s log in %s",
            name ? name : "someone", res ? res : "unknown");

        /* send a message */
        NotifyNotification *n = notify_notification_new(note,
                msg, NULL);
        notify_notification_set_urgency(n, NOTIFY_URGENCY_NORMAL);
        notify_notification_set_timeout(n, 3000); //3 seconds
        notify_notification_show (n, NULL);
        g_object_unref(G_OBJECT(n));

        free(name);
    }
}


One thing to know about the code above is that you won't get the right answer until you run it with the audit-2.7.6 library which should be released soon.(I found a bug in the library and corrected it while writing this example program.)

Conclusion
We have talked about using all of the various pieces of the auparse library. It should simplify writing analytical programs. Next time we will tie everything together with a short little project to send an email when certain events occur. I was going to do it in this post but decided it was long enough already.

Thursday, April 13, 2017

Writing a basic audispd plugin

In the last post we looked at how to write a basic auparse program. The program was suitable for searching historical records but was not up to par for realtime analysis. In this post we will look at the changes needed to make a program ready for realtime.

Realtime in no time
The audit system can transfer events in realtime to a dispatcher program. This program, audispd, can be thought of as a multiplexor. It takes one event record at a time and gives it to the plugins.

The plugins have to be designed a certain way.

  • They have to read events from stdin
  • They have to handle SIGHUP and SIGTERM
  • They must dequeue events as fast as possible
  • They must provide a plugin config file so that audispd knows how to start it.

The plugin config file has a specific format:

active = no
direction = out
path = /sbin/audisp-example
type = always
args = test
format = string

You can tell audispd if the plugin should be started by setting active to yes. The direction should always be out. (In the past there was the idea of being able to chain plugins together to transform events. But support for that has not become a priority.) Then you tell audispd what the path is to your plugin. You can tell it if the plugin type should always be running or its a builtin plugin. Since we are making our own, it should be always. You can pass up to 2 arguments. They are hard coded meaning that audispd does not do any kind of lookup but instead passes exactly what is on this line. It can be used to point to the location of a config file or anything else you can think up. And lastly, you can tell audispd what format the plugin expects its events to be in. There are 2 formats, the native audit structure and string. For any program that is designed to use auparse, always tell it string.

The design requirement for audispd plugins to read from stdin also provides the way to debug your application. During the development you may need to test different things. But if its running under a daemon how do you see what its doing?

The way I debug audispd programs is to capture some audit events in the raw format to a file. Then I "cat" the file into plugin on the bash command line.

# ausearch --start recent --raw > test.log
# cat test.log | ./audisp-example


OK. Let's get started. When events come in during realtime, it is one record at a time. This means you can't do anything until all records in the event have landed. So, the records need to be accumulated and the program must go back to waiting for the next piece so that it can process a whole event. The nice thing is that auparse has an event feed interface that takes care of this for you. The feed API is configured with a callback function that will be called as soon as an event is detected as complete. The callback function is handed an auparse_state_t variable that has one complete event in it. You cannot loop across events because there is only one. You can loop across records and fields.

The basic structure is:

Init auparse and tell it the source will be the feed API
Init the feed API by registering a callback function
Setup asynchronous reading via select, poll, or epoll
Read in the one record
Push record into feed API
If stdin has error, cleanup and exit

Let's make simple test program that simply writes the record type and how many fields that record contains to stdout. All error checking and signal support has been removed for clarity. In a real program these must be handled.

#define _GNU_SOURCE
#include <stdio.h>
#include <sys/select.h>
#include <string.h>
#include <errno.h>
#include <libaudit.h>
#include <auparse.h>


static void handle_event(auparse_state_t *au,
                auparse_cb_event_t cb_event_type, void *user_data)
{
        if (cb_event_type != AUPARSE_CB_EVENT_READY)
                return;

        auparse_first_record(au);
        do {
                int type = auparse_get_type(au);
                printf("record type %d(%s) has %d fields\n",
                        auparse_get_type(au),
                        audit_msg_type_to_name(auparse_get_type(au)),
                        auparse_get_num_fields(au));
        } while (auparse_next_record(au) > 0);
}

int main(int argc, char *argv[])
{
        auparse_state_t *au = NULL;
        char tmp[MAX_AUDIT_MESSAGE_LENGTH+1];

        /* Initialize the auparse library */
        au = auparse_init(AUSOURCE_FEED, 0);
        auparse_add_callback(au, handle_event, NULL, NULL);
        do {
                int retval = -1;
                fd_set read_mask;

                FD_ZERO(&read_mask);
                FD_SET(0, &read_mask);

                do {
                        retval = select(1, &read_mask, NULL, NULL, NULL);
                } while (retval == -1 && errno == EINTR);

                /* Now the event loop */
                 if (retval > 0) {
                        if (fgets_unlocked(tmp, MAX_AUDIT_MESSAGE_LENGTH,
                                stdin)) {
                                auparse_feed(au, tmp,
                                        strnlen(tmp, MAX_AUDIT_MESSAGE_LENGTH));
                        }
                } else if (retval == 0)
                        auparse_flush_feed(au);
                if (feof(stdin))
                        break;
        } while (1);

        /* Flush any accumulated events from queue */
        auparse_flush_feed(au);
        auparse_destroy(au);

        return 0;
}


You can ave the program as audisp-example.c and compile it like this:

gcc -o audisp-example audisp-example.c -lauparse -laudit


Now run the program as stated earlier by cat'ing some captured raw events out of a file.

# cat test.log | ./audisp-example
record type 1300(SYSCALL) has 27 fields
record type 1307(CWD) has 2 fields
record type 1302(PATH) has 11 fields
record type 1327(PROCTITLE) has 2 fields
record type 1300(SYSCALL) has 27 fields
record type 1307(CWD) has 2 fields
record type 1302(PATH) has 4 fields
record type 1327(PROCTITLE) has 2 fields


OK, great we have a working program. But its not too useful since what its doing is written to stdout. How about if we make a real program that sends a notification to the desktop anytime someone logs in? Wouldn't that be fun?


Login Alert Plugin
The first thing to know is that you cannot send a notification as root because it tries to connect to the system bus. Any program launched by audispd is started as root. So, here's the first challenge. Instead what you have to do is send it to the user's session bus that is logged in and you have to do it using their uid. We are going to cheat because working this out adds code and removes clarity of the auparse portion. You can google around for the missing pieces if you really want to develop this further. But we will cheat by creating a define that holds our account number. The program will not work unless you have the exact account number you are currently logged in with. On a default install, you are most likely to be account 1000. Adjust the define accordingly.


#define _GNU_SOURCE
#include <stdio.h>
#include <sys/select.h>
#include <string.h>
#include <errno.h>
#include <stdlib.h>
#include <unistd.h>
#include <libaudit.h>
#include <auparse.h>
#include <libnotify/notify.h>

#define MY_ACCOUNT 1000

const char *note = "Login Alert";

static void handle_event(auparse_state_t *au,
                auparse_cb_event_t cb_event_type, void *user_data)
{
        if (cb_event_type != AUPARSE_CB_EVENT_READY)
                return;

        auparse_first_record(au);
        do {
                int type = auparse_get_type(au);
                if (type == AUDIT_USER_LOGIN) {
                        char msg[256],  *name = NULL;
                        const char *res;

                        /* create a message */
                        if (!auparse_find_field(au, "acct")) {
                                auparse_first_record(au);
                                if (auparse_find_field(au, "auid"))
                                    name = strdup(auparse_interpret_field(au));
                        } else
                                name = strdup(auparse_interpret_field(au));
                        res = auparse_find_field(au, "res");
                        snprintf(msg, sizeof(msg), "%s log in %s",
                                name ? name : "someone", res);

                        /* send a message */
                        NotifyNotification *n = notify_notification_new(note,
                                        msg, NULL);
                        notify_notification_set_urgency(n, NOTIFY_URGENCY_NORMAL);
                        notify_notification_set_timeout(n, 3000); //3 seconds
                        notify_notification_show (n, NULL);
                        g_object_unref(G_OBJECT(n));

                        free(name);
                        return;
                }
        } while (auparse_next_record(au) > 0);
}

int main(int argc, char *argv[])
{
        auparse_state_t *au = NULL;
        char tmp[MAX_AUDIT_MESSAGE_LENGTH+1], bus[32];

        /* Initialize the auparse library */
        au = auparse_init(AUSOURCE_FEED, 0);
        auparse_add_callback(au, handle_event, NULL, NULL);

        /* Setup the notification stuff */

        notify_init(note); 
        snprintf(bus, sizeof(bus), "unix:path=/run/user/%d/bus", MY_ACCOUNT);
        setenv("DBUS_SESSION_BUS_ADDRESS", bus, 1);
        if (setresuid(MY_ACCOUNT, MY_ACCOUNT, MY_ACCOUNT))
                return 1;

        do {
                int retval;
                fd_set read_mask;

                FD_ZERO(&read_mask);
                FD_SET(0, &read_mask);

                do {
                        retval = select(1, &read_mask, NULL, NULL, NULL);
                } while (retval == -1 && errno == EINTR);

                /* Now the event loop */
                 if (retval > 0) {
                        if (fgets_unlocked(tmp, MAX_AUDIT_MESSAGE_LENGTH,
                                stdin)) {
                                auparse_feed(au, tmp,
                                        strnlen(tmp, MAX_AUDIT_MESSAGE_LENGTH));
                        }
                } else if (retval == 0)
                        auparse_flush_feed(au);
                if (feof(stdin))
                        break;
        } while (1);

        /* Flush any accumulated events from queue */
        auparse_flush_feed(au);
        auparse_destroy(au);

        return 0;
}


Save it as audisp-example2.c and compile as follows. You may need to install libnotify-devel and gdk-pixbuf2-devel before compiling:

gcc -o audisp-example2 audisp-example2.c \
 `pkg-config --cflags glib-2.0` `pkg-config --cflags gdk-pixbuf-2.0` \
 `pkg-config --libs glib-2.0` `pkg-config --libs gdk-pixbuf-2.0` \
 -lauparse -laudit -lnotify



Let's talk a bit about what the program is doing. If you look at the main function, you will see that reading events is almost exactly the same as in the first example program. We have to put the bus location into the environment for libnotify and change to the logged in user's account. But that's it for changes. The biggest difference from the previous program is in the callback function.

The callback function looks at the event type. If it is the user_login event, then we process it. If you used ausearch to grab a couple user_login events, you will notice that the event is different on success and failure. On failure we have an "acct" field with the user's account. But on success we have an "auid" field that has this information. This creates a problem that I'll show in the next blog post how to solve in a nicer way. I wanted to point this out now so that when we do this nicer in the next blog post it informs why we needed the auparse_normalizer interface.

But for now we'll look for "acct" using auparse_find_field(). If it fails, we'll reset to the beginning and scan for the "auid "field. It should be noted that if you scan for a field using auparse_find_field() and it fails to find the field, the internal cursor will have run "off the end". Auparse_find_field() will iterate across all fields and all records in an event. It will not cross the event boundary, though. Since this is a 1 record event, we can just call auparse_first_record() to place the internal cursor back at the beginning.

Creating a notice is very simple. We just call  notify_notification_new() with the message and associate it with the note that we initialized with. We can also add an icon if we wanted, but for simplicity we'll skip that. We set the urgency, give it a length of time to be shown, and then we show it. After that we have to clean up by releasing allocated memory.

In any event, let's give the program a spin. Copy it to /sbin and fill out a plugin config file as follows (we don't need args so comment it out):

active = yes
direction = out
path = /sbin/audisp-example2
type = always
#args = 1
format = string


The plugin config file should be copied to /etc/audisp/plugins.d/audisp-example2.conf. Then disable selinux enforcement by setenforce 0. Selinux doesn't know about our plugin and will block changing to your account and sending the notification. Stop auditd service and start it. Check its status.

[root@x2 ~]# setenforce 0
[root@x2 ~]# service auditd stop
Stopping logging:                                          [  OK  ]
[root@x2 ~]# service auditd start
Redirecting to /bin/systemctl start  auditd.service
[root@x2 ~]# service auditd status
Redirecting to /bin/systemctl status  auditd.service
● auditd.service - Security Auditing Service
   Loaded: loaded (/usr/lib/systemd/system/auditd.service; enabled; vendor prese
   Active: active (running) since Thu 2017-04-13 09:17:07 EDT; 5s ago
     Docs: man:auditd(8)
           https://github.com/linux-audit/audit-documentation
  Process: 6289 ExecStartPost=/sbin/augenrules --load (code=exited, status=0/SUC
  Process: 6283 ExecStart=/sbin/auditd (code=exited, status=0/SUCCESS)
 Main PID: 6284 (auditd)
    Tasks: 6 (limit: 4915)
   CGroup: /system.slice/auditd.service
           ├─6284 /sbin/auditd
           ├─6286 /sbin/audispd
           ├─6288 /sbin/audisp-example2
           └─6290 /usr/sbin/sedispatch


OK, looks good. Now ssh localhost and see what happens.

Conclusion
Once you understand the recipe of creating a realtime plugin, there are many things you can do. The audit system really is easy to work with. But as I hinted there are quirks that you have to know that perhaps you wished you didn't have to know. This is why the auparse_normalizer interface was created. Next blog post we will see how to use it in an audispd plugin to simplify event processing by creating a plugin that will send an email whenever it sees an event with a specific key.

Wednesday, April 12, 2017

Writing a basic auparse program

Auparse is the library distributed with the audit package to make accessing the logs easier. The main issue is that if you wanted to do anything with audit logs, the first thing that you would need to do is to parse the audit logs. That means learning rules about how events are structured and knowing quirks like interlaced events. In reality, its better to use a library meant for the task and just get on with whatever your project is.

Events
Events are denoted by a unit of text that have the same time stamp and serial number. Each record as logged by the audit daemon ends with a carriage return. A record has information about one or more aspects of something happening in the system.

Events can be simple or compound. A simple event has only one record. It is a complete thought that answers who did something to what and how did it turn out. A compound event is composed of multiple records that form a complete thought.

Each record is composed of multiple fields that are recorded as name=value. Anything that does not have an equal sign is superfluous information that can be discarded.

Navigating events
So, if we want to write a utility that processes audit events, the basic scheme is to iterate across each event, then iterate across each record, and then iterate across each field.

The auparse library has functions that allow you to do:
  • Basic initializations/cleanup
  • Iterate across events
  • Access event level informatio (timestamp, serial number, node)
  • Iterate across records
  • Access record level information (record type)
  • Iterate across each field
  • Access field level information (value, interpreted value, field type)
The best source to see the functions available at each level is to open the auparse.h file and look through it. The functions are grouped with a comment explaining the grouping. For details about a function, see its man page.

The most basic program that walks though logs or a file is as follows:

#include <auparse.h>

int main(int argc, char *argv[])
{
        auparse_state_t *au;

        if (argc == 2)
                au = auparse_init(AUSOURCE_FILE, argv[1]);
        else
                au = auparse_init(AUSOURCE_LOGS, NULL);

        auparse_first_record(au);
        do {
                do {
                        do {
                                const char *name = auparse_get_field_name(au);
                        } while (auparse_next_field(au) > 0);
                } while (auparse_next_record(au) > 0);
        } while (auparse_next_event(au) > 0);

        auparse_destroy(au);

        return 0;
}


Compile using:

# gcc -o basic  basic.c -lauparse


This doesn't do anything terribly useful, but it illustrates the basic anatomy. You call auparse_init to create a parsing state variable and to tell auparse where to expect the events to come from. This variable has to be passed to every function that the auparse library has. This is because auparse has been designed so that you can actually have 2 or more instances of the library working on logs simultaneously.

Next the program positions the internal cursor to the first record and then iterates over each event using auparse_next_event, then iterates across each record using auparse_next_record, and then each field using auparse_next_field.


A slightly more complicated example
The following program modifies the first program by using record and field level accessor functions to output the record name and the field name.

#include <stdio.h>
#include <auparse.h>


int main(int argc, char *argv[])
{
        auparse_state_t *au;

        if (argc == 2)
                au = auparse_init(AUSOURCE_FILE, argv[1]);
        else
                au = auparse_init(AUSOURCE_LOGS, NULL);
        if (au == NULL) {
                printf("You need to be root\n");
                return 1;
        }
        auparse_first_record(au);
        do {
                do {
                        char buf[32];
                        const char *type = auparse_get_type_name(au);
                        if (type == NULL) {
                                snprintf(buf, sizeof(buf), "%d",
                                        auparse_get_type(au));
                                type = buf;
                        }
                        printf("Record type: %s - ", type);
                        do {
                                const char *name = auparse_get_field_name(au);
                                printf("%s,", name);
                        } while (auparse_next_field(au) > 0);
                        printf("\b \n");
                } while (auparse_next_record(au) > 0);
        } while (auparse_next_event(au) > 0);

        auparse_destroy(au);

        return 0;
}


Save as audit-fields.c and compile using:

# gcc -o audit-fields audit-fields.c -lauparse


Running the program yields results like this:

# ./audit-fields /var/log/audit/audit.log | head
Record type: CWD - type,cwd
Record type: PATH - type,item,name,inode,dev,mode,ouid,ogid,rdev,obj,nametype
Record type: PROCTITLE - type,proctitle
Record type: SYSCALL - type,arch,syscall,success,exit,a0,a1,a2,a3,items,ppid,pid,auid,uid,gid,euid,suid,fsuid,egid,sgid,fsgid,tty,ses,comm,exe,subj,key
Record type: PROCTITLE - type,proctitle
Record type: SYSCALL - type,arch,syscall,success,exit,a0,a1,a2,a3,items,ppid,pid,auid,uid,gid,euid,suid,fsuid,egid,sgid,fsgid,tty,ses,comm,exe,subj,key
Record type: CWD - type,cwd
Record type: PATH - type,item,name,inode,dev,mode,ouid,ogid,rdev,obj,nametype
Record type: PROCTITLE - type,proctitle
Record type: SYSCALL - type,arch,syscall,success,exit,a0,a1,a2,a3,items,ppid,pid,auid,uid,gid,euid,suid,fsuid,egid,sgid,fsgid,tty,ses,comm,exe,subj,key
Record type: CWD - type,cwd
type,item,name,inode,dev,mode,ouid,ogid,rdev,obj,nametype
type,proctitle
Record type: SYSCALL - type,arch,syscall,success,exit,a0,a1,a2,a3,items,ppid,pid,auid,uid,gid,euid,suid,fsuid,egid,sgid,fsgid,tty,ses,comm,exe,subj,key
type,proctitle
Record type: SYSCALL - type,arch,syscall,success,exit,a0,a1,a2,a3,items,ppid,pid,auid,uid,gid,euid,suid,fsuid,egid,sgid,fsgid,tty,ses,comm,exe,subj,key
type,cwd
type,item,name,inode,dev,mode,ouid,ogid,rdev,obj,nametype
type,proctitle




Conclusion
Its really simple to create a basic program that walks the events in you logs or a file you specify. Any time you want to create a special purpose utility, you can use the source code given here to get your project off to a quick start.

There is another aspect to examining events. Sometimes you don't want to do historical retrieval, but would rather look at things in real time. This requires changing the basic program somewhat. We will look at how to write a simple program suitable to get events from audispd in the next post.

Thursday, March 16, 2017

The Three Level Sankey Diagram

A couple days ago we talked about a 2 level Sankey Diagram. The chart shows the relationship between items with a line who's width reflects the strength or volume of that relationship. This is useful in gauging the quantity of times something happens. I promised a tweak to the diagram that will make it even more useful. Here it is...

The three level Sankey Diagram
I wanted to give you a detailed theory of how the thing works in the last blog. To make the explanation clear, I gave you a tool that is somewhat limiting. But it is easier to follow along than a 3 level diagram.

The way that you make a 3 level diagram is to have a left, middle, and right column of data. Then you make the connections between left and middle, then middle and right. Then you plot it. So, rather than tell you again how it works, I'm just going to give you the code so that you have it and can play with it. Start up RStudio, open a new file, copy the following into it. I will assume at this point you know how to make the audit.csv file.

library(plyr)
library(dplyr)
library(networkD3)

# This script will make a 3 level Sankey diagram
left_field <- as.character("EVENT")
right_field <- as.character("RESULT")
middle_field <- as.character("OBJ_KIND")

# Subset the data based on this expression
operand1 <- as.character("")
operation <- as.character("")
operand2 <- as.character("")
expr <- paste(operand1, operation, operand2)

# Read in the data and don't let strings become factors
audit <- read.csv("~/R/audit-data/audit.csv", header=TRUE, stringsAsFactors = FALSE)
audit$one <- rep(1,nrow(audit))

# Subset the audit information depending on the expression
if (expr != "  ") {
  audit <- filter_(audit, expr)
}

# Make 2 dataframes for a 3 level Sankey
left = data.frame(audit[left_field], audit[middle_field], audit$one)
colnames(left) = c("Source", "Target", "Num")
right = data.frame(audit[middle_field], audit[right_field], audit$one)
colnames(right) = c("Source", "Target", "Num")

# Now summarize and collapse to unique values to calculate edges
l <- ddply(left, .(Source,Target), summarize, Value=sum(Num))
r <- ddply(right, .(Source,Target), summarize, Value=sum(Num))

# Calculate Nodes lookup table
nodes <- c(as.character(l$Source), as.character(l$Target), as.character(r$Target))
nodes <- data.frame(unique(as.factor(nodes)))
colnames(nodes) = c("Source")
nodes$ID <- seq.int(from = 0, to = nrow(nodes) - 1)
nodes <- nodes[,c("ID","Source")]

# Now map Node lookup table numbers to source and target
# Merge index onto Source
l_edges <- merge(l, nodes, by.x = "Source")
l_edges$source = l_edges$ID
r_edges <- merge(r, nodes, by.x = "Source")
r_edges$source = r_edges$ID

# Merge index onto Target
names(nodes) = c("ID", "Target")
l_edges2 <- l_edges[,c("Source","Target","Value","source")]
r_edges2 <- r_edges[,c("Source","Target","Value","source")]
l_edges <- merge(l_edges2, nodes, by.x = "Target")
r_edges <- merge(r_edges2, nodes, by.x = "Target")

# rename everything so its nice and neat
names(l_edges) <- c("osrc", "otgt", "value", "source", "target")
names(r_edges) <- c("osrc", "otgt", "value", "source", "target")
names(nodes) = c("ID", "name")

# Combine into one big final data frame
edges <- rbind(l_edges, r_edges)

sankeyNetwork(Links = edges, Nodes = nodes,
              Source = "source", Target = "target",
              Value = "value", NodeID = "name",
              fontSize = 16, nodeWidth = 30,
              height = 2000, width = 1800)


There are 3 variables left_field, middle_field, and right_field that defines what will get charted. This program has them set to "EVENT", "OBJ_KIND", and "RESULT". (If you need a reminder of what fields are available and what they mean, see this blog post.) This shows you what kind of objects are associated with the event and what the final result is. This is what it looks like on my system:




I would also recommend you play around with it. I would also suggest trying these:

ACTION
RESULT
OBJ_KIND




HOW
OBJ_KIND
RESULT




SUBJ_PRIME
RESULT
OBJ_KIND




EVENT
ACTION
OBJ_KIND




You should play around with it. You might see something in your events you hadn't noticed before. One last tip, too much data makes it hard to see things. You might experiment with subsetting the information either by adding command line switches to ausearch so that it narrows down what's collected or by using R code.

Wednesday, March 15, 2017

Account Names

In the last blog we looked at the 2 level Sankey Diagram. I had promised to show you an update to it that made it much more useful, but I decided to save that until next time. The reason is that I thought of a common security problem that warrants discussion. Since this is also a security blog, I thought we can take a pause on the data science and talk security.

The problem
If you have a computer on the internet it's just a fact of life that your system is being pounded constantly by people trying to login. So, what accounts do they go after?

To solve this problem, we need a script to gather information.

#!/bin/sh
tfile="acct.csv"
echo "ACCT" > $tfile

for log in /var/log/btmp*
do
        lastb -w -f $log | head -n -2 | awk '{ printf "%s\n", $1 }' >> $tfile
done


Save it and run it as root.

[root@webserver]# vi get-acct
[root@webserver]# chmod +x ./get-acct
[root@webserver]# ./get-acct
[root@webserver]# wc -l acct.csv
942453 acct.csv

OK. So, let's pull this into RStudio for some analysis. What we want to do is see what the most often used accounts are. Let's run the following script:

library(dplyr)

a <- read.csv("~/R/audit-data/acct.csv", header=TRUE)
a$one <- rep(1,nrow(a))

acct <- aggregate(a$one, by=list(a$ACCT), FUN=length)
colnames(acct) = c("acct", "tries")
account <- arrange(acct, desc(tries))


Run it. Now let's see what the 50 most popular accounts are:

> head(account, n=50)
            acct  tries
1           root 866755
2        support  10315
3          admin   7856
4           user   1904
5           ubnt   1469
6              a   1436
7           test   1324
8          guest    966
9       postgres    525
10            pi    496
11        oracle    463
12       ftpuser    425
13       service    369
14        nagios    314
15       monitor    287
16 administrator    246
17        backup    240
18           git    213
19     teamspeak    212
20          sshd    193
21       manager    184
22     minecraft    180
23           ftp    179
24         super    169
25       student    168
26        ubuntu    166
27        tomcat    160
28         ADMIN    158
29        zabbix    158
30           ts3    154
31      testuser    152
32          uucp    150
33           adm    145
34      operator    144
35      PlcmSpIp    144
36          alex    140
37    teamspeak3    140
38        client    138
39       default    138
40          info    126
41        telnet    126
42           www    124
43        hadoop    123
44        upload    120
45           fax    118
46            ts    118
47     webmaster    118
48       richard    112
49        debian    110
50      informix    110


So, what does this mean?

1) Do not allow root logins. Ever. Period.
2) Do not make an account based on a job function
3) Do not make an account based on a service name
4) Make sure all service accounts cannot be logged into
5) Do not make an account based on your first name

So, how can we check for active accounts on the system? First, let's make sure everything uses shadowed passwords:

# awk -F: '$2 != "x" { print $1 }' < /etc/passwd


Any problems here should be fixed. Next we can check which accounts are active:

# egrep -v '.*:\*|:\!' /etc/shadow | awk  -F: '{ print $1 }'


If you see any services listed or simple names and the system is hooked up to the internet 24x7, you might want to look into it. If you use two factor authentication or keys, then you are also likely in good shape.

The real point of this was to show you how to check what accounts are getting hammered the hardest by people trying to get in.