Upgrading your R libraries after OS upgrade

I recently posted about how to do an upgrade to Fedora 26 while maintaining the Nvidia drivers. So, what are you supposed to do with your R libraries you downloaded? If you try to use audit-explorer in RStudio, you'll get an error because system library versions have changed.

There are instructions on the web about how to do this. They basically say to run:

update.packages(ask=FALSE, checkBuilt = TRUE)

This basically works fine in some cases, but in our case the OS was upgraded and R can't tell that anything needs to be done because our R libraries were up to date before upgrading the OS.

We can run a script to remove and re-install the old libraries. The script works because the R libraries on the system were upgraded when you went to F26. All we need to do is rebuild the ones kept in your home dir.

The following script should be self explanatory.

# Ask R where it keeps its libraries
all <- .libPaths()

# Ask R where its system libraries are
site <- .Library.site

# Now subtract one from the other to get your home dir libraries
loc <- setdiff(all, site)

# Take a look at loc to ensure it only point to the R directory in your home dir
loc

# Ask R for the list of packages in your home dir
plist <- installed.packages(lib.loc = loc)
df <- as.data.frame(plist)
# Take a look at the packages to ensure this looks about right
#View(df)

# Get rid of the old ones
for (p in df$Package) { remove.packages(p) }

# Reinstall the packages
for (p in df$Package) { install.packages(p) }

When you run this, single step one line at a time. Do not run the whole thing. Single step it to the point where it outputs 'loc'. Does it look like a directory in your home dir? I get

"/home/sgrubb/R/x86_64-redhat-linux-gnu-library/3.4"

OK. Now single step down to the View(df) call. Uncomment that if you want. It shows a real nice table of all the package info.

Now its time for the scary part...deleting all the packages. Step over the first for loop. You will see a whole bunch of red text scroll in the Console pane. This is normal.

Now to put it all back, step over the last for loop. RStudio  will ask you if you want to restart R prior to installing. Tell it yes. It will ask again. This time cancel it by clicking on the X in the corner of the dialog. The dialog will pop up again and again. Click the X to close it. At some point it thinks it finished your for loop and it didn't. You can tell because you see an empty cursor ready to use in the console pane.

Fear not. Run the last for loop again. This time it will be unimpeded and will proceed to download and install all of your packages.

Whatever you do, do not exit RStudio until it after the second run of the for loop finishes. This can take 10 or 20 minutes depending on how many libraries you have. Exiting before the building finishes will surely lose the list of packages. You can tell its done because the Console pane is ready to use again.


Conclusion
When you upgrade the OS, sometimes R libraries won't work and upgrading doesn't work because you are on the latest version. The solution is running a script. It is not without danger, but it does the trick.

Upgrading to F26 while running nvidia drivers

Originally, I planned to post a much bigger article. I run with the rpmfusion-nonfree-updates-testing.repo enabled. I would not recommend that for most people. The reason being is that the released version of the nvidia drivers is 375.66. This is cuda 8. If you run with the testing repo you will get version 384.90. This one is cuda 9. That means redoing your whole environment. So, we'll save that for another blog post. Meanwhile let's go over how to do the upgrade.

Upgrading to F26
Upgrading to F26 from F25 was pretty smooth. I had to uninstall RStudio, but I already had the recipe to rebuild it. I followed the normal fedora upgrade instruction except I took one small deviation. If you use nvidia drivers, you probably have noticed that when you install a new kernel, akmods builds a couple new rpms and installs them on system shutdown. This way when you boot back up, the new kernel has matching drivers.

My concern was how does this work when upgrading the whole OS via dnf? What I did was:

1. Let dnf download all the packages as normal.
2. Reboot the system per dnf instructions so it can install them.
3. After it had installed all the new F26 packages, I watched carefully for the grub boot menu to appear. I edited the first entry to add a 3 at the end. This causes it to boot into runlevel 3 which is non-graphical.
4. Then I logged in, looked around to see how the upgrade went, and then did a reboot to go to try graphical mode. Doing a test boot into text mode was just in case it needed to build the rpms for the new F26 kernel during shutdown.

Sure enough, that is what happened. It started a shutdown job and built the new kernel modules and installed them. It came back up in graphical mode just fine.

In the near future, I write about switching to cuda 9. If you don't have to, I wouldn't for now.

Sometimes it takes two objects

Work is progressing on an upcoming release of the audit user space software. During the work to create text representations of the event, I found that some ideas just can't be adequately captured in the normalized view. For example, if admin mounted a disk drive all we could say is that a disk was mounted. But in truth, the admin mounted the disk to a specific place in the file system. How do we capture that? It is important.

After a while, I decided that sometimes there are simply two objects. The admin mounted this(1) to that(2). To address this the auparse library will assign fields to object2 (which is formally called primary2) whenever it sees the following:

1) Files are renamed by using the rename* syscalls
2) Files receive permission or ownership changes
3) Files get symlinked to
4) Disk partitions get mounted to a directory
5) Whenever uid or gid changes as a result of calling set*uid or set*gid syscalls

There may be other cases, so don't consider this the final specification. As I see more events, I'll add to this when necessary. Or if you have some ideas about when there might be a second object, leave a comment or email me.

Because this is a sparse column, it will not be enabled by default when the csv format is selected. To get it, you will need to pass --extra-obj2 to the ausearch program.

If however, you are a software developer, then you can get access to the normalized output via a new auparse_normalize_object_primary2 function. The way that it would be used in practice is similar to any of the other normalizer accessor functions. You would do something like this:

    rc = auparse_normalize_object_primary2(au);
    if (rc == 1) {
            const char *val;

            if (auparse_get_field_type(au) == AUPARSE_TYPE_ESCAPED_FILE)
                    val = auparse_interpret_realpath(au);
            else
                    val = auparse_interpret_field(au);
            printf("%s", val);
    }

This new function is not yet available unless you use the source code from github. This will be in the next release, audit-2.8, which should be out in the next week or two. Which reminds me...if you know of any issues in the audit code, now would be a good time to report it.