We’re talking about ways in order to elevate privileges on the Linux platform and the discussion today will also be of use for anybody who’s worried about privilege elevation on other UNIX platforms. Basically, whenever you’re worried with privilege elevation on UNIX, there are basically two paths that you should be looking at or for elevating privileges.

Watch a video version instead

The first such path is a kernel exploit. Innocence, your attacker will attempt gain execution in the kernel by hacking some user mode interface to a kernel service. Once that hacker has gained execution in the kernel, his goal will be to grant root privileges to a running shell or backdoor process through poking in the kernel’s data structures. That’s fortunately very easy to detect. Once this has happened we just have to have a scanner of all running processes and to take a look at their user IDs. Whenever the user ID of a running process changes by decreasing thereby suggesting that the process is increasing its own privileges it goes against a golden rule of the UNIX security model, which is that privileges for a given process can be reduced but can never be augmented. That’s pretty easy to detect that flashes is a big red light on our dashboards and there is little that a hacker can do in this space that will go beyond our surveillance.

The alternative from exploiting the kernel is to hack a highly privileged user mode application. In this case you want to gain execution in some daemon. Either through a good old memory exploit or more generally by exploiting a fault in the way this daemon handles it in books or a fault in the protocol for handling inputs either from the design or the implementation. An alternative on UNIX, you can hack the input parameters to a setuid program.

A setuid program is basically the tool of the devil. It’s on the other hand the only way for a user mode program to go from an unprivileged program to a high privilege program. In a sense, setuid programs, when they are started, get the privileges of the owner of the program rather than inherit the privileges of the parent process. This means that if a setuid program belongs to root when you run it, this program runs as root. Even if it was run by good old me who is not root. These things are evil, it’s the only way to properly implement the normal privileged elevation programs such as su or pseudo but many systems administrators are hell-bent on having a non delegating certain systems management tasks using setuid program and this is a very very bad idea and most privileged elevations go through these setuid programs.

Watch a video version instead

What you do when you are hacking a frivolous user mode application? Essentially, you coax it into forking a shell or a backdoor for you. This is hard to detect, surprisingly. Why? It’s not so much that it is difficult to detect whenever a certain program for child process that suddenly has a higher level of privileges, it’s just that this pattern of events is frequent. Many high privilege applications will forge into high privilege child processes on a regular basis thereby granting many false positives to any test checking that. What do you do in this case? All the alleged stuff you try to whitelist, you will inspect, you will audit your typical privilege elevation pathways on a system but you will only whitelist them at your own peril as the upcoming examples are about to show.

Two examples here. One is about a situation where we get the sense of a privilege elevation but there is none and the other is an actual good old privilege elevation problem.

Let’s take a look first at this investigation here. What we have here is this process called polkit-agent-helper-1 which is now running as root but has a parent that was polkit-gnome-authentication-agent-1 that was running as user Victor. User Victor was able to fork a new process called polkit-agent-helper-1 from the polkit-gnome-authentication-agent-1.

What’s at stake here? When you’re in the know as to how that gnome interface systems work, you quickly learn that gnome has its own the wrapper around the pseudo process. This wrapper basically handles the authentication checking and then once the identity of the user has been checked and been known to match the set of users who can run pseudo, then it has its own setuid helper process, in this case it’s called polkit-agent-helper-1 in order to get it to elevate privileges on a running gnome process.

When you go and look at what is happening here right so this is the raw data stream sent by the machine on which this elevated process was found to run.

When you go into the the past for this, you see that the user was running aptd likely through the systems update tools enabled by his Ubuntu machine.

When you’re running aptd and you call for a system update, then of course you need to elevate your privileges in order to get this updated system. This is what happens here and after that the system the starts downloading new packages and if we keep digging down and exclude everything that’s concerned with Chrome we see that stuff starts deleting an old version of libc and then it starts decompressing and unarchiving new packages, in this case, apparently, go1.8.linux-amd64.gar.gz and we see other traces through new processes that talk about the update of the system.

Watch a video version instead

In this investigation, it is clear that the process elevation from the gnome identification agent when the high privilege process created is polkit-agent-helper-1 this is clearly a legitimate setuid process granting legitimate privileged elevation.

All right, onwards to our next example. Here we have an investigation where the essential label is that privilege process ifstat with unprivileged parent bash. okay Let’s take a look at this guy.

Here we have a process in path /usr/local/bin/ifstat that’s been started by the good old shell from user hamelin, that was me yeah and that I did prepare for this it doesn’t look like it. This process is running as root.

What can we expect from this? What is the smell that I get? Well, clearly this program ifstat has been put into /usr/local/bin. Typically, people deploy programs in that directory when its programs that are not managed through the system’s packaging system. This means that likely this ifstat program is a homegrown piece of software or something built and put there through a hand managed package.

I go through to the systems administrator and ask, “What is ifstat?”. As the systems administrator I say, “It is a wrapper around ifconfig”. The ifconfig program for context is one by which you can query and modify the network configuration for a machine. So me again as the systems administrator tell you it’s because I wanted ifconfig to only be runnable by root and so I have written this wrapper around ifconfig called ifstat to enable users to use ifconfig only to query the status of network interfaces. “Ah” says I, “That is interesting. So how do you manage to get it to run ifconfig if your wrapper is meant to be run by lowly unprivileged users?” Me as a systems administrator say, “I have configured it as a setuid program. ” Then me as a hunter say to me a systems administrator, “You are a fool because look what this user have has done to you.”

Watch a video version instead

It has called your ifstat tool in order to get to query the status of the eth0 interface up to now that’s fine, but then what do you see? You see the typical signature of somebody hacking the inputs of a setuid program. In this case, the user has managed to introduce the possible and invocation of a shell. Basically, the inputs to this program is probably fed into the system libc function which starts a shell in order to run a child program and when you do so and do not properly sanitize the inputs that you set up to be run by system, then you run the risk of your users putting in inputs that do way more than you intended to do with your wrapper.

Let’s dig down the little from there in order to figure out whether the user actually managed to elevate his privileges in this instance. The same way I did with the first instance, I go to the raw data stream for this machine around the instance of the detected privileged elevation. Clearly this has happened soon after starting the system, so I go down I am looking for an invocation of the ifstat program. Here it is, ifstat.

This code identity piece of telemetry was about hashing this program and then here I see it, oh I see a new process here where a process which was the parent the parent was ifstat which we saw run from a shell initially and now this program ifstat is launching a shell in order to run ifconfig eth0 followed by bin/dash.

This does not smell good because what happens after that. It runs this shell through ifconfig who then runs ifconfig after that starts a dash process. Dash is a shell on the system and this dash process is indeed running as root.

There is indeed in this instance the actual elevation of a process through an illegitimate mechanism. This privilege elevation event requires immediate incident response because once this dash shell was launched, it had root privileges so it was able to do everything on the system including deploying a backdoor, including deploying a rootkit.

I thank you very much for your attention and we’ll get to make another show like this in about a month thank you very much cheers and goodbye.

Watch a video version instead

There are multiple ways for a program to seek privilege elevation, but they all are based on one simple principle: high privileges are granted by other high privilege programs. Such programs can be classified across two categories:

1. the operating system kernel;
2. highly privileged user-mode programs.

The signs of exploitation differ depending on whether the attacker is exploiting the kernel or a user-mode program. Let’s start with the latter.

Exploiting high-privilege user-mode programs

There are basically two categories of exploits at play here:

1. forking off of a running highly privileged program;
2. starting a highly privileged program from a lowly privileged program.

Case (B) is the easiest to monitor for malware detection. The typical scenario considered here is that the program just started was a set-UID or set-GID program. Any process under UNIX is running as a certain user and group, and its set of privileges derive from them. The usual privilege transfer protocol stipulates that the user and group of a process are inherited from the parent process that forks it off; the child process then executes a new program, which does not alter this set of privileges. However, set-UID and set-GID executables change this protocol: after the child process has been forked off, when it executes this program, the user (set-UID) or group (set-GID) it runs as becomes that which owns the program. Therefore, if the executable is owned by a user with a high level of privilege — typically the root user — the process runs with high privileges.

Many common UNIX programs, such as su and sudo, necessarily leverage this privilege transfer rule. However, many systems administrators tolerate other set-UID or set-GID programs in order to facilitate the execution or delegation of certain tasks. This is a dangerous vulnerability, especially if the set-UID/GID programs are relatively complex. Indeed, many hackers know how to supply parameter sets to such programs in order to trick them to run arbitrary shell commands.

Case (A) is quite a bit harder to monitor. In this case, the exploit consists of submitting specially crafted input to a highly privileged program so as to either take advantage of a design flaw, or to exploit a vulnerable bug. In both cases, the end result is that the attacker can trick the program into forking a process from which the exploit provides means to run a chosen program. The child process from the forking operation naturally inherits its parent’s high privileges. This case is harder to monitor, because many service daemons run with a high level of privilege, and naturally spawn off legitimate highly privileged child processes. A good example is the sshd daemon, which handles SSH connections to the machine: whenever the root user is permitted to log on the machine through SSH, sshd is legitimately able to fork off root-privileged interactive shells and commands.

In both cases, the solution for detecting malware activity is to raise an alert whenever a privileged process spawns off a child process with lower or equal privileges. We can consider privileged any process that runs with a UID smaller or equal to N: on many servers, N can be 0 (root); certain specialized Linux systems (e.g. Android) attribute important privileges to non-root users up to UID = 9999. This effectively captures any set-UID program execution (we whitelist the common legitimate set-UID programs su and sudo), as well as any privileged process spawned from a privileged daemon. We obviously advise IT administrators in customer organizations to minimize the number of uncommon set-UID programs and “root-spawns-root” processes (which result in false positive alerts). Our strategy of choice is to locally whitelist the latter processes that the customer insists on deploying on his servers.

Exploiting the operating system kernel

While many hacks leverage vulnerabilities of user-mode processes, others attack kernel bugs to get execution therein. Since user-mode processes are a much easier place to implement common computing tasks, such as TCP/IP communications, kernel exploits often look to raise the privileges of a shell (or other program) already being run by the attacker. In the Linux kernel, this is rather simple, as the address to the list of task structures is made public to all user mode processes. Once simply has to walk this list and set the user and group of the attacker’s shell to 0 (root).

However, such an exploit breaks an important rule of the lifecycle of processes: a process can set its own UID in order to lower its privileges, but never to raise them. By setting a process’ UID or GID to a lower value than it already had, the attacker poses an uncommon gesture that is easy to look for. In this case, SNOW keeps a running snapshot of all processes and the user and group each is running as. It updates this snapshot at randomized time intervals, and when it does, it compares with the previous snapshot: if any of them bears a lower UID than it did previously, this raises an alert. Since the lowering of a process’ UID is not a legitimate metadata shift, this detection heuristic generates no false positive alert.

Summary

In essence, looking for privilege elevation requires tracking two things on a system:

1. the apparition of any child process with UID/GID <= N;
2. the lowering of the UID/GID of any running process.

Heuristic #1 is the only of the two that can generate false positive alerts. Most of these may be eliminated through more security-conscious system configuration and administrative processes. The few instances that cannot be discarded are hence easily whitelisted, but should be clearly identified as known potential security holes.

Recording of the webinar right away! The webinar is recorded from here so if you lose your internet connection or need to step away from your keyboard, don’t worry we will send the recording out once  the webinar is finished.

Watch a video version instead

I’m Benoit Hamelin, CTO at Arcadia and I’m standing in for Justin Seitz today. Basically the point and the goal of this webinar is to discuss process chains and how we can use them in order to detect malware so that we can respond to data breaches in good time. Let’s start with a short introduction to what I mean by process chains. It’s no secret to most users of modern computers, oh but I’m getting ahead of myself, maybe just a little plan for the meeting today and so my intent is basically to be chattering away like this for about 20 minutes and then will have a 10 minute and more time allocated for questions and answers. I will also be available for a half an hour past the beginning of the webinar, this means I will be available up until 2 pm EDT, and therefore just a bit of chatter on my end, just a bit of courses and then you drive the show by asking the questions.

I was talking about process chains and I was saying that basically it’s no secret to anybody familiar with the usage of computers or servers that programs are started by other programs. In a more technical parlance, processes are started by other processes and this who starts whom relationship is basically the note that in terms of parent-child relationships. There is an ancestral relationship between processes and walking on these ancestral links we can identify process chains basically built by a process related to all its parent processes down to the initial user mode process started by the kernel when the machine boots.

What is this with process change that can help us fight malware? When you run a computer you’re using a workstation of your own or you’re running a server that deploys a certain application or set of applications, the programs are generally started in rather regular patterns. If you take a look at process chains on a certain machine you can identify common process chains. So what happens when this machine is infiltrated? Malware will try to leverage the spawning of or forking of new processes. Why would it do that? Well, often your user mode processes will be running into certain sandboxes, so by exiting to another process malware can exit the sandbox. It will also do that in order to elevate its own privileges by pushing a computation unto privilege process. It can also use that in order to bootstrap code injection or just to run its malware activity away from the site of intrusion. These various activities that malware does, involve process change that do not follow the typical pattern on the host. It generates what we could consider anomalous process chains. This begets a detection heuristic stated by whenever we detect anomalous process chains we should be investigating the machine.

In order to make sense of this principle let us look at two cases of anomalous process chains that I have set up on this demo account on SNOWboard.

The first case concerns a Windows machine where we can we can see that malware has typical process chain patterns all its own and that is quite distinct from legitimate processing chain patterns. Here, let me open the investigation and now that I think about it I had it open here in full screen tab.

When we think about malware and the way it can show itself in terms of process chains, what do we think about? One of the very typical process chain patterns that malware does is the so-called drop execute pattern, by which a certain piece of malware will intrude through say a browser or PDF reader and the shellcode will contact in-situ server get a binary and write this binary to file just to execute it right away. This is the drop execute pattern.

Watch a video version instead

Another thing that it can do is to infiltrate a common program, then just spawn out of it by injecting a certain piece of code or by spawning an interpreted program through a shell of some sort. Very typical examples are once again the web browser, acrobat reader, Macromedia flash or the dreaded Microsoft Office macros. In principle, generally, these guys these very common workstation programs they are not supposed to be spawning shells. They’re not supposed to shell out.

Let’s take a look at this investigation here. A single lead has been generated for this investigation for this Windows machine that I control basically. So what happens on it, well the summary is that in a sense when Word.exe has a child process three generations over of WSCRIPT.exe. W.script exe as you may recall, is a JavaScript interpreter that lives in the windows system 32 directory.

Let us go over the details. In essence the root program of this process chains is a Microsoft Word, the WINWord.exe executable. When Word has a parent but we don’t really care about it, we care more about its children processes. Its first step child is something called AppData\Roaming\Microsoft\Word\MSword.exe.

This is something that spells out like a Microsoft Word executable, but that has been deployed in the user Foobar\AppData\Roaming directory. That does not look good at all. If we dig further, this MSWord.exe program has itself spawned a child process and that child process is the WScript.exe windows script host binary and we even see from the command line that it’s running a JavaScript process also deployed into the appdata roaming directory of the Foobar user. This is where things get really bizarre, it does that from the Xeyqyn directory and the name of the script is fuarh.js. I don’t know about you guys, but I know a few foreign languages and I’m pretty damn sure that seeking is not anything foreign at all. If anything, it’s probably the result of generating a random directory name.

Same thing with the fuarh.js word. So very likely the javascript file has been written there in a randomly generated directory in the AppData\Roaming directory of user Foobar. Now random directories and file names are common in the temporary directory on Windows, but not quite that common on C:\users\Foobar\AppData especially if it’s been started by this weird MSWord.exe binary that’s also been deployed into C:\users\Foobar\AppData. To me this smells really really bad. So, how do I go about investigating? If we are running a JavaScript file and where Microsoft Word is not supposed to be running a JavaScript file it should be interesting to just go grab this javascript file and see what’s in it. The first thing that we should do is open the command, actually we don’t do it that way, rather we select this and indicate that we want to get this file in this host.

Watch a video version instead

This takes a short while because this machine is not in constant contact with my interface, it beacons only at randomized regular intervals so this would take a little time if I were to send the command but as an all great cooking show, I already have done it beforehand and what I got from it is I actually got the file in the previous execution of this piece of malware and I got it from there. I’m then able to download this JavaScript file and try to reverse-engineer its purpose.

I could also do the same so if we go back to analyzing the lead, we saw that there were two levels of weird processes going on here. The first one spawned by WINWORD.exe and another one called MSWord.exe, I would also want to download it. In addition, if there is a smell a strong smell as it happens here of malware being deployed on this machine. I would get a full snapshot of everything that happens on this machine. How do I do that? I go to commands and I use the update info command, I just type in the investigation and host ID here, so the investigation is 677, the host ID is this, and then apply and the result is as we saw here, a bunch of fresh information about what’s going on on this machine.

So I get a full snapshot of all processes running on the machine, refreshed host information, list of services, list of drivers, and what kind of tickles me right now, a list of autoruns.

I’m not a big windows expert here, but what I can tell you is that every other autorun key on this machine is about stuff deployed on C:\windows\something\something except this one here which is also deployed into the AppData subdirectory for user Foobar. This again smells bad especially given that it’s something that calls itself as svchost update, I’m pretty sure that Microsoft whenever it updates its svchost will not go through the AppData local of some user. I would likely grab this as well and reverse-engineer it and find that this is actually a stage 0 component for the turla malware. If I go and take a look at these guys, I see here okay something about VMware, something about our ISO, Userinit, that’s smells good, stuff in when logon shell, isolated commands, yada yada yada, very common stuff, alternate shell, this is probably something legit.

In order to analyze these autorun dumps more quickly, we’re also working on a way of computing a diff of autorun snapshots between a previous version which presumably would have happened before the malware deployed itself and this version and this diff would likely show me this one entry added and would also tell me an idea of the time frame at a which when this new autorun has been added. That completes my investigation of the first case.

We saw that certain, not certain, but malware in general especially on Windows have a very common process chains of their own and so singling out those process chains help in detecting the threat.

The second case I wanted to go over, and I see that I will over spend my time, so please feel free to tell me to shut up if you really want to ask questions right now. The second case is about the fact that often especially on unix and linux systems, malware forks a new process in order to elevate its privileges. So here we’re confronted with a much simpler example. On unix there is such a thing that we called setuid programs. So what do they eat in winter? Typically when you run a program on unix or on linux this program runs with the privileges of the user that starts it. If I run an editor program this program runs at benoit hamelin with my limited privileges, but setuid program accepted this move instead of starting with the privileges of the invoker of the program, they start with the privileges of the owner of the program files and since most program files are owned by root on any given unix system, setuid programs owned by root will effectively start with root privileges even though they are not started by the root uses. This means that when these programs start they provide a free privileged elevation to the user that starts. In general it’s very well known that setuid programs are a huge security hole because many of these programs are more complex than the analyst or programmers will bother to remark and from this complexity they can be exploited to run arbitrary computations with root privileges. In order to make sense of what happens with this setuid program, let’s take a look at this investigation now.

Watch a video version instead

What I see here, let’s close those keys, we see that I have a parent process – a simple bash script that I was running on there and what did I do there, I just type the name of setuid program. Now project yourself as a hunter who just got an alert of this type to investigate. So it’s not me, it’s somebody that says okay, this user they’re called hamelin has been running this program here and this program has user ID 0, whereas its parent has user ID 1000.

If we take a look at usernames 1000 means that this user named hamelin was doing something and all of a sudden his computation runs as root. There’s two possibilities here, first possibility is that this guy is running a privilege elevation exploit. I would want to check out the content of this test privilege program.

The other possibility is that he’s running a setuid program. The first thing I will do here is simply go check the rights with which this file has been done and for that I sent a simple ls-la command. We see from here that yes, this file belongs to root and yes, from this little S here by the RW we see that it’s a setuid program.

This is a program that could be used against the owner of the work station or against the owner of the machine in order to elevate privileges if it can be exploited. Two things I want to do here first is I get a copy of this file to go and audit in order to you know if I can find an exploit for it myself. If that’s so, I will sound the alarm to IT that they have to take it out right away. In any case, I will go and warn IT of this security hole and suggest a better approach for providing elevated privileges when running this program just to certain users or under certain circumstances by using things like the pseudo program or something else like this. That also concludes the study of this second case.

In conclusion before I start taking questions, I want to do point out again that the typical usage of computers will result in common process chain patterns. Your malware will tend to generate off pattern process chains and we must look for these. What do these things look like? They will look like a typical parent processing like in our MSWord example. They will look like for instance, signed processes that spawned unsigned programs. We did not see an example about this. Another thing we can see is child processes having higher privileges than their parents processes. This is also something that should raise your attention, you should investigate as soon as you can. Right now we’re also working on an artificial intelligence engine to identify the most common process chains on all machines that we monitor and eventually being able to detect anomalous process chains more generally than we do already.

All right, thank you and on this I will start taking your questions.

Watch a video version instead