Welcome to our webinar. Today we’re going to talk about looking for threats through statistical outliers and statistical analysis. So a few housekeeping items off the top as some of you might have heard already.  Number one, if you lose your connection or you need to step away from this webinar, it will be recorded and you’re going to receive a copy of that recording at the end. Number two, you’re all on mute so don’t worry you’re not going to make any accidental mistakes and talk into your microphone so that the whole world hears it, but that also means that you can use that question and answers box or the chat box.

Watch a video version instead

I prefer question-and-answer box if you need to ask anything or if you’re having trouble hearing or anything and I can address that, so please feel free to use that both throughout the webinar and towards the end when we get more into the questions and answers. We are only going to run this for about 15 to 30 minutes usually again depending on how much coffee I’ve had this morning really depends on how long the webinars will run and more than anything we really want to thank you for taking the time to come today. We know you’re all busy people you all have things to do so we really appreciate your time and we certainly intend to make the most out of it.

As I mentioned earlier we’re going to be talking about doing some statistical analysis using the SNOW platform from Arc4dia. Now how we do this is actually through this interface that you all should be seeing now which is called SNOWboard. SNOWboard is our window into all of the data that our SNOW agents are collecting every minute of every day across our customer networks and we even monitor our own networks and of course we monitor our test network that you’re going to see today.

For those of you who are new I’m going to walk you through a little bit about how SNOWboard works and kind of how the data comes in and how we look at it because that’s important but also going to show you how we move into more statistical analysis and how we can use it as well to look at the network at a higher level and also look at how infections might be spreading.

Hopefully you all can see my screen at this point. If not, we have major technical problems but you should be seeing is this kind of list of investigations and this is exactly what myself and the other hunters on Arc4dia do every single day as well as some of our customers is we get into SNOWboard and we take a look at what’s going on on our customer networks and each one of these is what’s called an investigation that’s pretty easy to understand.

If we take a look at just picking one here, what this investigation gets triggered by something that our agent has identified that looks suspicious. So this could be any number of things. In this case there’s process hollowing, which we covered in a previous webinar encourage you to go back and watch that. We also see some suspicious threads.

This investigation is going to embody a lot of information and it also includes a lot of tools that are useful for both hunting as an individual and hunting as a team. One of the things will often do is we’ll use the logging mechanism where we can actually say “Hey this is Justin, you know I pulled down a binary for analysis and Ida”. That’s going to tell my other co-workers “Hey, you know there’s no point be pulling on this binary, Justin is taking a look at it i’ll put my notes back in and we can have a conversation about what’s going on in this particular investigation.

This is also useful so that customers can actually collaborate with us so they can see what we’re doing maybe even provide input. Lots of our customers are actually really technical people who understand a lot of what we’re talking about. They might also be seeing things on other hosts that they’ll pop in and say “Hey, you know, are you guys noticing this, or I saw this on another host, or I saw something else they’re all alert. This is really useful.

The other thing that’s really powerful when working in this environment, is the notion of clustering. We have the ability as a team to look at a particular investigation and look at some of the data points which we’re going to get into and say well how can we actually save ourselves some time in the future by writing these powerful clustering rules that will automatically say okay if we see this particular process at this particular time on this host in the future we want to pull that into this investigation so that this investigation can kind of become the main point for investigating similar pieces of information. This is really powerful, this is so that we’re not duplicating work continually investigating the same things over and over. This is exactly what this clustering tab is for and again we’re probably going to do a webinar just on clustering at some point in the future.

The commands panel here you’re gonna see a bit of it in action later but we have the ability to speak to any one of our agents in the network and get them to perform tasks for us. This could be anything from retrieving a file, to retrieving memories, or performing other forensic operations and this is really powerful because it allows us to kind of expand our intelligence collecting and say okay if we have something going on this host we want to pull back some evidence, we want to collect some things. We also have the ability to trigger some of that stuff automatically, which you’re going to see very shortly.

The next thing down is the leads list. Just like if a police officer was investigating a homicide, you have an investigation which is the actual murder itself and then you have the leads that are coming in which are all the pieces of information and people that contribute to that investigation. As much the same thing and SNOWboard, that is kind of logically how we think is humans and we’ve applied that here. We say, okay we have these events that have happened around this time on this host, let’s pull them all into this investigation so we have a deeper view of what’s going on on this particular machine around this time.

We have the ability to look at these leads individually and we can see that each one of them has slightly different information. We see process following on the first three or four leads and then we see a suspicious thread found at the bottom.  By clicking on either one of these, what were able to do is actually see highly detailed technical information about what is going on and why that lead was included in the investigation. What this means is that we have looked at “okay what is going on here and what is relevant” and so we also have the ability then to capture this information and begin to say well what else can we look at around this time or what else can we look at from other tools.

If we look at the top here we actually have the ability to also do some automated forensic collection. This means that when this particular lead was brought in it automatically triggered some follow-up friends of work to happen and this means reading some process memory to extract this particular thread that was injected. So we’ve actually extracted the code of memory so that we can now do analysis on it.

We’ve also triggered a process memory map, which is gonna tell us here is what the memory layout look like for this process at the time that this was triggered. This is really useful for us because it gives us that information, and as you know malware has the ability to clear memory or kill itself and do things very quickly and so we want to collect as much evidence and as much intelligence as we can, as fast as we can, so we have all the information we need in order to do some analytical work because once it’s destroyed it’s destroyed, and so that is really key. All of this is rolled into this lead.

One of the other powerful things we have the ability to do is say well if we have this particular event that’s occurred now, what’s happened around the time of this event. By clicking on these little arrows right here we actually pull up a really neat thing called the event stream. This is all of the events that agent has collected around the time that that particular lead came in.

You can see that we’ve triggered some there’s some TCP connections, we’ve seen the thread injection here. We see some more TCP connections. We’re also looking at what is the behavior both before and after this lead has come in. That’s gonna tell us potentially the infection vector so, are we seeing Internet Explorer being run and then we’re seeing some Russian malware domain being accessed and then we see all of a sudden now we have a thread injection and a bunch of suspicious stuff happening after that. It helps us to work backwards to figure out “okay where did this happen, what occurred, are we seeing a known vulnerability that’s being used one against one of our customers or did they download a binary that they shouldn’t have?” That really helps us to figure this stuff out and this is a really powerful and useful feature again we could probably do an entire webinar just on the event stream but we live and die going through our investigations looking at this stream to figure out what’s going on around the time of that infection or the time of that suspicious behavior.

This is kind of the main investigation view, again this will give you kind of the basis for where a lot of our investigations start, but today of course, we are looking at how do we look at the network or look at host at a larger level and say what is a statistical outlier or can we look at things that are suspicious just based on the fact that their reputation count is low, meaning we haven’t seen them around a lot, we haven’t seen a particular binary get executed a lot around the network.

How we do that is through the host view and I already have a host few loaded up here and so this is an actual host view for any one of our agents.

We have the ability to pull this information up for any one of our agents across the entire network. This is really useful because when we’re looking at infections or potential outbreaks, we also going to stand back and look at what are they targeting a particular windows version. In this case it’s a windows 7 machine, are they targeting a particular subnet inside of the network, is there a way that it can be contained that way. This host information although granular, meaning we can drive down into one host, it also allows us to kind of look at that 30,000 foot view and say are we seeing commonalities or patterns or are we seeing an exploit being used against only a certain version of windows or a certain set of users and this is really useful.

The other thing to note, is that our customers actually have the ability, when our agents are deployed to segments networks based on utility. So inside of our SNOW system, we have the ability to say anything in this particular subnet is actually the executive team, so it’s the CEO, the CFO, the COO and we can segment those away from the development teams, away from the servers, and so that also allows us to look at what’s going on in a particular subnet, what’s going on with a particular group of people, and those logical divisions are really important when you’re looking at kind of higher-level statistical analysis of threats.

If we start to take a look at what’s available inside the host view, one of the cool things is we have all the latest events from that host. It is literally a running tally, a running audit trail. Here’s all the things that’s been going on in that particular host, starting with the newest stuff at the top, going oldest down. We can see all the events and that means you know unique DNS requests, we can see that we ran some commands, or we can see a code identity event, which means that our agent is actually looked at a binary and extracted its code signing certificate to take a look at it. This is really useful for us because we can see kind of a high level view of what’s going on that particular machine.

We can look at the event stream again, which is what we looked at more specifically to an investigation. We can do the same for a host, or we can begin to drive into the host objects. Inside of the SNOW platform, one of the coolest features is the fact we have this object database. It’s this big database of all the processes, the files, everything we see across an entire network, all gets rolled into this database so that we can actually do analytical work on it and that also means so as you’ll see later, we can query for very specific certain pieces of information so that we can dig into them. This object database is extremely useful and again is something that’s right hunters find incredibly useful when they’re digging into a particular process and they want to say you know where else have we seen this or they want to cross-reference a particular binary against virustotal but you’re also going to see that some of the statistical analysis here is already done for us.

Watch a video version instead

You can already see this very handy outliers link. I have loaded it up in another tab and we’re going to click on that. What the outliers is, if you never heard the term before is basically anything that is statistically outside of the normal. If you’ve ever looked at those graphs where there’s a whole bunch of little thoughts following a line on a graph, and then there’s that one dot up all lonely to the left in the top left of a graph, that’s an outlier, that’s a typical out like Google for a graph outlier that’s typically what you’ll see. It’s stuff that doesn’t fit the normal baseline which is of course where a lot of us live and die in the security world. We want to look at what is the baseline, what are things that we see all the time, and what are things that don’t fit within that baseline.

The beautiful thing about SNOWboard is actually boils down into something that’s really easy for you to look at and understand. We can look at something as simple as the number of locations that we’ve seen that particular binary or dll and say, hey if we have a thousand machines that are all be actively monitor in this particular subnet and we only ever see this file once on all of those machines, we probably want to go take a look at it because this is statistically unusual.

How we know that is we can actually reverse sort this and begin to take a look at what are things that are common. We can see common DLLs that are being used, we can see common processes the further we scroll down so things like svchost which is a critical windows process that runs on every machine, we can see hey there’s you know over 1,300 examples of this particular process running on the subnet that’s not that suspicious. This is really critical and this is often where we will start when we’re looking at what’s going on a host from an outliers perspective. Is anything that’s got a low number of locations in it, is a potential hit for a piece of malware or other threat that could be analyzed.

We’re looking at things like, you know while the chrome updater, this was only executed twice could it be for example that this is just a brand-new version of Chrome that’s being rolled out well.Very easily have the ability to query the database as you will see very shortly to actually look and say well what is this binary, where we’ve seen it before and what’s relevant about it. This is one of the very first things in the fundamental things we do when we’re looking at statistical analysis is the outliers and this is really critical.

What else can we actually look at? Even if we identify that one of these things look suspicious or hey this low number of execution counts. For example, we see malware demo windows, if you ever see malware demo windows running on one of your machines you typically want to go look at it, unless you’re doing one of these webinars. What else can we take a look at? Another thing we can look at are the autoruns. Anything that is going to do some form of persistence is usually most commonly going to target the autoruns keys in the registry and say okay every time this machine boots up or every time this user logs in, we want to make sure that our malware continues to run, that’s really critical.

When we’re looking at autoruns again, we see scrolling down we see things that are really common, we see things that are starting up all the time, and then looking at this particular host we see that there’s some things that look pretty strange. When there’s only for executions out of a network of a thousand-plus machines, that’s pretty strange to see something in the autoruns that is only available on four machines.  Again, if you have some logical separation your network, maybe it’s not that abnormal if you only have four machines because then it wouldn’t mean that that statistically irrelevant, it’s a hundred percent of them. In this case we have over a thousand machines, it is absolutely something that stands out.

What can we actually do to say okay what is this thing and what’s going on? The cool thing is we can just highlight this path and you’ll see this little fly out come up down here, we can click on the bullseye, and say search and objects.

I’ll close our little thing here. When this is done is actually query the database and said okay we want to know more about this particular path.

We actually have a couple of things that have come back from the database. One is a file path, so it’s saying hey we recognize this file path from somewhere. We can look at it and say okay there was four locations, interesting, we don’t have any associated parents of children for this particular file, but then we can actually see the hosts where this particular path showed up. This immediately is going to for us hunters, we’re gonna look at this and say uh oh, there’s four other machines here as it showed the location, we should probably include these roll these into the investigation because they too may have some suspicious behavior going on or they might already be in our investigation list we want to roll the stuff all together into one larger investigation. This might mean that we have a bit of an outbreak on our hands, or again this was for executives could also mean that there’s a spearfishing campaign that has been successful against all four executives and we want to go take a look at it. This is really useful that were able to query this database, take a look at it and say, well what is this thing, where we finding it and that also gives us the ability to go and take a look at what else we potentially need to do to contain the threat.

Same thing with autoruns, being able to click on that object in the database, look at the autoruns information and say, okay where else is it the auto runs?  We see it’s the same for hosts, so not only are we seeing the same file path showing up, we’re also seeing that it’s also persisted, so it’s also inserted itself into the auto runs on these four hosts. We probably want to go take a look at those four machines double hard now because this is really really suspicious. Once we’ve reached this kind of consensus that something very strange is going on, how do we actually begin to react to this? What do we actually do for next steps?

As I mentioned before one of the great things that we can actually highlight this path and we can use it and just say, okay we want to go and retrieve this file just by using our commands box down here, I’m not going to do that today. We reach out and can grab that file back, we can meet of course, depending on the customer, we can begin to work with them to say. hey I think there’s an incident here let’s start talking about implementing your incident response plan, assuming you have one, hopefully everyone listening here today has one, or a number of other things. We can begin reverse-engineering, taking a look at that particular file, we can also begin to look at other things that are going on that host.

This is where we will also look at okay well what have we seen any weird domains coming up in the last while or at all time that looks suspicious or don’t seem to fit, yet there’s some weird stuff we’re seeing here what is going on? Often will also see stuff like people trying to access tor onion, like hidden services through a regular web browser and were often alerted like that’s really weird what is this, what is this person up, when they’re it’s nine o’clock in the morning at their day job. Stuff like that that were able to actually look at and say, well what other things are actually contributing to the overall kind of threat or the overall reason why we would raise kind of the alert level for this particular organization or these sets of hosts.

This is the great thing is that we can dive down into any one of these things on a host, begin to do some statistical analysis on it and then bring that back into more of the investigative viewpoint, so we can always start from an investigative point and work our way up. or we can start from a statistical standpoint and work our way down to an actual investigation. Often what we’ll find is that customers tell us, you know here are critical machines or they might even put them into that logical subnet and we’re able to say, okay you know let’s go out and just start looking at outliers and this is something myself and other hunters on the team will do and we’ll look at it and say, okay well here’s some interesting outliers on these machines, you know what’s interesting or not? We’ll hear this is a weird powershell execution that we don’t see anywhere else, let’s take a look at powershell script and see what it is, ok it’s a homegrown system men you know who’s a written a thing to kind of purge a mail queue or whatever it is. We have the ability to immediately look at those things and start that investigation from 30,000 feet and begin to drive downwards into okay how do we actually acquire some evidence here begin some reverse engineering or looking at you know what other what else is going on that machine.

That’s kind of the beauty of having this data at your fingertips is that you can work from both ends something has been triggered an investigation, work your way up to let’s look at the statistical analysis of this to see if it’s spread or to see if there’s other things that are causing it. You can also start from the other side and work down towards opening an investigation begin rolling some leads together to figure out what’s actually going on.

Watch a video version instead

Today the topic is responding to incidents, which of course could be a 10 day class or longer, but we are going to try and distill it down into about 30 minutes. What I’m going to talk to you a bit about us hunters at Arc4dia approach investigations, how we use some of the tools that we have that Arc4dia built to respond to these incidents to investigate them and work with our customers to try to contain the incident.

SNOWboard is the interface to our agents that are deployed in our customer’s networks. This is where we, the hunters, spend all of our time looking at what’s going on in our customer networks, what investigations are happening, what kinds of stuff we see going on, and of course anything that looks interesting. 

Watch a video version instead

Now today what we’re going to do is focus on a particular host that has had some interesting stuff going on. I’m going to walk you through why we would determine this to be interesting of course this is a piece of malware that we custom build and deploy that actually helps us to train people and helps us to make sure we know exactly how to respond to these things. That’s also what you get with the Arc4dia team, not only defensive mind of people, but offensive mind of people. People who have deep experience working on offensive operations and looking at these things from the other side because it’s absolutely critical to understand how an attacker thinks or how they’re going to write malware and even to determine potential next steps they would take.

We see here this is our investigation list. So, every morning this is exactly what an Arc4dia hunter is going to be looking at, although much larger – this is a demo network. We are going to start looking at things that look potentially suspicious or things that look out of place.

Looking at this host here, and when clicking on this investigation is going to open up this particular screen and this screen is actually our investigation view.

An investigation is made of leads. So just when you see a detective on TV investigating a murder, that murder case is going to have leads that come together to give context for entire investigation. This is exactly how we structure SNOWboard to display information to us that we have leads that make up this investigation.

One of the cool things we have the ability to do is collaborate with one another via log. We can actually work together to write messages, say it’s time to reverse engineer this thing we found, we can leave notes for one another and our customers. This allows us to all work collaboratively as a team to investigate these incidents.

Further down into the lead list. Starting at the bottom, take a look at what we’re seeing. We have some information on the left that’s saying at this particular time on this host we had one suspicious thread that was found. This is on its own, could be potentially malicious, it could be something that actually completely benign, but the important thing is that we want to dive into the technical information. So, we are not interested in looking at fancy pie graphs or other information that doesn’t give us context. We’re interested in the raw technical data that’s going to allow us to make judgement calls based on our experience as responders and as attackers to think about what does this actually mean.

We can take a look at the information on the right hand side and say Ok, there’s a suspicious thread in vmtoolsd.exe.

VM tools is a process that is used to help make your virtual machines run a little nicer when you’re using them. This is suspicious that we see something being injected into vmtoolsd.exe.

We can see some pretty technical information here about what was injected, how big the memory was of the thread, what process it was, the threat ID, and where this thread was actually started – the base address and start address of that particular thread.

What we’re also able to do is, which is critical in an incident, is look at the context around when this thread injection happened. We can do this by clicking the fancy arrows that will bring up what’s called the event stream.

The event stream allows us to take a look at what’s going on before this event and what’s going on afterwards. You might see someone spawn Internet Explorer, then shortly thereafter there’s another process that gets executed and then you see a chain of events happening after that. This is really important because without context it’s extremely difficult to respond to an incident because you are kind of operating in the dark. This little tool here allows us to very quickly scroll through and see what’s going on. This is a very detailed view.

If we want only wanted to, for example, see all of the new processes created, around this time, we can apply a filter very easily to show us the new processes and we can filter it out and see all the new processes we have in this time range before and after. We can also expand the time range or contract it and adjust it based on how we want to investigate this particular incident.

Scrolling through, we’re building up a picture of what’s actually going on and it allows us to really become familiar on what is happening on this machine at this particular time. Often if you don’t have this information it’s very difficult to make judgement calls because sometimes you will see things that might look suspicious initially, but after you dig into them and look through things that are going on around it, it’s really not that suspicious. There’s a lot of genuinely non-invasive software out there that does sometimes behave a little suspiciously.

Scrolling through and some point we see this THREAD_INJECTION occur.

We also see some other events like a READPROCESSMEMORY, and will get into that. That’s part of SNOW’s automatic forensics capabilities that’s going to automatically a hunter or an analyst by collecting information for us.

We can see the injection happens here

We can see that we trigger a SUSPICIOUS_FILE_COLLECTION that we’ve pulled down this particular file again.

We see some HOLLOWING after that.

If you were on our previous webinar, we talked about process hollowing where there’s a piece of malware that actually hollows out a chunk of memory in a process and shoves code in it and executes it.

We can see some other stuff like a TCP_CONNECTION

We’re starting to build a picture around this time that something definitely looks suspicious, something is definitely up with this machine.

When we’re talking about a machine, we’re talking about hosts in SNOWboard. What we would also want to do in this case is say, what else can we learn about this host to help aid in this investigation.

Clicking in the box, the host view will pop up.

Any host in the network that we have the SNOW agent deployed on, were able to dive in to see what’s installed on that machine and what investigations that machine has triggered. We can also take a look at things that are more detailed like what processes we see running, unique modules (dlls or binaries that are actually executing on that machine). We can see the installed services, which can be really useful when you see something that is using a Window service for persistence.

Often we’ll also see that there are things being installed in the autoruns of that particular machine.

This is where malware will use registry keys or other techniques to gain persistence so when you reboot the machine the malware is going to get restarted so that it can continue to speak back to its command and control or it can continue to suck up keyboard keystrokes or whatever it’s designed to do.

Clicking on that view actually has got a lot of data that might not be apparent at first. The first thing we always look at is the column on the left hand side which is the number of locations that we’ve seen this particular file being executed. When we see this and we see a very low number like four out of hundreds of thousands of deployed agents, that tells us that something is definitely weird with this that we haven’t seen this very often and we’re seeing this now on this particular machine.

As you scroll down, the reputation of these binaries is actually going up. That’s exactly what are number of locations is. It’s effectively a reputational score telling you that the more times you’ve seen this thing, the more likely it’s OK or that we can easily explain where it comes from. The lower the number the worse it is for that particular host.

Immediately we’re looking at this and we’re saying, so we have this strange thing in program data. Program data is not a standard place that we see VMware installing things so this looks a little suspicious.

What we are able to do is then query our own database to see what we know about this particular process and can we learn from other hosts.

Watch a video version instead

One of the cool things we have is the ability to highlight any binaries we want, in the context menu saying search inobjects. It allows us to search this vast database we have of processes, modules, and file names. It will come back and tell us if we have some records for that particular object. We’ve only seen this process a couple of times on a couple of hosts. We can actually look specifically at the file name. This is also telling us that there’s more than this one host that may have been potentially infected or have these suspicious processes running. This is something that will tell us there’s more context here. We would probably want to make a note in this investigation to add these hosts also have this exact same processes, maybe we need to hash the files and see what’s going on.

This object database is extremely powerful because it allows us to easily drill down into what’s happening on that machine and look at exactly every binary that’s involved in this investigation, every process and say where else is it located and have we seen some of these things before.

If we were looking at this particular file we saw it was pretty rare and it wasn’t located on a lot of other machines. We can also see part of the investigation was vmtoolsd.exe as we saw in the leads. We can see it was executed in 252 different locations. This is also telling us some information that potentially this piece of malware is injecting threads directly into vmtoolsd.exe and then is attempting to hide itself as a vmware product in a different directory and in the autoruns of that machine.

What can we do from a response perspective? One of the things we can do is to reach out to that machine and say we’re a little suspicious of this, what we want to do is grab that file and make sure we have a copy of it. By highlighting the file path and then selecting the hand, this gives us a small list of commands that we can run. We have a larger list of commands that I will show you shortly.

This allows us to very simply in a couple of clicks, grab this file from this host. The investigation here is blank because I’m doing it from the host view, if we were doing this from the investigation view it automatically populates this in.

What we want this to do, is when we task this agent and bring us back this file, we want this file to automatically be assigned to the investigation we are working on so we can keep all the information in one place and so  hunters can being doing tasks and sharing the information with each other.

Click apply and that’s going to immediately send that command off to that agent. That agent is now going to work on pulling that file and sending it back to us and assigning it to that investigation. Often this is now where as hunters we’re also reverse engineers, so we’re going to take that file were going to pull it down and run it through some of the tools we’ve developed as well as traditional reverse engineer tools such as debuggers, IDA Pro, or hex editors, so we can begin to determine is this something we’ve seen before, is this one of our customer’s being explicitly targeted, is this a piece of malware designed explicitly for this customer, or is this something that’s part of larger group of ransomware family or some other piece of malware that has been repurposed so that the hash is different. This is also another value added thing you are going to get. As hunters, we’re also skilled at being able to pull apart this code and analyze it to see what’s going on behind the scenes.

We’ve done a little bit of investigation into the host, the other thing we can do is now go back and take a look a bit more information in surrounding what else is going in this particular lead that we can gain some information from. We’ve only really looked at the one lead and there’s other ones we can look at.

One cool thing that really assists hunters and our customers, is the ability for SNOW to automatically collect information when something weird has happened. Very easily we can expand the key at top, and we can see that a few things are already available for us as analysts or hunters.

We see a process memory map, this gives us a memory map of the process in question telling us who owns what chunk of memory, are there any pieces of memory that are flagged that are suspicious or look a little weird to us. We’re able to grab segments of memory with the click of a mouse. If we wanted to pull it back and analyze that memory and do a bit of forensics work on it and see what’s going on with it. This is all done automatically, we didn’t have to interact with anything, this is part of the intelligence of the SNOW system that is automatically pulling this information back for us. It’s saving us time and giving us highly contextual information. As malware executes, if you get there much later and haven’t had an opportunity to run tools or something else has impacted your ability to analyze it, having some very contextualized information at the time that there was something suspicious detected, is really important.

We see the process memory map, another thing that it did was automatically collect the little chunk of memory where the thread is executing. Down here where we see this thread executing, it’s actually reached out and grabbed that chunk of memory back so we that we can reverse engineer it. This is really critical because as you know memory is not persistent, when something writes something to disk there’s a level of persistence that you are able to retrieve. If something writes something into memory and that process exits at some point, that memory is gone and you have no ability to perform forensics on it or do any analysis. Whereas with this system, we able to get that memory right away so that we can come back much later if needed and retrieve it and still look at what’s going on in that chunk of memory. This is really important and important when dealing with incidents are able to do that.

This is critical stuff that allows us to build a much broader and detailed investigation. Working with our customers we are able to decide how do you want us to proceed. Sometimes it’s decided in advance or customers are very pro-active want to be involved.

Now that we’ve identified something weird going on, how can we actually work to stop what it’s doing for now, completely terminate it, or what do we actually have to do to get rid of this. This is going to vary depending on customer’s operating policies and is also going to depend on the particular piece of malware we’re dealing with.

What we can do is we actually have a list of very detailed commands that allows us to interact with that machine. I’ve already pulled up this list of commands – this is accessible through our dashboard by clicking on commands – and have populated the terminate thread command already with the thread ID that we identified as well as the host.

This is actually a real subtle but powerful technique. For example, if we identify that there is a suspicious thread running in a critical process, what we shown and done in the past is we can actually isolate that thread and suspend just that thread and that allows the critical process to continue executing. Sometimes malware will actually corrupt the process and it will blow up and die, other times if they’ve done it right, they will actually have the thread there sitting in memory executing and we’re able to come along and just suspend that thread and continue executing without destroying that critical process or destroying any evidence. In this case, if I wanted to completely terminate the thread, I can signal to our agent that the thread ID I’ve identified is being kind of malicious, we want to terminate that thread completely and get rid of it so that’s it’s not executing anymore.

This is actually a really cool command that we have the ability to do and do this to any process on the machine and we send that off for tasking.

Some of the other ones we demonstrated already such as getting files, we can get some information about the file. Also, if we’ve identified the vmware update file that was in the autoruns, if we identify that that’s a bad file and we want to get rid of it, what we can simply do is “delete file” and we can tell the remote agent to delete the file. That allows us to just completely remove that file off of that machine and we already have a copy of it because we reached out the machine and pulled it back and we already have a copy of it locally that we can operate off of.

In some cases, you will do this if it’s a known piece of malware. Other cases you might not want to do that, you might want to do some network isolation or some other things. This is really driven by the customer, we don’t make decisions on our own, we work very closely with our customers when there are incidents to make sure that they are as much as part of the team as anyone else.

Those are some of the file operations. Some of the traditional stuff like you can suspend a process so we can tell it to hang there while we investigate other pieces on the machine, we don’t want to kill the process, but we want to hang it and continue investigating around it and look around on the machine. We might even want to resume the malware so we can identify where it’s command and control is, or we want to completely suspend it and then decide to kill this process all together and have it completely dead and perform clean-up on that machine.

This is also one of those things that there are many different best practices and no two incidents are the same, but having strong policies helps teams approach these things in a systematic way. Sometimes we’re able to clean up a Stage 0 infection very easily all on our own with a little input from the customer. Other times there’s much more detailed things that have to happen, and we work to write reports that say all the things we’ve discovered about it, and the reverse engineering report surrounding what happened with this particular piece of malware. This varies from incident to incident, but is a very collaborative approach between Arc4dia and our customers and that’s exactly how we handle these things.

Watch a video version instead