In Linux Everything Is a File. In Threat Detection, Everything Is a Stream.

OK, so we all love how computers get older, right? We're getting some computer history now, like all that old-school UNIXy stuff. However, as is a theme in this newsletter, they're not even that old, electronic computers—maybe 90 years or so. In the grand scheme of things, electronic computers are still very young. Yet, despite their relative youth, computing can sometimes give the impression of being old. Some technologies have been around for multiple generations, leading us to forget certain things, discard other technologies, and build upon those older ideas. We've done a lot in those 90 years.

In that sense, if we go back in computing history, the idea of "everything is a file" was, for a while, a new way of thinking; it was a new and defining concept.

Files in UNIX are arranged in a hierarchical, tree shaped structure. There are two types of object: files, and directories. A directory is actually no more than a file, but its contents are controlled by the system, and the contents are names of other files. (A directory is sometimes called a catalog in other systems.) - Dennis Ritchie

Say we have some file.

/etc/some-file

This filename is actually a piece of metadata about an "inode" on the filesystem. We could easily read that file in the way we would expect, like opening it in an editor or cat. It's a file; it behaves like a file. We are used to that behaviour.

$ touch /etc/some-file
$ stat /etc/some-file
File: /etc/some-file
Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
Device: 802h/2050d	Inode: 22345       Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-01-20 14:52:54.760000000 +0000
Modify: 2024-01-20 14:52:54.760000000 +0000
Change: 2024-01-20 14:52:54.760000000 +0000
Birth: 2024-01-20 14:52:54.760000000 +0000

Another example in Linux is the /proc virtual filesystem, which provides an interface to the data structures in the Linux kernel. These data structures are not files, but with the /proc filesystem, they respond like files, and we can treat them as files.

$ cat /proc/1/status | grep Name
Name:	systemd

So now I can inspect the processes running in the kernel by inspecting a file. Even though systemd is not a file, it's a process. Neat.

Historically, UNIX was the first operating system to abstract all I/O under such a unified concept and small set of primitives. At the time, most operating systems were providing a distinct API for each device or device family. Some early microcomputer operating systems even required you to use multiple user commands to copy files – each one dedicated to a specific floppy disk size! - In UNIX Everything is a File

Imagine having to use a specific command to copy a file. Yeesh. Obviously, this idea was a useful abstraction in that we could just treat most things as files with a single API.

Difference Between a Stream and a File

So what's the difference between a stream and a file?

Files typically exist over a longer period of time and store data such as documents, images, logs, and so on. They don't have to be static, but we tend to think of them that way—certainly, when we open a file, we get the contents of that file at that time. If someone or something writes to the file after we've opened it, we don't see those changes; we just have our snapshot of the file at that moment.

Conversely, a stream is a sequence of data items made available over time; it's a flow of data. When we open a stream, we expect new events and new changes. Unlike files, streams don't usually have a fixed size. Typically, the sequential processing of streams occurs from a source such as a network connection. A new event comes in, we process it, and then we move on to the next event. We process data as it becomes available. We can see how this might be a "cloud syscall": a log of a cloud API call or set of API calls, what happened, who performed the action, etc. Not exactly a system call, but in the same ballpark.

In Threat Detection, Everything is a Stream

Threat detection is about determining what's happening RIGHT NOW and whether what's happening is malicious—whether it's a threat. What is happening is actually a series of events, which we can think of as a stream. We need to monitor those streams. But how do we do that? That's the big question. I want to make it clear that threat detection is a topic that covers a lot of ground. There is a lot of work that has to be done in threat detection and much analysis at many levels, but at the very beginning, we have to monitor what is happening; we have to get into those streams. We can do all kinds of analysis after we start watching, but we have to start watching.

Enter open-source tools like the Cloud Native Computing Foundation's Falco project.

If you know Falco, you probably know that it analyzes Linux kernel system calls and applies deterministic threat detection rules to them. Well, that is what it did in the beginning. But now Falco has evolved into a tool that can also analyze any streaming log file via a plugin-based architecture. So now, instead of just system calls,we can apply the same fast, deterministic rules-based threat detection model to any stream.

At the implementation level, the plugin system sends events to Falco using the same protocol that is used for syscalls, but the encoding and decoding of the payload is performed by the plugin itself according to its own data format representation. Every plugin exposes the necessary functionality to seamlessly integrate its events with Falco. - Falco.org

So not only can Falco look at system calls, it can also look at streaming log files, like the ones that come from public clouds, or the Kubernetes audit log, or software-as-a-service (SaaS) tools like Okta and Github. All of these have some sort of streaming log that we can analyze for maliciousness.

In this respect, we could say that "cloud logs are the new syscall."

Currently, Falco has plugins for streams such as those below (there are more):

• Kubernetes audit log
• Cloudtrail
• Seccomp Agent
• Github
• Okta

Example - The Falco Plugin for Okta

If we build the Okta plugin and install it and Falco as well, of course, then we can configure Falco to apply threat detection rules to the Okta event log.

For the sake of simplicity, I've setup only one Okta rule—a rule that notifies me when someone has logged in via Okta.

Here is a Falco rule specifically designed for Okta. It will alert when a user has logged in.

$ cat okta_rules.yaml
- rule: User logged in to OKTA
  desc: Detect the user login in to OKTA
  condition: okta.evt.type = "user.session.start"
  output: "A user has logged in toOKTA (user=%okta.actor.name, ip=%okta.client.ip)"
  priority: NOTICE
  source: okta
  tags: [okta]

As well, Falco has been configured to load that plugin and read that rules file (which only contains one rule, but normally would hold many).

$ grep "name\: okta" falco.yaml -A 7
  - name: okta
    library_path: /usr/share/falco/plugins/libokta.so
    init_config:
      organization: dev-<HIDDEN>
      api_token: <SECRET>
      cache_expiration: 84600 #24h
      cache_usermaxsize: 200
      refresh_interval: 10 #in seconds

Now, once a user has logged into Okta, we will get a notice from Falco.

# An entry from /var/log/syslog
Jan 20 16:15:08 falco falco: 16:14:06.659000000: Notice A user has logged in toOKTA (user=Curtis Collicutt, ip=192.168.0.10)

Streamin'

I've shown a very basic configuration for using Falco to monitor the Okta audit log. Falco provides various plugins to monitor many different types of streaming logs, and through the plugin system, you could monitor any stream (as long as you're willing to write the code to do it).

For threat detection, we need to be able to watch for all kinds of events, and those events will come in via a time-based stream. What's more, there's going to be a lot of them. Okta. Github. Public cloud audit logs. And as SaaS software becomes more popular and more distributed across enterprises, we're going to need to watch those audit logs for events.