Thomas Labarussias May 30, 2023

Monitoring your EKS clusters audit logs

This blog post is an update of a post of November 2022

At the beginning of the year 2022, Falco introduced a game changing feature, the Falco Plugins. They allow Falco to monitor and trigger alerts for any kind of event. Since the launch of the new plugin framework the Falco community has collaborated to create plugins for Github, AWS CloudTrail and Okta. A plugin has also replaced the way Falco consumes the Audit Logs generated by a K8s API server through a dedicated plugin. With these plugins, Falco covers more in depth the aspects of your infrastructure and allows you to use a single syntax for rules.

For months (okay, maybe years...), our adopters have asked us for a way to monitor K8s Audit Logs. The previous implementation used an internal web server to receive the logs from the Kubernetes API, although it was functional, it was a very manual process to install and manage clusters. This method didn't support clusters managed by cloud providers, such as EKS, AKS, or GKE as they had to capture the Audit Logs for their own usage and then add them to their log aggregators.

This situation is now solved thanks to the plugin framework and we're proud to announce the first release of the plugin for EKS Audit Logs: k8saudit-eks!!!

How it works

AWS captures the Audit Logs and exposes them in the CloudWatch Logs service. We have made available libs to create a clean session with the AWS API and pull logs from the relevant Cloudwatch Logs Stream. You can reuse these libs for any plugin you'd like to create for any Amazon service.

Usage

The configuration for the usage of the plugin is:

plugins:
  - name: k8saudit-eks
    library_path: libk8saudit-eks.so
    init_config:
      region: "{REGION}"
      profile: "{AWS_PROFILE}"
      shift: 10
      polling_interval: 10
      use_async: false
      buffer_size: 500
    open_params: "{CLUSTER_NAME}"
  - name: json
    library_path: libjson.so
    init_config: ""

load_plugins: [k8saudit-eks, json]

With:

init_config:
- profile: The Profile to use to create the session, env var AWS_PROFILE if present
- region: The Region of your EKS cluster, env var AWS_REGION is used if present
- use_async: If true then async extraction optimization is enabled (Default: true)
- polling_interval: Polling Interval in seconds (default: 5s)
- shift: Time shift in past in seconds (default: 1s)
- buffer_size: Buffer Size (default: 200)
open_params: A string which contains the name of your EKS Cluster (required).

If you run Falco inside an EKS cluster with a setup of an OIDC provider, the profile and region parameters can be omitted in favor of a service account + IAM Role (see the official docs).

IAM

Whatever the method you use for the authentication to the AWS API, you need to set up this minimal policy to your user/role:

{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Sid":"ReadAccessToCloudWatchLogs",
      "Effect":"Allow",
      "Action":[
        "logs:Describe*",
        "logs:FilterLogEvents",
        "logs:Get*",
        "logs:List*"
      ],
      "Resource":[
        "arn:aws:logs:${REGION}:${ACCOUNT_ID}:log-group:/aws/eks/${CLUSTER_NAME}/cluster:*"
      ]
    }
  ]
}

With:

REGION: The Region of your EKS cluster
ACCOUNT_ID: The ID of the account running the EKS cluster
CLUSTER_NAME: The name of your cluster EKS cluster (same value than in your plugin configuration)

Default Rules

A good thing about Kubernetes is that it brings standards into our industry. Despite a few differences, the cluster works in the same way and produces the same format of logs. This helps us enormously. By creating the k8saudit plugin we declared the fields to extract, as well as some default rules, which we can reuse for any plugin that consumes the same Audit Logs. It is a time saver for both, developers and adopters.

You can find the proposed default rules here.

Installation

For the installation, we'll use falcoctl. We can use it as a CLI tool for a locale installation or as a sidecar for Kubernetes. See this blog post for more information.

Local installation

In this example, we'll see how to install the k8saudit-eks in a local host.

First, add the index of artifacts for falcoctl:

sudo falcoctl index add falcosecurity https://falcosecurity.github.io/falcoctl/index.yaml

Install the latest version of the plugins:

sudo falcoctl artifact install k8saudit-eks

sudo falcoctl artifact install json

As the output notices, the plugins will be now available in /usr/share/plugins:

 INFO  Reading all configured index files from "/root/.config/falcoctl/indexes.yaml"
 INFO  Resolving dependencies ...
 INFO  Installing the following artifacts: [ghcr.io/falcosecurity/plugins/plugin/k8saudit-eks:latest]
 INFO  Preparing to pull "ghcr.io/falcosecurity/plugins/plugin/k8saudit-eks:latest"
 INFO  Pulling b2daf90a878e: ############################################# 100%
 INFO  Pulling 60b0846d1e18: ############################################# 100%
 INFO  Pulling 12c7f0f5f00e: ############################################# 100%
 INFO  Artifact successfully installed in "/usr/share/plugins"

 INFO  Reading all configured index files from "/root/.config/falcoctl/indexes.yaml"
 INFO  Resolving dependencies ...
 INFO  Installing the following artifacts: [ghcr.io/falcosecurity/plugins/plugin/json:latest]
 INFO  Preparing to pull "ghcr.io/falcosecurity/plugins/plugin/json:latest"
 INFO  Pulling 087dba2e76d6: ############################################# 100%
 INFO  Pulling c4cb35bb528a: ############################################# 100%
 INFO  Pulling 9b50ea237bc6: ############################################# 100%
 INFO  Artifact successfully installed in "/usr/share/plugins"

After installing the k8saudit-eks plugin, we also need to install the rules that it uses. (Those rules are the same as the k8saudit plugin rules.):

sudo falcoctl artifact install k8saudit-rules --resolve-deps=false

And we get the rules in /etc/falco:

 INFO  Reading all configured index files from "/root/.config/falcoctl/indexes.yaml"
 INFO  Installing the following artifacts: [ghcr.io/falcosecurity/plugins/ruleset/k8saudit:latest]
 INFO  Preparing to pull "ghcr.io/falcosecurity/plugins/ruleset/k8saudit:latest"
 INFO  Pulling b8c6b1def3eb: ############################################# 100%
 INFO  Pulling 6b6f3231f2b2: ############################################# 100%
 INFO  Pulling 91b2b7a9944d: ############################################# 100%
 INFO  Artifact successfully installed in "/etc/falco"

By default, the installation of a ruleset comes with the associated plugins. In this blog, we have installed them on our own, this is why --resolve-deps is set to false.

We now have the plugins and rules in our system. The last step is to configure Falco to use them. Modify your /etc/yaml to make it looks like the following (with the relevant values for REGION, ACCOUNT_ID, and CLUSTER_NAME):

rules_files:
  ...
  - /etc/k8s_audit_rules.yaml

plugins:
  - name: k8saudit-eks
    library_path: libk8saudit-eks.so
    init_config:
      region: "{REGION}"
      profile: "{AWS_PROFILE}"
      shift: 10
      polling_interval: 10
      use_async: false
      buffer_size: 500
    open_params: "{CLUSTER_NAME}"
  - name: json
    library_path: libjson.so
    init_config: ""

load_plugins: ["k8saudit-eks","json"]
...

You can now start Falco ( sudo /usr/bin/falco --disable-source syscall) and monitor the audit logs of your EKS cluster:

Wed May 10 14:07:18 2023: Falco version: 0.34.1 (x86_64)                                                                                                                                                    
Wed May 10 14:07:18 2023: Falco initialized with configuration file: /etc/yaml                                                                                                                  
Wed May 10 14:07:18 2023: Loading plugin 'k8saudit-eks' from file /usr/share/plugins/libk8saudit-eks.so                                                                                               
Wed May 10 14:07:18 2023: Loading plugin 'json' from file /usr/share/plugins/libjson.so                                                                                                               
Wed May 10 14:07:18 2023: Loading rules from file /etc/k8s_audit_rules.yaml                                                                                                                           
Wed May 10 14:07:18 2023: Loading rules from file /etc/rules.d/override-k8saudit.yaml                                                                                                                 
Wed May 10 14:07:18 2023: Starting health webserver with threadiness 4, listening on port 8765                                                                                                             
Wed May 10 14:07:18 2023: Enabled event sources: k8s_audit                                                                                                                                                  
Wed May 10 14:07:18 2023: Opening capture with plugin 'k8saudit-eks'

In this example, we also disable the collection of the syscalls with the option --disable-source syscall. The k8saudit-eks plugin works with a pull model, you should have only one Falco instance collecting the audit logs to avoid any duplication of alerts. We dedicate that instance to that role.

And get alerts:

14:09:17.699479000: Informational K8s Serviceaccount Created (user=system:node:ip-192-168-34-135.ec2.internal serviceaccount=default ns=default resource=serviceaccounts resp=201 decision=allow reason=)
14:09:24.046826000: Notice Attach/Exec to pod (user=kubernetes-admin pod=kubecon resource=pods ns=default action=exec command=sh)

Kubernetes installation

Next, we will see how to install the k8saudit-eks plugin with the falcoctl tool in a Kubernetes setup. We'll use the official Helm chart with the following values.yaml file:

serviceAccount:
  create: true
  annotations:
    - eks.amazonaws.com/role-arn: {ROLE_ARN}

falco:
  rules_files:
    - /etc/falco/k8s_audit_rules.yaml
    - /etc/falco/rules.d
  plugins:
    - name: k8saudit-eks
      library_path: libk8saudit-eks.so
      init_config:
        shift: 10
        polling_interval: 10
        use_async: false
        buffer_size: 500
      open_params: "{CLUSTER_NAME}"
    - name: json
      library_path: libjson.so
      init_config: ""
  load_plugins: [k8saudit-eks, json]

falcosidekick:
  enabled: true

driver:
  enabled: false
collectors:
  enabled: false

controller:
  kind: deployment
  deployment:
    replicas: 1

falcoctl:
  indexes:
  - name: falcosecurity
    url: https://falcosecurity.github.io/falcoctl/index.yaml
  artifact:
    install:
      enabled: true
    follow:
      enabled: true
  config:
    artifact:
      allowedTypes:
        - plugin
        - rulesfile
      install:
        resolveDeps: false
        refs: [k8saudit-rules:0, k8saudit-eks:0, json:0]
      follow:
        refs: [k8saudit-rules:0]

Let's look into this configuration section by section:

serviceAccount:
  create: true
  annotations:
    - eks.amazonaws.com/role-arn: {ROLE_ARN}

With EKS, we can use an OIDC provider and annotations on the service accounts to allow access to the AWS API ( see the official docs). This section allows you to set which ROLE_ARN to use.

falco:
  rules_files:
    - /etc/falco/k8s_audit_rules.yaml
    - /etc/falco/rules.d
  plugins:
    - name: k8saudit-eks
      library_path: libk8saudit-eks.so
      init_config:
        shift: 10
        polling_interval: 10
        use_async: false
        buffer_size: 500
      open_params: "{CLUSTER_NAME}"
    - name: json
      library_path: libjson.so
      init_config: ""
  load_plugins: [k8saudit-eks, json]

As the local installation, this section sets the used plugins.

driver:
  enabled: false
collectors:
  enabled: false

controller:
  kind: deployment
  deployment:
    replicas: 1

To collect and monitor all the syscalls, Falco must be deployed as a DaemontSet in your cluster, to have one instance per Kernel. In this example, we're deploying Falco with the k8saudit-eks plugin, which relies on a pull model. To avoid several Falco pods to collect the same audit logs and duplicate the alerts, we MUST install Falco with the k8saudit-eks plugin only once. The Helm chart allows to use a one replica deployment and to disable the syscalls and K8s metdadata collections, this is the purpose of this section.

falcoctl:
  indexes:
  - name: falcosecurity
    url: https://falcosecurity.github.io/falcoctl/index.yaml
  artifact:
    install:
      enabled: true
    follow:
      enabled: true
  config:
    artifact:
      allowedTypes:
        - plugin
        - rulesfile
      install:
        resolveDeps: false
        refs: [k8saudit-rules:0, k8saudit-eks:0, json:0]
      follow:
        refs: [k8saudit-rules:0]

The last section sets up falcoctl. It will install the requested plugins and rules and track new versions of the rules.

Now you can run your helm install command:

helm install falco-k8saudit-eks -n falco falcosecurity/falco -f ./values.yml

And see the result:

$ kubectl get pods -n falco

NAME                                      READY   STATUS    RESTARTS        AGE
falco-k8saudit-eks-96d7f6f99-vwjgc        2/2     Running   0               70m

$ kubectl logs falco-k8saudit-eks-96d7f6f99-vwjgc -n falco -c falco
Wed May 10 14:07:18 2023: Falco version: 0.34.1 (x86_64)
Wed May 10 14:07:18 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Wed May 10 14:07:18 2023: Loading plugin 'k8saudit-eks' from file /usr/share/falco/plugins/libk8saudit-eks.so
Wed May 10 14:07:18 2023: Loading plugin 'json' from file /usr/share/falco/plugins/libjson.so
Wed May 10 14:07:18 2023: Loading rules from file /etc/falco/k8s_audit_rules.yaml
Wed May 10 14:07:18 2023: Loading rules from file /etc/falco/rules.d/override-k8saudit.yaml
Wed May 10 14:07:18 2023: Starting health webserver with threadiness 4, listening on port 8765
Wed May 10 14:07:18 2023: Enabled event sources: k8s_audit
Wed May 10 14:07:18 2023: Opening capture with plugin 'k8saudit-eks'
{"hostname":"falco-k8saudit-eks-96d7f6f99-vwjgc","output":"14:09:17.699479000: Informational K8s Serviceaccount Created (user=system:node:ip-192-168-34-135.ec2.internal serviceaccount=default ns=default resource=serviceaccounts resp=201 decision=allow reason=)","priority":"Informational","rule":"K8s Serviceaccount Created","source":"k8s_audit","tags":["k8s"],"time":"2023-05-10T14:09:17.699479000Z", "output_fields": {"evt.time":1683727757699479000,"ka.auth.decision":"allow","ka.auth.reason":"","ka.response.code":"201","ka.target.name":"default","ka.target.namespace":"default","ka.target.resource":"serviceaccounts","ka.user.name":"system:node:ip-192-168-34-135.ec2.internal"}}
{"hostname":"falco-k8saudit-eks-96d7f6f99-vwjgc","output":"14:09:24.046826000: Notice Attach/Exec to pod (user=kubernetes-admin pod=kubecon resource=pods ns=default action=exec command=sh)","priority":"Notice","rule":"Attach/Exec Pod","source":"k8s_audit","tags":["k8s"],"time":"2023-05-10T14:09:24.046826000Z", "output_fields": {"evt.time":1683727764046826000,"ka.target.name":"kubecon","ka.target.namespace":"default","ka.target.resource":"pods","ka.target.subresource":"exec","ka.uri.param[command]":"sh","ka.user.name":"kubernetes-admin"}}

Disclaimer

Our tests noticed some latencies between the presence of the logs in the CloudWatch Logs Stream and their evaluation by Falco. This is more visible with highly requested API servers. The solution is to adapt the size of your nodes where Falco runs, considering a minimal size of 2xlarge as a safe option.

Conclusion

With this first Plugin for a managed K8s solution, we hope to open the door for new contributions from the community for other flavors like GKE and AKS. If you need to create a plugin for another AWS service, take also a look at the libs we created to help the developers.

You can find us in the Falco community. Please feel free to reach out to us for any questions, suggestions, or even for a friendly chat!

If you would like to find out more about Falco:

Get started in Falco.org
Check out the Falco project in GitHub.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

←Previous

You are viewing documentation for Falco version: v0.39.2