This article is based on my experience studying for and passing the Certified Kubernetes Security Specialist exam. I passed the exam on my first attempt in Sep 2021.
I passed the Certified Kubernetes Application Developer exam back in Feb 2020, followed by Certified Kubernetes Administrator in March 2020.
The Certified Kubernetes Security Specialist or CKS exam was released around November, 2020, but I didn’t have a chance to take that exam before Sep 2021.
As a bit of background information, I have been working with Kubernetes for the past 3 years almost on a day-to-day basis and that experience was an added advantage in helping me pass the CKS.
In this article, I’ll share some resources that should help you study for and pass the exam, along with a helpful cheatsheet you can use while preparing. I’ll also share some advice that should help you along the way.
What is Kubernetes?
Kubernetes is the most evolved and feature-rich Container Orchestration system out there, and it keeps getting better.
It has an enormous community to support, and it’s always building new features and resolving issues. Kubernetes is certainly evolving at a breakneck pace, and it becomes a challenge to keep up with its pace of development. This makes it the best bet for a container orchestration solution.
Table of Contents:
- Resources for the CKS Exam
- Kubernetes Cheat Sheet
- kubectl run command
- How to generate yaml spec from an existing Pod
- kubectl pod commands
- How to Print logs and export them
- How to create configmaps and secrets
- Helpful commands for debugging
- Rolling updates and Rollouts
- Scale and Autoscale command
- Network Policy
- Static Analysis using Kubesec
- Vulnerability scanning using Trivvy
- Hwo to remove unwanted services
- Runtime Classes
- RBAC Commands
- Cluster Maintenance
- CKS Exam Tips
- CKS Exam Topics
- How to secure and harden container images
- How to minimise OS Footprint
- How to limit node access
- SSH Hardening
- How to restrict network access
- Linux Syscalls
- AquaSec Tracee
- How to restict Syscalls with Seccomp
- Linux Capabilities
- How to Prepare for the Exam
- Practice, Practice and Practice!
Resources for the CKS Exam
The following are a few awesome resources available on passing the CKS exam:
- Certified Kubernetes Security Specialist by Killer.sh
- Certified Kubernetes Security Specialist (CKS) by KodeKloud
- Walid Shaari has gathered some indispensable materials for the CKS exam
- Abdennour’s References for CKS Exam Objectives
- Ibrahim Jelliti’s collection of resources to prepare for the Certified Kubernetes Security Specialist (CKSS) exam
- Let’s Learn CKS Scenarios by Saiyam Pathak
The courses for KodeKloud and Killer.sh provide mock exam simulators which are very helpful in preparing for the exam, and provide a pretty good idea of what the exam looks like. I strongly suggest enrolling in one or both courses.
Purchasing the exam from Linux Foundation provides you with 2 free attempts to exam simulator from killer.sh that way if you are well versed with the contents of the curriculum you can skip the courses and directly go for the exam simulator provided with the exam.
The exam costs $375 but there are offers and deals available, and if you look for them you might be able to get a better price. The duration of the exam is 2 hours and is valid for 2 years, unlike the CKA and CKAD which are valid for 3 years.
The CKS is a performance-based exam where you are provided with an exam simulator in which you have to work out the problems. You are allowed to open only one tab apart from the exam tab.
Since this exam requires you to write a lot of commands, I figured early on that I’d have to rely on aliases to reduce the number of keystrokes to save time.
I used the vi editor during the exam, here I will share some useful tips for this editor.
vi defaults for ~/.vimrc:
kubectl defaults for ~/.bashrc:
kubectl get command provides short catchy names for accessing resources and like
persistentstorageclaim, these can help save a lot of keystrokes and valuable time during the exam.
- po for
- rs for
- deploy for
- svc for
- ns for
- netpol for
- pv for
- pvc for
- sa for
Kubernetes Cheat Sheet
kubectl run command
kubectl run command provides a flag
--restart which allows for the creation of different kinds of Kubernetes objects from a Deployment to CronJob. The below snippet shows the different options available for the
How to generate yaml spec from an existing Pod
Sometimes it is easier to generate a spec from an existing Pod and make changes to it than create a new one from scratch. The
kubectl get pod command provides us with the required flags to output the pod spec in the format we want.
kubectl pod commands
kubectl run command provides a lot of options like specifying requests and limits a Pod is supposed to use or the commands a container should run once created.
How to Print logs and export them
Logs are the fundamental source of information when it comes to debugging an application. The
kubectl logs command provides the functionality to check the logs of a given Pod. The below commands can be used to check the logs of a given Pod.
Apart from just looking at logs, we can also export logs to a file for further debugging of sharing the same with anyone.
How to create configmaps and secrets
kubectl create command provides us with the capability to create ConfigMaps and Secrets from the command line, we can also use the YAML file to create the same resources and by using
kubectl apply -f <filename> we can apply the commands.
Helpful commands for debugging
Debugging is a very important skill while facing issues and errors both in our day jobs while solving problems in the CKS exam.
Apart from the ability to output logs from a container the
kubectl exec commands allow us to log in to a running container and debug issues. While inside the container we can also use utilities like
nslookup to diagnose networking-related issues.
Rolling updates and Rollouts
kubectl rollout command provides the ability to check for the status of updates and if required roll back to a previous version.
Scale and Autoscale command
kubectl scale command provides the functionality to scale up or scale down Pods in a given deployment.
kubectl autoscale command we can define the minimum number of Pods that should be running for a given deployment and the maximum numbers of Pods the deployment can scale to along with the scaling criteria like CPU percentage.
In a Kubernetes cluster, all pods can communicate with all pods by default, which can be a security issue in some implementations.
To get around this issue, Kubernetes introduced Network Policies to allow or deny traffic to and from pods based on pod labels which are part of the pod spec.
The below example denies both the Ingress and Egress traffic for pods running in all namespaces.
The below example denies both the Ingress and Egress traffic for Pods running in all namespaces. But allows access to DNS resolution services running on port 53.
The below example denies Egress access to the metadata server running on IP address
169.256.169.256 in AWS EC2 Instances.
The below example allows Egress access to the metadata server running on IP address
169.256.169.256 in AWS EC2 Instances.
Static Analysis using Kubesec
Kubesec is a Static Analysis tool for analyzing the YAML files to find issues with the files.
Vulnerability scanning using Trivvy
Trivvy is a Vulnerability Scanning tool that scans container images for security issues.
How to remove unwanted services
systemctl exposes the capabilities to start, stop, enable, disable and list services running on a Linux Virtual Machine.
Kubernetes introduced the RuntimeClass feature in version
v1.12 for selecting the container runtime configuration. The container runtime configuration is used to run a Pod’s underlying containers.
Most Kubernetes clusters use the
dockershim as the Runtime class for the running containers but you can use different container Runtimes.
dockershim has been deprecated in Kubernetes version
v1.20, and will be removed in
Creating a Runtime Class:
Use a runtime class for any given Pod:
Role-based access control (RBAC) commands provide a method of regulating access to Kubernetes resources based on the roles of individual users or service accounts. (Source)
Create a role
Create a role binding
Create a cluster role
Create a Clusterrole Binding association with a service account
You use the
kubectl drain command to remove all running workloads (pods) from a given Node.
You use the
kubectl cordon command to cordon a node to mark it as schedulable.
Ands you use the
kubectl uncordon command to set the node as schedulable, meaning the Controller Manager can schedule new pods to the given node.
Draining a node of all pods
Drain a node and ignore daemonsets
Mark a node un schedulable, no new pods can be scheduled on this node
Mark a node schedulable
CKS Exam Tips
kubectl get command provides the user with an output flag the
--output which helps us in formatting the output in the form of json, yaml, wide or custom-columns.
JSON and JSONPath
Outputs the contents of all the pods in the form of a JSON Object:
The JSONPath outputs a specific key from the JSON Object
.items[*] is used where we have multiple objects for instance multiple containers with a Pod config:
The command returns the internal IP of a Node using JSONPath
The command checks for Equality on a specific key:
Custom Columns are helpful in order to output specific fields:
CKS Exam Topics
The CKS exam covers delve on topics related to security in the Kubernetes ecosystem. Kubernetes security is a vast topic to cover in one article, this article contains some of the topics covered in the exam.
How to secure and harden container images
While designing container images to run your code pay special attention to securing and hardening measures in order to prevent hacks and privilege escalation attacks. Keep the below points in mind while building the container images:
- Use specific package versions like
- Don’t run as root, use the
USER <username>to block root access.
- Make filesystem read-only in the
- Remove shell access using
RUN rm -rf /bin/*
How to minimise OS Footprint
ADD create container layers. Other instructions create temporary intermediate images and do not increase the size of the build. Instructions that create layers add to the size of the resulting image.
A typical Dockerfile looks like the one given below, it adds a single layer using the
Multi Stage Builds
Multi-Stage builds leverage multiple
FROM statements in the Dockerfile. The
FROM instruction marks a new stage in the build, combining multiple
FROM statements allow to leverage from the previous build in order to selectively copy binaries over to the new build stage omitting the unnecessary binaries. The resulting Docker image is considerably smaller in size with a drastically reduced attack surface.
How to limit node access
Access Control files contain sensitive information about users/groups in the Linux OS.
Disable user account helps in securing access to a Node by disabling login to a given user account.
root user account is of special significance as the root account has all the capabilities.
Add a user with a home directory and shell
Delete the user account
Delete a Group
Add user to group
Remove user from a Group
Set a password for the user
Elevate a user to Sudoer
How to disable SSH
The configuration is given in the
/etc/ssh/sshd_config can be leveraged to secure SSH access to Linux nodes. Setting the
no disables the root login on a node. To enforce using a key to login and disabling login using passwords to nodes the
PasswordAuthentication can be set to
Set no login for the root user:
SSH Copy user key / Passwordless SSH:
How to remove obsolete packages and services
List all services running on a Ubuntu machine.
Stop and disable and remove a service
How to restrict kernel modules
In Linux, Kernel modules are pieces of code that can be loaded and unloaded into the kernel upon demand. They extend the functionality of the kernel without the need to reboot the system. A module can be configured as built-in or loadable.
List all Kernel Modules
Manually Load Modules into Kernel
Blacklist a module: (Reference: CIS Benchmarks -> 3.4 Uncommon Network Protocols)
How to identify and disable open ports
Check for Open Ports:
Check Port usage
Reference Doc for list of open ports: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#control-plane-node-s
How to restrict network access
How to identity a service running on port
Uncomplicated Fire Wall(UFW) is a tool for managing firewall rules in Arch Linux, Debian, or Ubuntu. UFW provides the functionality to allow and block traffic on a given port and from a given source.
Installing UFW Firewall
Allow all outbound and inbound connections
Enable / Activate Firewall
Linux Syscalls are used to make requests from user space into the Linux kernel. For instance, while creating a file the userspace makes a request to the Linux Kernel to create the file.
Kernel Space has the following:
- Kernel Code
- Kernel Extensions
- Device Drivers
How to trace Syscalls using Strace
Tracing syscalls using strace
Get PID of service
List all syscalls made during an operation
Consolidated listing syscalls: (Count and summarise)
Follow a PID and consolidate
AquaSec Tracee was created by Aqua Security which uses eBPF to trace events in containers. Tracee makes use of eBPF (Extended Berkeley Packet Filter) at runtime directly in the kernel space without interfering with the kernel source or loading any kernel modules.
- Binary stored at
- Needs access to the following, in read-only mode if run using a container with
/tmp/tracee-> Default workspace
/lib/modules-> Kernel Headers
/usr/src-> Kernel Headers
Running Tracee in Docker container
How to restrict Syscalls with Seccomp
SECCOMP - Secure Computing Mode is a Linux Kernel level feature that can be used to sandbox applications to only use the syscalls they need.
Check support for seccomp:
Test to change system time:
Check seccomp status for any PID:
- Mode 0: Disabled
- Mode 1: Strict
- Mode 2: Filtered
The following configuration is used to whitelist syscalls. While list profile is secure but syscalls have to be selectively enabled as it blocks all syscalls by default.
The following configuration is used to blacklist syscalls. The black list profile has a greater attack surface than the white list.
The Docker seccomp profile blocks 60 of the 300+ syscalls on the x86 architecture.
Using seccomp profiles with docker
Allow all syscalls with the container
Docker container to get container runtime related information:
Seccomp in Kubernetes
Secure computing mode (SECCOMP) is a Linux kernel feature. You can use it to restrict the actions available within the container. Seccomp documentation
Run amicontained in Kubernetes:
As of version
v1.20 Kubernetes does not implement seccomp by default.
Seccomp ‘RuntimeDefault’ docker profile in Kubernetes:
Default seccomp location in kubelets
Create a seccomp profile in node
Local seccomp profile, this file should exist locally on a node to be able to work
The above profile will enable syscalls to be saved to a file
Mapping syscall numbers to syscall name
AppArmor is a Linux security module that is used to confine a program to a limited set of resources.
Install AppArmor utils
Check if AppArmor is running and enabled
The AppArmor profiles are stored at
Listing AppArmor profiles
Deny all file write profile
Deny write to
Deny remount root FS
Check profile status
Profile load modes
Enforce, monitor and enforce the rules
Complain, will not enforce the rules but logs them as events
Unconfined, will not enforce or log events
Check if a profile is valid
Disable a profile
Generate a profile and answer the series of questions that follow
Generate profile for a command
Disable profile from logs
How to use AppArmor in Kubernetes
To be used with Kubernetes the following prerequisites must be met:
- Kubernetes version should be greater than
- AppArmor Kernel module should be enabled
- AppArmor profile should be loaded in the kernel
- Container runtime should be supported
Sample usage in Kubernetes:
Note: The container should run in a node containing the AppArmor profile.
The Linux capabilities feature breaks up the privileges available to processes run as the
root user into smaller groups of privileges. This way a process running with
root privilege can be limited to get only the minimal permissions it needs to perform its operation.
Docker supports the Linux capabilities as part of the Docker run command: with
--cap-drop. By default, a container is started with several capabilities that are allowed by default and can be dropped. Other permissions can be added manually.
--cap-drop support the ALL value, to allow or drop all capabilities. By default Docker containers run with 14 capabilities.
- Kernel < 2.2
- Privileged Process
- Unprivileged Process
- Kernel >= 2.2
- Privileged Process
- Privileged Process
Check what capabilities a command needs
Get process capabilities
Add security capabilities
How to prepare for the Exam
CKS is considered a pretty tough exam. But based on my experience I think that, given good enough practice and if you understand the concepts the exam covers, it’ll be pretty manageable within two hours.
You definitely need to understand the underlying Kubernetes concepts. And since a prerequisite for CKS is to pass the CKA exam, you should have a strong understanding of Kubernetes and how it functions before attempting the CKS.
In addition, to pass the CKS, you need to understand the threats and security implications introduced by container orchestration.
The introduction of the CKS exam is an indication that the security of containers should not be taken lightly. Security mechanisms should always be in place to thwart attacks on Kubernetes clusters.
The Tesla cryptocurrency hack that was thanks to an unprotected Kubernetes dashboard brings to light the risks associated with Kubernetes or any other container orchestration engine. Hackerone has a Kubernetes bounty page listing the source code repos used in a standard Kubernetes cluster.
Practice, Practice, and Practice!
Practice is the key to cracking the exam, I personally found that the exam simulators by KodeKloud and Killer.sh were immensely helpful for me.
I didn’t have as much time to prepare for the CKS exam as I had for the CKA exam, but I was working on Kubernetes in my day job so I’d become really comfortable with it.
Practice is the key to success. Best of luck with the exam!
Subscribe to Faizan Bashir
Get the latest posts delivered right to your inbox