AppArmor and SELinux

Docker's AppArmor profile, a custom one, and how to tell a security module refused you.

mount: mounting none on /mnt/x failed: Permission denied, from a container that was started with --cap-add SYS_ADMIN for exactly that mount. The capability is there, and Docker's seccomp profile allows mount once CAP_SYS_ADMIN is added ("seccomp: filter system calls"). The kernel log has nothing to say about it. The refusal comes from a third layer, a Linux Security Module, and this one doesn't log. Docker works with two of these modules, AppArmor and SELinux. You'll see what Docker's AppArmor profile blocks and how to prove it was the cause. Then you'll write, load and debug a profile of your own. Last comes what an SELinux host needs before its labels apply to containers.

Mandatory access control in one paragraph

Ordinary file permissions are discretionary. The owner of a file decides, and root passes every check. A Linux Security Module (LSM) adds hooks in the kernel that run after those checks. They can refuse an operation even for root with every capability. That's mandatory access control: a policy the host admin loads, which processes can't change. AppArmor writes its policy as per-program profiles of paths and permissions. It ships enabled on Ubuntu, Debian and their derivatives. SELinux labels every process and file with a type and allows access by type. It ships enforcing on Fedora, RHEL and its rebuilds. A host runs one of the two as its main MAC module.

docker-default

On an AppArmor host, Docker generates a profile called docker-default and loads it into the kernel when the daemon starts. It applies that profile to every container that doesn't ask for something else:

docker info lists apparmor with profile=default. Inside the container, /proc/self/attr/apparmor/current reads docker-default (enforce). On kernels with several stacked LSMs that path is the reliable one. The older /proc/self/attr/current still works on this host. docker inspect shows the profile in AppArmorProfile while SecurityOpt stays empty. On the host, aa-status lists the profile and every confined process, here the container's busybox running sleep.

The profile is generated from a template in the moby/profiles repository. It allows ordinary file and network access and leaves capability decisions to Docker's capability set. What it adds is a set of denials for things containers should never do, even when someone hands them a capability. It denies mounting filesystems and writing to most of /proc (sysrq-trigger, kcore, sys/kernel and others) and to /sys. It also denies tracing processes outside the profile. --privileged replaces the profile with unconfined.

A refusal that leaves no log line

The mount fails, and you won't find an AppArmor denial for docker-default in the last minute of the kernel log. That's not a logging problem. docker-default refuses mounts with an explicit deny mount rule. AppArmor doesn't log explicit deny rules unless the rule is written audit deny. The third command gives you the diagnosis. The same container with --security-opt apparmor=unconfined mounts the tmpfs. If removing the profile changes the result and nothing else changed, the profile was the cause.

Run that A/B test only on a disposable host or a throwaway container, and treat it as a measurement, never as a fix. With the capability still added, the unconfined container could mount anything its other settings allow. That's the escape path "Capability and kernel escapes" describes. The same reasoning sorts out any Permission denied or Operation not permitted you get from a container:

A profile of your own

docker-default is broad so that it fits every image. In a profile for one workload, you say what the process may do and refuse the rest. Those refusals are logged. This one lets a container read anything, run busybox, and write only under /data:

abi <abi/4.0>,
include <tunables/global>
 
# lab-app: read the image, run busybox, write only under /data.
# Anything not allowed here is denied, and AppArmor logs the denial.
profile lab-app flags=(attach_disconnected,mediate_deleted) {
  include <abstractions/base>
 
  /** r,
  /bin/busybox mrix,
  /data/** w,
 
  signal (receive) peer=unconfined,
}

abi <abi/4.0> pins the policy language version and tunables/global defines variables such as @{PROC}. attach_disconnected makes paths resolve inside the container's own mount namespace, which every container profile needs. abstractions/base covers what almost every program touches, such as shared libraries, /dev/null and locale files. /** r allows reading everything. /bin/busybox mrix allows mapping and executing busybox, and ix keeps the new program in the same profile. /data/** w is the only write rule. signal (receive) peer=unconfined lets the container runtime deliver docker stop. Everything else is refused. That includes every capability, because the profile has no capability rule, and all networking, because it has no network rule.

Docker doesn't load custom profiles. You load them with apparmor_parser. Keep the file in /etc/apparmor.d and the AppArmor service loads it again at boot:

The container runs under lab-app (enforce). Writing /data/out works. Running apk, a separate binary, fails, and so does creating /etc/lab-x. Unlike docker-default's explicit denies, these refusals come from rules the profile doesn't have. AppArmor logs each one with the operation (exec, mknod), the profile and the path. That's what you need to decide whether a rule is missing. Without auditd the kernel rate-limits these messages, so a large burst of denials can lose lines. Hosts running auditd keep them in /var/log/audit/audit.log. The same command under docker-default runs apk and writes /etc/lab-x, which confirms the profile made the difference.

Writing a profile is an iterative job, and complain mode is the tool for it. A profile in complain mode logs what it would refuse, and allows it. apparmor_parser -C loads a profile in complain mode:

In complain mode the write succeeds, and the log shows apparmor="ALLOWED" for each step of it (mknod, open, file_perm). So the method goes like this. Load a new profile in complain mode and run the app through its real workload. Turn the logged accesses that are legitimate into rules (aa-logprof from apparmor-utils can propose them). Then reload it in enforce mode, which the last command of that step did. In Compose the setting is security_opt: [apparmor=lab-app]. The failure you'll meet first with custom profiles is a name the kernel doesn't know. That happens, for example, after a host was rebuilt and the file was never copied to /etc/apparmor.d:

runc can't switch the container to a profile that isn't loaded, so the container isn't created. The useful part of the message is unable to apply apparmor profile. Check aa-status on that host. Clean up, which unloads the profile with apparmor_parser -R and removes its file:

SELinux hosts

The lab VMs run Ubuntu, so SELinux is described here from the Docker and Fedora docs, not demonstrated. With SELinux, container processes run as the type container_t, and files meant for containers carry container_file_t. The policy allows container_t to touch little else. Each container also gets its own pair of MCS categories, such as s0:c12,c34. So two containers with the same UID still can't read each other's files. ps -eZ shows the process labels and ls -Z the file labels.

Docker Engine from Docker's own packages doesn't enable SELinux support by default. Until the daemon runs with it, containers aren't given container_t labels, and the :z and :Z volume options relabel nothing:

{
  "selinux-enabled": true
}

Distro packages such as Fedora's moby-engine enable it already, so check docker info. An SELinux-enabled daemon lists selinux under Security Options. You then need to label a bind mount before a container can use it. :z gives the content a shared container label and :Z a label private to one container. "Volumes, bind mounts and tmpfs in practice" in Docker in depth covers which to use, and the host paths never to relabel. A refusal shows up as an AVC record, which you can find with ausearch -m avc -ts recent on hosts running auditd. For a single container, --security-opt label=disable is the equivalent of apparmor=unconfined. It belongs in the same A/B test, never in production. setenforce 0 turns SELinux off for the whole host and every workload on it. It's not a fix for one container.

Hands-on

On secopslog-docker-sec, before you run the clean-up step, repeat the custom command with echo x > /tmp/lab-x in place of the /etc/lab-x write. Read the profile first and predict the result from its single write rule, /data/** w. Wait a few seconds, as the lab does, then rerun the custom-log command and check whether a new mknod line names /tmp/lab-x. Clean up afterwards so the profile doesn't stay loaded in the VM.

Takeaway

An empty kernel log doesn't clear AppArmor. docker-default refused your tmpfs mount with a count of 0 DENIED lines, and only the apparmor=unconfined A/B run named the layer. Where this goes wrong is the test flag staying in a run command or Compose file as the "fix". The other common slip is a custom profile that never reached /etc/apparmor.d on a rebuilt host, so the container fails with unable to apply apparmor profile.

Related