Chapter 7: Integration with systemd

Introduction

The introduction explains why Podman works well with systemd for managing containerized services and previews the integrations covered in the chapter.

• Systemd process tracking: Explain how Podman’s fork/exec model lets systemd track and manage the processes of a containerized service, unlike a client-server container engine.
  Why it matters: This explains why systemd can manage a Podman container as a service.
• Container services as unit files: Describe why Podman encourages systemd unit files containing Podman commands for starting containerized services at boot, and identify service unit files as a way to define, manage, and share those services.
  Why it matters: Unit files are the chapter’s starting point for putting systemd in charge of a containerized service.
• Chapter integration roadmap: Describe the integrations the chapter previews: running systemd inside a container, using journald for logs and events, and using systemd with Podman for service lifecycle features such as socket activation, notifications, restarting, and updating.
  Why it matters: The roadmap shows the different ways systemd participates in container service management.

Key terms: fork/exec model, scope, cgroup hierarchy, PID cgroup, client-server methodology, systemd unit files, service files, Podman commands, systemd, journald

7.1 Running systemd within a container

This section contrasts single-service containers with containers that run systemd, explains why an init process matters, and identifies when Podman enters systemd mode.

• Single-service and systemd containers: Distinguish a microservice container, where one service runs as PID 1 and writes logs to stdout and stderr, from a container where systemd runs as PID 1 and starts one or more services. Explain why existing systemd unit files and local communication between services can make the latter useful when moving a multiservice application into a container.
  Why it matters: Podman supports both approaches, so choosing between them depends on how the application is organized.
• PID 1 zombie reaping: Explain how an exited process becomes a zombie when its parent does not collect its exit status, and why the container's initial process must reap such processes. State that systemd can do this and that Podman's --init option offers a tiny init program for reaping.
  Why it matters: An init process prevents exited processes from lingering in the container.
• Automatic systemd mode detection: Describe how Podman examines the container's command and automatically uses systemd mode when it launches init or a command whose path ends in systemd. Give the ubi8-init image as an example intended for this mode.
  Why it matters: The container's launch command determines whether Podman prepares it to run systemd.

Key terms: microservice, PID 1, systemd unit files, multiservice application, stdout, stderr, SIGCHLD, exit status, zombie process, --init

7.1.1 Containerized systemd requirements

This section explains how Podman prepares a nonprivileged container so systemd can start without mounting filesystems itself.

• Systemd container environment requirements: Explain why systemd can fail in a nonprivileged container and how Podman prevents that failure. Systemd expects tmpfs mounts on /run and /tmp; Podman checks whether the image starts systemd and prepares the container environment so systemd does not need to mount them itself.
  Why it matters: This explains how systemd can run inside a locked-down container without granting it mounting privileges.

Key terms: systemd, tmpfs, /run, /tmp, CAP_SYS_ADMIN, entry point

7.1.2 Podman container in systemd mode

This subsection shows how to inspect a Podman container’s systemd-mode environment and stop a container running systemd.

• Systemd mode environment: Explain how --systemd=always lets you inspect a container’s systemd-mode environment even before systemd is running. Describe the tmpfs mounts at /run and /tmp, the container environment variable, and how systemd can be started as PID 1.
  Why it matters: These checks show what Podman prepares for systemd inside a container.
• Systemd container shutdown signal: Explain why Ctrl-C does not stop the demonstrated ubi8-init container and how stopping it through Podman sends systemd its configured StopSignal, 37 (SIGRTMIN+3), for shutdown.
  Why it matters: A systemd-based container needs the signal systemd expects in order to shut down.

Key terms: --systemd=always, systemd mode, PID1, tmpfs, /run, /tmp, ubi8-init, StopSignal, 37, SIGRTMIN+3

7.1.3 Running an Apache service within a systemd container

This section builds and runs an Apache image with systemd inside the container, then shows how that setup changes where to find service logs.

• Apache systemd image construction: Describe how the Containerfile uses ubi8-init as its base, installs httpd with dnf, cleans up installation files, and enables httpd.service so systemd starts Apache when the container runs.
  Why it matters: Enabling the service in the image lets systemd start Apache without a custom process-launch script.
• Apache container launch and check: Describe building the image as my-systemd, running it with host port 8080 mapped to container port 80 and ./html mounted at /var/www/html, then checking the page in a browser.
  Why it matters: The port mapping and mounted content make the containerized Apache service visible and testable from the host.
• Systemd container log location: Explain why podman logs shows no output in this example and where to look instead for Apache problems.
  Why it matters: An empty podman logs result does not mean the service has no diagnostic information.

Key terms: Containerfile, ubi8-init, httpd, dnf, systemctl enable httpd.service, my-systemd, podman build, --p 8080:80, -v ./html/:/var/www/html:Z, index.html

7.2 Journald for logging and events

Introduces journald as Linux’s logging service and explains why Podman uses it by default.

• Journald and Podman logging: Explain that journald is a systemd service that collects and stores logs, and that Podman uses it by default for logging data. Describe its persistent records and built-in log rotation.
  Why it matters: This gives you the context for understanding where Podman logs go and how they are retained.

Key terms: systemd journal, journald, log rotation

7.2.1 Log driver

This section explains how Podman log drivers affect log retention and rotation, and shows how to select and check the journald driver.

• Log driver defaults and availability: Explain that Podman defaults to journald when systemd is the init system, falls back to a file driver when running inside a container without systemd, and that some rootless users cannot use journald.
  Why it matters: The available log driver depends on where and how Podman runs.
• Log retention and rotation tradeoffs: Distinguish journald from k8s-file by explaining that container stdout and stderr can produce substantial logs, journald has systemd-provided rotation and retains records after container removal, while k8s-file risks filling storage and loses its log file when the container is removed.
  Why it matters: Retention and rotation determine whether logs remain useful without exhausting disk space.
• Journald selection and retention check: Describe checking the system's default log driver, overriding it through the log_driver option in a user containers.conf.d configuration file, and checking the journal after a --rm container exits to confirm its launch record remains.
  Why it matters: The check makes the effect of the log-driver choice observable after container removal.

Key terms: journald, file driver, rootless, systemd, stdout, stderr, journald driver, k8s-file driver, log rotation, log_driver

7.2.2 Events

This section explains how Podman records container lifecycle events and why journald is often preferable to a file for storing them.

• Event recording and configuration: Explain that Podman events record steps in a container’s lifecycle, such as a start event, and describe how to inspect and configure the events logger.
  Why it matters: Events let you see what happened to a container, while the logger setting determines where that record goes.
• File and journald tradeoffs: Explain the risks of the file events logger, the benefits of journald, and the section’s recommendation for aligning event and log drivers.
  Why it matters: Choosing the events backend affects disk usage and whether event history survives a reboot.

Key terms: Podman events, Podman info, events_logger, containers.conf, file event logger, $XDG_RUNTIME_DIR, rootless users, tmpfs, journald, rotation

7.3 Starting containers at boot

This section introduces why boot-time containerized services need systemd when Podman runs without a daemon.

• Daemonless boot-time startup: Explain why Podman cannot rely on a daemon to start containers automatically at boot and why another service manager is needed.
  Why it matters: This explains the problem that systemd integration addresses.
• Systemd service management: Describe systemd's proposed role in installing, running, and managing containerized applications, including launching applications shipped as container images through service unit files.
  Why it matters: Service unit files are the link between containerized applications and systemd management.

Key terms: daemon, boot time, systemd, container images, systemd service unit files, containerized services

7.3.1 Restarting containers

This section explains how Podman restart policies and systemd services restart containers, including after a reboot, and why systemd unit files offer another approach.

• Container restart policy: Explain that podman run --restart sets whether a container restarts when it was not stopped by a user, such as after a crash or system reboot.
  Why it matters: A restart policy determines whether Podman brings a container back after an interruption.
• Automatic restarts at boot: Describe how systemd runs Podman's restart services at boot to start containers marked --restart=always, with separate services for rootful and rootless containers.
  Why it matters: This connects Podman's restart setting to what happens when the host boots.
• Restart policy limitations: Explain that --restart=always requires a container already created on the system and will restart containers even if they fail; identify a systemd service unit file as the alternative introduced next.
  Why it matters: These limitations explain why the chapter next turns to systemd unit files for containerized services.

Key terms: podman run, --restart, restart policy, systemd, --restart=always, podman-restart.service, rootful, rootless, systemd service unit file

7.3.2 Podman containers as systemd services

This section shows how to manage an existing Podman container as a systemd service, why detached runs can cause trouble, and how generated unit files help.

• Systemd and conmon responsibilities: Explain that systemd monitors the processes in its service unit, including the container processes, while conmon monitors the container, records its exit code, and cleans up its environment.
  Why it matters: The distinction explains how systemd can manage a container without needing to understand containers itself.
• Detached run service pitfall: Explain why running `podman run --detach` inside a systemd unit can cause systemd to consider the service finished and take it down.
  Why it matters: This is the configuration mistake the generated unit file is meant to avoid.
• Generated container service workflow: Describe the section's workflow: create a container from the image, generate a systemd service unit for that container, reload systemd so it notices the unit, then start, check, and stop the service. Explain that the generated unit's `ExecStart` and `ExecStop` commands start and stop the existing container.
  Why it matters: Generating the unit supplies workable defaults for managing the container like a host service.
• Existing container unit portability: Distinguish a unit generated for a specific existing container from the more portable unit file produced with `podman generate systemd --new`.
  Why it matters: The distinction matters when a service file must work on another person's machine.

Key terms: systemd unit file, conmon, container processes, exit code, podman run --detach, service, podman generate systemd, myapp.service, ExecStart, ExecStop

7.3.3 Distributing systemd unit files to manage Podman containers

This section shows how `podman generate systemd --new` produces unit files that create, stop, and remove containers, making a containerized service distributable to other machines.

• New unit container lifecycle: Trace how a unit generated with `--new` manages a container differently from a unit for an existing container: `ExecStart` runs it, `ExecStop` stops it, and `ExecStopPost` removes it.
  Why it matters: The unit file manages the container’s full lifecycle rather than depending on a container created beforehand.
• Distributable container service units: Explain why a unit generated with `--new` can be shared with another user or machine: starting the service can pull the image and create and run the container without the recipient creating it first.
  Why it matters: A shareable unit file lets the same containerized service be started on multiple machines without manually preparing each container.

Key terms: podman generate systemd, --new, ExecStart, ExecStop, ExecStopPost, --cidfile, myapp-new.service, systemctl, pull the image, auto-update

7.3.4 Automatically updating Podman containers

This section shows how Podman and systemd can detect updated container images, recreate services, and run updates unattended.

• Automatic update purpose: Explain why auto-update is useful when an image changes, especially for services deployed across many or intermittently connected nodes.
  Why it matters: It connects image updates and vulnerability fixes to the operational cost of maintaining many machines.
• Auto-update service setup: Describe the requirements for an automatically updated container: the io.containers.autoupdate=registry label and a systemd unit generated with podman generate systemd --new. Explain that the revised unit must be loaded by systemd and the service started.
  Why it matters: Podman needs both the marked container and a systemd service that can recreate it.
• Registry update lifecycle: Trace what podman auto-update does when a newer image is available in the registry and how restarting the systemd unit replaces the running container.
  Why it matters: The update is a service restart and container recreation, not an in-place modification.
• Daily systemd update scheduling: Describe how Podman's systemd timer and service units run auto-update once per day, with separate rootful and rootless units.
  Why it matters: The timers make registry checks possible without someone manually invoking the update command.
• Update checks and failure safeguards: Distinguish the --dry-run option, which checks for needed updates without applying them, from --roll-back, which requests a return to the previous image if an update fails. State why a broken update across many nodes makes service readiness important.
  Why it matters: Unattended updates need ways to inspect pending changes and deal with failures.

Key terms: auto-update, container registry, edge computing, io.containers.autoupdate=registry, podman generate systemd --new, daemon-reload, podman auto-update, ExecStopPost, podman stop, podman rm

7.4 Running containers in notify unit files

This section explains how systemd notify units wait for a containerized service to become ready, how Podman passes the notification socket into the container, and how failed readiness checks can trigger rollback after an update.

• Service readiness notifications: Explain why launching a service process is not the same as having a ready service, and how a notify unit lets the service tell systemd when it is fully up and running.
  Why it matters: Readiness notifications keep dependent services from starting before the services they rely on can accept connections.
• Container notification socket access: Describe how a containerized service can send its readiness notification to systemd: Podman needs to mount the notification socket into the container and pass it the NOTIFY_SOCKET environment variable.
  Why it matters: The service inside the container needs access to the socket that systemd listens on for readiness notifications.
• Readiness failure and rollback: Explain that systemd marks a service failed if it does not notify readiness within the specified time, and that a failed readiness check after a Podman auto-update can cause an automatic rollback to the previous container.
  Why it matters: A readiness check can detect an updated container that starts but never becomes usable.

Key terms: notify, sd-notify, systemd, NOTIFY_SOCKET, systemd socket, volume mount, Podman auto-update, rollback

7.5 Rolling back failed containers after update

The section explains how Podman rolls back a failed auto-update and introduces socket activation as another way systemd can manage containers.

• Failed auto-update rollback: Trace what happens when an updated containerized service fails to send an sd-notify message within the time limit, and explain why rollback is useful.
  Why it matters: Rollback keeps the previous service available while you diagnose the failure and prepare another image.
• Systemd socket activation: Describe the socket-activation option introduced here and relate it to systemd managing containers on a single system.
  Why it matters: Socket activation offers a way to start a container only when a request arrives.

Key terms: sd-notify, notify socket, podman auto-update, rollback, systemd, container orchestrator, socket activation, unit file

7.6 Socket-activated Podman containers

This section explains how systemd can start a Podman container on demand when a connection arrives at a socket, rather than keeping the service running while idle.

• Socket activation lifecycle: Explain how systemd listens on an IP or UNIX domain socket without starting a service, activates the service when a connection arrives, and can start it again after it exits. Describe how this speeds boot and saves resources.
  Why it matters: The listening socket makes a service available without requiring its process to run continuously.
• Container connection handoff: Trace how a connection to a systemd-managed socket activates Podman, which launches a container and passes the socket connection to the service inside it. Describe what the section's browser test shows before connection, after connection, and when the service is stopped.
  Why it matters: This shows how socket activation reaches a containerized service, not just an ordinary host service.

Key terms: systemd, socket-activated services, IP sockets, UNIX domain sockets, Podman, myapp.socket, podman.socket, podman.service, containerized services

7.7 Summary

The summary recaps how Podman integrates with systemd and journald to run services, manage containers, and handle logs and events.