INTRODUCTION
In modern architectures, containers are everywhere—running microservices, distributed HPC jobs, or ephemeral tasks. Logging can quickly get out of hand when you have hundreds (or thousands) of containers. systemd journald unifies these logs for easier correlation and search.
MONITORING DOCKER CONTAINERS
Below is a snippet that uses Docker’s Python SDK to gather container metrics, then logs them into journald:
import docker
import systemd.journal
import logging
import queue
import threading
import time
from systemd.journal import JournalHandler
class StructuredLogger:
def __init__(self, name):
self.logger = logging.getLogger(name)
self.logger.setLevel(logging.DEBUG)
self.logger.addHandler(JournalHandler(SYSLOG_IDENTIFIER=name))
def log(self, level, message, **kwargs):
self.logger.log(level, message, extra=kwargs)
class ContainerMonitor:
def __init__(self):
self.client = docker.from_env()
self.logger = StructuredLogger("container.monitor")
self.metrics_queue = queue.Queue()
self.running = True
def collect_metrics(self, container):
stats = container.stats(stream=False)
# Basic CPU usage calculation
cpu_delta = stats['cpu_stats']['cpu_usage']['total_usage'] - \
stats['precpu_stats']['cpu_usage']['total_usage']
system_delta = stats['cpu_stats']['system_cpu_usage'] - \
stats['precpu_stats']['system_cpu_usage']
cpu_percent = (cpu_delta / system_delta) * 100.0 if system_delta else 0.0
return {
"cpu_percent": cpu_percent,
"memory_usage": stats['memory_stats']['usage'],
"memory_limit": stats['memory_stats']['limit'],
"container_name": container.name,
"container_id": container.id[:12]
}
def monitor_container(self, container):
while self.running:
try:
metrics = self.collect_metrics(container)
self.metrics_queue.put(metrics)
except Exception as e:
self.logger.log(logging.ERROR, f"Error: {e}",
CONTAINER_NAME=container.name,
ALERT_TYPE="monitoring_error")
time.sleep(5)
def log_metrics(self):
while self.running:
try:
metrics = self.metrics_queue.get(timeout=1)
self.logger.log(logging.INFO, "Container stats",
ALERT_TYPE="container_stats", **metrics)
# Additional checks for high CPU or memory
if metrics["cpu_percent"] > 90:
self.logger.log(logging.WARNING, "High CPU usage", ALERT_TYPE="high_cpu", **metrics)
except queue.Empty:
continue
def run(self):
threads = []
for container in self.client.containers.list():
t = threading.Thread(target=self.monitor_container, args=(container,))
t.start()
threads.append(t)
logging_thread = threading.Thread(target=self.log_metrics)
logging_thread.start()
threads.append(logging_thread)
return threads
if __name__ == "__main__":
monitor = ContainerMonitor()
threads = monitor.run()
try:
for t in threads:
t.join()
except KeyboardInterrupt:
monitor.running = False
PERFORMANCE CONSIDERATIONS
• Batching Logs: If you log at high frequency, consider using an in-memory queue to batch entries and reduce I/O overhead.
• Log Levels: Keep production systems at WARNING or INFO unless you’re actively debugging.
• CPU Overhead: Over-logging can starve CPU cycles if you’re scraping stats too frequently.
SECURITY AND COMPLIANCE IN JOURNALD
• Permissions: Only users in the systemd-journal group can read logs.
• Data Masking/Sanitization: If you handle PII (Personal Identifiable Information) or sensitive tokens, mask or encrypt these fields.
• Rotation and Retention: Configure journald’s size and retention policies so you don’t run out of space.
EXAMPLE: LIVE DEBUGGING OF UNHEALTHY CONTAINERS
Quickly isolate container logs in real time:
journalctl ALERT_TYPE=container_stats CONTAINER_NAME=myapp -f
Combine this with time-based queries to see a historical spike or crash.