Structured Logging with Python and systemd Journald

INTRODUCTION
Plain-text logs can be tough to parse programmatically at scale, especially if you’re correlating multiple services or HPC jobs. Journald allows structured logging—attaching metadata fields to each log entry—giving you powerful search and filtering capabilities. Python integrates cleanly with journald, letting you attach custom keys like ALERT_LEVEL or CPU_PERCENT.
BASIC INTEGRATION WITH PYTHON
Below is a simplified code snippet illustrating both traditional and structured logging. It uses Python’s logging and systemd.journal:

import time
import psutil
import systemd.journal
import logging
from systemd.journal import JournalHandler

logger = logging.getLogger("demo.basic")
logger.setLevel(logging.DEBUG)
logger.addHandler(JournalHandler())

def log_system_stats():
    while True:
        try:
            cpu_percent = psutil.cpu_percent()
            memory = psutil.virtual_memory()

            # Traditional text-based log
            logger.info(f"System stats - CPU: {cpu_percent}%, Memory: {memory.percent}%")

            # Structured logging
            logger.info("System stats collected", extra={
                "cpu_percent": cpu_percent,
                "memory_percent": memory.percent,
                "MONITOR_TYPE": "system_stats",
            })

            if cpu_percent > 90:
                logger.error("High CPU usage detected", extra={
                    "cpu_percent": cpu_percent,
                    "ALERT_LEVEL": "critical",
                })

        except Exception as e:
            logger.exception("Error collecting system stats", extra={
                "error_type": type(e).__name__,
                "ALERT_LEVEL": "error"
            })

        time.sleep(5)

if __name__ == "__main__":
    logger.info("System monitoring started", extra={"MONITOR_TYPE": "startup"})
    log_system_stats()

BENEFITS OF STRUCTURED LOGGING
• Query by Key-Value: “journalctl MONITOR_TYPE=system_stats cpu_percent=’>90’”
• Consistent Metadata: Fields like ALERT_LEVEL=critical or error_type=ValueError make analysis straightforward.
• Flexible Integration: You can attach any custom field relevant to your application or scientific workflow.
BEST PRACTICES
• Consistent Field Names: Tag them with uppercase (e.g., CPU_PERCENT, MONITOR_TYPE) for clarity.
• Code Efficiency at Scale: Use lazy evaluation or placeholders to avoid building strings for logs that won’t be emitted if log level is high.
• Error Reporting: Use logger.exception(…) along with structured fields so you can quickly filter error vs info logs in journald.
ADVANCED EXAMPLE: CONTAINER MONITORING
If you have Docker containers running HPC or microservices, you can watch each container’s CPU, memory, and network usage, then log structured events. This helps with root-cause analysis of container crashes or resource spikes. (See Blog Post 4 for more details.)
CONCLUSION
Structured logging in Python harnesses journald’s power, making large-scale or HPC-level debugging much simpler. The next post dives into container monitoring and security.

← Previous Post Next Post →