Linux journalctl Command: Find Timeout Errors Tutorial
Description
How to Use Linux journalctl Command for Log Analysis?
Quick Answer: Use journalctl -b to view current boot logs, journalctl -f to follow live logs, and journalctl -u servicename to check specific services. Filter with grep for errors: journalctl -b | grep "error". Essential for systemd log management and troubleshooting.
Essential journalctl Command Examples
# View current boot logs
journalctl -b
# Follow live logs (like tail -f)
journalctl -f
# Check specific service logs
journalctl -u nginx.service
# Show only error messages
journalctl -p err
# Find timeout errors
journalctl -b | grep "Timed out waiting"
# View logs from last hour
journalctl --since "1 hour ago"
# Show last 50 entries
journalctl -n 50
# Check kernel messages
journalctl -k
# View logs in reverse order (newest first)
journalctl -r
# Filter by time range
journalctl --since "2025-01-15 09:00:00" --until "2025-01-15 17:00:00"
# Follow specific service logs live
journalctl -u ssh.service -f
# Find failed services
journalctl -b | grep "Failed to start"
Frequently Asked Questions
Q: What does journalctl command do in Linux? A: journalctl is the systemd journal query tool that displays and filters system logs. It replaces traditional log file viewing and provides centralized access to all system, kernel, and service logs.
Q: How do I view current boot logs with journalctl? A: Use journalctl -b to show logs from the current boot session only, or journalctl -b -1 for the previous boot. This helps isolate current session issues from historical problems.
Q: How do I follow live logs like tail -f? A: Use journalctl -f to follow logs in real-time, similar to tail -f. Add -u servicename to follow a specific service: journalctl -u nginx -f.
Q: How do I check logs for a specific service? A: Use journalctl -u servicename to view logs for a specific systemd service. Example: journalctl -u ssh.service shows SSH service logs.
Q: How do I filter journalctl by time? A: Use --since and --until options with time expressions: journalctl --since "1 hour ago" or journalctl --since "2025-01-15 09:00:00".
Essential Steps to Master journalctl Log Analysis
- Start with boot logs: Use
journalctl -bto view current session logs and identify system startup issues - Check service status: Use
journalctl -u servicenameto investigate specific service problems and failures - Filter by priority: Use
journalctl -p errto focus on error-level messages and critical issues - Use time filters: Apply
--sinceand--untilto narrow down logs to specific time periods - Follow live logs: Use
journalctl -ffor real-time monitoring during troubleshooting sessions - Combine with grep: Pipe output to grep for specific patterns:
journalctl -b | grep "error"
Most Important journalctl Commands
| Command | Purpose | Use Case |
|---|---|---|
journalctl -b | Current boot logs | System startup issues |
journalctl -f | Follow live logs | Real-time monitoring |
journalctl -u service | Service-specific logs | Service troubleshooting |
journalctl -p err | Error messages only | Critical issue detection |
journalctl --since "1h ago" | Time-filtered logs | Recent problem analysis |
What Are the Most Important journalctl Command Options?
Essential Viewing Options
# Current boot session logs
journalctl -b
# Previous boot session logs
journalctl -b -1
# Follow logs in real-time
journalctl -f
# Show last N entries
journalctl -n 50
# Reverse order (newest first)
journalctl -r
Service and Unit Filtering
# Specific service logs
journalctl -u nginx.service
# Multiple services
journalctl -u nginx.service -u mysql.service
# All systemd units
journalctl -u "*.service"
# Kernel messages only
journalctl -k
How Do You Filter journalctl Logs by Time?
Time-Based Filtering Options
# Logs from last hour
journalctl --since "1 hour ago"
# Logs from last 30 minutes
journalctl --since "30 minutes ago"
# Logs from today
journalctl --since today
# Logs from yesterday
journalctl --since yesterday
# Logs from specific date
journalctl --since "2025-01-15"
Advanced Time Filtering
# Specific date and time
journalctl --since "2025-01-15 09:00:00"
# Time range filtering
journalctl --since "2 hours ago" --until "1 hour ago"
# Date range filtering
journalctl --since "2025-01-15" --until "2025-01-16"
# Combine with other filters
journalctl -u nginx.service --since "1 hour ago"
Time Format Examples
# Relative times
journalctl --since "2 days ago"
journalctl --since "1 week ago"
journalctl --since "10 minutes ago"
# Absolute times
journalctl --since "2025-01-15 14:30:00"
journalctl --since "Jan 15 14:30:00"
journalctl --since "15:30"
How Do You Filter journalctl by Priority and Message Type?
Priority Level Filtering
# Emergency messages only
journalctl -p emerg
# Alert level and above
journalctl -p alert
# Critical errors and above
journalctl -p crit
# Error messages and above
journalctl -p err
# Warning messages and above
journalctl -p warning
# Informational and above
journalctl -p info
# Debug messages (all)
journalctl -p debug
Message Type Filtering
# Kernel messages only
journalctl -k
# User session messages
journalctl --user
# System messages only
journalctl --system
# Messages from specific process
journalctl _PID=1234
# Messages from specific executable
journalctl _COMM=sshd
Advanced Pattern Filtering
# Find timeout errors
journalctl -b | grep "Timed out waiting"
# Find failed services
journalctl -b | grep "Failed to start"
# Find memory errors
journalctl -k | grep -i "memory"
# Find disk errors
journalctl -k | grep -i "disk"
# Find network errors
journalctl | grep -E "(network|eth0|connection)"
What Are Advanced journalctl Command Techniques?
Output Format Options
# JSON output for parsing
journalctl -o json
# JSON output (pretty printed)
journalctl -o json-pretty
# Short format (syslog style)
journalctl -o short
# Verbose format (all fields)
journalctl -o verbose
# Export format for backup
journalctl -o export
# CAT format (message only)
journalctl -o cat
Field-Based Filtering
# Filter by specific systemd unit
journalctl _SYSTEMD_UNIT=nginx.service
# Filter by process ID
journalctl _PID=1234
# Filter by user ID
journalctl _UID=1000
# Filter by executable name
journalctl _COMM=sshd
# Filter by hostname
journalctl _HOSTNAME=server01
Disk Usage and Maintenance
# Check journal disk usage
journalctl --disk-usage
# Verify journal integrity
journalctl --verify
# Vacuum old logs (keep 1 week)
sudo journalctl --vacuum-time=1week
# Vacuum by size (keep 1GB)
sudo journalctl --vacuum-size=1G
# Vacuum by number of files
sudo journalctl --vacuum-files=10
When Should You Use Different journalctl Commands?
System Troubleshooting Scenarios
# Boot problems - check current boot
journalctl -b -p err
# Service failures - check specific service
journalctl -u failed-service.service -n 100
# Performance issues - check recent system logs
journalctl --since "1 hour ago" -p warning
# Hardware problems - check kernel messages
journalctl -k --since "1 hour ago"
Development and Debugging
# Application debugging - follow app logs
journalctl -u myapp.service -f
# Web server issues - check access patterns
journalctl -u nginx.service --since "30 minutes ago"
# Database problems - monitor database service
journalctl -u mysql.service -p err
# Container issues - check container runtime
journalctl -u docker.service --since today
Security Analysis
# Authentication failures
journalctl | grep "authentication failure"
# SSH login attempts
journalctl -u ssh.service | grep "Failed password"
# Sudo usage tracking
journalctl | grep "sudo.*COMMAND"
# System security events
journalctl -p warning | grep -E "(security|auth|login)"
What Are All journalctl Command Options?
| Option | Description | Example |
|---|---|---|
-b | Boot session logs | journalctl -b |
-f | Follow live logs | journalctl -f |
-u | Specific unit/service | journalctl -u nginx |
-p | Priority level | journalctl -p err |
-n | Number of entries | journalctl -n 50 |
-r | Reverse order | journalctl -r |
-k | Kernel messages | journalctl -k |
-x | Add explanatory text | journalctl -x |
--since | Time filter start | journalctl --since "1h ago" |
--until | Time filter end | journalctl --until "now" |
What Are Essential journalctl Log Analysis Practices?
Daily Monitoring Routine
# Check for critical errors
journalctl -p crit --since today
# Review service failures
journalctl -b | grep "Failed to start"
# Monitor system performance
journalctl -k --since "1 hour ago" | grep -E "(error|warning)"
# Check authentication logs
journalctl --since today | grep -E "(login|auth)"
Automated Log Analysis
# Create daily error report
journalctl -p err --since today > /tmp/daily_errors.log
# Monitor specific service health
journalctl -u critical-service.service --since "5 minutes ago" -p warning
# Check for disk issues
journalctl -k | grep -E "(disk|filesystem|mount)" | tail -20
# Network connectivity monitoring
journalctl --since "10 minutes ago" | grep -E "(network|connection|timeout)"
Performance Optimization
# Limit journal size
sudo journalctl --vacuum-size=100M
# Clean old entries
sudo journalctl --vacuum-time=2weeks
# Persistent storage configuration
sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald
How Can You Create Useful journalctl Aliases and Scripts?
Essential Aliases
# Add to ~/.bashrc
alias jb='journalctl -b' # Current boot
alias jf='journalctl -f' # Follow logs
alias je='journalctl -p err' # Errors only
alias jt='journalctl --since "1 hour ago"' # Last hour
alias jk='journalctl -k' # Kernel only
Advanced Functions
# Service log checker
jservice() {
if [ -z "$1" ]; then
echo "Usage: jservice <service-name>"
return 1
fi
journalctl -u "$1.service" -n 50 --no-pager
}
# Error finder with context
jerrors() {
local since=${1:-"1 hour ago"}
echo "=== Errors since $since ==="
journalctl --since "$since" -p err --no-pager
}
# Boot analysis
jboot() {
local boot=${1:-0}
echo "=== Boot Analysis (boot $boot) ==="
journalctl -b "$boot" -p warning --no-pager | head -20
}
# Service monitor
jmonitor() {
if [ -z "$1" ]; then
echo "Usage: jmonitor <service-name>"
return 1
fi
echo "Monitoring $1.service (Ctrl+C to stop)..."
journalctl -u "$1.service" -f --no-pager
}
Diagnostic Scripts
#!/bin/bash
# system_health_check.sh - Comprehensive log analysis
echo "=== System Health Check $(date) ==="
echo -e "\n1. Critical Errors:"
journalctl -p crit --since today --no-pager | head -10
echo -e "\n2. Failed Services:"
journalctl -b | grep "Failed to start" --no-pager | head -10
echo -e "\n3. Recent Timeouts:"
journalctl -b | grep "Timed out waiting" --no-pager | head -10
echo -e "\n4. Kernel Warnings:"
journalctl -k --since "1 hour ago" -p warning --no-pager | head -10
echo -e "\n5. Authentication Failures:"
journalctl --since today | grep "authentication failure" --no-pager | head -10
echo -e "\n6. Disk Usage:"
journalctl --disk-usage
What Commands Are Related to journalctl?
systemctl– Control systemd services and view statusdmesg– Display kernel ring buffer messagestail– View end of files (traditional log viewing)grep– Search text patterns in logsawk– Advanced text processing and filteringless– Paginated viewing of log fileslogrotate– Manage log file rotationrsyslog– Traditional system logging daemonsystemd-analyze– Analyze system boot performance
Common journalctl Command Problems and Solutions
“No journal files were found” Error
Problem: journalctl shows no logs or journal files missing
Diagnosis:
# Check if systemd-journald is running
systemctl status systemd-journald
# Check journal directory
ls -la /var/log/journal/
# Check for runtime journal
ls -la /run/log/journal/
# Check journal configuration
cat /etc/systemd/journald.conf
Solutions:
# Create persistent journal directory
sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald
# Enable persistent storage
sudo sed -i 's/#Storage=auto/Storage=persistent/' /etc/systemd/journald.conf
sudo systemctl restart systemd-journald
# Check disk space
df -h /var/log
# Fix permissions
sudo chown root:systemd-journal /var/log/journal
sudo chmod 2755 /var/log/journal
“Failed to determine timestamp” Error
Problem: Corrupted journal files or timestamp issues
Diagnosis:
# Verify journal integrity
journalctl --verify
# Check for corrupted files
sudo journalctl --verify --quiet
# Check system time
timedatectl status
# Check for clock synchronization
systemctl status systemd-timesyncd
Solutions:
# Rebuild journal database
sudo systemctl stop systemd-journald
sudo rm /var/log/journal/*/*
sudo systemctl start systemd-journald
# Fix system time
sudo timedatectl set-ntp true
sudo timedatectl set-timezone America/New_York
# Vacuum corrupted entries
sudo journalctl --vacuum-time=1d
# Force journal rotation
sudo systemctl kill --kill-who=main --signal=SIGUSR2 systemd-journald
Permission Denied Accessing Logs
Problem: Cannot view system logs as regular user
Diagnosis:
# Check current user groups
groups
# Check journal file permissions
ls -la /var/log/journal/
# Check systemd-journal group membership
getent group systemd-journal
# Test with sudo
sudo journalctl -b
Solutions:
# Add user to systemd-journal group
sudo usermod -a -G systemd-journal $USER
# Re-login or use newgrp
newgrp systemd-journal
# Use sudo for system logs
sudo journalctl -b
# Check adm group membership (alternative)
sudo usermod -a -G adm $USER
journalctl Command Hanging
Problem: journalctl command appears to hang or freeze
Diagnosis:
# Check if pager is active
ps aux | grep less
# Check journal size
journalctl --disk-usage
# Check system load
top | head -5
# Check for large journal files
find /var/log/journal -type f -size +100M
Solutions:
# Disable pager
journalctl --no-pager
# Use head to limit output
journalctl -n 100
# Vacuum large journals
sudo journalctl --vacuum-size=100M
# Use time limits
journalctl --since "1 hour ago"
# Kill hanging process
pkill -f journalctl
Logs Not Appearing in Real Time
Problem: Live log following (-f) not showing new entries
Diagnosis:
# Check if service is active
systemctl is-active systemd-journald
# Check if logs are being written
ls -la /var/log/journal/*/system.journal
# Test with different service
journalctl -u systemd-journald -f
# Check inotify limits
cat /proc/sys/fs/inotify/max_user_watches
Solutions:
# Restart journald service
sudo systemctl restart systemd-journald
# Increase inotify limits
echo 'fs.inotify.max_user_watches=524288' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
# Use polling mode
journalctl -f --interval=1
# Check specific service
journalctl -u your-service.service -f
High Memory Usage by journald
Problem: systemd-journald consuming excessive memory
Diagnosis:
# Check journald memory usage
ps aux | grep systemd-journald
# Check journal size
journalctl --disk-usage
# Check rate limiting
journalctl | grep "rate"
# Monitor memory in real-time
top -p $(pgrep systemd-journald)
Solutions:
# Configure journal size limits
sudo tee /etc/systemd/journald.conf << EOF
[Journal]
SystemMaxUse=100M
RuntimeMaxUse=50M
MaxFileSec=1week
MaxRetentionSec=1month
EOF
# Restart journald
sudo systemctl restart systemd-journald
# Vacuum old logs
sudo journalctl --vacuum-size=50M
sudo journalctl --vacuum-time=1week
# Enable rate limiting
sudo tee -a /etc/systemd/journald.conf << EOF
RateLimitInterval=30s
RateLimitBurst=10000
EOF
Missing Service Logs
Problem: Specific service logs not appearing in journal
Diagnosis:
# Check if service is using journal
systemctl status your-service.service
# Check service configuration
systemctl cat your-service.service
# Look for syslog configuration
grep -r "syslog" /etc/your-service/
# Check for custom logging
ps aux | grep your-service
Solutions:
# Ensure service uses systemd logging
sudo systemctl edit your-service.service
# Add:
# [Service]
# StandardOutput=journal
# StandardError=journal
# Check for service-specific logs
tail -f /var/log/your-service.log
# Force service to use journal
sudo systemctl daemon-reload
sudo systemctl restart your-service.service
# Check journal configuration
journalctl -u your-service.service --since "10 minutes ago"
Journal File Corruption
Problem: Corrupted journal files preventing log access
Diagnosis:
# Verify journal integrity
journalctl --verify
# Check for specific errors
sudo journalctl --verify 2>&1 | grep -i error
# Identify corrupted files
find /var/log/journal -name "*.journal" -exec journalctl --file {} --verify \;
# Check disk health
sudo fsck /var
Solutions:
# Backup existing journals
sudo cp -r /var/log/journal /var/log/journal.backup
# Remove corrupted files (WARNING: data loss)
sudo rm /var/log/journal/*/system@*.journal~
# Force journal rotation
sudo systemctl kill --signal=SIGUSR2 systemd-journald
# Rebuild journal completely (last resort)
sudo systemctl stop systemd-journald
sudo rm -rf /var/log/journal/*
sudo systemctl start systemd-journald
# Restore from backup if needed
sudo cp -r /var/log/journal.backup/* /var/log/journal/
Mastering the journalctl command enables efficient system log analysis, troubleshooting, and monitoring in modern Linux environments using systemd’s centralized logging system.