Linux journalctl Command: Find Timeout Errors Tutorial
Description
How to Use Linux journalctl Command for Log Analysis?
Quick Answer: Use journalctl -b
to view current boot logs, journalctl -f
to follow live logs, and journalctl -u servicename
to check specific services. Filter with grep for errors: journalctl -b | grep "error"
. Essential for systemd log management and troubleshooting.
Essential journalctl Command Examples
# View current boot logs
journalctl -b
# Follow live logs (like tail -f)
journalctl -f
# Check specific service logs
journalctl -u nginx.service
# Show only error messages
journalctl -p err
# Find timeout errors
journalctl -b | grep "Timed out waiting"
# View logs from last hour
journalctl --since "1 hour ago"
# Show last 50 entries
journalctl -n 50
# Check kernel messages
journalctl -k
# View logs in reverse order (newest first)
journalctl -r
# Filter by time range
journalctl --since "2025-01-15 09:00:00" --until "2025-01-15 17:00:00"
# Follow specific service logs live
journalctl -u ssh.service -f
# Find failed services
journalctl -b | grep "Failed to start"
Frequently Asked Questions
Q: What does journalctl command do in Linux? A: journalctl
is the systemd journal query tool that displays and filters system logs. It replaces traditional log file viewing and provides centralized access to all system, kernel, and service logs.
Q: How do I view current boot logs with journalctl? A: Use journalctl -b
to show logs from the current boot session only, or journalctl -b -1
for the previous boot. This helps isolate current session issues from historical problems.
Q: How do I follow live logs like tail -f? A: Use journalctl -f
to follow logs in real-time, similar to tail -f
. Add -u servicename
to follow a specific service: journalctl -u nginx -f
.
Q: How do I check logs for a specific service? A: Use journalctl -u servicename
to view logs for a specific systemd service. Example: journalctl -u ssh.service
shows SSH service logs.
Q: How do I filter journalctl by time? A: Use --since
and --until
options with time expressions: journalctl --since "1 hour ago"
or journalctl --since "2025-01-15 09:00:00"
.
Essential Steps to Master journalctl Log Analysis
- Start with boot logs: Use
journalctl -b
to view current session logs and identify system startup issues - Check service status: Use
journalctl -u servicename
to investigate specific service problems and failures - Filter by priority: Use
journalctl -p err
to focus on error-level messages and critical issues - Use time filters: Apply
--since
and--until
to narrow down logs to specific time periods - Follow live logs: Use
journalctl -f
for real-time monitoring during troubleshooting sessions - Combine with grep: Pipe output to grep for specific patterns:
journalctl -b | grep "error"
Most Important journalctl Commands
Command | Purpose | Use Case |
---|---|---|
journalctl -b | Current boot logs | System startup issues |
journalctl -f | Follow live logs | Real-time monitoring |
journalctl -u service | Service-specific logs | Service troubleshooting |
journalctl -p err | Error messages only | Critical issue detection |
journalctl --since "1h ago" | Time-filtered logs | Recent problem analysis |
What Are the Most Important journalctl Command Options?
Essential Viewing Options
# Current boot session logs
journalctl -b
# Previous boot session logs
journalctl -b -1
# Follow logs in real-time
journalctl -f
# Show last N entries
journalctl -n 50
# Reverse order (newest first)
journalctl -r
Service and Unit Filtering
# Specific service logs
journalctl -u nginx.service
# Multiple services
journalctl -u nginx.service -u mysql.service
# All systemd units
journalctl -u "*.service"
# Kernel messages only
journalctl -k
How Do You Filter journalctl Logs by Time?
Time-Based Filtering Options
# Logs from last hour
journalctl --since "1 hour ago"
# Logs from last 30 minutes
journalctl --since "30 minutes ago"
# Logs from today
journalctl --since today
# Logs from yesterday
journalctl --since yesterday
# Logs from specific date
journalctl --since "2025-01-15"
Advanced Time Filtering
# Specific date and time
journalctl --since "2025-01-15 09:00:00"
# Time range filtering
journalctl --since "2 hours ago" --until "1 hour ago"
# Date range filtering
journalctl --since "2025-01-15" --until "2025-01-16"
# Combine with other filters
journalctl -u nginx.service --since "1 hour ago"
Time Format Examples
# Relative times
journalctl --since "2 days ago"
journalctl --since "1 week ago"
journalctl --since "10 minutes ago"
# Absolute times
journalctl --since "2025-01-15 14:30:00"
journalctl --since "Jan 15 14:30:00"
journalctl --since "15:30"
How Do You Filter journalctl by Priority and Message Type?
Priority Level Filtering
# Emergency messages only
journalctl -p emerg
# Alert level and above
journalctl -p alert
# Critical errors and above
journalctl -p crit
# Error messages and above
journalctl -p err
# Warning messages and above
journalctl -p warning
# Informational and above
journalctl -p info
# Debug messages (all)
journalctl -p debug
Message Type Filtering
# Kernel messages only
journalctl -k
# User session messages
journalctl --user
# System messages only
journalctl --system
# Messages from specific process
journalctl _PID=1234
# Messages from specific executable
journalctl _COMM=sshd
Advanced Pattern Filtering
# Find timeout errors
journalctl -b | grep "Timed out waiting"
# Find failed services
journalctl -b | grep "Failed to start"
# Find memory errors
journalctl -k | grep -i "memory"
# Find disk errors
journalctl -k | grep -i "disk"
# Find network errors
journalctl | grep -E "(network|eth0|connection)"
What Are Advanced journalctl Command Techniques?
Output Format Options
# JSON output for parsing
journalctl -o json
# JSON output (pretty printed)
journalctl -o json-pretty
# Short format (syslog style)
journalctl -o short
# Verbose format (all fields)
journalctl -o verbose
# Export format for backup
journalctl -o export
# CAT format (message only)
journalctl -o cat
Field-Based Filtering
# Filter by specific systemd unit
journalctl _SYSTEMD_UNIT=nginx.service
# Filter by process ID
journalctl _PID=1234
# Filter by user ID
journalctl _UID=1000
# Filter by executable name
journalctl _COMM=sshd
# Filter by hostname
journalctl _HOSTNAME=server01
Disk Usage and Maintenance
# Check journal disk usage
journalctl --disk-usage
# Verify journal integrity
journalctl --verify
# Vacuum old logs (keep 1 week)
sudo journalctl --vacuum-time=1week
# Vacuum by size (keep 1GB)
sudo journalctl --vacuum-size=1G
# Vacuum by number of files
sudo journalctl --vacuum-files=10
When Should You Use Different journalctl Commands?
System Troubleshooting Scenarios
# Boot problems - check current boot
journalctl -b -p err
# Service failures - check specific service
journalctl -u failed-service.service -n 100
# Performance issues - check recent system logs
journalctl --since "1 hour ago" -p warning
# Hardware problems - check kernel messages
journalctl -k --since "1 hour ago"
Development and Debugging
# Application debugging - follow app logs
journalctl -u myapp.service -f
# Web server issues - check access patterns
journalctl -u nginx.service --since "30 minutes ago"
# Database problems - monitor database service
journalctl -u mysql.service -p err
# Container issues - check container runtime
journalctl -u docker.service --since today
Security Analysis
# Authentication failures
journalctl | grep "authentication failure"
# SSH login attempts
journalctl -u ssh.service | grep "Failed password"
# Sudo usage tracking
journalctl | grep "sudo.*COMMAND"
# System security events
journalctl -p warning | grep -E "(security|auth|login)"
What Are All journalctl Command Options?
Option | Description | Example |
---|---|---|
-b | Boot session logs | journalctl -b |
-f | Follow live logs | journalctl -f |
-u | Specific unit/service | journalctl -u nginx |
-p | Priority level | journalctl -p err |
-n | Number of entries | journalctl -n 50 |
-r | Reverse order | journalctl -r |
-k | Kernel messages | journalctl -k |
-x | Add explanatory text | journalctl -x |
--since | Time filter start | journalctl --since "1h ago" |
--until | Time filter end | journalctl --until "now" |
What Are Essential journalctl Log Analysis Practices?
Daily Monitoring Routine
# Check for critical errors
journalctl -p crit --since today
# Review service failures
journalctl -b | grep "Failed to start"
# Monitor system performance
journalctl -k --since "1 hour ago" | grep -E "(error|warning)"
# Check authentication logs
journalctl --since today | grep -E "(login|auth)"
Automated Log Analysis
# Create daily error report
journalctl -p err --since today > /tmp/daily_errors.log
# Monitor specific service health
journalctl -u critical-service.service --since "5 minutes ago" -p warning
# Check for disk issues
journalctl -k | grep -E "(disk|filesystem|mount)" | tail -20
# Network connectivity monitoring
journalctl --since "10 minutes ago" | grep -E "(network|connection|timeout)"
Performance Optimization
# Limit journal size
sudo journalctl --vacuum-size=100M
# Clean old entries
sudo journalctl --vacuum-time=2weeks
# Persistent storage configuration
sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald
How Can You Create Useful journalctl Aliases and Scripts?
Essential Aliases
# Add to ~/.bashrc
alias jb='journalctl -b' # Current boot
alias jf='journalctl -f' # Follow logs
alias je='journalctl -p err' # Errors only
alias jt='journalctl --since "1 hour ago"' # Last hour
alias jk='journalctl -k' # Kernel only
Advanced Functions
# Service log checker
jservice() {
if [ -z "$1" ]; then
echo "Usage: jservice <service-name>"
return 1
fi
journalctl -u "$1.service" -n 50 --no-pager
}
# Error finder with context
jerrors() {
local since=${1:-"1 hour ago"}
echo "=== Errors since $since ==="
journalctl --since "$since" -p err --no-pager
}
# Boot analysis
jboot() {
local boot=${1:-0}
echo "=== Boot Analysis (boot $boot) ==="
journalctl -b "$boot" -p warning --no-pager | head -20
}
# Service monitor
jmonitor() {
if [ -z "$1" ]; then
echo "Usage: jmonitor <service-name>"
return 1
fi
echo "Monitoring $1.service (Ctrl+C to stop)..."
journalctl -u "$1.service" -f --no-pager
}
Diagnostic Scripts
#!/bin/bash
# system_health_check.sh - Comprehensive log analysis
echo "=== System Health Check $(date) ==="
echo -e "\n1. Critical Errors:"
journalctl -p crit --since today --no-pager | head -10
echo -e "\n2. Failed Services:"
journalctl -b | grep "Failed to start" --no-pager | head -10
echo -e "\n3. Recent Timeouts:"
journalctl -b | grep "Timed out waiting" --no-pager | head -10
echo -e "\n4. Kernel Warnings:"
journalctl -k --since "1 hour ago" -p warning --no-pager | head -10
echo -e "\n5. Authentication Failures:"
journalctl --since today | grep "authentication failure" --no-pager | head -10
echo -e "\n6. Disk Usage:"
journalctl --disk-usage
What Commands Are Related to journalctl?
systemctl
– Control systemd services and view statusdmesg
– Display kernel ring buffer messagestail
– View end of files (traditional log viewing)grep
– Search text patterns in logsawk
– Advanced text processing and filteringless
– Paginated viewing of log fileslogrotate
– Manage log file rotationrsyslog
– Traditional system logging daemonsystemd-analyze
– Analyze system boot performance
Common journalctl Command Problems and Solutions
“No journal files were found” Error
Problem: journalctl shows no logs or journal files missing
Diagnosis:
# Check if systemd-journald is running
systemctl status systemd-journald
# Check journal directory
ls -la /var/log/journal/
# Check for runtime journal
ls -la /run/log/journal/
# Check journal configuration
cat /etc/systemd/journald.conf
Solutions:
# Create persistent journal directory
sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald
# Enable persistent storage
sudo sed -i 's/#Storage=auto/Storage=persistent/' /etc/systemd/journald.conf
sudo systemctl restart systemd-journald
# Check disk space
df -h /var/log
# Fix permissions
sudo chown root:systemd-journal /var/log/journal
sudo chmod 2755 /var/log/journal
“Failed to determine timestamp” Error
Problem: Corrupted journal files or timestamp issues
Diagnosis:
# Verify journal integrity
journalctl --verify
# Check for corrupted files
sudo journalctl --verify --quiet
# Check system time
timedatectl status
# Check for clock synchronization
systemctl status systemd-timesyncd
Solutions:
# Rebuild journal database
sudo systemctl stop systemd-journald
sudo rm /var/log/journal/*/*
sudo systemctl start systemd-journald
# Fix system time
sudo timedatectl set-ntp true
sudo timedatectl set-timezone America/New_York
# Vacuum corrupted entries
sudo journalctl --vacuum-time=1d
# Force journal rotation
sudo systemctl kill --kill-who=main --signal=SIGUSR2 systemd-journald
Permission Denied Accessing Logs
Problem: Cannot view system logs as regular user
Diagnosis:
# Check current user groups
groups
# Check journal file permissions
ls -la /var/log/journal/
# Check systemd-journal group membership
getent group systemd-journal
# Test with sudo
sudo journalctl -b
Solutions:
# Add user to systemd-journal group
sudo usermod -a -G systemd-journal $USER
# Re-login or use newgrp
newgrp systemd-journal
# Use sudo for system logs
sudo journalctl -b
# Check adm group membership (alternative)
sudo usermod -a -G adm $USER
journalctl Command Hanging
Problem: journalctl command appears to hang or freeze
Diagnosis:
# Check if pager is active
ps aux | grep less
# Check journal size
journalctl --disk-usage
# Check system load
top | head -5
# Check for large journal files
find /var/log/journal -type f -size +100M
Solutions:
# Disable pager
journalctl --no-pager
# Use head to limit output
journalctl -n 100
# Vacuum large journals
sudo journalctl --vacuum-size=100M
# Use time limits
journalctl --since "1 hour ago"
# Kill hanging process
pkill -f journalctl
Logs Not Appearing in Real Time
Problem: Live log following (-f) not showing new entries
Diagnosis:
# Check if service is active
systemctl is-active systemd-journald
# Check if logs are being written
ls -la /var/log/journal/*/system.journal
# Test with different service
journalctl -u systemd-journald -f
# Check inotify limits
cat /proc/sys/fs/inotify/max_user_watches
Solutions:
# Restart journald service
sudo systemctl restart systemd-journald
# Increase inotify limits
echo 'fs.inotify.max_user_watches=524288' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
# Use polling mode
journalctl -f --interval=1
# Check specific service
journalctl -u your-service.service -f
High Memory Usage by journald
Problem: systemd-journald consuming excessive memory
Diagnosis:
# Check journald memory usage
ps aux | grep systemd-journald
# Check journal size
journalctl --disk-usage
# Check rate limiting
journalctl | grep "rate"
# Monitor memory in real-time
top -p $(pgrep systemd-journald)
Solutions:
# Configure journal size limits
sudo tee /etc/systemd/journald.conf << EOF
[Journal]
SystemMaxUse=100M
RuntimeMaxUse=50M
MaxFileSec=1week
MaxRetentionSec=1month
EOF
# Restart journald
sudo systemctl restart systemd-journald
# Vacuum old logs
sudo journalctl --vacuum-size=50M
sudo journalctl --vacuum-time=1week
# Enable rate limiting
sudo tee -a /etc/systemd/journald.conf << EOF
RateLimitInterval=30s
RateLimitBurst=10000
EOF
Missing Service Logs
Problem: Specific service logs not appearing in journal
Diagnosis:
# Check if service is using journal
systemctl status your-service.service
# Check service configuration
systemctl cat your-service.service
# Look for syslog configuration
grep -r "syslog" /etc/your-service/
# Check for custom logging
ps aux | grep your-service
Solutions:
# Ensure service uses systemd logging
sudo systemctl edit your-service.service
# Add:
# [Service]
# StandardOutput=journal
# StandardError=journal
# Check for service-specific logs
tail -f /var/log/your-service.log
# Force service to use journal
sudo systemctl daemon-reload
sudo systemctl restart your-service.service
# Check journal configuration
journalctl -u your-service.service --since "10 minutes ago"
Journal File Corruption
Problem: Corrupted journal files preventing log access
Diagnosis:
# Verify journal integrity
journalctl --verify
# Check for specific errors
sudo journalctl --verify 2>&1 | grep -i error
# Identify corrupted files
find /var/log/journal -name "*.journal" -exec journalctl --file {} --verify \;
# Check disk health
sudo fsck /var
Solutions:
# Backup existing journals
sudo cp -r /var/log/journal /var/log/journal.backup
# Remove corrupted files (WARNING: data loss)
sudo rm /var/log/journal/*/system@*.journal~
# Force journal rotation
sudo systemctl kill --signal=SIGUSR2 systemd-journald
# Rebuild journal completely (last resort)
sudo systemctl stop systemd-journald
sudo rm -rf /var/log/journal/*
sudo systemctl start systemd-journald
# Restore from backup if needed
sudo cp -r /var/log/journal.backup/* /var/log/journal/
Mastering the journalctl command enables efficient system log analysis, troubleshooting, and monitoring in modern Linux environments using systemd’s centralized logging system.