Linux Monitoring Scripts Automation: Alerts System Health Checks Linux Mastery Series

Bottom Line Up Front: Linux monitoring scripts automation enables proactive system management by continuously tracking CPU, memory, disk, and network metrics, triggering instant alerts when thresholds are breached—reducing downtime by up to 90% and preventing critical failures before they impact production environments.

Table of Contents

• How to Create Automated System Monitoring Scripts?
• What Are the Best Practices for Linux Monitoring Scripts Automation?
• How to Set Up Alert Notification Systems?
• What Metrics Should You Monitor on Linux Systems?
• How to Integrate Email and Slack Alerts?
• Troubleshooting Common Monitoring Script Issues

---

How to Create Automated System Monitoring Scripts?

Building effective linux monitoring scripts automation begins with understanding your system’s critical resources. Consequently, identifying which metrics require continuous observation forms the foundation of proactive system administration.

System Resource Monitoring Script

The following bash script demonstrates comprehensive resource monitoring with automated threshold checking:

#!/bin/bash
#
# system_monitor.sh - Comprehensive Linux System Resource Monitor
# Purpose: Track CPU, memory, and disk usage with alert generation
#

# Configuration variables
CPU_THRESHOLD=80
MEMORY_THRESHOLD=85
DISK_THRESHOLD=90
LOG_FILE="/var/log/system_monitor.log"
ALERT_EMAIL="admin@example.com"

# Function to send alert notifications
send_alert() {
    local alert_message="$1"
    local alert_subject="$2"
    
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] ALERT: ${alert_message}" >> "${LOG_FILE}"
    echo "${alert_message}" | mail -s "${alert_subject}" "${ALERT_EMAIL}"
}

# Monitor CPU usage
monitor_cpu() {
    local cpu_usage
    cpu_usage=$(top -bn1 | grep "Cpu(s)" | sed "s/.*, *\([0-9.]*\)%* id.*/\1/" | awk '{print 100 - $1}')
    
    if (( $(echo "${cpu_usage} > ${CPU_THRESHOLD}" | bc -l) )); then
        send_alert "High CPU usage detected: ${cpu_usage}%" "CPU Alert - System Monitor"
        return 1
    fi
    
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] CPU usage: ${cpu_usage}%" >> "${LOG_FILE}"
    return 0
}

# Monitor memory usage
monitor_memory() {
    local memory_usage
    memory_usage=$(free | grep Mem | awk '{printf "%.2f", $3/$2 * 100.0}')
    
    if (( $(echo "${memory_usage} > ${MEMORY_THRESHOLD}" | bc -l) )); then
        send_alert "High memory usage detected: ${memory_usage}%" "Memory Alert - System Monitor"
        return 1
    fi
    
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] Memory usage: ${memory_usage}%" >> "${LOG_FILE}"
    return 0
}

# Monitor disk usage
monitor_disk() {
    local disk_usage
    disk_usage=$(df -h / | awk 'NR==2 {print $5}' | sed 's/%//g')
    
    if [ "${disk_usage}" -gt "${DISK_THRESHOLD}" ]; then
        send_alert "High disk usage detected: ${disk_usage}%" "Disk Alert - System Monitor"
        return 1
    fi
    
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] Disk usage: ${disk_usage}%" >> "${LOG_FILE}"
    return 0
}

# Main monitoring execution
main() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] Starting system health check..." >> "${LOG_FILE}"
    
    monitor_cpu
    monitor_memory
    monitor_disk
    
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] Health check completed." >> "${LOG_FILE}"
    echo "---" >> "${LOG_FILE}"
}

# Execute main function
main

This comprehensive monitoring script provides real-time system health tracking, moreover it can be scheduled via cron for continuous automated surveillance.

Scheduling with Cron

To enable automated monitoring, additionally configure cron to execute the script at regular intervals:

# Edit crontab
crontab -e

# Add monitoring job - runs every 5 minutes
*/5 * * * * /usr/local/bin/system_monitor.sh

# Alternative: hourly monitoring
0 * * * * /usr/local/bin/system_monitor.sh

For detailed cron scheduling techniques, reference our guide on Cron Jobs and Task Scheduling.

---

What Are the Best Practices for Linux Monitoring Scripts Automation?

Implementing robust linux monitoring scripts automation requires following industry-standard methodologies. Subsequently, adhering to these practices ensures reliable and maintainable monitoring infrastructure.

Best Practice Guidelines

1. Modular Function Design: Create reusable functions for each monitoring component
2. Centralized Configuration: Store thresholds and settings in separate config files
3. Comprehensive Logging: Maintain detailed logs with timestamps for audit trails
4. Error Handling: Implement proper error checking and graceful failure recovery
5. Resource Efficiency: Optimize scripts to minimize system resource consumption
6. Security Considerations: Restrict script permissions and secure credential storage

Advanced Logging Configuration

#!/bin/bash
#
# advanced_logger.sh - Enhanced logging with rotation support
#

LOG_DIR="/var/log/monitoring"
MAX_LOG_SIZE=10485760  # 10MB in bytes
LOG_RETENTION_DAYS=30

# Create log directory if it doesn't exist
mkdir -p "${LOG_DIR}"

# Function for log rotation
rotate_logs() {
    local log_file="$1"
    local log_size
    
    if [ -f "${log_file}" ]; then
        log_size=$(stat -f%z "${log_file}" 2>/dev/null || stat -c%s "${log_file}")
        
        if [ "${log_size}" -gt "${MAX_LOG_SIZE}" ]; then
            mv "${log_file}" "${log_file}.$(date +%Y%m%d_%H%M%S)"
            touch "${log_file}"
            
            # Clean old logs
            find "${LOG_DIR}" -name "*.log.*" -mtime +"${LOG_RETENTION_DAYS}" -delete
        fi
    fi
}

# Enhanced logging function
log_message() {
    local level="$1"
    local message="$2"
    local log_file="${LOG_DIR}/system_monitor.log"
    
    rotate_logs "${log_file}"
    
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [${level}] ${message}" >> "${log_file}"
    
    # Also output to stderr for critical messages
    if [ "${level}" = "CRITICAL" ] || [ "${level}" = "ERROR" ]; then
        echo "[${level}] ${message}" >&2
    fi
}

# Usage examples
log_message "INFO" "System monitoring initiated"
log_message "WARNING" "CPU usage approaching threshold"
log_message "CRITICAL" "Disk space critically low"

Therefore, implementing proper logging ensures comprehensive system health tracking and simplifies troubleshooting efforts.

---

How to Set Up Alert Notification Systems?

Effective alert notification systems transform passive monitoring into active incident response. Furthermore, integrating multiple notification channels guarantees alerts reach administrators regardless of their location.

Email Alert Configuration

#!/bin/bash
#
# email_alerting.sh - Email notification system for monitoring alerts
#

# Email configuration
SMTP_SERVER="smtp.example.com"
SMTP_PORT="587"
SMTP_USER="alerts@example.com"
ALERT_RECIPIENTS="admin@example.com,oncall@example.com"

# Function to send email with attachment
send_email_alert() {
    local subject="$1"
    local body="$2"
    local attachment="$3"
    local priority="$4"  # High, Normal, Low
    
    local email_body="To: ${ALERT_RECIPIENTS}
From: ${SMTP_USER}
Subject: ${subject}
X-Priority: ${priority:-Normal}
Content-Type: text/html; charset=UTF-8

<html>
<body>
<h2 style='color: #c0392b;'>${subject}</h2>
<p>${body}</p>
<hr>
<p><small>Generated by Linux Monitoring System at $(hostname)</small></p>
<p><small>Timestamp: $(date '+%Y-%m-%d %H:%M:%S %Z')</small></p>
</body>
</html>"
    
    if [ -n "${attachment}" ]; then
        echo "${email_body}" | mail -s "${subject}" -a "${attachment}" "${ALERT_RECIPIENTS}"
    else
        echo "${email_body}" | mail -s "${subject}" "${ALERT_RECIPIENTS}"
    fi
    
    return $?
}

# Example usage with system metrics
generate_alert_report() {
    local report_file="/tmp/system_report_$(date +%Y%m%d_%H%M%S).txt"
    
    {
        echo "=== System Health Report ==="
        echo ""
        echo "CPU Usage:"
        top -bn1 | grep "Cpu(s)"
        echo ""
        echo "Memory Usage:"
        free -h
        echo ""
        echo "Disk Usage:"
        df -h
        echo ""
        echo "Top Processes:"
        ps aux --sort=-%mem | head -n 10
    } > "${report_file}"
    
    send_email_alert \
        "System Alert: Resource Threshold Exceeded" \
        "Critical system resources have exceeded defined thresholds. Please review the attached report." \
        "${report_file}" \
        "High"
    
    rm -f "${report_file}"
}

Accordingly, email alerting provides detailed incident reports with historical context for informed decision-making.

Slack Webhook Integration

Modern linux monitoring scripts automation benefits from instant messaging integration. Similarly, Slack webhooks provide real-time notifications to team channels:

#!/bin/bash
#
# slack_alerting.sh - Slack webhook integration for monitoring alerts
#

SLACK_WEBHOOK_URL="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
SLACK_CHANNEL="#monitoring-alerts"
SLACK_USERNAME="Linux Monitor Bot"

# Function to send Slack notification
send_slack_alert() {
    local title="$1"
    local message="$2"
    local color="$3"  # good, warning, danger
    local hostname=$(hostname)
    
    local payload=$(cat <<EOF
{
    "channel": "${SLACK_CHANNEL}",
    "username": "${SLACK_USERNAME}",
    "icon_emoji": ":warning:",
    "attachments": [
        {
            "color": "${color}",
            "title": "${title}",
            "text": "${message}",
            "fields": [
                {
                    "title": "Server",
                    "value": "${hostname}",
                    "short": true
                },
                {
                    "title": "Timestamp",
                    "value": "$(date '+%Y-%m-%d %H:%M:%S %Z')",
                    "short": true
                }
            ],
            "footer": "Linux Monitoring System",
            "footer_icon": "https://platform.slack-edge.com/img/default_application_icon.png"
        }
    ]
}
EOF
)
    
    curl -X POST \
        -H 'Content-Type: application/json' \
        --data "${payload}" \
        "${SLACK_WEBHOOK_URL}"
}

# Alert severity levels
send_info_alert() {
    send_slack_alert "$1" "$2" "good"
}

send_warning_alert() {
    send_slack_alert "$1" "$2" "warning"
}

send_critical_alert() {
    send_slack_alert "$1" "$2" "danger"
}

# Usage example
check_disk_space() {
    local disk_usage=$(df -h / | awk 'NR==2 {print $5}' | sed 's/%//g')
    
    if [ "${disk_usage}" -gt 90 ]; then
        send_critical_alert \
            "Critical Disk Space Alert" \
            "Disk usage is at ${disk_usage}% on root filesystem. Immediate action required!"
    elif [ "${disk_usage}" -gt 75 ]; then
        send_warning_alert \
            "Disk Space Warning" \
            "Disk usage is at ${disk_usage}% on root filesystem. Please monitor closely."
    fi
}

For comprehensive network monitoring techniques, explore our article on Network Performance Monitoring.

---

What Metrics Should You Monitor on Linux Systems?

Selecting appropriate metrics for linux monitoring scripts automation directly impacts system reliability. Therefore, prioritizing critical system indicators ensures efficient resource utilization and early problem detection.

Essential System Metrics

Metric Category	Key Indicators	Recommended Thresholds
CPU Performance	Usage percentage, load average, process count	80% sustained usage
Memory Resources	Used memory, available memory, swap usage	85% memory utilization
Disk Operations	Disk space, I/O wait, read/write rates	90% capacity, 5% iowait
Network Activity	Bandwidth usage, packet loss, connection count	80% bandwidth, >1% loss
System Health	Temperature, failed services, error logs	Hardware limits, service failures

Comprehensive Metrics Collection Script

#!/bin/bash
#
# metrics_collector.sh - Comprehensive system metrics collection
#

METRICS_DIR="/var/lib/monitoring/metrics"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

# Create metrics directory
mkdir -p "${METRICS_DIR}"

# Collect CPU metrics
collect_cpu_metrics() {
    local output_file="${METRICS_DIR}/cpu_${TIMESTAMP}.json"
    
    local cpu_usage=$(top -bn1 | grep "Cpu(s)" | sed "s/.*, *\([0-9.]*\)%* id.*/\1/" | awk '{print 100 - $1}')
    local load_avg=$(uptime | awk -F'load average:' '{print $2}' | awk '{print $1 $2 $3}')
    local cpu_cores=$(nproc)
    
    cat > "${output_file}" <<EOF
{
    "timestamp": "${TIMESTAMP}",
    "cpu_usage_percent": ${cpu_usage},
    "load_average": "${load_avg}",
    "cpu_cores": ${cpu_cores},
    "hostname": "$(hostname)"
}
EOF
}

# Collect memory metrics
collect_memory_metrics() {
    local output_file="${METRICS_DIR}/memory_${TIMESTAMP}.json"
    
    local mem_info=$(free -b | grep Mem)
    local total=$(echo ${mem_info} | awk '{print $2}')
    local used=$(echo ${mem_info} | awk '{print $3}')
    local free=$(echo ${mem_info} | awk '{print $4}')
    local usage_percent=$(echo "scale=2; ${used} * 100 / ${total}" | bc)
    
    cat > "${output_file}" <<EOF
{
    "timestamp": "${TIMESTAMP}",
    "total_bytes": ${total},
    "used_bytes": ${used},
    "free_bytes": ${free},
    "usage_percent": ${usage_percent},
    "hostname": "$(hostname)"
}
EOF
}

# Collect disk metrics
collect_disk_metrics() {
    local output_file="${METRICS_DIR}/disk_${TIMESTAMP}.json"
    
    df -B1 | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{print $6}' | while read mount_point; do
        local disk_info=$(df -B1 "${mount_point}" | tail -1)
        local filesystem=$(echo ${disk_info} | awk '{print $1}')
        local size=$(echo ${disk_info} | awk '{print $2}')
        local used=$(echo ${disk_info} | awk '{print $3}')
        local available=$(echo ${disk_info} | awk '{print $4}')
        local usage_percent=$(echo ${disk_info} | awk '{print $5}' | sed 's/%//g')
        
        cat >> "${output_file}" <<EOF
{
    "timestamp": "${TIMESTAMP}",
    "mount_point": "${mount_point}",
    "filesystem": "${filesystem}",
    "size_bytes": ${size},
    "used_bytes": ${used},
    "available_bytes": ${available},
    "usage_percent": ${usage_percent},
    "hostname": "$(hostname)"
}
EOF
    done
}

# Collect network metrics
collect_network_metrics() {
    local output_file="${METRICS_DIR}/network_${TIMESTAMP}.json"
    
    local interfaces=$(ip -br link | awk '{print $1}' | grep -vE 'lo')
    
    for iface in ${interfaces}; do
        if [ -d "/sys/class/net/${iface}/statistics" ]; then
            local rx_bytes=$(cat /sys/class/net/${iface}/statistics/rx_bytes)
            local tx_bytes=$(cat /sys/class/net/${iface}/statistics/tx_bytes)
            local rx_packets=$(cat /sys/class/net/${iface}/statistics/rx_packets)
            local tx_packets=$(cat /sys/class/net/${iface}/statistics/tx_packets)
            local rx_errors=$(cat /sys/class/net/${iface}/statistics/rx_errors)
            local tx_errors=$(cat /sys/class/net/${iface}/statistics/tx_errors)
            
            cat >> "${output_file}" <<EOF
{
    "timestamp": "${TIMESTAMP}",
    "interface": "${iface}",
    "rx_bytes": ${rx_bytes},
    "tx_bytes": ${tx_bytes},
    "rx_packets": ${rx_packets},
    "tx_packets": ${tx_packets},
    "rx_errors": ${rx_errors},
    "tx_errors": ${tx_errors},
    "hostname": "$(hostname)"
}
EOF
        fi
    done
}

# Main execution
main() {
    collect_cpu_metrics
    collect_memory_metrics
    collect_disk_metrics
    collect_network_metrics
    
    # Clean old metrics (older than 7 days)
    find "${METRICS_DIR}" -type f -mtime +7 -delete
}

main

Additionally, structured metrics collection enables trend analysis and capacity planning for future infrastructure needs.

---

How to Integrate Email and Slack Alerts?

Multi-channel alert distribution ensures critical notifications reach administrators through their preferred communication platforms. Furthermore, redundant alerting prevents single points of failure in notification delivery.

Unified Alert Dispatcher

#!/bin/bash
#
# unified_alerting.sh - Multi-channel alert distribution system
#

# Source configuration
source /etc/monitoring/alert_config.sh

# Alert levels
ALERT_INFO="INFO"
ALERT_WARNING="WARNING"
ALERT_CRITICAL="CRITICAL"

# Main alert dispatcher function
dispatch_alert() {
    local level="$1"
    local title="$2"
    local message="$3"
    local metric_data="$4"
    
    # Determine color/priority based on level
    local email_priority
    local slack_color
    
    case "${level}" in
        "${ALERT_INFO}")
            email_priority="Normal"
            slack_color="good"
            ;;
        "${ALERT_WARNING}")
            email_priority="High"
            slack_color="warning"
            ;;
        "${ALERT_CRITICAL}")
            email_priority="High"
            slack_color="danger"
            ;;
        *)
            email_priority="Normal"
            slack_color="good"
            ;;
    esac
    
    # Send to all configured channels
    send_email_notification "${title}" "${message}" "${email_priority}"
    send_slack_notification "${title}" "${message}" "${slack_color}"
    
    # Log alert
    log_alert "${level}" "${title}" "${message}"
}

# Email notification function
send_email_notification() {
    local subject="$1"
    local body="$2"
    local priority="$3"
    
    # Generate HTML email body
    local html_body="<html>
<head>
    <style>
        body { font-family: Arial, sans-serif; }
        .alert-box { 
            border: 2px solid #c0392b; 
            padding: 15px; 
            background-color: #fadbd8;
            border-radius: 5px;
        }
        .info { border-color: #3498db; background-color: #d6eaf8; }
        .warning { border-color: #f39c12; background-color: #fdebd0; }
        .critical { border-color: #c0392b; background-color: #fadbd8; }
    </style>
</head>
<body>
    <div class='alert-box ${priority,,}'>
        <h2>${subject}</h2>
        <p>${body}</p>
        <hr>
        <p><strong>Server:</strong> $(hostname)</p>
        <p><strong>Timestamp:</strong> $(date '+%Y-%m-%d %H:%M:%S %Z')</p>
    </div>
</body>
</html>"
    
    echo "${html_body}" | mail -s "[${priority}] ${subject}" -a "Content-Type: text/html" "${EMAIL_RECIPIENTS}"
}

# Slack notification function
send_slack_notification() {
    local title="$1"
    local message="$2"
    local color="$3"
    
    local payload=$(cat <<EOF
{
    "username": "Linux Monitor",
    "icon_emoji": ":computer:",
    "attachments": [
        {
            "color": "${color}",
            "title": "${title}",
            "text": "${message}",
            "fields": [
                {
                    "title": "Server",
                    "value": "$(hostname)",
                    "short": true
                },
                {
                    "title": "Time",
                    "value": "$(date '+%H:%M:%S')",
                    "short": true
                }
            ]
        }
    ]
}
EOF
)
    
    curl -s -X POST -H 'Content-Type: application/json' \
        --data "${payload}" \
        "${SLACK_WEBHOOK_URL}" > /dev/null
}

# Alert logging function
log_alert() {
    local level="$1"
    local title="$2"
    local message="$3"
    
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [${level}] ${title}: ${message}" >> /var/log/monitoring/alerts.log
}

# Example: CPU threshold monitoring with unified alerts
monitor_cpu_with_alerts() {
    local cpu_usage=$(top -bn1 | grep "Cpu(s)" | sed "s/.*, *\([0-9.]*\)%* id.*/\1/" | awk '{print 100 - $1}')
    
    if (( $(echo "${cpu_usage} > 90" | bc -l) )); then
        dispatch_alert \
            "${ALERT_CRITICAL}" \
            "Critical CPU Usage Alert" \
            "CPU usage has reached ${cpu_usage}%, which exceeds the critical threshold of 90%. Immediate investigation required."
    elif (( $(echo "${cpu_usage} > 75" | bc -l) )); then
        dispatch_alert \
            "${ALERT_WARNING}" \
            "CPU Usage Warning" \
            "CPU usage is at ${cpu_usage}%, approaching critical levels. Monitor closely."
    fi
}

Consequently, unified alert dispatching simplifies notification management and ensures consistent message formatting across all channels.

For related monitoring setup, review our guide on Setting up Prometheus and Grafana on Linux.

---

Troubleshooting Common Monitoring Script Issues

Even well-designed linux monitoring scripts automation encounters operational challenges. Therefore, understanding common failure patterns enables rapid resolution and maintains monitoring reliability.

Common Issues and Solutions

Issue 1: Script Execution Permissions

Symptom: Cron jobs fail silently without generating alerts

Solution:

# Verify script permissions
ls -l /usr/local/bin/system_monitor.sh

# Set correct permissions
chmod 755 /usr/local/bin/system_monitor.sh

# Ensure script is owned by appropriate user
chown root:root /usr/local/bin/system_monitor.sh

# Test script execution
sudo -u root /usr/local/bin/system_monitor.sh

Issue 2: Mail Command Not Found

Symptom: Email alerts fail with “mail: command not found”

Solution:

# Install mailutils on Debian/Ubuntu
sudo apt update
sudo apt install mailutils

# Install mailx on RHEL/CentOS
sudo yum install mailx

# Configure mail server
sudo dpkg-reconfigure postfix

# Test email functionality
echo "Test message" | mail -s "Test Subject" admin@example.com

Issue 3: Threshold Detection Inaccuracy

Symptom: False positive or false negative alerts

Solution:

#!/bin/bash
#
# accurate_threshold_check.sh - Improved threshold detection
#

# Use floating-point arithmetic with bc
check_threshold_accurately() {
    local current_value="$1"
    local threshold="$2"
    
    # Ensure values are numeric
    if ! [[ "${current_value}" =~ ^[0-9.]+$ ]] || ! [[ "${threshold}" =~ ^[0-9.]+$ ]]; then
        echo "ERROR: Invalid numeric value" >&2
        return 2
    fi
    
    # Use bc for accurate comparison
    local result=$(echo "${current_value} > ${threshold}" | bc -l)
    
    if [ "${result}" -eq 1 ]; then
        return 0  # Threshold exceeded
    else
        return 1  # Within threshold
    fi
}

# Example usage
cpu_usage="85.7"
threshold="80.0"

if check_threshold_accurately "${cpu_usage}" "${threshold}"; then
    echo "CPU usage ${cpu_usage}% exceeds threshold ${threshold}%"
fi

Issue 4: Log File Growth

Symptom: Monitoring logs consume excessive disk space

Solution:

#!/bin/bash
#
# log_management.sh - Automated log rotation and cleanup
#

LOG_DIR="/var/log/monitoring"
MAX_LOG_SIZE_MB=50
RETENTION_DAYS=30

# Function to rotate logs
rotate_large_logs() {
    find "${LOG_DIR}" -type f -name "*.log" | while read log_file; do
        local size_mb=$(du -m "${log_file}" | cut -f1)
        
        if [ "${size_mb}" -gt "${MAX_LOG_SIZE_MB}" ]; then
            local backup_name="${log_file}.$(date +%Y%m%d_%H%M%S)"
            mv "${log_file}" "${backup_name}"
            gzip "${backup_name}"
            touch "${log_file}"
            echo "Rotated large log: ${log_file}"
        fi
    done
}

# Function to clean old logs
cleanup_old_logs() {
    find "${LOG_DIR}" -type f \( -name "*.log.*" -o -name "*.gz" \) -mtime +"${RETENTION_DAYS}" -delete
    echo "Cleaned logs older than ${RETENTION_DAYS} days"
}

# Execute maintenance
rotate_large_logs
cleanup_old_logs

# Schedule this script via cron
# 0 2 * * * /usr/local/bin/log_management.sh

Issue 5: Network-Dependent Alerts Failing

Symptom: Slack/webhook notifications fail intermittently

Solution:

#!/bin/bash
#
# resilient_alerting.sh - Network-resilient alert delivery
#

ALERT_QUEUE_DIR="/var/spool/monitoring/alerts"
MAX_RETRY_ATTEMPTS=3
RETRY_DELAY=60

# Create queue directory
mkdir -p "${ALERT_QUEUE_DIR}"

# Queue alert for later delivery
queue_alert() {
    local alert_data="$1"
    local queue_file="${ALERT_QUEUE_DIR}/alert_$(date +%s)_$$.json"
    
    echo "${alert_data}" > "${queue_file}"
}

# Send alert with retry logic
send_alert_with_retry() {
    local webhook_url="$1"
    local payload="$2"
    local attempts=0
    
    while [ ${attempts} -lt ${MAX_RETRY_ATTEMPTS} ]; do
        if curl -s -o /dev/null -w "%{http_code}" \
            -X POST \
            -H 'Content-Type: application/json' \
            --data "${payload}" \
            --max-time 10 \
            "${webhook_url}" | grep -q "200"; then
            return 0
        fi
        
        attempts=$((attempts + 1))
        sleep ${RETRY_DELAY}
    done
    
    # Queue for later if all retries fail
    queue_alert "${payload}"
    return 1
}

# Process queued alerts
process_alert_queue() {
    find "${ALERT_QUEUE_DIR}" -type f -name "alert_*.json" | while read queued_alert; do
        local payload=$(cat "${queued_alert}")
        
        if send_alert_with_retry "${SLACK_WEBHOOK_URL}" "${payload}"; then
            rm -f "${queued_alert}"
            echo "Successfully sent queued alert"
        fi
    done
}

# Run queue processor periodically via cron
# */10 * * * * /usr/local/bin/process_alert_queue.sh

Moreover, implementing comprehensive error handling ensures monitoring systems remain operational even during partial infrastructure failures.

---

FAQ: Linux Monitoring Scripts Automation

How often should monitoring scripts run?

Critical system resources should be monitored every 5-15 minutes, while less volatile metrics can be checked hourly. However, adjust frequencies based on your specific workload patterns and infrastructure requirements.

What’s the difference between monitoring and logging?

Monitoring actively tracks system metrics in real-time and triggers alerts when thresholds are exceeded. In contrast, logging passively records events for historical analysis and troubleshooting purposes.

Can monitoring scripts impact system performance?

Well-designed scripts have minimal performance impact (typically <1% CPU usage). Nevertheless, excessive monitoring frequency or inefficient scripts can affect performance, so always profile your monitoring tools.

How do I monitor multiple servers simultaneously?

Implement centralized monitoring using tools like Prometheus, Nagios, or custom solutions with remote execution capabilities. Moreover, consider using SSH-based parallel execution with tools like parallel-ssh or Ansible for distributed monitoring.

Should I use custom scripts or enterprise monitoring tools?

Custom scripts excel for specific use cases and learning purposes, while enterprise tools provide comprehensive features, scalability, and support. Therefore, many organizations use hybrid approaches combining both solutions.

How can I reduce false positive alerts?

Implement intelligent thresholding with baseline analysis, use time-windowed averaging instead of instant values, and establish alert suppression during maintenance windows. Additionally, implement alert escalation policies with different severity levels.

What authentication method is best for email alerts?

Use application-specific passwords or OAuth tokens rather than standard account passwords. Furthermore, consider relay servers or cloud-based email services with API keys for improved security and reliability.

How do I secure monitoring scripts and credentials?

Store credentials in separate configuration files with restricted permissions (600), use environment variables, or implement secret management tools like HashiCorp Vault. Never hard-code credentials directly in scripts.

---

Additional Resources

Official Documentation

• Advanced Bash-Scripting Guide – Comprehensive bash scripting reference
• Cron Format Documentation – Official cron scheduling syntax
• SystemD Timer Units – Modern alternative to cron jobs

Monitoring Tools and Frameworks

• Prometheus Documentation – Modern monitoring and alerting toolkit
• Nagios Core – Industry-standard infrastructure monitoring
• Zabbix Documentation – Enterprise-class monitoring solution

Community Resources

• r/linuxadmin – System administration community
• Server Fault – Q&A for system administrators
• Linux Questions Monitoring Forum – Community support

Related LinuxTips.pro Articles

• Bash Scripting Basics: Your First Scripts
• System Performance Monitoring with top and htop
• Introduction to Ansible for Linux Automation
• Linux Performance Troubleshooting Methodology

---

Conclusion

Linux monitoring scripts automation represents the cornerstone of proactive system administration, enabling administrators to maintain robust, reliable infrastructure through continuous surveillance and instant alerting. By implementing the comprehensive monitoring strategies outlined in this guide, you’ve acquired the knowledge to build sophisticated monitoring systems that prevent downtime, optimize performance, and ensure business continuity.

Remember that effective monitoring evolves with your infrastructure—regularly review thresholds, update alerting logic, and expand monitoring coverage as your systems grow. Start with the foundational scripts provided here, then customize them to match your specific operational requirements and organizational policies.

The investment in automated monitoring pays immediate dividends through reduced incident response times, improved system availability, and enhanced operational visibility across your entire Linux infrastructure.

Ready to take your monitoring to the next level? Explore our advanced guide on Setting up Prometheus and Grafana for enterprise-grade monitoring dashboards and visualization.

---

Last Updated: October 2025 | Article #50 in the Linux Mastery 100 Series