How to view, analyze, and rotate log files in Linux system log management?

Linux system log management is an important part of troubleshooting and security auditing. Mastering log analysis skills can quickly locate problems.

Log file locations:

• /var/log/messages: main system log (CentOS/RHEL)
• /var/log/syslog: system log (Debian/Ubuntu)
• /var/log/auth.log: authentication log (Debian/Ubuntu)
• /var/log/secure: security log (CentOS/RHEL)
• /var/log/kern.log: kernel log
• /var/log/boot.log: system boot log
• /var/log/dmesg: kernel boot messages
• /var/log/cron: scheduled task log
• /var/log/maillog: mail log
• /var/log/nginx/: Nginx log directory
• /var/log/apache2/: Apache log directory
• /var/log/mysql/: MySQL log directory

Log viewing tools:

• cat: view entire file content
• less: page view, supports scrolling up/down and searching
• tail: view end of file, tail -f for real-time viewing
• head: view beginning of file
• grep: search for specific content
• zcat: view compressed log files (.gz)
• zgrep: search compressed log files
• journalctl: systemd log viewing tool

Common journalctl commands:

• journalctl: view all logs
• journalctl -f: view logs in real-time
• journalctl -u service: view logs for a specific service
• journalctl -u service -f: view service logs in real-time
• journalctl --since today: view today's logs
• journalctl --since "2024-01-01" --until "2024-01-02": view logs for a specific time period
• journalctl -p err: view error-level logs
• journalctl -p warning: view warning-level logs
• journalctl -b: view logs for current boot
• journalctl -b -1: view logs for previous boot
• journalctl --disk-usage: view log disk usage
• journalctl --vacuum-size=1G: limit log size to 1GB

Log levels:

• emerg: emergency
• alert: alert
• crit: critical
• err: error
• warning: warning
• notice: notice
• info: information
• debug: debug

Log rotation (logrotate):

• Configuration files: /etc/logrotate.conf and /etc/logrotate.d/
• Configuration example:

  shell
  /var/log/nginx/*.log {
      daily
      missingok
      rotate 14
      compress
      delaycompress
      notifempty
      create 0640 www-data adm
      sharedscripts
      postrotate
          [ -f /var/run/nginx.pid ] && kill -USR1 `cat /var/run/nginx.pid`
      endscript
  }

• Manual execution: logrotate -f /etc/logrotate.conf
• Test configuration: logrotate -d /etc/logrotate.conf

Log analysis techniques:

• Count errors: grep -i error /var/log/syslog | wc -l
• Find logs for a specific time period: grep "2024-01-01" /var/log/syslog
• Find access from a specific IP: grep "192.168.1.1" /var/log/nginx/access.log
• Count most visited IPs: awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10
• Find failed login attempts: grep "Failed password" /var/log/auth.log
• Find system reboot records: last reboot

Log management best practices:

• Regularly clean old logs: use logrotate for automatic rotation
• Monitor log size: set alert thresholds
• Centralized log management: use ELK (Elasticsearch, Logstash, Kibana) or Graylog
• Log backup: regularly backup important logs
• Log permissions: ensure log file permissions are correct to prevent unauthorized access
• Structured logging: use JSON format for easier parsing and analysis

Remote log collection:

• rsyslog: system log service, supports remote log collection
• syslog-ng: powerful log management tool
• fluentd: unified log collection layer
• filebeat: lightweight log collector

Troubleshooting process:

1. Confirm the time when the problem occurred
2. View main system logs (/var/log/messages or /var/log/syslog)
3. View related service logs
4. Use grep to search for error information
5. Analyze log context
6. Combine with other monitoring tools (top, vmstat, iostat) for comprehensive judgment