Linux / UNIX: Monitoring the operating system memory, cpu, hard drive, performance and other resources.
One of the more powerfull features of any Linux/Unix system are the monitoring features and capabilities of the OS. UNIX systems in general have been around for decades and in that time the tools available to users and admins alike on both systems have grown tramendously. In fact, the log files, tools and applications under UNIX / Linux allow you to really drill down into the heart of the operating system and diagnose a problem with the potential to even fix the issue yourself with the right know how or if the problem isn't really complicated. Further, the open source community has grown tramendously recently so finding support online for errors you find is no longer as difficult as it originally was. And support is no longer esoteric to developers either. :)
In this case we will look at all the associated Linux commands that can help you identify, debug and possibly even fix potential problems:
Here is a list of some of the more useful tools and a brief introduction to using them:
- Log files under ‘/var/log’
- Network monitoring ‘netstat’ command.
- Process table lookup using ‘ps’
- Realtime resource monitoring using ‘top’
- Simple system health checker using ‘uptime’
- Disk information and settings using ‘hdparm’
- Disk temperature checking using ‘hddtemp’
- Disk SMART monitoring using ’smartctl’
- Memory checking using ‘free’ & ‘cat /proc/meminfo’
- View process tree using ‘pstree’
- UNIX / Linux concept of Load.
- System / boot messages using ‘dmesg’
- System Activity Reporting using ’sar’
- Linux / UNIX NIC (Network Card) statistics using ‘ifconfig’
- Processor statistics using ‘mpstat’
- Disk statistics using ‘iostat’
- Memory statistics using ‘vmstat’
- List Open Files using ‘lsof’
- Linux / UNIX Program Core (Dump) files.
Generally ‘/var/log/’ is where Linux/UNIX systems store log files. This is especially true for old applications, especially written by Linux/UNIX core developers. In recent years, however, many developers, especially those coming from the Windows development world, have seen these log files and folders being placed under the application path folders. (NOTE: Unix systems compartmentalize their files by type rather then application as Windows does. In Windows applications are typically installed in this manner: ‘C:\Program Files\Application\All files for application go here’. Under UNIX you have ‘/etc/’ for config files of the application, ‘/var/log’ for log files application generates when running, ‘/bin/’ and ‘/usr/bin’ where the executable files of the application reside, for all applications. So under your Linux box, all log files will be under ‘/var/log’ including the most important system log files:
- /var/log/messages – Main Linux / Unix system log file. Holds the bulk of the system log information.
- /var/log/secure – Main secure log file. Saves network / firewall and other NET related log information.
- /var/log/dmsg – BOOT log file. Essentially what happened during boot up time. Great for first diagnosis when something doesn’t work during boot up.
- /var/log/maillog – If you use sendmail or any other mail server/service on UNIX this log file will hold much of the information you need.
- /var/log/acpid – Power state and power management log file.
- /var/log/cron – Scheduled cron jobs and any jobs that have been scheduled to run through the Unix / Linux ‘cron’ utility job scheduler.
Of course, the log files are only as good as the application writing to them. If the application doesn’t write log files detailed enough to diagnose problems, obviously, the log files won’t help you much when you run into issues. The above utilities/locations will tell you plenty about your system health. One command line terminal program that I find usefull in checking into my system workings is ‘konsole‘. This is just one variation of many dozens of GUI based Linux consoles, however I find this one most useful due to it’s behavior like a browser allowing you to open multiple windows to the system in tabs and that's 'konsole' is scriptable.