{"id":10,"date":"2007-12-24T04:02:29","date_gmt":"2007-12-24T08:02:29","guid":{"rendered":"http:\/\/www.microworkshop.com\/WordPress\/?p=10"},"modified":"2012-12-01T19:44:10","modified_gmt":"2012-12-02T00:44:10","slug":"linux-unix-monitoring-the-operating-system-memory-cpu-hard-drive-performance-and-other-resources","status":"publish","type":"post","link":"https:\/\/microdevsys.com\/wp\/linux-unix-monitoring-the-operating-system-memory-cpu-hard-drive-performance-and-other-resources\/","title":{"rendered":"Linux \/ UNIX: Monitoring the operating system memory, cpu, hard drive, performance and other resources."},"content":{"rendered":"

\n\tOne of the more powerfull features of any Linux\/Unix system are the monitoring features and capabilities of the OS. UNIX<\/strong> systems in general have been around for decades and in that time the tools available to users and admins alike on both systems have grown tramendously. In fact, the log files, tools and applications under UNIX \/ Linux<\/strong> allow you to really drill down into the heart of the operating system and diagnose a problem with the potential to even fix the issue yourself with the right know how or if the problem isn't really complicated.  Further, the open source community has grown tramendously recently so finding support online for errors you find is no longer as difficult as it originally was.  And support is no longer esoteric to developers either.  \ud83d\ude42\n<\/p>\n

\n\tIn this case we will look at all the associated Linux commands that can help you identify, debug and possibly even fix potential problems:\n<\/p>\n

\n\t
\n\tHere is a list of some of the more useful tools and a brief introduction to using them:\n<\/p>\n

    \n
  1. \n\t\tLog files under ‘\/var\/log’<\/strong>\n\t<\/li>\n
  2. \n\t\tNetwork monitoring ‘netstat’<\/strong> command.\n\t<\/li>\n
  3. \n\t\tProcess table lookup using ‘ps’<\/strong>\n\t<\/li>\n
  4. \n\t\tRealtime resource monitoring using ‘top’<\/strong>\n\t<\/li>\n
  5. \n\t\tSimple system health checker using ‘uptime’<\/strong>\n\t<\/li>\n
  6. \n\t\tDisk information and settings using ‘hdparm’<\/strong>\n\t<\/li>\n
  7. \n\t\tDisk temperature checking using ‘hddtemp’<\/strong>\n\t<\/li>\n
  8. \n\t\tDisk SMART monitoring using ’smartctl’<\/strong>\n\t<\/li>\n
  9. \n\t\tMemory checking using ‘free’<\/strong> & ‘cat \/proc\/meminfo’<\/strong>\n\t<\/li>\n
  10. \n\t\tView process tree using ‘pstree’<\/strong>\n\t<\/li>\n
  11. \n\t\tUNIX \/ Linux concept of Load<\/em>.\n\t<\/li>\n
  12. \n\t\tSystem \/ boot messages using ‘dmesg’<\/strong>\n\t<\/li>\n
  13. \n\t\tSystem Activity Reporting using ’sar’<\/strong>\n\t<\/li>\n
  14. \n\t\tLinux \/ UNIX NIC (Network Card) statistics using ‘ifconfig’<\/strong>\n\t<\/li>\n
  15. \n\t\tProcessor statistics using ‘mpstat’<\/strong>\n\t<\/li>\n
  16. \n\t\tDisk statistics using ‘iostat’<\/strong>\n\t<\/li>\n
  17. \n\t\tMemory statistics using ‘vmstat’<\/strong>\n\t<\/li>\n
  18. \n\t\tList Open Files using ‘lsof’<\/strong>\n\t<\/li>\n
  19. \n\t\tLinux \/ UNIX Program Core<\/strong> (Dump) files.\n\t<\/li>\n
  20. \n\t\tEXAMPLE!<\/strong>\n\t<\/li>\n<\/ol>\n

    \n\tGenerally ‘\/var\/log\/’<\/strong> is where Linux\/UNIX systems store log files. This is especially true for old applications, especially written by Linux\/UNIX core developers. In recent years, however, many developers, especially those coming from the Windows development world, have seen these log files and folders being placed under the application path folders. (NOTE<\/span>: Unix systems compartmentalize their files by type rather then application as Windows does. In Windows applications are typically installed in this manner: ‘C:\\Program Files\\Application\\All files for application go here’<\/strong>. Under UNIX you have ‘\/etc\/’<\/strong> for config files of the application, ‘\/var\/log’<\/strong> for log files application generates when running, ‘\/bin\/’<\/strong> and ‘\/usr\/bin’<\/strong> where the executable files of the application reside, for all applications. So under your Linux box, all log files will be under ‘\/var\/log’<\/strong> including the most important system log files:\n<\/p>\n

      \n
    1. \n\t\t\/var\/log\/messages<\/strong> – Main Linux \/ Unix system log file. Holds the bulk of the system log information.\n\t<\/li>\n
    2. \n\t\t\/var\/log\/secure<\/strong> – Main secure log file. Saves network \/ firewall and other NET related log information.\n\t<\/li>\n
    3. \n\t\t\/var\/log\/dmsg<\/strong> – BOOT log file. Essentially what happened during boot up time. Great for first diagnosis when something doesn’t work during boot up.\n\t<\/li>\n
    4. \n\t\t\/var\/log\/maillog<\/strong> – If you use sendmail or any other mail server\/service on UNIX this log file will hold much of the information you need.\n\t<\/li>\n
    5. \n\t\t\/var\/log\/acpid<\/strong> – Power state and power management log file.\n\t<\/li>\n
    6. \n\t\t\/var\/log\/cron<\/strong> – Scheduled cron jobs and any jobs that have been scheduled to run through the Unix \/ Linux ‘cron’ utility job scheduler.\n\t<\/li>\n<\/ol>\n

      \n\tOf course, the log files are only as good as the application writing to them. If the application doesn’t write log files detailed enough to diagnose problems, obviously, the log files won’t help you much when you run into issues.  The above utilities\/locations will tell you plenty about your system health. One command line terminal program that I find usefull in checking into my system workings is ‘konsole‘<\/strong>. This is just one variation of many dozens of GUI based Linux consoles<\/a>, however I find this one most useful due to it’s behavior like a browser allowing you to open multiple windows to the system in tabs and that's 'konsole'<\/strong> is scriptable.\n<\/p>\n

      \n\t\n<\/p>\n

      \n\tNETSTAT<\/strong><\/u>
      \n\tNetstat is an extensive network traffic diagnostic utility printing local\/foreign IP addresses port and other connections within and from outside to your workstation. The most common I use is ‘netstat -apntee‘<\/strong> which prints information as below:\n<\/p>\n

      \n\t$ netstat -apntee<\/strong>
      \n\tActive Internet connections (servers and established)
      \n\tProto Recv-Q Send-Q Local Address Foreign Address State User Inode PID\/Program name
      \n\ttcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 0 6535 2076\/portmap
      \n\ttcp 0 0 127.0.0.1:50000 0.0.0.0:* LISTEN 0 6965 2228\/hpiod
      \n\ttcp 0 0 127.0.0.1:50002 0.0.0.0:* LISTEN 0 6997 2233\/python
      \n\ttcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 0 7112 2277\/vsftpd
      \n\ttcp 0 0 192.168.1.1:53 0.0.0.0:* LISTEN 25 6500 2058\/named
      \n\ttcp 0 0 192.168.0.4:53 0.0.0.0:* LISTEN 25 6498 2058\/named
      \n\ttcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 25 6496 2058\/named
      \n\ttcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 0 7019 2244\/cupsd
      \n\ttcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 0 7204 2306\/sendmail: acce
      \n\ttcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 25 6503 2058\/named
      \n\ttcp 68 0 192.168.0.4:55890 74.208.30.232:21 CLOSE_WAIT 0 10234 2883\/gftp-gtk
      \n\ttcp 0 0 :::80 :::* LISTEN 0 7286 2328\/httpd
      \n\ttcp 0 0 :::22 :::* LISTEN 0 7054 2256\/sshd
      \n\t$\n<\/p>\n

      \n\tHere is the brakedown of what the values mean (You can get more detailed meaning form using ‘netstatp –help‘<\/strong> or ‘man netstat‘<\/strong>)\n<\/p>\n

      \n\t‘-n‘<\/strong> This means print entries in numeric format (IP addresses instead of host numbers)
      \n\t‘-a‘<\/strong> Show listening and non listening sockets.
      \n\t‘-p‘<\/strong> Show program name and PID of process owning the connection.
      \n\t‘-t‘<\/strong> Do not trim long addresses (Do NOT truncate on output).
      \n\t‘-e‘<\/strong> Print additional detailed information. Use ‘-ee‘ for maximum detail.\n<\/p>\n

      \n\tThe ‘-c‘<\/strong> option with ‘netstat‘<\/strong> is also useful if you want to display the information on the screen and have it refreshed periodically. An alternative is to use ‘watch -n 5 “netstat -apntee”‘<\/strong> instead which will refresh output and run ‘netstat -apntee‘<\/strong> every 5 seconds per the ‘-n‘<\/strong> option. This utility is very useful if you want to scan your system for connections coming into your system and coming out. This is a very useful command to use in the hosting industry.<\/p>\n

      \tPS<\/strong><\/u>
      \n\tThis is probably one of the more usefull and powerfull UNIX commands available and virtually combines many most common and useful features of other Linux commands.  However, it's power is often overlooked, and only simple variants of this command are used. This command checks the process table on a Unix host and prints information on it. The command can show you everything running on a host and often what is going on with a process, how it started, with what commands it started with and from where to list just a few of the details it can produce on a process running on a UNIX \/ Linux os. The best way to see what it can do is to actually view a few examples of the command:\n<\/p>\n

      \n\t‘ps -axfwweo pid,user,cmd=‘<\/strong> Print ALL processes (-a<\/strong>), processes without controlling tty’s (-x<\/strong>), show parent-child relationship (-f<\/strong>), print extended information (wide format) (-w<\/strong>), show environment (-e<\/strong>) and define\/customize output as follows:
      \n\t‘pid‘<\/strong> Process PID.
      \n\t‘user‘<\/strong> User that ran the process.
      \n\t‘cmd=‘<\/strong> Command used to run process with.\n<\/p>\n

      \n\tHere are a few more variations you may find usefull.\n<\/p>\n

      \n\t‘ps -axfeo pid,user,pcpu,pmem,ppid,psr,etime,cputime,cp,nice,rtprio,state,vsz,size,cmd=‘<\/strong>\n<\/p>\n

      \n\tThe new options to ‘-o‘<\/strong> for above command include:\n<\/p>\n

      \n\t‘pmem‘<\/strong> Percent of memory used by a process.
      \n\t‘pcpu‘<\/strong> Percent of CPU used by a process.
      \n\t‘ppid‘<\/strong> Parent process ID that started\/spawned this process.
      \n\t‘psr‘<\/strong> Processor the process is running on.
      \n\t‘etime‘<\/strong> Elapsed time since process was running.
      \n\t‘cputime‘<\/strong> Cumulative CPU time.
      \n\t‘cp‘<\/strong> Per millisecond CPU usage.
      \n\t‘nice‘<\/strong> Priority with which the process is running.
      \n\t‘rtprio‘<\/strong> Real time priority.
      \n\t‘state‘<\/strong> The state the process is in (From ‘man‘ pages):\n<\/p>\n

      \n\tD<\/strong> Uninterruptible sleep (usually IO)
      \n\tR<\/strong> Running or runnable (on run queue)
      \n\tS<\/strong> Interruptible sleep (waiting for an event to complete)
      \n\tT<\/strong> Stopped, either by a job control signal or because it is being traced.
      \n\tW<\/strong> paging (not valid since the 2.6.xx kernel)
      \n\tX<\/strong> dead (should never be seen)
      \n\tZ<\/strong> Defunct (”zombie”) process, terminated but not reaped by its parent.\n<\/p>\n

      \n\t‘vsz‘<\/strong> Virtual memory size of process in KiB (In multiples of 1024)
      \n\t‘size‘<\/strong> Memory size in KiB (In multiples of 1024) NOTE: This number is an estimate and is very rough and therefore not exact.\n<\/p>\n

      \n\t‘ps‘<\/strong> has an extensive man page (Type ‘man ps’ for a full list of options possible). An example from the man pages is:\n<\/p>\n

      \n\t‘ps -eo euser,ruser,suser,fuser,f,comm,label‘<\/strong>\n<\/p>\n

      \n\twhich returns the security level of running processes. The ‘ps‘<\/strong> utility is one of the utilities you can use to drill down or get very detailed information on a system problem short of doing a memory dump on a process that may be causing you issues.\n<\/p>\n

      \n\tTOP<\/strong><\/u>
      \n\tIn it’s simplest form ‘top‘<\/strong> ran without parameters, will give you process information and top system resource users for your system. A parametarized version is ‘top -cbn1‘<\/strong> which will print a single snapshot<\/em> of a top output can be used if you do not need to see constantly updated information. Once in ‘top’<\/strong> press ‘?’<\/strong> to get a list of commands you can use with ‘top’<\/strong> while it is running. Use ‘man top‘<\/strong> for command line option list or ‘top –help‘<\/strong> for a brief list of options.\n<\/p>\n

      \n\tUPTIME<\/strong><\/u>
      \n\tThis command, unlike the above, has only one option, -V<\/strong> to get version, and is usually ran without options on a command line to get a health report of a Unix system in the form of a ‘load average’<\/strong> number. The command is usefull for early and quick reporting of system health on systems either too busy or when other utilities fail to run sufficiently quickly.\n<\/p>\n

      \n\tHDPARM<\/u><\/strong>
      \n\tShow\/set drive information\/settings respectively. To get read statistics on a drive to find out how busy it is, use ‘hdparm -tT <DEVICE>‘<\/strong>. To get drive information including to check ‘udma‘<\/strong> settings use ‘hdparm -i <DEVICE>‘<\/strong>. To set or change a ‘udma‘<\/strong> setting you can use something like this ‘hdparm -Xudma5 -d1 <DEVICE>‘<\/strong>. <DEVICE><\/strong> stands in for ‘\/dev\/hda‘<\/strong>, ‘\/dev\/hdb‘<\/strong> etc. or the actual hard drive you wish to check on on your system.\n<\/p>\n

      \n\tHDDTEMP<\/u><\/strong>
      \n\tShow\/report drive temperature (if drive has sensor for temperature to begin with). Example run without parameters:\n<\/p>\n

      \n\t$ hddtemp \/dev\/hdb<\/strong>
      \n\t\/dev\/hdb: WDC WD1200JB-00GVA0: 42°C
      \n\t$\n<\/p>\n

      \n\tUse ‘hddtemp –help‘<\/strong> for more available options.\n<\/p>\n

      \n\tSMARTCTL<\/u><\/strong>
      \n\tReport on \/ show S.M.A.R.T. controller information\/settings for a drive. Use ‘smartctl -a <DEVICE>‘<\/strong> where <DEVICE><\/strong> is the actual name of your device you wish to check on. This also prints all errors the device has encountered so is a very useful tool in determining if you should be replacing a hard drive soon. In my case it reported 32 errors on a drive telling me I will need to be changing it soon. \ud83d\ude42\n<\/p>\n

      \n\tFREE Memory using ‘free’ & ‘cat \/PROC\/MEMINFO’<\/strong><\/u>
      \n\tThe command ‘free‘<\/strong> printed system memory usage including swap space utilization:\n<\/p>\n

      \n\t$ free<\/strong>
      \n\ttotal used free shared buffers cached
      \n\tMem: 515716 445576 70140 0 38368 176252
      \n\t-\/+ buffers\/cache: 230956 284760
      \n\tSwap: 2530196 0 2530196
      \n\t$\n<\/p>\n

      \n\tAlternately, typing ‘cat \/proc\/meminfo‘ displays detailed memory information. This holds more detailed memory usage analysis\/brake down then does ‘free‘ including categorization as is done on typical UNIX \/ Linux hosts.\n<\/p>\n

      \n\tPSTREE<\/u><\/strong>
      \n\t‘pstree‘<\/strong> in it’s simplest format prints commands and their parent-child relationship to each other. This command is usefull if there is a runaway process spawning too many instances of another process. The command has a limited subset of options available by running ‘pstree –help‘<\/strong>.  It is a simplistic command primarily designed to show process associations.\n<\/p>\n

      \n\tLOAD<\/strong><\/u>
      \n\tThis is not a command but a concept. Load is an index starting at 0.00<\/strong> that indicates system activity. On a clean system this should not go over 1.00<\/strong> however in some applications, this value may be tolerated when significantly higher. Values of 1 generally means users of a system will see noticeable delays in their tasks. One command that prints load is ‘uptime‘<\/strong> typed without parameters that was discussed earlier.<\/p>\n

      \tDMESG<\/strong><\/u>
      \n\tTyping ‘dmesg‘<\/strong> on a command line prints boot up error\/messages encountered when the system was started up.\n<\/p>\n

      \n\tSAR<\/strong><\/u>
      \n\t‘sar‘<\/strong> is a System Activity Reporting tool\/command. Typed without parameters, it shows analysis of system processes and resource usage history. This is especially useful when determining the root causes of a slowdown on a Unix\/Linux host. As with ‘ps‘<\/strong> this utility has an extensive documentation and comes with many options. A variation is:\n<\/p>\n

      \n\t$ sar -n DEV<\/strong>\n<\/p>\n

      \n\tWhich shows network activity on a host for a range of time. ‘sar‘ is used extensively in the industry and has a detailed man page of available options.\n<\/p>\n

      \n\tIFCONFIG<\/u><\/strong>
      \n\tShows network card (NIC \/ Network Interface Card) statistics and settings including errors on the card. Example:\n<\/p>\n

      \n\t$ ifconfig<\/strong>
      \n\tlo Link encap:Local Loopback
      \n\tinet addr:127.0.0.1 Mask:255.0.0.0
      \n\tinet6 addr: ::1\/128 Scope:Host
      \n\tUP LOOPBACK RUNNING MTU:16436 Metric:1
      \n\tRX packets:1931 errors:0 dropped:0 overruns:0 frame:0
      \n\tTX packets:1931 errors:0 dropped:0 overruns:0 carrier:0
      \n\tcollisions:0 txqueuelen:0
      \n\tRX bytes:3039216 (2.8 MiB) TX bytes:3039216 (2.8 MiB)
      \n\t$\n<\/p>\n

      \n\tThis is especially usefull when you are experiencing network issues. If there are problems with your network card, the errors can be visible after typing this command.\n<\/p>\n

      \n\tMPSTAT<\/strong><\/u>
      \n\tShows processor\/CPU activity on a host for a specified time. ‘mpstat‘<\/strong> without parameters shows the current snapshot of CPU related usage information. A variation of the command is ‘mpstat –a 1 1‘<\/strong>. Basic usage information is:\n<\/p>\n

      \n\tUsage: mpstat [ options… ] [ [ ] ]\n<\/p>\n

      \n\tIOSTAT<\/strong><\/u>
      \n\t‘iostat‘<\/strong> works similarly to ‘mpstat‘<\/strong> but reports usage statistics on a drive. ‘iostat \/dev\/hdb ALL 1 1′<\/strong> shows more detailed information taken at interval of 1<\/strong> second. Basic usage and options are:\n<\/p>\n

      \n\t[ -c | -d ] [ -k | -m ] [ -t ] [ -V ] [ -x ]
      \n\t[ [ … ] | ALL ] [ -p [ | ALL ] ]\n<\/p>\n

      \n\tMore detailed information is available by ‘man iostat‘, ‘info iostat‘ or ‘iostat –help’.\n<\/p>\n

      \n\tVMSTAT<\/strong><\/u>
      \n\t‘vmstat‘<\/strong> works similar to ‘iostat‘<\/strong> however reports information on memory utilization. Without any options ‘vmstat‘ <\/strong>prints memory utilization and may be considered more accurate in it’s reporting then ‘free‘<\/strong>. A number of options are available and can be seen by running ‘man vmstat‘<\/strong>.\n<\/p>\n

      \n\tLSOF<\/strong><\/u>
      \n\t‘lsof‘<\/strong> is short for ‘ls’<\/strong> (list) Open Files. This command shows all currently open files on a host. This command is ideal if you want to know which process has what files open. This is especially useful in identifying commands or files being used outside their allowed locations or command using old outdated files that could be causing problems. This command is very useful since it links PID or processes with files they have open, information which is not readily available from other commands. ‘lsof‘<\/strong> has a huge number of parameters that rivals ‘ps‘<\/strong>.\n<\/p>\n

      \n\tCORE FILES<\/u><\/strong>
      \n\tThis feature of UNIX \/ Linux systems is essentially a memory\/library dump of running jobs that could be causing issues. The core files are extensively use in pin pointing problems in crashing or hung applications and are essential to diagnosis by developers. The topic of core files<\/strong> is far too extensive to cover here and will be covered in a separate topic in the future.\n<\/p>\n

      \n\tEXAMPLE<\/u><\/strong>
      \n\tAs an example of a case that you may run into, which coincidently I have ran into on Linux distros as far back as I can remember is with the Flash Plug in technology and available browsers in Linux. Once you install the flash plugin following the directions in the readme file that comes with the Flash plugin, you’ll, unfortunately, notice the plugin is significantly less powerfull for Linux then Windows. For one, the version of flash available for Linux will be older then the latest one available for Windows. The second limitation, is that the other player from Adobe, Shockwave, is not available for FireFox at the time of this writing. So often enough, you’ll run into situations online where you can’t view the latest flash of a site despite the fact that you have flash installed. And here is where the issue may popup for you on your Linux distribution. At the point where flash is starting up, FirsFox would repeatedly freeze on me to the point where it’s unusuable. Coincidently, so does my entire system. Since this happened too often for me, I followed this procedure:\n<\/p>\n