The Dutch Prutser's Blog

By: Harald van Breederode

  • Disclaimer

    The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.
  • Subscribe

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 248 other followers

Understanding Linux Load Average – Part 2

Posted by Harald van Breederode on May 5, 2012

In part 1 we performed a series of experiments to explore the relation between CPU utilization and Linux load average. We came to the conclusion that CPU utilization clearly influences the load average. In part 2 we will continue our experiments and take a look if disk I/O also influences the Linux load average.

Disk I/O and load average

The first experiment is starting 2 processes performing disk I/O on an otherwise idle system to measure the amount I/O issued, the load average and CPU utilization using the sar command. BTW: My load-gen script uses the dd command to generate disk I/O.

$ load-gen disk 2
$ sar –b 30 6
Linux 2.6.32-300.20.1.el5uek (roger.example.com) 	04/21/2012

09:26:59 PM       tps      rtps      wtps   bread/s   bwrtn/s
09:27:29 PM  24881.59      0.00  24881.59      0.00  46776.38
09:27:59 PM  26894.34      0.00  26894.34      0.00  50553.80
09:28:29 PM  26772.19      0.10  26772.09      0.80  50327.21
09:28:59 PM  27366.10      0.00  27366.10      0.00  51438.92
09:29:29 PM  25126.06      0.20  25125.86      1.61  47241.72
09:29:59 PM  22815.88      0.00  22815.88      0.00  42897.59
Average:     25643.28      0.05  25643.23      0.40  48207.04

The -b command line option given to sar tells it to report disk I/O statistics. The above output tells us that on average 48207 blocks per second were written to disk and almost nothing was read. What effect does this have on the load average?

$ sar –q 30 6
Linux 2.6.32-300.20.1.el5uek (roger.example.com) 	04/21/2012

09:26:59 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
09:27:29 PM         4       293      0.89      1.70      1.75
09:27:59 PM         3       293      1.33      1.73      1.76
09:28:29 PM         1       293      1.59      1.75      1.77
09:28:59 PM         3       293      1.89      1.81      1.78
09:29:29 PM         3       293      1.93      1.83      1.79
09:29:59 PM         2       291      2.02      1.86      1.80
Average:            3       293      1.61      1.78      1.77

The run-queue utilization varies because a process is not on the run-queue while it is waiting for its I/O to complete. However the 1-minute load average climbs towards the number of processes issuing disk I/O, 2 in this case, which is a clear indication that performing disk I/O influences the load average. Before jumping to conclusions too soon we also have to take a look at the CPU utilization.

$ sar –P ALL –u 30 6
Linux 2.6.32-300.20.1.el5uek (roger.example.com) 	04/21/2012

09:26:59 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
09:27:29 PM       all      0.89      0.00     27.62     58.21      0.00     13.28
09:27:29 PM         0      0.93      0.00     41.46     57.57      0.00      0.03
09:27:29 PM         1      0.84      0.00     14.28     58.81      0.00     26.07

09:27:29 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
09:27:59 PM       all      0.76      0.00     30.93     59.04      0.00      9.27
09:27:59 PM         0      0.70      0.00     45.57     53.63      0.00      0.10
09:27:59 PM         1      0.82      0.00     16.59     64.37      0.00     18.22

09:27:59 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
09:28:29 PM       all      0.84      0.00     30.19     58.33      0.00     10.64
09:28:29 PM         0      0.70      0.00     44.67     54.56      0.00      0.07
09:28:29 PM         1      0.97      0.00     16.25     61.96      0.00     20.82

09:28:29 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
09:28:59 PM       all      0.79      0.00     30.48     57.33      0.00     11.39
09:28:59 PM         0      0.67      0.00     45.35     53.98      0.00      0.00
09:28:59 PM         1      0.94      0.00     16.47     60.48      0.00     22.10

09:28:59 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
09:29:29 PM       all      0.81      0.00     26.43     55.30      0.00     17.46
09:29:29 PM         0      0.70      0.00     40.33     58.93      0.00      0.03
09:29:29 PM         1      0.90      0.00     13.53     51.93      0.00     33.64

09:29:29 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
09:29:59 PM       all      0.62      0.00     22.09     53.08      0.00     24.21
09:29:59 PM         0      0.57      0.00     35.11     64.33      0.00      0.00
09:29:59 PM         1      0.64      0.00     10.17     42.84      0.00     46.35

Average:          CPU     %user     %nice   %system   %iowait    %steal     %idle
Average:          all      0.78      0.00     27.93     56.86      0.00     14.43
Average:            0      0.71      0.00     42.08     57.16      0.00      0.04
Average:            1      0.85      0.00     14.51     56.58      0.00     28.06

On average the CPU was about 29% busy, 14% idle and 57% was spent waiting for I/O completion (IOWAIT). During IOWAIT the system is actually idle but because there is at least 1 outstanding I/O it is reported as IOWAIT. IOWAIT is a common misunderstood CPU state! See the mpstat manual page for a good description of the various CPU states on a Linux system.

What does top have to report?

top - 21:30:00 up 11:14,  1 user,  load average: 2.02, 1.86, 1.80
Tasks: 191 total,   1 running, 190 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.6%us, 15.5%sy,  0.0%ni, 24.3%id, 53.1%wa,  0.9%hi,  5.7%si,  0.0%st
Mem:   3074820k total,  2550432k used,   524388k free,   220404k buffers
Swap:  5144568k total,        0k used,  5144568k free,  1160732k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
27576 hbreeder  20   0 63168  612  512 D 13.9  0.0   0:04.19 dd                 
27575 hbreeder  20   0 63168  612  512 D 13.1  0.0   0:03.95 dd                 
27542 hbreeder  20   0 12756 1188  836 R  0.1  0.0   0:00.23 top                 

The top output above confirms that although the CPU utilization is quite low, the load average is about 2. Since there was nothing else running on my system during this period we can safely conclude that the Linux load average is also influenced by the number of processes issuing disk I/O.

Demystifying IOWAIT

As stated above IOWAIT is basically the same as idle time. The only difference is that during IOWAIT there is at least 1 outstanding I/O. Had there been another process (or thread) waiting for the CPU it would use the CPU and the IOWAIT would be reported as one of the other CPU states such as USER, NICE or SYSTEM. We can easily prove this by performing another experiment by starting 2 CPU load processes while the 2 disk load processes are still running.

$ load-gen cpu 2
$ sar –P ALL –u 30 6
Linux 2.6.32-300.20.1.el5uek (roger.example.com) 	04/21/2012

09:30:00 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
09:30:30 PM       all     63.29      0.00     36.41      0.30      0.00      0.00
09:30:30 PM         0     30.11      0.00     69.29      0.60      0.00      0.00
09:30:30 PM         1     96.47      0.00      3.53      0.00      0.00      0.00

09:30:30 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
09:31:00 PM       all     61.33      0.00     38.52      0.15      0.00      0.00
09:31:00 PM         0     28.28      0.00     71.42      0.30      0.00      0.00
09:31:00 PM         1     94.40      0.00      5.60      0.00      0.00      0.00

09:31:00 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
09:31:30 PM       all     62.60      0.00     37.40      0.00      0.00      0.00
09:31:30 PM         0     29.67      0.00     70.33      0.00      0.00      0.00
09:31:30 PM         1     95.53      0.00      4.47      0.00      0.00      0.00

09:31:30 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
09:32:00 PM       all     64.29      0.00     35.71      0.00      0.00      0.00
09:32:00 PM         0     30.84      0.00     69.16      0.00      0.00      0.00
09:32:00 PM         1     97.70      0.00      2.30      0.00      0.00      0.00

09:32:00 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
09:32:30 PM       all     63.77      0.00     36.23      0.00      0.00      0.00
09:32:30 PM         0     31.40      0.00     68.60      0.00      0.00      0.00
09:32:30 PM         1     96.13      0.00      3.87      0.00      0.00      0.00

09:32:30 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
09:33:00 PM       all     63.49      0.00     36.51      0.00      0.00      0.00
09:33:00 PM         0     31.97      0.00     68.03      0.00      0.00      0.00
09:33:00 PM         1     95.00      0.00      5.00      0.00      0.00      0.00

Average:          CPU     %user     %nice   %system   %iowait    %steal     %idle
Average:          all     63.13      0.00     36.80      0.08      0.00      0.00
Average:            0     30.38      0.00     69.47      0.15      0.00      0.00
Average:            1     95.87      0.00      4.13      0.00      0.00      0.00

Voila, the IOWAIT is gone! If one of the disk I/O load processes has to wait for I/O completion, one of the CPU load processes can use the CPU and therefore the system does not enter the IOWAIT state. Isn’t that a nice way to “tune” a system that is “suffering” high amounts of IOWAIT? ;-)

To further prove that disk I/O is just another factor that influences the Linux load average, just as CPU utilization does, we can take a look at the sar run-queue statistics:

$ sar –q 30 6
Linux 2.6.32-300.20.1.el5uek (roger.example.com) 	04/21/2012

09:30:00 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
09:30:30 PM         4       295      2.80      2.06      1.87
09:31:00 PM         4       295      3.27      2.24      1.93
09:31:30 PM         4       295      3.70      2.44      2.01
09:32:00 PM         4       295      3.82      2.59      2.07
09:32:30 PM         4       295      3.89      2.72      2.13
09:33:00 PM         4       289      3.93      2.84      2.19
Average:            4       294      3.57      2.48      2.03

The load average has increased by 2. Hopefully this matches your expectation. If not, please read part 1 again!

Does the current load average have an impact on how well the system is performing? We can check this by the amount of I/O the system is able to perform using sar:

$ sar –b 30 6
Linux 2.6.32-300.20.1.el5uek (roger.example.com) 	04/21/2012

09:30:00 PM       tps      rtps      wtps   bread/s   bwrtn/s
09:30:30 PM  19960.52      0.00  19960.52      0.00  37556.49
09:31:00 PM  20967.83      0.00  20967.83      0.00  39447.60
09:31:30 PM  20959.73      0.00  20959.73      0.00  39421.50
09:32:00 PM  20634.82      0.10  20634.72      0.80  38803.77
09:32:30 PM  20117.79      0.00  20117.79      0.00  37849.28
09:33:00 PM  19797.07      0.10  19796.97      0.80  37237.61
Average:     20406.31      0.03  20406.28      0.27  38386.08

The above output shows that, under the current load average of 4, on average the system is able to write 38386 blocks per second to disk. This is considerable less compared to the average of 48207 blocks per second we were able to write before with a load average of 2.

We will take a look at the corresponding top output before we come to a conclusion.

top - 21:33:01 up 11:17,  1 user,  load average: 3.93, 2.84, 2.19
Tasks: 195 total,   5 running, 190 sleeping,   0 stopped,   0 zombie
Cpu(s): 63.7%us, 19.6%sy,  0.0%ni,  0.0%id,  0.0%wa,  3.0%hi, 13.8%si,  0.0%st
Mem:   3074820k total,  2553332k used,   521488k free,   221216k buffers
Swap:  5144568k total,        0k used,  5144568k free,  1160828k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
27581 hbreeder  20   0 63836 1064  908 R 82.3  0.0   1:47.34 busy-cpu           
27582 hbreeder  20   0 63836 1068  908 R 59.4  0.0   2:18.79 busy-cpu           
27621 hbreeder  20   0 63168  616  512 R 23.9  0.0   0:07.19 dd                 
27620 hbreeder  20   0 63168  616  512 R 23.8  0.0   0:07.14 dd                 
27588 hbreeder  20   0 12756 1192  836 R  0.0  0.0   0:00.13 top                

The above top output confirms that the load average is about 4, the CPUs are fully utilized and that there is no IOWAIT. It also shows the processes that are active on the system.

Conclusion

So far we have proven that both CPU utilization and disk I/O influences the load average on a Linux system. The question remains if they are the only factors driving the load average? Or are there other factors to consider?
Stay tuned for part three to get the answer!
-Harald

About these ads

9 Responses to “Understanding Linux Load Average – Part 2”

  1. Admin said

    Hi Harald,

    Thanks for the nice topic. I have two findings. The first is that, while running only the two processes performing disk I/O, the sar output shows the runq-sz varying from 1 to 4. I should expect max 2 processes on the run queue.

    The second is that, while running both the cpu and I/O bound scripts, the runq-sz is constantly on 4 processes on the run queue. So the I/O based processes seem to be on the run queue all the time

    Best regards,
    Marco

    • Harald van Breederode said

      Hi Marco,

      The dd processes generating the disk I/O are driven by a shell script similar to:
      while :
      dd of=file.$$ …
      rm file.$$
      done
      So there are actually 4 processes involvedduring the first experiment, which explains why the run-queue utilization varies between 1 and 4.

      • Jenny said

        Hi Harald,

        This is very nice article to explain the Load average concept. Thanks for sharing this. But I have question that why on the load average of 4 the number of writing blocks are less than the one on load average of 2.

        Thanks,

  2. izzol said

    Because the Linux Scheduler is fuckup :) Try the same on FreBSD for example :)

  3. Reblogged this on Raheel's Blog and commented:
    Interesting series Part 2: Understanding Linux Load Average

  4. […] Understanding Linux Load Average – Part 2 […]

  5. […] Understanding Linux Load Average 谢谢 @jametong 参考:part1 part2 […]

  6. Rushi said

    Don’t you think disk I/O processes are contributing to run-queue length only because they too are utilizing CPU (14% in above example) to perform I/O?If not please explain why.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

Join 248 other followers

%d bloggers like this: