Understanding Linux Load Average – Part 2
Posted by Harald van Breederode on May 5, 2012
In part 1 we performed a series of experiments to explore the relation between CPU utilization and Linux load average. We came to the conclusion that CPU utilization clearly influences the load average. In part 2 we will continue our experiments and take a look if disk I/O also influences the Linux load average.
Disk I/O and load average
The first experiment is starting 2 processes performing disk I/O on an otherwise idle system to measure the amount I/O issued, the load average and CPU utilization using the sar command. BTW: My load-gen script uses the dd command to generate disk I/O.
$ load-gen disk 2 $ sar –b 30 6 Linux 2.6.32-300.20.1.el5uek (roger.example.com) 04/21/2012 09:26:59 PM tps rtps wtps bread/s bwrtn/s 09:27:29 PM 24881.59 0.00 24881.59 0.00 46776.38 09:27:59 PM 26894.34 0.00 26894.34 0.00 50553.80 09:28:29 PM 26772.19 0.10 26772.09 0.80 50327.21 09:28:59 PM 27366.10 0.00 27366.10 0.00 51438.92 09:29:29 PM 25126.06 0.20 25125.86 1.61 47241.72 09:29:59 PM 22815.88 0.00 22815.88 0.00 42897.59 Average: 25643.28 0.05 25643.23 0.40 48207.04
The -b command line option given to sar tells it to report disk I/O statistics. The above output tells us that on average 48207 blocks per second were written to disk and almost nothing was read. What effect does this have on the load average?
$ sar –q 30 6 Linux 2.6.32-300.20.1.el5uek (roger.example.com) 04/21/2012 09:26:59 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 09:27:29 PM 4 293 0.89 1.70 1.75 09:27:59 PM 3 293 1.33 1.73 1.76 09:28:29 PM 1 293 1.59 1.75 1.77 09:28:59 PM 3 293 1.89 1.81 1.78 09:29:29 PM 3 293 1.93 1.83 1.79 09:29:59 PM 2 291 2.02 1.86 1.80 Average: 3 293 1.61 1.78 1.77
The run-queue utilization varies because a process is not on the run-queue while it is waiting for its I/O to complete. However the 1-minute load average climbs towards the number of processes issuing disk I/O, 2 in this case, which is a clear indication that performing disk I/O influences the load average. Before jumping to conclusions too soon we also have to take a look at the CPU utilization.
$ sar –P ALL –u 30 6 Linux 2.6.32-300.20.1.el5uek (roger.example.com) 04/21/2012 09:26:59 PM CPU %user %nice %system %iowait %steal %idle 09:27:29 PM all 0.89 0.00 27.62 58.21 0.00 13.28 09:27:29 PM 0 0.93 0.00 41.46 57.57 0.00 0.03 09:27:29 PM 1 0.84 0.00 14.28 58.81 0.00 26.07 09:27:29 PM CPU %user %nice %system %iowait %steal %idle 09:27:59 PM all 0.76 0.00 30.93 59.04 0.00 9.27 09:27:59 PM 0 0.70 0.00 45.57 53.63 0.00 0.10 09:27:59 PM 1 0.82 0.00 16.59 64.37 0.00 18.22 09:27:59 PM CPU %user %nice %system %iowait %steal %idle 09:28:29 PM all 0.84 0.00 30.19 58.33 0.00 10.64 09:28:29 PM 0 0.70 0.00 44.67 54.56 0.00 0.07 09:28:29 PM 1 0.97 0.00 16.25 61.96 0.00 20.82 09:28:29 PM CPU %user %nice %system %iowait %steal %idle 09:28:59 PM all 0.79 0.00 30.48 57.33 0.00 11.39 09:28:59 PM 0 0.67 0.00 45.35 53.98 0.00 0.00 09:28:59 PM 1 0.94 0.00 16.47 60.48 0.00 22.10 09:28:59 PM CPU %user %nice %system %iowait %steal %idle 09:29:29 PM all 0.81 0.00 26.43 55.30 0.00 17.46 09:29:29 PM 0 0.70 0.00 40.33 58.93 0.00 0.03 09:29:29 PM 1 0.90 0.00 13.53 51.93 0.00 33.64 09:29:29 PM CPU %user %nice %system %iowait %steal %idle 09:29:59 PM all 0.62 0.00 22.09 53.08 0.00 24.21 09:29:59 PM 0 0.57 0.00 35.11 64.33 0.00 0.00 09:29:59 PM 1 0.64 0.00 10.17 42.84 0.00 46.35 Average: CPU %user %nice %system %iowait %steal %idle Average: all 0.78 0.00 27.93 56.86 0.00 14.43 Average: 0 0.71 0.00 42.08 57.16 0.00 0.04 Average: 1 0.85 0.00 14.51 56.58 0.00 28.06
On average the CPU was about 29% busy, 14% idle and 57% was spent waiting for I/O completion (IOWAIT). During IOWAIT the system is actually idle but because there is at least 1 outstanding I/O it is reported as IOWAIT. IOWAIT is a common misunderstood CPU state! See the mpstat manual page for a good description of the various CPU states on a Linux system.
What does top have to report?
top - 21:30:00 up 11:14, 1 user, load average: 2.02, 1.86, 1.80 Tasks: 191 total, 1 running, 190 sleeping, 0 stopped, 0 zombie Cpu(s): 0.6%us, 15.5%sy, 0.0%ni, 24.3%id, 53.1%wa, 0.9%hi, 5.7%si, 0.0%st Mem: 3074820k total, 2550432k used, 524388k free, 220404k buffers Swap: 5144568k total, 0k used, 5144568k free, 1160732k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27576 hbreeder 20 0 63168 612 512 D 13.9 0.0 0:04.19 dd 27575 hbreeder 20 0 63168 612 512 D 13.1 0.0 0:03.95 dd 27542 hbreeder 20 0 12756 1188 836 R 0.1 0.0 0:00.23 top
The top output above confirms that although the CPU utilization is quite low, the load average is about 2. Since there was nothing else running on my system during this period we can safely conclude that the Linux load average is also influenced by the number of processes issuing disk I/O.
Demystifying IOWAIT
As stated above IOWAIT is basically the same as idle time. The only difference is that during IOWAIT there is at least 1 outstanding I/O. Had there been another process (or thread) waiting for the CPU it would use the CPU and the IOWAIT would be reported as one of the other CPU states such as USER, NICE or SYSTEM. We can easily prove this by performing another experiment by starting 2 CPU load processes while the 2 disk load processes are still running.
$ load-gen cpu 2 $ sar –P ALL –u 30 6 Linux 2.6.32-300.20.1.el5uek (roger.example.com) 04/21/2012 09:30:00 PM CPU %user %nice %system %iowait %steal %idle 09:30:30 PM all 63.29 0.00 36.41 0.30 0.00 0.00 09:30:30 PM 0 30.11 0.00 69.29 0.60 0.00 0.00 09:30:30 PM 1 96.47 0.00 3.53 0.00 0.00 0.00 09:30:30 PM CPU %user %nice %system %iowait %steal %idle 09:31:00 PM all 61.33 0.00 38.52 0.15 0.00 0.00 09:31:00 PM 0 28.28 0.00 71.42 0.30 0.00 0.00 09:31:00 PM 1 94.40 0.00 5.60 0.00 0.00 0.00 09:31:00 PM CPU %user %nice %system %iowait %steal %idle 09:31:30 PM all 62.60 0.00 37.40 0.00 0.00 0.00 09:31:30 PM 0 29.67 0.00 70.33 0.00 0.00 0.00 09:31:30 PM 1 95.53 0.00 4.47 0.00 0.00 0.00 09:31:30 PM CPU %user %nice %system %iowait %steal %idle 09:32:00 PM all 64.29 0.00 35.71 0.00 0.00 0.00 09:32:00 PM 0 30.84 0.00 69.16 0.00 0.00 0.00 09:32:00 PM 1 97.70 0.00 2.30 0.00 0.00 0.00 09:32:00 PM CPU %user %nice %system %iowait %steal %idle 09:32:30 PM all 63.77 0.00 36.23 0.00 0.00 0.00 09:32:30 PM 0 31.40 0.00 68.60 0.00 0.00 0.00 09:32:30 PM 1 96.13 0.00 3.87 0.00 0.00 0.00 09:32:30 PM CPU %user %nice %system %iowait %steal %idle 09:33:00 PM all 63.49 0.00 36.51 0.00 0.00 0.00 09:33:00 PM 0 31.97 0.00 68.03 0.00 0.00 0.00 09:33:00 PM 1 95.00 0.00 5.00 0.00 0.00 0.00 Average: CPU %user %nice %system %iowait %steal %idle Average: all 63.13 0.00 36.80 0.08 0.00 0.00 Average: 0 30.38 0.00 69.47 0.15 0.00 0.00 Average: 1 95.87 0.00 4.13 0.00 0.00 0.00
Voila, the IOWAIT is gone! If one of the disk I/O load processes has to wait for I/O completion, one of the CPU load processes can use the CPU and therefore the system does not enter the IOWAIT state. Isn’t that a nice way to “tune” a system that is “suffering” high amounts of IOWAIT? ;-)
To further prove that disk I/O is just another factor that influences the Linux load average, just as CPU utilization does, we can take a look at the sar run-queue statistics:
$ sar –q 30 6 Linux 2.6.32-300.20.1.el5uek (roger.example.com) 04/21/2012 09:30:00 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 09:30:30 PM 4 295 2.80 2.06 1.87 09:31:00 PM 4 295 3.27 2.24 1.93 09:31:30 PM 4 295 3.70 2.44 2.01 09:32:00 PM 4 295 3.82 2.59 2.07 09:32:30 PM 4 295 3.89 2.72 2.13 09:33:00 PM 4 289 3.93 2.84 2.19 Average: 4 294 3.57 2.48 2.03
The load average has increased by 2. Hopefully this matches your expectation. If not, please read part 1 again!
Does the current load average have an impact on how well the system is performing? We can check this by the amount of I/O the system is able to perform using sar:
$ sar –b 30 6 Linux 2.6.32-300.20.1.el5uek (roger.example.com) 04/21/2012 09:30:00 PM tps rtps wtps bread/s bwrtn/s 09:30:30 PM 19960.52 0.00 19960.52 0.00 37556.49 09:31:00 PM 20967.83 0.00 20967.83 0.00 39447.60 09:31:30 PM 20959.73 0.00 20959.73 0.00 39421.50 09:32:00 PM 20634.82 0.10 20634.72 0.80 38803.77 09:32:30 PM 20117.79 0.00 20117.79 0.00 37849.28 09:33:00 PM 19797.07 0.10 19796.97 0.80 37237.61 Average: 20406.31 0.03 20406.28 0.27 38386.08
The above output shows that, under the current load average of 4, on average the system is able to write 38386 blocks per second to disk. This is considerable less compared to the average of 48207 blocks per second we were able to write before with a load average of 2.
We will take a look at the corresponding top output before we come to a conclusion.
top - 21:33:01 up 11:17, 1 user, load average: 3.93, 2.84, 2.19 Tasks: 195 total, 5 running, 190 sleeping, 0 stopped, 0 zombie Cpu(s): 63.7%us, 19.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 3.0%hi, 13.8%si, 0.0%st Mem: 3074820k total, 2553332k used, 521488k free, 221216k buffers Swap: 5144568k total, 0k used, 5144568k free, 1160828k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 27581 hbreeder 20 0 63836 1064 908 R 82.3 0.0 1:47.34 busy-cpu 27582 hbreeder 20 0 63836 1068 908 R 59.4 0.0 2:18.79 busy-cpu 27621 hbreeder 20 0 63168 616 512 R 23.9 0.0 0:07.19 dd 27620 hbreeder 20 0 63168 616 512 R 23.8 0.0 0:07.14 dd 27588 hbreeder 20 0 12756 1192 836 R 0.0 0.0 0:00.13 top
The above top output confirms that the load average is about 4, the CPUs are fully utilized and that there is no IOWAIT. It also shows the processes that are active on the system.
Conclusion
So far we have proven that both CPU utilization and disk I/O influences the load average on a Linux system. The question remains if they are the only factors driving the load average? Or are there other factors to consider?
Stay tuned for part three to get the answer!
-Harald
Admin said
Hi Harald,
Thanks for the nice topic. I have two findings. The first is that, while running only the two processes performing disk I/O, the sar output shows the runq-sz varying from 1 to 4. I should expect max 2 processes on the run queue.
The second is that, while running both the cpu and I/O bound scripts, the runq-sz is constantly on 4 processes on the run queue. So the I/O based processes seem to be on the run queue all the time
Best regards,
Marco
Harald van Breederode said
Hi Marco,
The dd processes generating the disk I/O are driven by a shell script similar to:
while :
dd of=file.$$ …
rm file.$$
done
So there are actually 4 processes involvedduring the first experiment, which explains why the run-queue utilization varies between 1 and 4.
Jenny said
Hi Harald,
This is very nice article to explain the Load average concept. Thanks for sharing this. But I have question that why on the load average of 4 the number of writing blocks are less than the one on load average of 2.
Thanks,
izzol said
Because the Linux Scheduler is fuckup :) Try the same on FreBSD for example :)
Harald van Breederode said
Hi Izzol,
Thanx for your constructive feedback …
-Harald
Raheel Syed said
Reblogged this on Raheel's Blog and commented:
Interesting series Part 2: Understanding Linux Load Average
Veritas Volume Manager (VxVM) command line examples | IT World said
[…] Understanding Linux Load Average – Part 2 […]
Understanding Linux CPU Load 资料汇总 | 系统技术非业余研究 said
[…] Understanding Linux Load Average 谢谢 @jametong 参考：part1 part2 […]
Rushi said
Don’t you think disk I/O processes are contributing to run-queue length only because they too are utilizing CPU (14% in above example) to perform I/O?If not please explain why.