Search This Blog

Tuesday, January 15, 2013

Keeping an eye on I/O waits

With the growth in use of shared storage and databases running in virtual machines, a common problem over the last couple of years has been slow disk I/O.
System administrators tend to keep an eye on CPU resources in virtual systems, and CPU starvation for a virtual machine is one of the first thinks looked at, however storage is less so.

This manifests itself in system problems:
  • at a particular time of the day (say every evening at 10:00pm)
  • systems do perform ok, then deteriorate.
  • systems that are migrated to newer hardware, but perform poorly.
Apart from lots of memory, databases really need fast consistent disk I/O.
When looking at a poorly performing system you should ask:
  • How much memory (relative to the size of the database) does the system have?
  • What is the load - number of concurrent sessions, number of active sessions?
  • What is the disk I/O sub system?
As an example when looking at a Unix/Linux system you should check exactly what the storage system is even if the disk devices look like the standard (sda, sdb, sdc etc).
Some configurations of virtual machines and shared storage do present standard hard disk devices even if they are virtualised.

Linux/Unix Systems.

Run iostat over time to see what the busy devices are, an in particular what the %iowait statistic is:
A 'healthy' system:
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    1.60    9.22    0.20   88.98
A system to investigate:
avg-cpu:  %user   %nice    %sys %iowait   %idle
           4.00    0.00    1.10   49.25   45.65

IOStat gets its data from the kernel iostats module, which writes to /proc/doskstats.

The fields are:

Field 1 # of reads completed This is the total number of reads completed successfully.
Field 2 # of reads merged, field 6 # of writes merged Reads and writes which are adjacent to each other may be merged for efficiency. Thus two 4K reads may become one 8K read before it is ultimately handed to the disk, and so it will be counted (and queued) as only one I/O. This field lets you know how often this was done.
Field 3 # of sectors read This is the total number of sectors read successfully.
Field 4 # of milliseconds spent reading This is the total number of milliseconds spent by all reads (as measured from __make_request() to end_that_request_last()).
Field 5 # of writes completed This is the total number of writes completed successfully.
Field 7 # of sectors written This is the total number of sectors written successfully.
Field 8 # of milliseconds spent writing This is the total number of milliseconds spent by all writes (as measured from __make_request() to end_that_request_last()).
Field 9 # of I/Os currently in progress The only field that should go to zero. Incremented as requests are given to appropriate struct request_queue and decremented as they finish.
Field 10 # of milliseconds spent doing I/Os This field increases so long as field 9 is nonzero.
Field 11 weighted # of milliseconds spent doing I/Os This field is incremented at each I/O start, I/O completion, I/O merge, or read of these stats by the number of I/Os in progress (field 9) times the number of milliseconds spent doing I/O since the last update of this field. This can provide an easy measure of both I/O completion time and the backlog that may be accumulating.


In performance manager look at the % Disk Time, Current Disk Queue Length counters. The Split I/O counter is also useful to see if there is a data block mismatch between Windows and the physical device.

No comments:

Post a Comment