Here is the disk latency graph of my server. The server has two 5400 RPM 32MB Cache SATA 3.0Gb/s hard drives mounted with software RAID 1 configuration. No special changes has happend in those days and as you can see in the next graph, IO shows normal fluctuations. (The first peak is caused by a weekly corn)
It serves 130,000 unique visitors per day with over 2 million page views/day mostly served by memcache.
I can't figure out what causes the problem.