I'm taking care of a few servers that make relatively heavy use of a NFS filer.
At crunch time a single server sends about 2k reads + 2k writes per sec to the filer. According to nfsiostat at these times the read and write RTTs increase from the usual 5-10ms to 50-100ms for both r/w. The average execution time according to nfsiostat though is drastically different for reads and writes: reads ca 200ms, writes between 100-200sec.
Now from the little I understand average exec time in nfsiostat is essentially RTT + kernel time (RPC queuing etc). Talking about RPC - I see the exact same behaviour on RH5.8 (RPC backlog ca 5k at crunch) and RH6.3 (RPC backlog 0 at all times).
So - question: what could be the reason for NFS read/write execution time being so vastly different?
(The NFS calls are made by proprietary app into which I don't have much visibility. Same applies to the filer (NetApp - outsourced). Kernel tracing is probably not going to be an option either)
Thank you very much in advance.
EDIT: to clarify and because I get some answers dealing with NFS performance in general - the only part the really interests me is: according to nfsiostat doco RTT is "... the duration from the time that client's kernel sends the RPC request until the time it receives the reply." and Avg Exe is "... the duration from the time that NFS client does the RPC request to its kernel until the RPC request is completed, this includes the RTT time above." (http://man7.org/linux/man-pages/man8/nfsiostat.8.html).
My (maybe naive) understanding is that after RTT the server is done, out of the equation. So what does the NFS client do that Avg Exe are so different for reads/writes when RTT is the same?