• 15

A PHP Error was encountered

Severity: Notice

Message: Undefined index: userid

Filename: views/question.php

Line Number: 191


File: /home/prodcxja/public_html/questions/application/views/question.php
Line: 191
Function: _error_handler

File: /home/prodcxja/public_html/questions/application/controllers/Questions.php
Line: 433
Function: view

File: /home/prodcxja/public_html/questions/index.php
Line: 315
Function: require_once

name Punditsdkoslkdosdkoskdo

COMSTAR and ZFS zvol giving bursty write pattern

I've just built a small NAS/SAN box and am doing some testing with it. Configuration is:

  • 2GB RAM
  • 2x Opteron 265s
  • Iwill DK8N
  • 5x500GB SATA drives (1 spare, rest mirrored) = 1TB usable space
  • Emulex LPC10000DC
  • NexentaCore 3.0.2

The first thing I'm testing is COMSTAR performance over fibre-channel. I created a 700g zvol and exposed it over COMSTAR. Write-caching is enabled. My FC initiator is a Windows 7 box with a Q-Logic QLA2340 card in it.

Everything works great, except for write speeds. For some reason, I'm only seeing ~30MB/s, whereas I can peak at about 90MB/s read speeds, using IOMeter. Doing some hunting around, I ran a zpool iostat command and saw the following pattern:

$ zpool iostat tank 1
tank        88.2G   840G      0  5.47K      0  43.4M
tank        88.2G   840G      0  16.1K      0   128M
tank        88.4G   840G      1  6.69K  8.93K  52.6M
tank        88.4G   840G     52  4.89K   110K  38.3M
tank        88.4G   840G    151      0   280K      0
tank        88.4G   840G    112      0   253K      0
tank        88.4G   840G      1  9.89K  16.0K  78.6M
tank        88.4G   840G      0  14.3K      0   113M
tank        88.6G   839G    150  2.99K   318K  20.2M
tank        88.6G   839G     15      0  45.0K      0
tank        88.6G   839G      3      0  17.8K      0
tank        88.6G   839G      0      0      0      0
tank        88.6G   839G      0      0      0      0
tank        88.6G   839G      0  11.3K      0  89.9M
tank        88.6G   839G      0  13.8K      0   110M
tank        88.8G   839G     90  7.91K   184K  61.0M
tank        88.9G   839G    105     91   220K   140K
tank        88.9G   839G     42      0  84.6K      0
tank        88.9G   839G      0      0      0      0
tank        88.9G   839G      0  6.52K  7.99K  51.8M
tank        88.9G   839G      0  15.6K   1011   124M
tank        88.9G   839G     91  4.30K   193K  31.9M
tank        89.1G   839G      1  9.48K  15.8K  74.7M
tank        89.1G   839G     41      0  99.9K      0
tank        89.1G   839G    166      0   374K      0
tank        89.1G   839G     87      0   162K      0
tank        89.1G   839G      0      0  7.92K      0

It appears to be writing a lot, then stopping.

What's going on here?

That's normal for ZFS. What's happening is it's coalescing the writes in memory and then committing them to disk at the same time. This a great wiki about what it's doing and how you can tune it:


  • 1
Reply Report
      • 2
    • Interesting. I guess my concern was around the speed seen by the COMSTAR initiator - Bonnie++ and these results indicate that 4 spindles can do ~125M/s quite easily, seeing only 30MB/s at the other end of a 2GB FC link made me question what I was doing wrong. Also, the fact that I see a similar bursting pattern with IOMeter on the client gave cause for concern. If it were caching properly, I'd expect to see a fairly constant IO level on the client. I'll read up on that link and see if I can get total throughput up anyhow.

Trending Tags