• 5
name

A PHP Error was encountered

Severity: Notice

Message: Undefined index: userid

Filename: views/question.php

Line Number: 191

Backtrace:

File: /home/prodcxja/public_html/questions/application/views/question.php
Line: 191
Function: _error_handler

File: /home/prodcxja/public_html/questions/application/controllers/Questions.php
Line: 433
Function: view

File: /home/prodcxja/public_html/questions/index.php
Line: 315
Function: require_once

name Punditsdkoslkdosdkoskdo

Why is writing to my NAS in small chunks so slow?

I just discovered that the bottleneck of a Python script was writing a text file unbuffered line by line to our QNAP NAS. See attached Python snippet. Is this necessarily so slow or is there something wrong with our QNAP/network setup?

# local SSD: 2 seconds
with open(r'C:\Daten\numbers.txt', 'w') as f:

# local SSD with buffering: 2 seconds
with open(r'C:\Daten\numbers.txt', 'w', buffering=2**20) as f:

# Share on QNAP NAS: ### 36 ... 61 seconds! ###
with open(r'I:\numbers.txt', 'w') as f:

# Share on QNAP NAS with buffering: 2 ... 3 seconds
with open(r'I:\numbers.txt', 'w', buffering=2**20) as f:
    for i in range(1000000):
        print(i, file=f)

Any small I/O is going to be expensive compared to larger transfer. In this case, the situation is worsened by:

  • using unbuffered I/O
  • sending data down to the network rather than a fast local bus
  • using a remote filesystem protocol (ie: CIFS or NFS).

While you can somewhat improve performance by fine-tuning the NFS/CIFS/SMB server behavior, I strongly suggest you to avoid generic unbuffered I/O in favor of buffered I/O + explicit flush points (if required by your application).

  • 2
Reply Report
      • 2
    • On top the OP should read up on the difference performance characteristic of SSD and HDD and on the speed of his network ;)

Trending Tags