We will have a machine at work, that on peak performance, should be able to push 50 ("write heads") x 75GB of data per hour. That's peak performance of ~1100MB/s write speed. To get that from the machine, it requires two 10GBi lines. My question is what kind of server+technology can handle/store such data flow ?
Currently for data storage we work with ZFS, although write speeds were never a question. (we are not even close to these speeds) Would ZFS (zfs on linux) be an option ? We also need to store a lot of data, the "IT guide" suggests somewhere between 50-75 TB in total. So it probably can't be all SSDs unless we want to offer our first-born child.
Some additions based on the excellent replies :
- the maximum is 50x75GB/hour during peak which is less than 24h (most likely <6h)
- We don't expect this to happen soon, most likely we will run 5-10x75GB/hour
- it's a pre-alpha machine, however requirements should be met (even though a lot of question marks are in play)
- we would use NFS as connection from the machine to the server
- layout : generating machine -> storage (this one) -> (safe raid 6) -> compute cluster
- so read speed is not essential, but it would be nice to use it from the compute cluster (but this is completely optional)
- most likely it's going to be large data files (not many small)