on
TIL: dd over netcat
I found out that you can transfer raw bytes as generated by the dd
command
over the network using netcat
, which improves throughput compared to
transfer over SSH using scp
. Thanks to NDCHost for a
comprehensive writeup.
On the server (serverB.example.net), start a process listing for raw data packets:
nc -l 19000 | bzip2 -d | dd bs=16M of=/ptah/to/output/file
And on the client, send data packets:
dd bs=16M if=/path/to/input/file | bzip2 -c | nc serverB.example.net 19000
Since I seems to have lost the USB ports on my NAS, probably due to a surge caused by a faulty power supply, I can now use this method from another computer as the first step of the ingestion script. I then mount this newly create dd image locally and run the rsync commands to complete the ingestion process.
Here are some more blog posts with benchmark of different network transfer methods for context: 1 and 2.