TIL: dd over netcat

I found out that you can transfer raw bytes as generated by the ddcommand over the network using netcat, which improves throughput compared to transfer over SSH using scp. Thanks to NDCHost for a comprehensive writeup.

On the server (serverB.example.net), start a process listing for raw data packets:

nc -l 19000 | bzip2 -d | dd bs=16M of=/ptah/to/output/file

And on the client, send data packets:

dd bs=16M if=/path/to/input/file | bzip2 -c | nc serverB.example.net 19000

Since I seems to have lost the USB ports on my NAS, probably due to a surge caused by a faulty power supply, I can now use this method from another computer as the first step of the ingestion script. I then mount this newly create dd image locally and run the rsync commands to complete the ingestion process.

Here are some more blog posts with benchmark of different network transfer methods for context: 1 and 2.