TIL: dd over netcat
Written by Sebastian Dümcke on
Tags: TIL
I found out that you can transfer raw bytes as generated by the
dd
command over the network using netcat
, which improves throughput
compared to transfer over SSH using scp
. Thanks to NDCHost for a
comprehensive writeup.
On the server (serverB.example.net), start a process listing for raw data packets:
nc -l 19000 | bzip2 -d | dd bs=16M of=/ptah/to/output/file
And on the client, send data packets:
dd bs=16M if=/path/to/input/file | bzip2 -c | nc serverB.example.net 19000
Since I seems to have lost the USB ports on my NAS, probably due to a surge caused by a faulty power supply, I can now use this method from another computer as the first step of the ingestion script. I then mount this newly create dd image locally and run the rsync commands to complete the ingestion process.
Here are some more blog posts with benchmark of different network transfer methods for context: 1 and 2.