> But the people you responded to were talking about slowdowns that exist in general, not just ones that apply directly to the post.
I think that's incorrect though. These slowdowns do not exist in general (see my next reply where I run rsync and it immadiately maxes out my 10 Gbit/s).
I think original poster digiown is right with "Note there is no intrinsic reason running multiple streams should be faster than one [EDIT: 'at this scale']. It almost always indicates some bottleneck in the application". In this case it's the user running rsync as a serially-reading program reading from a network mount.
> rsync having trouble doing >1Gbps over the network
rsync copies at 10 Gbit/s without problem between my machines.
Though I have to give `-e 'ssh -c aes256-gcm@openssh.com'` or aes128-gcm, otherwise encryption bottlenecks at 5 Gbit/s with the default `chacha20-poly1305@openssh.com`.
> I don't see why you're saying this.
Because of the part you agreed making sense: It read each file with the sequence `open()/read()/.../read()/close()`, but those files are on the network mount ("/Volumes/mercury"), so each `read()` of size `#define IO_BUFFER_SIZE (32*1024)` is a network roundtrip.
I see, so you're saying the file end of rsync is forced to wait for the network because the filesystem itself waits, not the network end of rsync. That makes sense.
Though I wonder what the actual delay is. The numbers in the post implied several milliseconds, enough to maybe account for 30 seconds of the 8 minutes. But maybe changing files resets the transfer speed a bunch.
To be clear, when I said "delay" in my last post I meant the per-file penalty, not the round-trip latency.
Radxa Orion O6/.DS_Store, 6KB at 4.8MBps, that means it took 1.3ms to transfer
Radxa Orion O6/Micro Center Visit Details.pages, 141KB at 9.8MBps, 14ms
Radxa Orion O6/Radxa Orion O6.md, 19KB at 1.9MBps, 10ms
We know the link can do well over 100MBps, so that time is almost all overhead. But it doesn't seem to be a simple delay. Perhaps a fresh TCP window scaling up on a per-file basis? That would be an unfortunate filesytem design.
Since the two 4MB files both get up to ~100MBps, the same speed as the 250MB file, it seems like the maximum impact from switching files isn't much more than 15ms. If the average is below 10ms then we're looking at half a minute wasted over 3564 files. If the average is 20ms then switching files is responsible for 71 seconds wasted.
By that estimate the file-level serialization is a real issue, but the bigger issue is whatever's preventing that 250MB file from ramping up all the way to 10Gbps.
I guess until we know what that network share is and how it works, we cannot really progress. Taking 10 ms to fetch the 19 KB file is definitely not great, it would be ~50 LAN roundtrips.
> whatever's preventing that 250MB file from ramping up all the way to 10Gbps
Probably that network share does small reads if small reads are requested.
Might be good if the author ran `dd` or a similar tool where the read buffer size can be controlled.
I think that's incorrect though. These slowdowns do not exist in general (see my next reply where I run rsync and it immadiately maxes out my 10 Gbit/s).
I think original poster digiown is right with "Note there is no intrinsic reason running multiple streams should be faster than one [EDIT: 'at this scale']. It almost always indicates some bottleneck in the application". In this case it's the user running rsync as a serially-reading program reading from a network mount.
> rsync having trouble doing >1Gbps over the network
rsync copies at 10 Gbit/s without problem between my machines.
Though I have to give `-e 'ssh -c aes256-gcm@openssh.com'` or aes128-gcm, otherwise encryption bottlenecks at 5 Gbit/s with the default `chacha20-poly1305@openssh.com`.
> I don't see why you're saying this.
Because of the part you agreed making sense: It read each file with the sequence `open()/read()/.../read()/close()`, but those files are on the network mount ("/Volumes/mercury"), so each `read()` of size `#define IO_BUFFER_SIZE (32*1024)` is a network roundtrip.