IT - Which Is Faster: find | cpio -pdvm OR rsync?

To determine which is faster between find | cpio -pdvm and rsync for copying a large directory tree locally, we need to consider several factors: the nature of the operation, the tools' design, the system environment, and the specific use case. Let’s break this down based on the information provided in the web results and general knowledge about these tools.

---

Overview of the Tools

1. find | cpio -pdvm:
   • find: Recursively lists all files and directories in a given path.
   • cpio: A tool for copying files into or out of a cpio or tar archive. In this case, with the -pdvm options:
     • -p: Pass-through mode (copy files from one directory tree to another).
     • -d: Create directories as needed.
     • -v: Verbose mode (shows files being copied).
     • -m: Preserve modification times.
   • This method is a pipeline where find generates a list of files, and cpio copies them to the destination.
2. rsync:
   • A versatile tool for efficiently copying and synchronizing files, both locally and remotely.
   • By default, rsync compares source and destination files using metadata (e.g., file size, modification time) to determine which files need to be copied or updated. It can also use checksums for more thorough comparisons.
   • Common options for local copying include -a (archive mode, preserving permissions, timestamps, etc.), -v (verbose), and --progress (shows progress).

---

Key Factors Affecting Speed

1. Operation Type:
   • Initial Copy to an Empty Destination:
     • If the destination is empty, rsync has to perform checks (e.g., comparing metadata) to determine which files to copy, even though all files need to be copied. This overhead can make rsync slower than find | cpio for a one-time copy to an empty destination.
     • find | cpio doesn’t perform any comparison; it simply copies all files listed by find to the destination, which can be faster for an initial copy.
   • Synchronization (Incremental Copy):
     • If the destination already contains some files, rsync excels because it only copies files that have changed, using its delta-transfer algorithm to minimize data transfer. find | cpio would copy everything again, making it slower in this scenario.
2. File System and Disk I/O:
   • Both methods are disk I/O-bound for local copies, meaning the speed of the source and destination disks (e.g., HDD vs. SSD) plays a significant role.
   • find | cpio may have a slight edge in raw copying speed because it doesn’t perform comparisons, but this depends on the file system and how well it handles metadata operations (e.g., creating directories, setting timestamps).
3. File Size and Number of Files:
   • For a large number of small files, the overhead of rsync’s comparison can be significant, making find | cpio faster.
   • For large files, rsync can be more efficient if the files are already partially copied (due to its delta-transfer algorithm), but for an initial copy, find | cpio might still be faster.
4. Sparse Files and Special Files:
   • rsync handles sparse files efficiently with the -S option (as noted in linux - Copying a large directory tree locally? cp or rsync? - Server Fault - serverfault.com, web ID: 2), but this isn’t enabled by default.
   • cpio can have issues with very large files (>2GB) due to format limitations (as mentioned in cpio vs. rsync - Hewlett Packard Enterprise Community - community.hpe.com, web ID: 0), which could cause failures or slowdowns.
5. CPU Usage:
   • rsync can be CPU-intensive if using options like -z (compression, which is unnecessary for local copies) or checksums (--checksum). For local copies, these options should be avoided to maximize speed (as noted in usb - How to speed up rsync between two local disks? - Super User - superuser.com, web ID: 1).
   • find | cpio has lower CPU overhead since it doesn’t perform comparisons or compression.

---

Insights from Web Results

1. Initial Copy Performance:
   • cpio vs. rsync - Hewlett Packard Enterprise Community - community.hpe.com (web ID: 0) notes that rsync isn’t ideal for copying to an empty destination because its metadata checks are unnecessary when all files need to be copied. This suggests find | cpio could be faster for an initial copy.
   • linux - Copying a large directory tree locally? cp or rsync? - Server Fault - serverfault.com (web ID: 2) reports a test on a 70GB file system with 48GB of data, 385,000 files, and 427,730 directories, where cp was about 3 minutes faster than rsync and find | cpio. While this test compares cp to rsync, it implies that find | cpio is competitive with cp and likely faster than rsync for an initial copy.
2. Sparse Files and Large Files:
   • cpio vs. rsync - Hewlett Packard Enterprise Community - community.hpe.com (web ID: 0) warns that cpio may have issues with files larger than 2GB, which could cause failures. rsync doesn’t have this limitation and can handle sparse files efficiently with the -S option (web ID: 2).
3. Optimizing rsync:
   • usb - How to speed up rsync between two local disks? - Super User - superuser.com (web ID: 1) suggests that rsync can be as fast as cp for local copies if optimized (e.g., disabling compression with -z, which is only useful for slow networks). For example, adding -o Compression=no increased transfer speed from 10 MB/s to 40 MB/s in one test.
4. Alternative Tools:
   • The web results also mention alternatives like tar (e.g., tar --ignore-failed-read -C $SRC -cf - . | tar --ignore-failed-read -C $DEST -xf -) and vxdump/vxrestore, which can be faster than both find | cpio and rsync in some scenarios (web ID: 0, web ID: 2). However, these are less commonly used and may not be available in all environments.

---

Which Is Faster?

• For an Initial Copy to an Empty Destination:
  • find | cpio -pdvm is likely faster than rsync because it avoids the overhead of rsync’s metadata comparisons. The web results (web ID: 0, web ID: 2) support this, as rsync’s checks are unnecessary when the destination is empty, and find | cpio can copy files more directly.
  • Example: In the test from web ID: 2, cp (which is similar to find | cpio in terms of raw copying) was 3 minutes faster than rsync for a 70GB file system with many files.
• For Incremental Copies (Synchronization):
  • rsync is significantly faster than find | cpio because it only copies files that have changed, using its efficient delta-transfer algorithm. find | cpio would copy everything again, which is inefficient.
• Specific Use Case Considerations:
  • If you have very large files (>2GB), rsync is safer due to cpio’s limitations (web ID: 0).
  • If you have sparse files, rsync with the -S option is better (web ID: 2).
  • If you’re copying a large number of small files, find | cpio may have an edge due to lower overhead.

---

Recommendations

1. For an Initial Copy:
   • Use find | cpio -pdvm if the destination is empty and you’re confident that your files are within cpio’s limitations (e.g., no files larger than 2GB). Example command:
     bash

     cd /source
     find . -depth -print | cpio -pdvm /destination

   • This will likely be faster than rsync for a one-time copy.
2. For Incremental Copies:
   • Use rsync with appropriate options to minimize overhead. Example command:
     bash

     rsync -ahS --progress /source/ /destination/

     • -a: Archive mode (preserves permissions, timestamps, etc.).
     • -h: Human-readable sizes.
     • -S: Handle sparse files efficiently.
     • --progress: Show progress (optional).
     • Avoid -z (compression) for local copies, as it slows things down (web ID: 1).
3. Optimizing rsync for Initial Copies:
   • If you prefer rsync for its reliability and features, you can reduce its overhead for an initial copy by using options like --noatime (to avoid updating access times) and avoiding unnecessary checks:
     bash

     rsync -ahS --noatime --progress /source/ /destination/

   • However, even with these optimizations, find | cpio may still be faster for an initial copy.
4. Alternative: Use tar:
   • The web results suggest that tar can be faster than both find | cpio and rsync for local copies (web ID: 2). Example command:
     bash

     tar --ignore-failed-read -C /source -cf - . | tar --ignore-failed-read -C /destination -xf -

   • This method is worth considering if speed is critical and you’re open to alternatives.

---

Conclusion

• For an initial copy to an empty destination: find | cpio -pdvm is generally faster than rsync due to lower overhead, as supported by the web results (web ID: 0, web ID: 2). However, be cautious of cpio’s limitations with large files.
• For incremental copies or synchronization: rsync is much faster and more efficient.
• For maximum speed: Consider using tar as an alternative, as it may outperform both find | cpio and rsync in some scenarios.

If you’re performing a one-time copy and speed is your primary concern, I recommend starting with find | cpio -pdvm. If you need reliability, incremental updates, or support for special files, rsync is the better choice. Would you like to test these commands on your system to compare their performance, or do you have a specific use case in mind that we can optimize for?