Copying Directories of Files: rsync vs cp: Difference between revisions

From Free Knowledge Base- The DUCK Project
Jump to navigation Jump to search
← Older edit
mNo edit summary
 
Line 7: Line 7:
A common task is to back up a directory of files using the '''cp''' command. An example command might look like this:
A common task is to back up a directory of files using the '''cp''' command. An example command might look like this:


  cp -adv ./SourceDirectory /media/user/BackupDestination/Subfolder
  cp -adv ./SourceDirectory /path/to/BackupDestination/


This command copies the directory "SourceDirectory" recursively to the specified destination path. The options used are:
This command copies the directory "SourceDirectory" (including the directory itself) recursively to the destination. The options used are:
* '''-a''': archive mode, which preserves symbolic links, permissions, timestamps, and other attributes
* '''-a''': archive mode, which preserves symbolic links, permissions, timestamps, and other attributes
* '''-d''': preserves symbolic links as links (similar to --no-dereference in some contexts)
* '''-d''': preserves symbolic links as links
* '''-v''': verbose mode, which displays the files being copied
* '''-v''': verbose mode, which displays the files being copied


This method works effectively for straightforward copies, providing feedback through verbose output. However, if the copy process is interrupted (for example, due to a network disconnection or accidental removal of the drive), it does not resume from the point of interruption, and partial progress may be lost.
This method works effectively for straightforward copies. However, if the copy process is interrupted, it does not resume, and partial progress may be lost.


The objectives in improving this process include ensuring the integrity of copied files, allowing resumption after interruptions, and providing visible progress information during the transfer.
The objectives include ensuring file copy integrity, allowing resumption after interruptions, providing visible progress, and correctly handling directory names that contain spaces.


=== Recommended rsync Equivalent ===
=== Recommended rsync Command ===


A more robust alternative is to use '''rsync'''. The equivalent command for the above scenario is:
The robust alternative is '''rsync'''. To copy a directory (including the directory name itself) and all its contents to a destination folder, use:


  rsync -av --progress --partial --whole-file ./SourceDirectory /media/user/BackupDestination/Subfolder
  rsync -av --progress --partial --whole-file "./Source Directory" /path/to/BackupDestination/


This command includes the following options:
This creates "/path/to/BackupDestination/Source Directory/" containing the full structure.
* '''-a''': archive mode, which recursively copies directories while preserving symbolic links, permissions, timestamps, ownership, and other file attributes in a comprehensive manner
* '''-v''': verbose mode, providing detailed output about the files being processed
* '''--progress''': displays progress information for individual files as well as an overall transfer summary
* '''--partial''': retains partially transferred files, enabling rsync to resume interrupted transfers efficiently
* '''--whole-file''': disables delta-transfer algorithm, which is unnecessary and slightly slower for initial full copies to an empty destination directory, thus optimizing speed in this use case


For maximum file integrity verification, the '''--checksum''' option could be added, which compares files based on checksums rather than just size and modification time. However, this introduces a significant performance penalty due to the computational overhead of calculating checksums for every file, making it unsuitable when transfer speed is a priority.
Key options:
* '''-a''': archive mode (recursive, preserves attributes)
* '''-v''': verbose output
* '''--progress''': shows per-file and overall progress
* '''--partial''': keeps partial files for resumption
* '''--whole-file''': optimizes speed for initial full copies by skipping delta transfers


=== Performance and Speed Comparison ===
Important notes on directory handling:
* Use quotes around the source path if the directory name contains spaces (e.g., "./Source Directory").
* Do '''not''' add a trailing slash to the source path — this ensures the directory itself is created at the destination.
* Adding a trailing slash to the source (e.g., "./Source Directory"/) would copy only the contents directly into the destination, without creating the base directory.
* The destination path may end with a slash or not; rsync will create necessary directories.


When performing an initial full copy to an empty destination directory:
For maximum integrity, '''--checksum''' can be added, but it significantly slows the process.
* The '''cp -adv''' command is typically slightly faster, with an advantage of approximately 5–20% in many cases. This is because '''cp''' copies files directly without additional preparatory steps.
* '''rsync''' incurs minor overhead from building a file list and performing quick checks based on file size and timestamps, even when all files need to be copied anew.
* In real-world scenarios involving a network-mounted source and a removable destination such as an SD card, the primary bottleneck is usually the input/output speed of the devices rather than the tool's overhead. As a result, the practical difference in transfer time is often small and may not be noticeable for most users.


To further reduce rsync's overhead in initial full copies, the '''--whole-file''' option (as included in the recommended command) is beneficial.
=== Performance Comparison ===
 
For initial full copies to an empty or partial destination:
* '''cp -adv''' is slightly faster (typically 5–20% less overhead) due to direct copying without file list checks.
* '''rsync''' has minor overhead from scanning, but device I/O (network and SD card) is usually the bottleneck, making differences small.
* '''--whole-file''' reduces rsync overhead in full-copy scenarios.


=== Advantages of Each Method ===
=== Advantages of Each Method ===


==== Advantages of cp -adv ====
==== Advantages of cp -adv ====
* Slightly higher speed for initial full copies due to minimal overhead
* Slightly faster for initial full copies
* Simpler syntax with fewer options required
* Simpler command syntax
* No preliminary scanning or file list generation
* Lower CPU usage
* Marginally lower CPU resource consumption during the operation
* No preliminary file scanning


==== Advantages of rsync -av --progress --partial --whole-file ====
==== Advantages of rsync -av --progress --partial --whole-file ====
* Automatic resumption of transfers interrupted by issues such as network failures, power loss, or removal of the storage device
* Resumes automatically after interruptions
* Detailed progress display, including per-file and overall statistics, which is helpful for monitoring long transfers
* Displays detailed progress
* Ability to safely re-execute the exact same command on subsequent runs; rsync will skip files that are already identical in the destination
* Safe to re-run; skips identical files
* Superior preservation and handling of file attributes, including permissions, timestamps, symbolic links, and special files
* Excellent handling of spaces in filenames and paths
* Option to perform dry runs (with '''--dry-run''') to verify what would be copied or to confirm that the destination matches the source without transferring data
* Robust preservation of attributes
* More informative verbose output, aiding in the identification of potential issues during the copy process
* Supports dry runs for verification (--dry-run)
* Retention of partially transferred files upon interruption, ensuring that completed portions are not lost
* Retains partial transfers


=== Conclusion and Recommendation ===
=== Recommendation ===


For one-time backups where interruptions are unlikely and maximum speed is desired, the '''cp -adv''' command remains a straightforward and efficient choice.
For one-time copies with no interruptions expected, '''cp -adv''' (with proper quoting for spaces) is simple and fast.


However, for recurring backups, large directory structures, or environments prone to interruptions (such as those involving network drives or removable media), '''rsync''' with the recommended options provides greater reliability, resumability, and user feedback, making it the preferred tool despite the minor performance trade-off.
For backups involving large directories, spaces in names, network sources, or removable media prone to disconnection, '''rsync''' with the above options and correct slash handling is far more reliable and practical.


== Related ==
== Related ==
Line 70: Line 76:
* [[Disk Imaging for Linux]]
* [[Disk Imaging for Linux]]
* [[Disk Archiving Linux Commands]]
* [[Disk Archiving Linux Commands]]


[[Category:Computer_Technology]]
[[Category:Computer_Technology]]
[[Category:Linux]]
[[Category:Linux]]

Latest revision as of 15:21, 24 December 2025

Copying Directories of Files: rsync vs cp

This page provides a detailed comparison between using the cp and rsync commands for backing up directories containing files, particularly in scenarios such as copying from a network-mounted drive to a removable storage device like an SD flash drive. The discussion is based on usage in Linux Mint 21.1 with the Cinnamon desktop environment.

Background and Example Scenario

A common task is to back up a directory of files using the cp command. An example command might look like this: 

cp -adv ./SourceDirectory /path/to/BackupDestination/

This command copies the directory "SourceDirectory" (including the directory itself) recursively to the destination. The options used are:

  • -a: archive mode, which preserves symbolic links, permissions, timestamps, and other attributes
  • -d: preserves symbolic links as links
  • -v: verbose mode, which displays the files being copied

This method works effectively for straightforward copies. However, if the copy process is interrupted, it does not resume, and partial progress may be lost.

The objectives include ensuring file copy integrity, allowing resumption after interruptions, providing visible progress, and correctly handling directory names that contain spaces.

Recommended rsync Command

The robust alternative is rsync. To copy a directory (including the directory name itself) and all its contents to a destination folder, use: 

rsync -av --progress --partial --whole-file "./Source Directory" /path/to/BackupDestination/

This creates "/path/to/BackupDestination/Source Directory/" containing the full structure.

Key options:

  • -a: archive mode (recursive, preserves attributes)
  • -v: verbose output
  • --progress: shows per-file and overall progress
  • --partial: keeps partial files for resumption
  • --whole-file: optimizes speed for initial full copies by skipping delta transfers

Important notes on directory handling:

  • Use quotes around the source path if the directory name contains spaces (e.g., "./Source Directory").
  • Do not add a trailing slash to the source path — this ensures the directory itself is created at the destination.
  • Adding a trailing slash to the source (e.g., "./Source Directory"/) would copy only the contents directly into the destination, without creating the base directory.
  • The destination path may end with a slash or not; rsync will create necessary directories.

For maximum integrity, --checksum can be added, but it significantly slows the process.

Performance Comparison

For initial full copies to an empty or partial destination:

  • cp -adv is slightly faster (typically 5–20% less overhead) due to direct copying without file list checks.
  • rsync has minor overhead from scanning, but device I/O (network and SD card) is usually the bottleneck, making differences small.
  • --whole-file reduces rsync overhead in full-copy scenarios.

Advantages of Each Method

Advantages of cp -adv

  • Slightly faster for initial full copies
  • Simpler command syntax
  • Lower CPU usage
  • No preliminary file scanning

Advantages of rsync -av --progress --partial --whole-file

  • Resumes automatically after interruptions
  • Displays detailed progress
  • Safe to re-run; skips identical files
  • Excellent handling of spaces in filenames and paths
  • Robust preservation of attributes
  • Supports dry runs for verification (--dry-run)
  • Retains partial transfers

Recommendation

For one-time copies with no interruptions expected, cp -adv (with proper quoting for spaces) is simple and fast.

For backups involving large directories, spaces in names, network sources, or removable media prone to disconnection, rsync with the above options and correct slash handling is far more reliable and practical.

Related

Retrieved from "http://wiki.robotz.com/index.php?title=Copying_Directories_of_Files:_rsync_vs_cp&oldid=24048"