From 157d7d45f501238ffb891a78b3eaeda81a42861c Mon Sep 17 00:00:00 2001 From: Nick Craig-Wood Date: Mon, 18 Apr 2016 18:30:29 +0100 Subject: [PATCH] Version v1.29 --- MANUAL.html | 156 ++++++++++++++++----- MANUAL.md | 208 ++++++++++++++++++++++------ MANUAL.txt | 222 ++++++++++++++++++++++++------ docs/content/changelog.md | 36 ++++- docs/content/downloads.md | 42 +++--- fs/version.go | 2 +- rclone.1 | 277 ++++++++++++++++++++++++++++++++------ 7 files changed, 755 insertions(+), 188 deletions(-) diff --git a/MANUAL.html b/MANUAL.html index 17b9054af..e463e9875 100644 --- a/MANUAL.html +++ b/MANUAL.html @@ -12,7 +12,7 @@

Rclone

Logo

@@ -51,10 +51,10 @@

Install

Rclone is a Go program and comes as a single binary file.

Download the relevant binary.

-

Or alternatively if you have Go installed use

+

Or alternatively if you have Go 1.5+ installed use

go get github.com/ncw/rclone
-

and this will build the binary in $GOPATH/bin. If you have built rclone before then you will want to update its dependencies first with this (remove -f if using go < 1.4)

-
go get -u -v -f github.com/ncw/rclone/...
+

and this will build the binary in $GOPATH/bin. If you have built rclone before then you will want to update its dependencies first with this

+
go get -u -v github.com/ncw/rclone/...

See the Usage section of the docs for how to use rclone, or run rclone -h.

linux binary downloaded files install example

unzip rclone-v1.17-linux-amd64.zip
@@ -148,36 +148,63 @@ rclone --dry-run --min-size 100M delete remote:path

That reads "delete everything with a minimum size of 100 MB", hence delete all files bigger than 100MBytes.

rclone check source:path dest:path

Checks the files in the source and destination match. It compares sizes and MD5SUMs and prints a report of files which don't match. It doesn't alter the source or destination.

+

--size-only may be used to only compare the sizes, not the MD5SUMs.

rclone dedupe remote:path

-

Interactively find duplicate files and offer to delete all but one or rename them to be different. Only useful with Google Drive which can have duplicate file names.

+

By default dedup interactively finds duplicate files and offers to delete all but one or rename them to be different. Only useful with Google Drive which can have duplicate file names.

+

The dedupe command will delete all but one of any identical (same md5sum) files it finds without confirmation. This means that for most duplicated files the dedupe command will not be interactive. You can use --dry-run to see what would happen without doing anything.

+

Here is an example run.

+

Before - with duplicates

+
$ rclone lsl drive:dupes
+  6048320 2016-03-05 16:23:16.798000000 one.txt
+  6048320 2016-03-05 16:23:11.775000000 one.txt
+   564374 2016-03-05 16:23:06.731000000 one.txt
+  6048320 2016-03-05 16:18:26.092000000 one.txt
+  6048320 2016-03-05 16:22:46.185000000 two.txt
+  1744073 2016-03-05 16:22:38.104000000 two.txt
+   564374 2016-03-05 16:22:52.118000000 two.txt
+

Now the dedupe session

$ rclone dedupe drive:dupes
-2016/01/31 14:13:11 Google drive root 'dupes': Looking for duplicates
-two.txt: Found 3 duplicates
-  1:       564374 bytes, 2016-01-31 14:07:22.159000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81
-  2:      1744073 bytes, 2016-01-31 14:07:12.490000000, md5sum 851957f7fb6f0bc4ce76be966d336802
-  3:      6048320 bytes, 2016-01-31 14:07:02.111000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36
+2016/03/05 16:24:37 Google drive root 'dupes': Looking for duplicates using interactive mode.
+one.txt: Found 4 duplicates - deleting identical copies
+one.txt: Deleting 2/3 identical duplicates (md5sum "1eedaa9fe86fd4b8632e2ac549403b36")
+one.txt: 2 duplicates remain
+  1:      6048320 bytes, 2016-03-05 16:23:16.798000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36
+  2:       564374 bytes, 2016-03-05 16:23:06.731000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81
+s) Skip and do nothing
+k) Keep just one (choose which in next step)
+r) Rename all to be different (by changing file.jpg to file-1.jpg)
+s/k/r> k
+Enter the number of the file to keep> 1
+one.txt: Deleted 1 extra copies
+two.txt: Found 3 duplicates - deleting identical copies
+two.txt: 3 duplicates remain
+  1:       564374 bytes, 2016-03-05 16:22:52.118000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81
+  2:      6048320 bytes, 2016-03-05 16:22:46.185000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36
+  3:      1744073 bytes, 2016-03-05 16:22:38.104000000, md5sum 851957f7fb6f0bc4ce76be966d336802
 s) Skip and do nothing
 k) Keep just one (choose which in next step)
 r) Rename all to be different (by changing file.jpg to file-1.jpg)
 s/k/r> r
 two-1.txt: renamed from: two.txt
 two-2.txt: renamed from: two.txt
-two-3.txt: renamed from: two.txt
-one.txt: Found 2 duplicates
-  1:         6579 bytes, 2016-01-31 14:05:01.235000000, md5sum 2b76c776249409d925ae7ccd49aea59b
-  2:         6579 bytes, 2016-01-31 12:50:30.318000000, md5sum 2b76c776249409d925ae7ccd49aea59b
-s) Skip and do nothing
-k) Keep just one (choose which in next step)
-r) Rename all to be different (by changing file.jpg to file-1.jpg)
-s/k/r> k
-Enter the number of the file to keep> 2
-one.txt: Deleted 1 extra copies
+two-3.txt: renamed from: two.txt

The result being

$ rclone lsl drive:dupes
-   564374 2016-01-31 14:07:22.159000000 two-1.txt
-  1744073 2016-01-31 14:07:12.490000000 two-2.txt
-  6048320 2016-01-31 14:07:02.111000000 two-3.txt
-     6579 2016-01-31 12:50:30.318000000 one.txt
+ 6048320 2016-03-05 16:23:16.798000000 one.txt + 564374 2016-03-05 16:22:52.118000000 two-1.txt + 6048320 2016-03-05 16:22:46.185000000 two-2.txt + 1744073 2016-03-05 16:22:38.104000000 two-3.txt +

Dedupe can be run non interactively using the --dedupe-mode flag.

+ +

For example to rename all the identically named photos in your Google Photos directory, do

+
rclone dedupe --dedupe-mode rename "drive:Google Photos"

rclone config

Enter an interactive configuration session.

rclone help

@@ -216,11 +243,16 @@ rclone sync /path/to/files remote:current-backup

--contimeout=TIME

Set the connection timeout. This should be in go time format which looks like 5s for 5 seconds, 10m for 10 minutes, or 3h30m.

The connection timeout is the amount of time rclone will wait for a connection to go through to a remote object storage system. It is 1m by default.

+

--dedupe-mode MODE

+

Mode to run dedupe command in. One of interactive, skip, first, newest, oldest, rename. The default is interactive. See the dedupe command for more information as to what these options mean.

-n, --dry-run

Do a trial run with no permanent changes. Use this to see what rclone would do without actually doing it. Useful when setting up the sync command which deletes files in the destination.

--ignore-existing

Using this option will make rclone unconditionally skip all files that exist on the destination, no matter the content of these files.

While this isn't a generally recommended option, it can be useful in cases where your files change due to encryption. However, it cannot correct partial transfers in case a transfer was interrupted.

+

-I, --ignore-times

+

Using this option will cause rclone to unconditionally upload all files regardless of the state of files on the destination.

+

Normally rclone would skip any files that have the same modification time and are the same size (or have the same checksum if using --checksum).

--log-file=FILE

Log all of rclone's output to FILE. This is not active by default. This can be useful for tracking down problems with syncs in combination with the -v flag.

--low-level-retries NUMBER

@@ -406,6 +438,7 @@ y/e/d> /file.jpg - matches "file.jpg" in the root directory of the remote - doesn't match "afile.jpg" - doesn't match "directory/file.jpg" +

Important Note that you must use / in patterns and not \ even if running on Windows.

A * matches anything but not a /.

*.jpg  - matches "file.jpg"
        - matches "directory/file.jpg"
@@ -626,7 +659,7 @@ file2.jpg
Backblaze B2 SHA1 -Partial +Yes No No @@ -652,7 +685,6 @@ file2.jpg

ModTime

The cloud storage system supports setting modification times on objects. If it does then this enables a using the modification times as part of the sync. If not then only the size will be checked by default, though the MD5SUM can be checked with the --checksum flag.

All cloud storage systems support some kind of date on the object and these will be set when transferring from the cloud storage system.

-

Backblaze B2 preserves file modification times on files uploaded and downloaded, but doesn't use them to decide which objects to sync.

Case Insensitive

If a cloud storage systems is case sensitive then it is possible to have two files which differ only in case, eg file.txt and FILE.txt. If a cloud storage system is case insensitive then that isn't possible.

This can cause problems when syncing between a case insensitive system and a case sensitive system. The symptom of this is that no matter how many times you run the sync it never completes fully.

@@ -665,7 +697,7 @@ file2.jpg

Most of the time this doesn't cause any problems as people tend to avoid files whose name differs only by case even on case sensitive systems.

Duplicate files

If a cloud storage system allows duplicate files then it can have two objects with the same name.

-

This confuses rclone greatly when syncing.

+

This confuses rclone greatly when syncing - use the rclone dedupe command to rename or remove duplicates.

Google Drive

Paths are specified as drive:path

Drive paths may be as deep as required, eg drive:directory/subdirectory.

@@ -750,11 +782,13 @@ y/e/d> y

Specific options

Here are the command line options specific to this cloud storage system.

--drive-chunk-size=SIZE

-

Upload chunk size. Must a power of 2 >= 256k. Default value is 256kB.

+

Upload chunk size. Must a power of 2 >= 256k. Default value is 8 MB.

+

Making this larger will improve performance, but note that each chunk is buffered in memory one per transfer.

+

Reducing this will reduce memory usage but decrease performance.

--drive-full-list

-

Use a full listing for directory list. More data but usually quicker. On by default, disable with --full-drive-list=false.

+

No longer does anything - kept for backwards compatibility.

--drive-upload-cutoff=SIZE

-

File size cutoff for switching to chunked upload. Default is 256kB.

+

File size cutoff for switching to chunked upload. Default is 8 MB.

--drive-use-trash

Send files to the trash instead of deleting permanently. Defaults to off, namely deleting files permanently.

--drive-auth-owner-only

@@ -1169,6 +1203,8 @@ y/e/d> y

Modified time

The modified time is stored as metadata on the object as X-Object-Meta-Mtime as floating point since the epoch accurate to 1 ns.

This is a defacto standard (used in the official python-swiftclient amongst others) for storing the modification time for an object.

+

Limitations

+

The Swift API doesn't return a correct MD5SUM for segmented files (Dynamic or Static Large Objects) so rclone won't check or use the MD5SUM for these.

Dropbox

Paths are specified as remote:path

Dropbox paths may be as deep as required, eg remote:directory/subdirectory.

@@ -1244,9 +1280,10 @@ y/e/d> y

Here are the command line options specific to this cloud storage system.

--dropbox-chunk-size=SIZE

Upload chunk size. Max 150M. The default is 128MB. Note that this isn't buffered into memory.

-

Limitations

+

Limitations

Note that Dropbox is case insensitive so you can't have a file called "Hello.doc" and one called "hello.doc".

There are some file names such as thumbs.db which Dropbox can't store. There is a full list of them in the "Ignored Files" section of this document. Rclone will issue an error message File name disallowed - not uploading if it attempt to upload one of those file names, but the sync won't fail.

+

If you have more than 10,000 files in a directory then rclone purge dropbox:dir will return the error Failed to purge: There are too many files involved in this operation. As a work-around do an rclone delete dropbix:dir followed by an rclone rmdir dropbox:dir.

Google Cloud Storage

Paths are specified as remote:bucket (or remote: for the lsd command.) You may put subdirectories in too, eg remote:bucket/path/to/dir.

The initial setup for google cloud storage involves getting a token from Google Cloud Storage which you need to do in your browser. rclone config walks you through it.

@@ -1430,7 +1467,7 @@ y/e/d> y

Files this size or more will be downloaded via their tempLink. This is to work around a problem with Amazon Cloud Drive which blocks downloads of files bigger than about 10GB. The default for this is 9GB which shouldn't need to be changed.

To download files above this threshold, rclone requests a tempLink which downloads the file through a temporary URL directly from the underlying S3 storage.

-

Limitations

+

Limitations

Note that Amazon cloud drive is case insensitive so you can't have a file called "Hello.doc" and one called "hello.doc".

Amazon cloud drive has rate limiting so you may notice errors in the sync (429 errors). rclone will automatically retry the sync up to 3 times by default (see --retries flag) which should hopefully work around this problem.

Amazon cloud drive has an internal limit of file sizes that can be uploaded to the service. This limit is not officially published, but all files larger than this will fail.

@@ -1518,7 +1555,7 @@ y/e/d> y

Above this size files will be chunked - must be multiple of 320k. The default is 10MB. Note that the chunks will be buffered into memory.

--onedrive-upload-cutoff=SIZE

Cutoff for switching to chunked upload - must be <= 100MB. The default is 10MB.

-

Limitations

+

Limitations

Note that One Drive is case insensitive so you can't have a file called "Hello.doc" and one called "hello.doc".

Rclone only supports your default One Drive, and doesn't work with One Drive for business. Both these issues may be fixed at some point depending on user demand!

There are quite a few characters that can't be in One Drive file names. These can't occur on Windows platforms, but on non-Windows platforms they are common. Rclone will map these names to and from an identical looking unicode equivalent. For example if a file has a ? in it will be mapped to ？ instead.

@@ -1596,8 +1633,9 @@ y/e/d> y

The modified time is stored as metadata on the object as X-Object-Meta-Mtime as floating point since the epoch accurate to 1 ns.

This is a defacto standard (used in the official python-swiftclient amongst others) for storing the modification time for an object.

Note that Hubic wraps the Swift backend, so most of the properties of are the same.

-

Limitations

-

Code to refresh the OpenStack token isn't done yet which may cause problems with very long transfers.

+

Limitations

+

This uses the normal OpenStack Swift mechanism to refresh the Swift API credentials and ignores the expires field returned by the Hubic API.

+

The Swift API doesn't return a correct MD5SUM for segmented files (Dynamic or Static Large Objects) so rclone won't check or use the MD5SUM for these.

Backblaze B2

B2 is Backblaze's cloud storage system.

Paths are specified as remote:bucket (or remote: for the lsd command.) You may put subdirectories in too, eg remote:bucket/path/to/dir.

@@ -1662,15 +1700,17 @@ y/e/d> y 
rclone sync /home/local/directory remote:bucket

Modified time

The modified time is stored as metadata on the object as X-Bz-Info-src_last_modified_millis as milliseconds since 1970-01-01 in the Backblaze standard. Other tools should be able to use this as a modified time.

-

Modified times are set on upload, read on download and shown in listings. They are not used in syncing as unfortunately B2 doesn't have an API method to set them independently of doing an upload.

+

Modified times are used in syncing and are fully supported except in the case of updating a modification time on an existing object. In this case the object will be uploaded again as B2 doesn't have an API method to set the modification time independent of doing an upload.

SHA1 checksums

The SHA1 checksums of the files are checked on upload and download and will be used in the syncing process. You can use the --checksum flag.

Versions

When rclone uploads a new version of a file it creates a new version of it. Likewise when you delete a file, the old version will still be available.

The old versions of files are visible in the B2 web interface, but not via rclone yet.

Rclone doesn't provide any way of managing old versions (downloading them or deleting them) at the moment. When you purge a bucket, all the old versions will be deleted.

+

Transfers

+

Backblaze recommends that you do lots of transfers simultaneously for maximum speed. In tests from my SSD equiped laptop the optimum setting is about --transfers 32 though higher numbers may be used for a slight speed improvement. The optimum number for you may vary depending on your hardware, how big the files are, how much you want to load your computer, etc. The default of --transfers 4 is definitely too low for Backblaze B2 though.

API

-

Here are some notes I made on the backblaze API while integrating it with rclone which detail the changes I'd like to see. With a couple of small tweaks Backblaze could enable rclone to not make a temporary copy of files when doing cloud to cloud copies and fully support modification times.

+

Here are some notes I made on the backblaze API while integrating it with rclone which detail the changes I'd like to see.

Yandex Disk

Yandex Disk is a cloud storage solution created by Yandex.

Yandex paths may be as deep as required, eg remote:directory/subdirectory.

@@ -1774,6 +1814,46 @@ nounc = true

This will use UNC paths on c:\src but not on z:\dst. Of course this will cause problems if the absolute path length of a file exceeds 258 characters on z, so only use this option if you have to.

Changelog