Version v1.31

This commit is contained in:
Nick Craig-Wood 2016-07-13 12:26:22 +01:00
parent 96e2271cce
commit 63f6827a0d
7 changed files with 1421 additions and 199 deletions

View File

@ -12,7 +12,7 @@
<div id="header">
<h1 class="title">rclone(1) User Manual</h1>
<h2 class="author">Nick Craig-Wood</h2>
<h3 class="date">Jun 18, 2016</h3>
<h3 class="date">Jul 13, 2016</h3>
</div>
<h1 id="rclone">Rclone</h1>
<p><a href="http://rclone.org/"><img src="http://rclone.org/img/rclone-120x120.png" alt="Logo" /></a></p>
@ -67,6 +67,16 @@ sudo chmod 755 /usr/sbin/rclone
sudo mkdir -p /usr/local/share/man/man1
sudo cp rclone.1 /usr/local/share/man/man1/
sudo mandb </code></pre>
<h2 id="installation-with-ansible">Installation with Ansible</h2>
<p>This can be done with <a href="https://github.com/stefangweichinger/ansible-rclone">Stefan Weichinger's ansible role</a>.</p>
<p>Instructions</p>
<ol style="list-style-type: decimal">
<li><code>git clone https://github.com/stefangweichinger/ansible-rclone.git</code> into your local roles-directory</li>
<li>add the role to the hosts you want rclone installed to:</li>
</ol>
<pre><code> - hosts: rclone-hosts
roles:
- rclone</code></pre>
<h2 id="configure">Configure</h2>
<p>First you'll need to configure rclone. As the object storage systems have quite complicated authentication these are kept in a config file <code>.rclone.conf</code> in your home directory by default. (You can use the <code>--config</code> option to choose a different config file.)</p>
<p>The easiest way to make the config is to run rclone with the config option:</p>
@ -108,6 +118,7 @@ destpath/two.txt</code></pre>
<pre><code>destpath/sourcepath/one.txt
destpath/sourcepath/two.txt</code></pre>
<p>If you are familiar with <code>rsync</code>, rclone always works as if you had written a trailing / - meaning &quot;copy the contents of this directory&quot;. This applies to all commands and whether you are talking about the source or destination.</p>
<p>See the <code>--no-traverse</code> option for controlling whether rclone lists the destination directory or not.</p>
<h3 id="rclone-sync-sourcepath-destpath">rclone sync source:path dest:path</h3>
<p>Sync the source to the destination, changing the destination only. Doesn't transfer unchanged files, testing by size and modification time or MD5SUM. Destination is updated to match source, including deleting files if necessary.</p>
<p><strong>Important</strong>: Since this can cause data loss, test first with the <code>--dry-run</code> flag to see exactly what would be copied and deleted.</p>
@ -115,9 +126,9 @@ destpath/sourcepath/two.txt</code></pre>
<p>It is always the contents of the directory that is synced, not the directory so when source:path is a directory, it's the contents of source:path that are copied, not the directory name and contents. See extended explanation in the <code>copy</code> command above if unsure.</p>
<p>If dest:path doesn't exist, it is created and the source:path contents go there.</p>
<h3 id="move-sourcepath-destpath">move source:path dest:path</h3>
<p>Moves the source to the destination.</p>
<p>If there are no filters in use this is equivalent to a copy followed by a purge, but may use server side operations to speed it up if possible.</p>
<p>If filters are in use then it is equivalent to a copy followed by delete, followed by an rmdir (which only removes the directory if empty). The individual file moves will be moved with server side operations if possible.</p>
<p>Moves the contents of the source directory to the destination directory. Rclone will error if the source and destination overlap.</p>
<p>If no filters are in use and if possible this will server side move <code>source:path</code> into <code>dest:path</code>. After this <code>source:path</code> will no longer longer exist.</p>
<p>Otherwise for each file in <code>source:path</code> selected by the filters (if any) this will move it into <code>dest:path</code>. If possible a server side move will be used, otherwise it will copy it (server side if possible) into <code>dest:path</code> then delete the original (if no errors on copy) in <code>source:path</code>.</p>
<p><strong>Important</strong>: Since this can cause data loss, test first with the --dry-run flag.</p>
<h3 id="rclone-ls-remotepath">rclone ls remote:path</h3>
<p>List all the objects in the path with size and path.</p>
@ -149,6 +160,8 @@ rclone --dry-run --min-size 100M delete remote:path</code></pre>
<h3 id="rclone-check-sourcepath-destpath">rclone check source:path dest:path</h3>
<p>Checks the files in the source and destination match. It compares sizes and MD5SUMs and prints a report of files which don't match. It doesn't alter the source or destination.</p>
<p><code>--size-only</code> may be used to only compare the sizes, not the MD5SUMs.</p>
<h3 id="rclone-cleanup-remotepath">rclone cleanup remote:path</h3>
<p>Clean up the remote if possible. Empty the trash or delete old file versions. Not supported by all remotes.</p>
<h3 id="rclone-dedupe-remotepath">rclone dedupe remote:path</h3>
<p>By default <code>dedup</code> interactively finds duplicate files and offers to delete all but one or rename them to be different. Only useful with Google Drive which can have duplicate file names.</p>
<p>The <code>dedupe</code> command will delete all but one of any identical (same md5sum) files it finds without confirmation. This means that for most duplicated files the <code>dedupe</code> command will not be interactive. You can use <code>--dry-run</code> to see what would happen without doing anything.</p>
@ -209,6 +222,16 @@ two-3.txt: renamed from: two.txt</code></pre>
<p>Enter an interactive configuration session.</p>
<h3 id="rclone-help">rclone help</h3>
<p>Prints help on rclone commands and options.</p>
<h2 id="copying-single-files">Copying single files</h2>
<p>rclone normally syncs or copies directories. However if the source remote points to a file, rclone will just copy that file. The destination remote must point to a directory - rclone will give the error <code>Failed to create file system for &quot;remote:file&quot;: is a file not a directory</code> if it isn't.</p>
<p>For example, suppose you have a remote with a file in called <code>test.jpg</code>, then you could copy just that file like this</p>
<pre><code>rclone copy remote:test.jpg /tmp/download</code></pre>
<p>The file <code>test.jpg</code> will be placed inside <code>/tmp/download</code>.</p>
<p>This is equivalent to specifying</p>
<pre><code>rclone copy --no-traverse --files-from /tmp/files remote: /tmp/download</code></pre>
<p>Where <code>/tmp/files</code> contains the single line</p>
<pre><code>test.jpg</code></pre>
<p>It is recommended to use <code>copy</code> when copying single files not <code>sync</code>. They have pretty much the same effect but <code>copy</code> will use a lot less memory.</p>
<h2 id="quoting-and-the-shell">Quoting and the shell</h2>
<p>When you are typing commands to your computer you are using something called the command line shell. This interprets various characters in an OS specific way.</p>
<p>Here are some gotchas which may help users unfamiliar with the shell rules</p>
@ -291,6 +314,9 @@ rclone sync /path/to/files remote:current-backup</code></pre>
<h3 id="no-gzip-encoding">--no-gzip-encoding</h3>
<p>Don't set <code>Accept-Encoding: gzip</code>. This means that rclone won't ask the server for compressed files automatically. Useful if you've set the server to return files with <code>Content-Encoding: gzip</code> but you uploaded compressed files.</p>
<p>There is no need to set this in normal operation, and doing so will decrease the network transfer efficiency of rclone.</p>
<h3 id="no-update-modtime">--no-update-modtime</h3>
<p>When using this flag, rclone won't update modification times of remote files if they are incorrect as it would normally.</p>
<p>This can be used if the remote is being synced with another tool also (eg the Google Drive client).</p>
<h3 id="q---quiet">-q, --quiet</h3>
<p>Normally rclone outputs stats and a completion message. If you set this flag it will make as little output as possible.</p>
<h3 id="retries-int">--retries int</h3>
@ -306,8 +332,8 @@ rclone sync /path/to/files remote:current-backup</code></pre>
<p>The default is <code>1m</code>. Use 0 to disable.</p>
<h3 id="delete-beforeduringafter">--delete-(before,during,after)</h3>
<p>This option allows you to specify when files on your destination are deleted when you sync folders.</p>
<p>Specifying the value <code>--delete-before</code> will delete all files present on the destination, but not on the source <em>before</em> starting the transfer of any new or updated files.</p>
<p>Specifying <code>--delete-during</code> (default value) will delete files while checking and uploading files. This is usually the fastest option.</p>
<p>Specifying the value <code>--delete-before</code> will delete all files present on the destination, but not on the source <em>before</em> starting the transfer of any new or updated files. This uses extra memory as it has to store the source listing before proceeding.</p>
<p>Specifying <code>--delete-during</code> (default value) will delete files while checking and uploading files. This is usually the fastest option. Currently this works the same as <code>--delete-after</code> but it may change in the future.</p>
<p>Specifying <code>--delete-after</code> will delay deletion of files until all new/updated files have been successfully transfered.</p>
<h3 id="timeouttime">--timeout=TIME</h3>
<p>This sets the IO idle timeout. If a transfer has started but then becomes idle for this long it is considered broken and disconnected.</p>
@ -377,6 +403,11 @@ c/u/q&gt;</code></pre>
<p><code>--no-check-certificate</code> controls whether a client verifies the server's certificate chain and host name. If <code>--no-check-certificate</code> is true, TLS accepts any certificate presented by the server and any host name in that certificate. In this mode, TLS is susceptible to man-in-the-middle attacks.</p>
<p>This option defaults to <code>false</code>.</p>
<p><strong>This should be used only for testing.</strong></p>
<h3 id="no-traverse">--no-traverse</h3>
<p>The <code>--no-traverse</code> flag controls whether the destination file system is traversed when using the <code>copy</code> or <code>move</code> commands.</p>
<p>If you are only copying a small number of files and/or have a large number of files on the destination then <code>--no-traverse</code> will stop rclone listing the destination and save time.</p>
<p>However if you are copying a large number of files, escpecially if you are doing a copy where lots of the files haven't changed and won't need copying then you shouldn't use <code>--no-traverse</code>.</p>
<p>It can also be used to reduce the memory usage of rclone when copying - <code>rclone --no-traverse copy src dst</code> won't load either the source or destination listings into memory so will use the minimum amount of memory.</p>
<h2 id="filtering">Filtering</h2>
<p>For the filtering options</p>
<ul>
@ -529,7 +560,7 @@ y/e/d&gt;</code></pre>
<li><code>secret17.jpg</code></li>
<li>non <code>*.jpg</code> and <code>*.png</code></li>
</ul>
<p>A similar process is done on directory entries before recursing into them. This only works on remotes which have a concept of directory (Eg local, drive, onedrive, amazon cloud drive) and not on bucket based remotes (eg s3, swift, google compute storage, b2).</p>
<p>A similar process is done on directory entries before recursing into them. This only works on remotes which have a concept of directory (Eg local, google drive, onedrive, amazon drive) and not on bucket based remotes (eg s3, swift, google compute storage, b2).</p>
<h2 id="adding-filtering-rules">Adding filtering rules</h2>
<p>Filtering rules are added with the following command line flags.</p>
<h3 id="exclude---exclude-files-matching-pattern"><code>--exclude</code> - Exclude files matching pattern</h3>
@ -579,6 +610,24 @@ file2.avi</code></pre>
file1.jpg
file2.jpg</code></pre>
<p>Then use as <code>--files-from files-from.txt</code>. This will only transfer <code>file1.jpg</code> and <code>file2.jpg</code> providing they exist.</p>
<p>For example, let's say you had a few files you want to back up regularly with these absolute paths:</p>
<pre><code>/home/user1/important
/home/user1/dir/file
/home/user2/stuff</code></pre>
<p>To copy these you'd find a common subdirectory - in this case <code>/home</code> and put the remaining files in <code>files-from.txt</code> with or without leading <code>/</code>, eg</p>
<pre><code>user1/important
user1/dir/file
user2/stuff</code></pre>
<p>You could then copy these to a remote like this</p>
<pre><code>rclone copy --files-from files-from.txt /home remote:backup</code></pre>
<p>The 3 files will arrive in <code>remote:backup</code> with the paths as in the <code>files-from.txt</code>.</p>
<p>You could of course choose <code>/</code> as the root too in which case your <code>files-from.txt</code> might look like this.</p>
<pre><code>/home/user1/important
/home/user1/dir/file
/home/user2/stuff</code></pre>
<p>And you would transfer it like this</p>
<pre><code>rclone copy --files-from files-from.txt / remote:backup</code></pre>
<p>In this case there will be an extra <code>home</code> directory on the remote.</p>
<h3 id="min-size---dont-transfer-any-file-smaller-than-this"><code>--min-size</code> - Don't transfer any file smaller than this</h3>
<p>This option controls the minimum size file which will be transferred. This defaults to <code>kBytes</code> but a suffix of <code>k</code>, <code>M</code>, or <code>G</code> can be used.</p>
<p>For example <code>--min-size 50k</code> means no files smaller than 50kByte will be transferred.</p>
@ -1161,6 +1210,40 @@ region = other-v2-signature</code></pre>
],
}</code></pre>
<p>Because this is a json dump, it is encoding the <code>/</code> as <code>\/</code>, so if you use the secret key as <code>xxxxxx/xxxx</code> it will work fine.</p>
<h3 id="minio">Minio</h3>
<p><a href="https://minio.io/">Minio</a> is an object storage server built for cloud application developers and devops.</p>
<p>It is very easy to install and provides an S3 compatible server which can be used by rclone.</p>
<p>To use it, install Minio following the instructions from the web site.</p>
<p>When it configures itself Minio will print something like this</p>
<pre><code>AccessKey: WLGDGYAQYIGI833EV05A SecretKey: BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF Region: us-east-1
Minio Object Storage:
http://127.0.0.1:9000
http://10.0.0.3:9000
Minio Browser:
http://127.0.0.1:9000
http://10.0.0.3:9000</code></pre>
<p>These details need to go into <code>rclone config</code> like this. Note that it is important to put the region in as stated above.</p>
<pre><code>env_auth&gt; 1
access_key_id&gt; WLGDGYAQYIGI833EV05A
secret_access_key&gt; BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF
region&gt; us-east-1
endpoint&gt; http://10.0.0.3:9000
location_constraint&gt;
server_side_encryption&gt;</code></pre>
<p>Which makes the config file look like this</p>
<pre><code>[minio]
env_auth = false
access_key_id = WLGDGYAQYIGI833EV05A
secret_access_key = BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF
region = us-east-1
endpoint = http://10.0.0.3:9000
location_constraint =
server_side_encryption = </code></pre>
<p>Minio doesn't support all the features of S3 yet. In particular it doesn't support MD5 checksums (ETags) or metadata. This means rclone can't check MD5SUMs or store the modified date. However you can work around this with the <code>--size-only</code> flag of rclone.</p>
<p>So once set up, for example to copy files into a bucket</p>
<pre><code>rclone --size-only copy /path/to/files minio:bucket</code></pre>
<h2 id="swift">Swift</h2>
<p>Swift refers to <a href="http://www.openstack.org/software/openstack-storage/">Openstack Object Storage</a>. Commercial implementations of that being:</p>
<ul>
@ -1224,6 +1307,8 @@ User domain - optional (v3 auth)
domain&gt; Default
Tenant name - optional
tenant&gt;
Tenant domain - optional (v3 auth)
tenant_domain&gt;
Region name - optional
region&gt;
Storage URL - optional
@ -1461,10 +1546,10 @@ y/e/d&gt; y</code></pre>
<p>To use a Service Account instead of OAuth2 token flow, enter the path to your Service Account credentials at the <code>service_account_file</code> prompt and rclone won't use the browser based authentication flow.</p>
<h3 id="modified-time-3">Modified time</h3>
<p>Google google cloud storage stores md5sums natively and rclone stores modification times as metadata on the object, under the &quot;mtime&quot; key in RFC3339 format accurate to 1ns.</p>
<h2 id="amazon-cloud-drive">Amazon Drive</h2>
<h2 id="amazon-drive">Amazon Drive</h2>
<p>Paths are specified as <code>remote:path</code></p>
<p>Paths may be as deep as required, eg <code>remote:directory/subdirectory</code>.</p>
<p>The initial setup for Amazon cloud drive involves getting a token from Amazon which you need to do in your browser. <code>rclone config</code> walks you through it.</p>
<p>The initial setup for Amazon Drive involves getting a token from Amazon which you need to do in your browser. <code>rclone config</code> walks you through it.</p>
<p>Here is an example of how to make a remote called <code>remote</code>. First run:</p>
<pre><code> rclone config</code></pre>
<p>This will guide you through an interactive setup process:</p>
@ -1520,26 +1605,26 @@ y/e/d&gt; y</code></pre>
<p>See the <a href="http://rclone.org/remote_setup/">remote setup docs</a> for how to set it up on a machine with no Internet browser available.</p>
<p>Note that rclone runs a webserver on your local machine to collect the token as returned from Amazon. This only runs from the moment it opens your browser to the moment you get back the verification code. This is on <code>http://127.0.0.1:53682/</code> and this it may require you to unblock it temporarily if you are running a host firewall.</p>
<p>Once configured you can then use <code>rclone</code> like this,</p>
<p>List directories in top level of your Amazon cloud drive</p>
<p>List directories in top level of your Amazon Drive</p>
<pre><code>rclone lsd remote:</code></pre>
<p>List all the files in your Amazon cloud drive</p>
<p>List all the files in your Amazon Drive</p>
<pre><code>rclone ls remote:</code></pre>
<p>To copy a local directory to an Amazon cloud drive directory called backup</p>
<p>To copy a local directory to an Amazon Drive directory called backup</p>
<pre><code>rclone copy /home/source remote:backup</code></pre>
<h3 id="modified-time-and-md5sums-1">Modified time and MD5SUMs</h3>
<p>Amazon cloud drive doesn't allow modification times to be changed via the API so these won't be accurate or used for syncing.</p>
<p>Amazon Drive doesn't allow modification times to be changed via the API so these won't be accurate or used for syncing.</p>
<p>It does store MD5SUMs so for a more accurate sync, you can use the <code>--checksum</code> flag.</p>
<h3 id="deleting-files-1">Deleting files</h3>
<p>Any files you delete with rclone will end up in the trash. Amazon don't provide an API to permanently delete files, nor to empty the trash, so you will have to do that with one of Amazon's apps or via the Amazon cloud drive website.</p>
<p>Any files you delete with rclone will end up in the trash. Amazon don't provide an API to permanently delete files, nor to empty the trash, so you will have to do that with one of Amazon's apps or via the Amazon Drive website.</p>
<h3 id="specific-options-3">Specific options</h3>
<p>Here are the command line options specific to this cloud storage system.</p>
<h4 id="acd-templink-thresholdsize">--acd-templink-threshold=SIZE</h4>
<p>Files this size or more will be downloaded via their <code>tempLink</code>. This is to work around a problem with Amazon Drive which blocks downloads of files bigger than about 10GB. The default for this is 9GB which shouldn't need to be changed.</p>
<p>To download files above this threshold, rclone requests a <code>tempLink</code> which downloads the file through a temporary URL directly from the underlying S3 storage.</p>
<h3 id="limitations-3">Limitations</h3>
<p>Note that Amazon cloud drive is case insensitive so you can't have a file called &quot;Hello.doc&quot; and one called &quot;hello.doc&quot;.</p>
<p>Amazon cloud drive has rate limiting so you may notice errors in the sync (429 errors). rclone will automatically retry the sync up to 3 times by default (see <code>--retries</code> flag) which should hopefully work around this problem.</p>
<p>Amazon cloud drive has an internal limit of file sizes that can be uploaded to the service. This limit is not officially published, but all files larger than this will fail.</p>
<p>Note that Amazon Drive is case insensitive so you can't have a file called &quot;Hello.doc&quot; and one called &quot;hello.doc&quot;.</p>
<p>Amazon Drive has rate limiting so you may notice errors in the sync (429 errors). rclone will automatically retry the sync up to 3 times by default (see <code>--retries</code> flag) which should hopefully work around this problem.</p>
<p>Amazon Drive has an internal limit of file sizes that can be uploaded to the service. This limit is not officially published, but all files larger than this will fail.</p>
<p>At the time of writing (Jan 2016) is in the area of 50GB per file. This means that larger files are likely to fail.</p>
<p>Unfortunatly there is no way for rclone to see that this failure is because of file size, so it will retry the operation, as any other failure. To avoid this problem, use <code>--max-size=50GB</code> option to limit the maximum size of uploaded files.</p>
<h2 id="microsoft-one-drive">Microsoft One Drive</h2>
@ -1775,20 +1860,65 @@ y/e/d&gt; y</code></pre>
<h3 id="sha1-checksums">SHA1 checksums</h3>
<p>The SHA1 checksums of the files are checked on upload and download and will be used in the syncing process. You can use the <code>--checksum</code> flag.</p>
<p>Large files which are uploaded in chunks will store their SHA1 on the object as <code>X-Bz-Info-large_file_sha1</code> as recommended by Backblaze.</p>
<h3 id="versions">Versions</h3>
<p>When rclone uploads a new version of a file it creates a <a href="https://www.backblaze.com/b2/docs/file_versions.html">new version of it</a>. Likewise when you delete a file, the old version will still be available.</p>
<p>The old versions of files are visible in the B2 web interface, but not via rclone yet.</p>
<p>Rclone doesn't provide any way of managing old versions (downloading them or deleting them) at the moment. When you <code>purge</code> a bucket, all the old versions will be deleted.</p>
<h3 id="transfers">Transfers</h3>
<p>Backblaze recommends that you do lots of transfers simultaneously for maximum speed. In tests from my SSD equiped laptop the optimum setting is about <code>--transfers 32</code> though higher numbers may be used for a slight speed improvement. The optimum number for you may vary depending on your hardware, how big the files are, how much you want to load your computer, etc. The default of <code>--transfers 4</code> is definitely too low for Backblaze B2 though.</p>
<h3 id="versions">Versions</h3>
<p>When rclone uploads a new version of a file it creates a <a href="https://www.backblaze.com/b2/docs/file_versions.html">new version of it</a>. Likewise when you delete a file, the old version will still be available.</p>
<p>Old versions of files are visible using the <code>--b2-versions</code> flag.</p>
<p>If you wish to remove all the old versions then you can use the <code>rclone cleanup remote:bucket</code> command which will delete all the old versions of files, leaving the current ones intact. You can also supply a path and only old versions under that path will be deleted, eg <code>rclone cleanup remote:bucket/path/to/stuff</code>.</p>
<p>When you <code>purge</code> a bucket, the current and the old versions will be deleted then the bucket will be deleted.</p>
<p>However <code>delete</code> will cause the current versions of the files to become hidden old versions.</p>
<p>Here is a session showing the listing and and retreival of an old version followed by a <code>cleanup</code> of the old versions.</p>
<p>Show current version and all the versions with <code>--b2-versions</code> flag.</p>
<pre><code>$ rclone -q ls b2:cleanup-test
9 one.txt
$ rclone -q --b2-versions ls b2:cleanup-test
9 one.txt
8 one-v2016-07-04-141032-000.txt
16 one-v2016-07-04-141003-000.txt
15 one-v2016-07-02-155621-000.txt</code></pre>
<p>Retreive an old verson</p>
<pre><code>$ rclone -q --b2-versions copy b2:cleanup-test/one-v2016-07-04-141003-000.txt /tmp
$ ls -l /tmp/one-v2016-07-04-141003-000.txt
-rw-rw-r-- 1 ncw ncw 16 Jul 2 17:46 /tmp/one-v2016-07-04-141003-000.txt</code></pre>
<p>Clean up all the old versions and show that they've gone.</p>
<pre><code>$ rclone -q cleanup b2:cleanup-test
$ rclone -q ls b2:cleanup-test
9 one.txt
$ rclone -q --b2-versions ls b2:cleanup-test
9 one.txt</code></pre>
<h3 id="specific-options-5">Specific options</h3>
<p>Here are the command line options specific to this cloud storage system.</p>
<h4 id="b2-chunk-size-valueesize">--b2-chunk-size valuee=SIZE</h4>
<p>When uploading large files chunk the file into this size. Note that these chunks are buffered in memory. 100,000,000 Bytes is the minimim size (default 96M).</p>
<h4 id="b2-upload-cutoffsize">--b2-upload-cutoff=SIZE</h4>
<p>Cutoff for switching to chunked upload (default 4.657GiB == 5GB). Files above this size will be uploaded in chunks of <code>--b2-chunk-size</code>. The default value is the largest file which can be uploaded without chunks.</p>
<h3 id="api">API</h3>
<p>Here are <a href="https://gist.github.com/ncw/166dabf352b399f1cc1c">some notes I made on the backblaze API</a> while integrating it with rclone.</p>
<h4 id="b2-test-modeflag">--b2-test-mode=FLAG</h4>
<p>This is for debugging purposes only.</p>
<p>Setting FLAG to one of the strings below will cause b2 to return specific errors for debugging purposes.</p>
<ul>
<li><code>fail_some_uploads</code></li>
<li><code>expire_some_account_authorization_tokens</code></li>
<li><code>force_cap_exceeded</code></li>
</ul>
<p>These will be set in the <code>X-Bz-Test-Mode</code> header which is documented in the <a href="https://www.backblaze.com/b2/docs/integration_checklist.html">b2 integrations checklist</a>.</p>
<h4 id="b2-versions">--b2-versions</h4>
<p>When set rclone will show and act on older versions of files. For example</p>
<p>Listing without <code>--b2-versions</code></p>
<pre><code>$ rclone -q ls b2:cleanup-test
9 one.txt</code></pre>
<p>And with</p>
<pre><code>$ rclone -q --b2-versions ls b2:cleanup-test
9 one.txt
8 one-v2016-07-04-141032-000.txt
16 one-v2016-07-04-141003-000.txt
15 one-v2016-07-02-155621-000.txt</code></pre>
<p>Showing that the current version is unchanged but older versions can be seen. These have the UTC date that they were uploaded to the server to the nearest millisecond appended to them.</p>
<p>Note that when using <code>--b2-versions</code> no file write operations are permitted, so you can't upload files or delete them.</p>
<h2 id="yandex-disk">Yandex Disk</h2>
<p><a href="https://disk.yandex.com">Yandex Disk</a> is a cloud storage solution created by <a href="http://yandex.com">Yandex</a>.</p>
<p>Yandex paths may be as deep as required, eg <code>remote:directory/subdirectory</code>.</p>
@ -1892,6 +2022,50 @@ nounc = true</code></pre>
<p>This will use UNC paths on <code>c:\src</code> but not on <code>z:\dst</code>. Of course this will cause problems if the absolute path length of a file exceeds 258 characters on z, so only use this option if you have to.</p>
<h2 id="changelog">Changelog</h2>
<ul>
<li>v1.31 - 2016-07-13
<ul>
<li>New Features</li>
<li>Reduce memory on sync by about 50%</li>
<li>Implement --no-traverse flag to stop copy traversing the destination remote.
<ul>
<li>This can be used to reduce memory usage down to the smallest possible.</li>
<li>Useful to copy a small number of files into a large destination folder.</li>
</ul></li>
<li>Implement cleanup command for emptying trash / removing old versions of files
<ul>
<li>Currently B2 only</li>
</ul></li>
<li>Single file handling improved
<ul>
<li>Now copied with --files-from</li>
<li>Automatically sets --no-traverse when copying a single file</li>
</ul></li>
<li>Info on using installing with ansible - thanks Stefan Weichinger</li>
<li>Implement --no-update-modtime flag to stop rclone fixing the remote modified times.</li>
<li>Bug Fixes</li>
<li>Fix move command - stop it running for overlapping Fses - this was causing data loss.</li>
<li>Local</li>
<li>Fix incomplete hashes - this was causing problems for B2.</li>
<li>Amazon Drive</li>
<li>Rename Amazon Cloud Drive to Amazon Drive - no changes to config file needed.</li>
<li>Swift</li>
<li>Add support for non-default project domain - thanks Antonio Messina.</li>
<li>S3</li>
<li>Add instructions on how to use rclone with minio.</li>
<li>Add ap-northeast-2 (Seoul) and ap-south-1 (Mumbai) regions.</li>
<li>Skip setting the modified time for objects &gt; 5GB as it isn't possible.</li>
<li>Backblaze B2</li>
<li>Add --b2-versions flag so old versions can be listed and retreived.</li>
<li>Treat 403 errors (eg cap exceeded) as fatal.</li>
<li>Implement cleanup command for deleting old file versions.</li>
<li>Make error handling compliant with B2 integrations notes.</li>
<li>Fix handling of token expiry.</li>
<li>Implement --b2-test-mode to set <code>X-Bz-Test-Mode</code> header.</li>
<li>Set cutoff for chunked upload to 200MB as per B2 guidelines.</li>
<li>Make upload multi-threaded.</li>
<li>Dropbox</li>
<li>Don't retry 461 errors.</li>
</ul></li>
<li>v1.30 - 2016-06-18
<ul>
<li>New Features</li>
@ -2562,6 +2736,18 @@ h='&#x67;&#x6d;&#x61;&#x69;&#108;&#46;&#x63;&#x6f;&#x6d;';a='&#64;';n='&#106;&#1
document.write('<a h'+'ref'+'="ma'+'ilto'+':'+e+'" clas'+'s="em' + 'ail">'+e+'<\/'+'a'+'>');
// -->
</script><noscript>&#106;&#114;&#x77;&#x39;&#x37;&#50;&#32;&#x61;&#116;&#32;&#x67;&#x6d;&#x61;&#x69;&#108;&#32;&#100;&#x6f;&#116;&#32;&#x63;&#x6f;&#x6d;</noscript></li>
<li>Antonio Messina <script type="text/javascript">
<!--
h='&#x67;&#x6d;&#x61;&#x69;&#108;&#46;&#x63;&#x6f;&#x6d;';a='&#64;';n='&#x61;&#110;&#116;&#x6f;&#110;&#x69;&#x6f;&#46;&#x73;&#46;&#x6d;&#x65;&#x73;&#x73;&#x69;&#110;&#x61;';e=n+a+h;
document.write('<a h'+'ref'+'="ma'+'ilto'+':'+e+'" clas'+'s="em' + 'ail">'+e+'<\/'+'a'+'>');
// -->
</script><noscript>&#x61;&#110;&#116;&#x6f;&#110;&#x69;&#x6f;&#46;&#x73;&#46;&#x6d;&#x65;&#x73;&#x73;&#x69;&#110;&#x61;&#32;&#x61;&#116;&#32;&#x67;&#x6d;&#x61;&#x69;&#108;&#32;&#100;&#x6f;&#116;&#32;&#x63;&#x6f;&#x6d;</noscript></li>
<li>Stefan G. Weichinger <script type="text/javascript">
<!--
h='&#x6f;&#x6f;&#112;&#x73;&#46;&#x63;&#x6f;&#46;&#x61;&#116;';a='&#64;';n='&#x6f;&#102;&#102;&#x69;&#x63;&#x65;';e=n+a+h;
document.write('<a h'+'ref'+'="ma'+'ilto'+':'+e+'" clas'+'s="em' + 'ail">'+e+'<\/'+'a'+'>');
// -->
</script><noscript>&#x6f;&#102;&#102;&#x69;&#x63;&#x65;&#32;&#x61;&#116;&#32;&#x6f;&#x6f;&#112;&#x73;&#32;&#100;&#x6f;&#116;&#32;&#x63;&#x6f;&#32;&#100;&#x6f;&#116;&#32;&#x61;&#116;</noscript></li>
</ul>
<h2 id="contact-the-rclone-project">Contact the rclone project</h2>
<p>The project website is at:</p>

391
MANUAL.md
View File

@ -1,6 +1,6 @@
% rclone(1) User Manual
% Nick Craig-Wood
% Jun 18, 2016
% Jul 13, 2016
Rclone
======
@ -72,6 +72,23 @@ linux binary downloaded files install example
sudo cp rclone.1 /usr/local/share/man/man1/
sudo mandb
Installation with Ansible
-------
This can be done with [Stefan Weichinger's ansible
role](https://github.com/stefangweichinger/ansible-rclone).
Instructions
1. `git clone https://github.com/stefangweichinger/ansible-rclone.git` into your local roles-directory
2. add the role to the hosts you want rclone installed to:
```
- hosts: rclone-hosts
roles:
- rclone
```
Configure
---------
@ -155,6 +172,9 @@ written a trailing / - meaning "copy the contents of this directory".
This applies to all commands and whether you are talking about the
source or destination.
See the `--no-traverse` option for controlling whether rclone lists
the destination directory or not.
### rclone sync source:path dest:path ###
Sync the source to the destination, changing the destination
@ -178,16 +198,18 @@ go there.
### move source:path dest:path ###
Moves the source to the destination.
Moves the contents of the source directory to the destination
directory. Rclone will error if the source and destination overlap.
If there are no filters in use this is equivalent to a copy followed
by a purge, but may use server side operations to speed it up if
possible.
If no filters are in use and if possible this will server side move
`source:path` into `dest:path`. After this `source:path` will no
longer longer exist.
If filters are in use then it is equivalent to a copy followed by
delete, followed by an rmdir (which only removes the directory if
empty). The individual file moves will be moved with server side
operations if possible.
Otherwise for each file in `source:path` selected by the filters (if
any) this will move it into `dest:path`. If possible a server side
move will be used, otherwise it will copy it (server side if possible)
into `dest:path` then delete the original (if no errors on copy) in
`source:path`.
**Important**: Since this can cause data loss, test first with the
--dry-run flag.
@ -262,6 +284,11 @@ don't match. It doesn't alter the source or destination.
`--size-only` may be used to only compare the sizes, not the MD5SUMs.
### rclone cleanup remote:path ###
Clean up the remote if possible. Empty the trash or delete old file
versions. Not supported by all remotes.
### rclone dedupe remote:path ###
By default `dedup` interactively finds duplicate files and offers to
@ -349,6 +376,34 @@ Enter an interactive configuration session.
Prints help on rclone commands and options.
Copying single files
--------------------
rclone normally syncs or copies directories. However if the source
remote points to a file, rclone will just copy that file. The
destination remote must point to a directory - rclone will give the
error `Failed to create file system for "remote:file": is a file not a
directory` if it isn't.
For example, suppose you have a remote with a file in called
`test.jpg`, then you could copy just that file like this
rclone copy remote:test.jpg /tmp/download
The file `test.jpg` will be placed inside `/tmp/download`.
This is equivalent to specifying
rclone copy --no-traverse --files-from /tmp/files remote: /tmp/download
Where `/tmp/files` contains the single line
test.jpg
It is recommended to use `copy` when copying single files not `sync`.
They have pretty much the same effect but `copy` will use a lot less
memory.
Quoting and the shell
---------------------
@ -592,6 +647,14 @@ uploaded compressed files.
There is no need to set this in normal operation, and doing so will
decrease the network transfer efficiency of rclone.
### --no-update-modtime ###
When using this flag, rclone won't update modification times of remote
files if they are incorrect as it would normally.
This can be used if the remote is being synced with another tool also
(eg the Google Drive client).
### -q, --quiet ###
Normally rclone outputs stats and a completion message. If you set
@ -629,12 +692,15 @@ The default is `1m`. Use 0 to disable.
This option allows you to specify when files on your destination are
deleted when you sync folders.
Specifying the value `--delete-before` will delete all files present on the
destination, but not on the source *before* starting the transfer
of any new or updated files.
Specifying the value `--delete-before` will delete all files present
on the destination, but not on the source *before* starting the
transfer of any new or updated files. This uses extra memory as it
has to store the source listing before proceeding.
Specifying `--delete-during` (default value) will delete files while checking
and uploading files. This is usually the fastest option.
Specifying `--delete-during` (default value) will delete files while
checking and uploading files. This is usually the fastest option.
Currently this works the same as `--delete-after` but it may change in
the future.
Specifying `--delete-after` will delay deletion of files until all new/updated
files have been successfully transfered.
@ -799,6 +865,24 @@ This option defaults to `false`.
**This should be used only for testing.**
### --no-traverse ###
The `--no-traverse` flag controls whether the destination file system
is traversed when using the `copy` or `move` commands.
If you are only copying a small number of files and/or have a large
number of files on the destination then `--no-traverse` will stop
rclone listing the destination and save time.
However if you are copying a large number of files, escpecially if you
are doing a copy where lots of the files haven't changed and won't
need copying then you shouldn't use `--no-traverse`.
It can also be used to reduce the memory usage of rclone when copying
- `rclone --no-traverse copy src dst` won't load either the source or
destination listings into memory so will use the minimum amount of
memory.
Filtering
---------
@ -1075,7 +1159,7 @@ This would exclude
A similar process is done on directory entries before recursing into
them. This only works on remotes which have a concept of directory
(Eg local, drive, onedrive, amazon cloud drive) and not on bucket
(Eg local, google drive, onedrive, amazon drive) and not on bucket
based remotes (eg s3, swift, google compute storage, b2).
## Adding filtering rules ##
@ -1182,6 +1266,41 @@ Prepare a file like this `files-from.txt`
Then use as `--files-from files-from.txt`. This will only transfer
`file1.jpg` and `file2.jpg` providing they exist.
For example, let's say you had a few files you want to back up
regularly with these absolute paths:
/home/user1/important
/home/user1/dir/file
/home/user2/stuff
To copy these you'd find a common subdirectory - in this case `/home`
and put the remaining files in `files-from.txt` with or without
leading `/`, eg
user1/important
user1/dir/file
user2/stuff
You could then copy these to a remote like this
rclone copy --files-from files-from.txt /home remote:backup
The 3 files will arrive in `remote:backup` with the paths as in the
`files-from.txt`.
You could of course choose `/` as the root too in which case your
`files-from.txt` might look like this.
/home/user1/important
/home/user1/dir/file
/home/user2/stuff
And you would transfer it like this
rclone copy --files-from files-from.txt / remote:backup
In this case there will be an extra `home` directory on the remote.
### `--min-size` - Don't transfer any file smaller than this ###
This option controls the minimum size file which will be transferred.
@ -1819,6 +1938,63 @@ removed).
Because this is a json dump, it is encoding the `/` as `\/`, so if you
use the secret key as `xxxxxx/xxxx` it will work fine.
### Minio ###
[Minio](https://minio.io/) is an object storage server built for cloud application developers and devops.
It is very easy to install and provides an S3 compatible server which can be used by rclone.
To use it, install Minio following the instructions from the web site.
When it configures itself Minio will print something like this
```
AccessKey: WLGDGYAQYIGI833EV05A SecretKey: BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF Region: us-east-1
Minio Object Storage:
http://127.0.0.1:9000
http://10.0.0.3:9000
Minio Browser:
http://127.0.0.1:9000
http://10.0.0.3:9000
```
These details need to go into `rclone config` like this. Note that it
is important to put the region in as stated above.
```
env_auth> 1
access_key_id> WLGDGYAQYIGI833EV05A
secret_access_key> BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF
region> us-east-1
endpoint> http://10.0.0.3:9000
location_constraint>
server_side_encryption>
```
Which makes the config file look like this
```
[minio]
env_auth = false
access_key_id = WLGDGYAQYIGI833EV05A
secret_access_key = BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF
region = us-east-1
endpoint = http://10.0.0.3:9000
location_constraint =
server_side_encryption =
```
Minio doesn't support all the features of S3 yet. In particular it
doesn't support MD5 checksums (ETags) or metadata. This means rclone
can't check MD5SUMs or store the modified date. However you can work
around this with the `--size-only` flag of rclone.
So once set up, for example to copy files into a bucket
rclone --size-only copy /path/to/files minio:bucket
Swift
----------------------------------------
@ -1891,6 +2067,8 @@ User domain - optional (v3 auth)
domain> Default
Tenant name - optional
tenant>
Tenant domain - optional (v3 auth)
tenant_domain>
Region name - optional
region>
Storage URL - optional
@ -2271,7 +2449,7 @@ Paths are specified as `remote:path`
Paths may be as deep as required, eg `remote:directory/subdirectory`.
The initial setup for Amazon cloud drive involves getting a token from
The initial setup for Amazon Drive involves getting a token from
Amazon which you need to do in your browser. `rclone config` walks
you through it.
@ -2344,21 +2522,21 @@ you to unblock it temporarily if you are running a host firewall.
Once configured you can then use `rclone` like this,
List directories in top level of your Amazon cloud drive
List directories in top level of your Amazon Drive
rclone lsd remote:
List all the files in your Amazon cloud drive
List all the files in your Amazon Drive
rclone ls remote:
To copy a local directory to an Amazon cloud drive directory called backup
To copy a local directory to an Amazon Drive directory called backup
rclone copy /home/source remote:backup
### Modified time and MD5SUMs ###
Amazon cloud drive doesn't allow modification times to be changed via
Amazon Drive doesn't allow modification times to be changed via
the API so these won't be accurate or used for syncing.
It does store MD5SUMs so for a more accurate sync, you can use the
@ -2369,7 +2547,7 @@ It does store MD5SUMs so for a more accurate sync, you can use the
Any files you delete with rclone will end up in the trash. Amazon
don't provide an API to permanently delete files, nor to empty the
trash, so you will have to do that with one of Amazon's apps or via
the Amazon cloud drive website.
the Amazon Drive website.
### Specific options ###
@ -2379,9 +2557,9 @@ system.
#### --acd-templink-threshold=SIZE ####
Files this size or more will be downloaded via their `tempLink`. This
is to work around a problem with Amazon Drive which blocks
downloads of files bigger than about 10GB. The default for this is
9GB which shouldn't need to be changed.
is to work around a problem with Amazon Drive which blocks downloads
of files bigger than about 10GB. The default for this is 9GB which
shouldn't need to be changed.
To download files above this threshold, rclone requests a `tempLink`
which downloads the file through a temporary URL directly from the
@ -2389,17 +2567,17 @@ underlying S3 storage.
### Limitations ###
Note that Amazon cloud drive is case insensitive so you can't have a
Note that Amazon Drive is case insensitive so you can't have a
file called "Hello.doc" and one called "hello.doc".
Amazon cloud drive has rate limiting so you may notice errors in the
Amazon Drive has rate limiting so you may notice errors in the
sync (429 errors). rclone will automatically retry the sync up to 3
times by default (see `--retries` flag) which should hopefully work
around this problem.
Amazon cloud drive has an internal limit of file sizes that can be
uploaded to the service. This limit is not officially published,
but all files larger than this will fail.
Amazon Drive has an internal limit of file sizes that can be uploaded
to the service. This limit is not officially published, but all files
larger than this will fail.
At the time of writing (Jan 2016) is in the area of 50GB per file.
This means that larger files are likely to fail.
@ -2782,20 +2960,6 @@ will be used in the syncing process. You can use the `--checksum` flag.
Large files which are uploaded in chunks will store their SHA1 on the
object as `X-Bz-Info-large_file_sha1` as recommended by Backblaze.
### Versions ###
When rclone uploads a new version of a file it creates a [new version
of it](https://www.backblaze.com/b2/docs/file_versions.html).
Likewise when you delete a file, the old version will still be
available.
The old versions of files are visible in the B2 web interface, but not
via rclone yet.
Rclone doesn't provide any way of managing old versions (downloading
them or deleting them) at the moment. When you `purge` a bucket, all
the old versions will be deleted.
### Transfers ###
Backblaze recommends that you do lots of transfers simultaneously for
@ -2806,6 +2970,64 @@ depending on your hardware, how big the files are, how much you want
to load your computer, etc. The default of `--transfers 4` is
definitely too low for Backblaze B2 though.
### Versions ###
When rclone uploads a new version of a file it creates a [new version
of it](https://www.backblaze.com/b2/docs/file_versions.html).
Likewise when you delete a file, the old version will still be
available.
Old versions of files are visible using the `--b2-versions` flag.
If you wish to remove all the old versions then you can use the
`rclone cleanup remote:bucket` command which will delete all the old
versions of files, leaving the current ones intact. You can also
supply a path and only old versions under that path will be deleted,
eg `rclone cleanup remote:bucket/path/to/stuff`.
When you `purge` a bucket, the current and the old versions will be
deleted then the bucket will be deleted.
However `delete` will cause the current versions of the files to
become hidden old versions.
Here is a session showing the listing and and retreival of an old
version followed by a `cleanup` of the old versions.
Show current version and all the versions with `--b2-versions` flag.
```
$ rclone -q ls b2:cleanup-test
9 one.txt
$ rclone -q --b2-versions ls b2:cleanup-test
9 one.txt
8 one-v2016-07-04-141032-000.txt
16 one-v2016-07-04-141003-000.txt
15 one-v2016-07-02-155621-000.txt
```
Retreive an old verson
```
$ rclone -q --b2-versions copy b2:cleanup-test/one-v2016-07-04-141003-000.txt /tmp
$ ls -l /tmp/one-v2016-07-04-141003-000.txt
-rw-rw-r-- 1 ncw ncw 16 Jul 2 17:46 /tmp/one-v2016-07-04-141003-000.txt
```
Clean up all the old versions and show that they've gone.
```
$ rclone -q cleanup b2:cleanup-test
$ rclone -q ls b2:cleanup-test
9 one.txt
$ rclone -q --b2-versions ls b2:cleanup-test
9 one.txt
```
### Specific options ###
Here are the command line options specific to this cloud storage
@ -2824,11 +3046,48 @@ Cutoff for switching to chunked upload (default 4.657GiB ==
`--b2-chunk-size`. The default value is the largest file which can be
uploaded without chunks.
### API ###
#### --b2-test-mode=FLAG ####
Here are [some notes I made on the backblaze
API](https://gist.github.com/ncw/166dabf352b399f1cc1c) while
integrating it with rclone.
This is for debugging purposes only.
Setting FLAG to one of the strings below will cause b2 to return
specific errors for debugging purposes.
* `fail_some_uploads`
* `expire_some_account_authorization_tokens`
* `force_cap_exceeded`
These will be set in the `X-Bz-Test-Mode` header which is documented
in the [b2 integrations
checklist](https://www.backblaze.com/b2/docs/integration_checklist.html).
#### --b2-versions ####
When set rclone will show and act on older versions of files. For example
Listing without `--b2-versions`
```
$ rclone -q ls b2:cleanup-test
9 one.txt
```
And with
```
$ rclone -q --b2-versions ls b2:cleanup-test
9 one.txt
8 one-v2016-07-04-141032-000.txt
16 one-v2016-07-04-141003-000.txt
15 one-v2016-07-02-155621-000.txt
```
Showing that the current version is unchanged but older versions can
be seen. These have the UTC date that they were uploaded to the
server to the nearest millisecond appended to them.
Note that when using `--b2-versions` no file write operations are
permitted, so you can't upload files or delete them.
Yandex Disk
----------------------------------------
@ -3012,6 +3271,42 @@ file exceeds 258 characters on z, so only use this option if you have to.
Changelog
---------
* v1.31 - 2016-07-13
* New Features
* Reduce memory on sync by about 50%
* Implement --no-traverse flag to stop copy traversing the destination remote.
* This can be used to reduce memory usage down to the smallest possible.
* Useful to copy a small number of files into a large destination folder.
* Implement cleanup command for emptying trash / removing old versions of files
* Currently B2 only
* Single file handling improved
* Now copied with --files-from
* Automatically sets --no-traverse when copying a single file
* Info on using installing with ansible - thanks Stefan Weichinger
* Implement --no-update-modtime flag to stop rclone fixing the remote modified times.
* Bug Fixes
* Fix move command - stop it running for overlapping Fses - this was causing data loss.
* Local
* Fix incomplete hashes - this was causing problems for B2.
* Amazon Drive
* Rename Amazon Cloud Drive to Amazon Drive - no changes to config file needed.
* Swift
* Add support for non-default project domain - thanks Antonio Messina.
* S3
* Add instructions on how to use rclone with minio.
* Add ap-northeast-2 (Seoul) and ap-south-1 (Mumbai) regions.
* Skip setting the modified time for objects > 5GB as it isn't possible.
* Backblaze B2
* Add --b2-versions flag so old versions can be listed and retreived.
* Treat 403 errors (eg cap exceeded) as fatal.
* Implement cleanup command for deleting old file versions.
* Make error handling compliant with B2 integrations notes.
* Fix handling of token expiry.
* Implement --b2-test-mode to set `X-Bz-Test-Mode` header.
* Set cutoff for chunked upload to 200MB as per B2 guidelines.
* Make upload multi-threaded.
* Dropbox
* Don't retry 461 errors.
* v1.30 - 2016-06-18
* New Features
* Directory listing code reworked for more features and better error reporting (thanks to Klaus Post for help). This enables
@ -3593,6 +3888,8 @@ Contributors
* Leigh Klotz <klotz@quixey.com>
* Romain Lapray <lapray.romain@gmail.com>
* Justin R. Wilson <jrw972@gmail.com>
* Antonio Messina <antonio.s.messina@gmail.com>
* Stefan G. Weichinger <office@oops.co.at>
Contact the rclone project
--------------------------

View File

@ -1,6 +1,6 @@
rclone(1) User Manual
Nick Craig-Wood
Jun 18, 2016
Jul 13, 2016
@ -75,6 +75,21 @@ linux binary downloaded files install example
sudo mandb
Installation with Ansible
This can be done with Stefan Weichinger's ansible role.
Instructions
1. git clone https://github.com/stefangweichinger/ansible-rclone.git
into your local roles-directory
2. add the role to the hosts you want rclone installed to:
- hosts: rclone-hosts
roles:
- rclone
Configure
First you'll need to configure rclone. As the object storage systems
@ -156,6 +171,9 @@ written a trailing / - meaning "copy the contents of this directory".
This applies to all commands and whether you are talking about the
source or destination.
See the --no-traverse option for controlling whether rclone lists the
destination directory or not.
rclone sync source:path dest:path
Sync the source to the destination, changing the destination only.
@ -179,15 +197,17 @@ go there.
move source:path dest:path
Moves the source to the destination.
Moves the contents of the source directory to the destination directory.
Rclone will error if the source and destination overlap.
If there are no filters in use this is equivalent to a copy followed by
a purge, but may use server side operations to speed it up if possible.
If no filters are in use and if possible this will server side move
source:path into dest:path. After this source:path will no longer longer
exist.
If filters are in use then it is equivalent to a copy followed by
delete, followed by an rmdir (which only removes the directory if
empty). The individual file moves will be moved with server side
operations if possible.
Otherwise for each file in source:path selected by the filters (if any)
this will move it into dest:path. If possible a server side move will be
used, otherwise it will copy it (server side if possible) into dest:path
then delete the original (if no errors on copy) in source:path.
IMPORTANT: Since this can cause data loss, test first with the --dry-run
flag.
@ -262,6 +282,11 @@ alter the source or destination.
--size-only may be used to only compare the sizes, not the MD5SUMs.
rclone cleanup remote:path
Clean up the remote if possible. Empty the trash or delete old file
versions. Not supported by all remotes.
rclone dedupe remote:path
By default dedup interactively finds duplicate files and offers to
@ -350,6 +375,34 @@ rclone help
Prints help on rclone commands and options.
Copying single files
rclone normally syncs or copies directories. However if the source
remote points to a file, rclone will just copy that file. The
destination remote must point to a directory - rclone will give the
error
Failed to create file system for "remote:file": is a file not a directory
if it isn't.
For example, suppose you have a remote with a file in called test.jpg,
then you could copy just that file like this
rclone copy remote:test.jpg /tmp/download
The file test.jpg will be placed inside /tmp/download.
This is equivalent to specifying
rclone copy --no-traverse --files-from /tmp/files remote: /tmp/download
Where /tmp/files contains the single line
test.jpg
It is recommended to use copy when copying single files not sync. They
have pretty much the same effect but copy will use a lot less memory.
Quoting and the shell
When you are typing commands to your computer you are using something
@ -589,6 +642,14 @@ compressed files.
There is no need to set this in normal operation, and doing so will
decrease the network transfer efficiency of rclone.
--no-update-modtime
When using this flag, rclone won't update modification times of remote
files if they are incorrect as it would normally.
This can be used if the remote is being synced with another tool also
(eg the Google Drive client).
-q, --quiet
Normally rclone outputs stats and a completion message. If you set this
@ -628,10 +689,13 @@ deleted when you sync folders.
Specifying the value --delete-before will delete all files present on
the destination, but not on the source _before_ starting the transfer of
any new or updated files.
any new or updated files. This uses extra memory as it has to store the
source listing before proceeding.
Specifying --delete-during (default value) will delete files while
checking and uploading files. This is usually the fastest option.
Currently this works the same as --delete-after but it may change in the
future.
Specifying --delete-after will delay deletion of files until all
new/updated files have been successfully transfered.
@ -790,6 +854,24 @@ This option defaults to false.
THIS SHOULD BE USED ONLY FOR TESTING.
--no-traverse
The --no-traverse flag controls whether the destination file system is
traversed when using the copy or move commands.
If you are only copying a small number of files and/or have a large
number of files on the destination then --no-traverse will stop rclone
listing the destination and save time.
However if you are copying a large number of files, escpecially if you
are doing a copy where lots of the files haven't changed and won't need
copying then you shouldn't use --no-traverse.
It can also be used to reduce the memory usage of rclone when copying -
rclone --no-traverse copy src dst won't load either the source or
destination listings into memory so will use the minimum amount of
memory.
Filtering
@ -1064,7 +1146,7 @@ This would exclude
A similar process is done on directory entries before recursing into
them. This only works on remotes which have a concept of directory (Eg
local, drive, onedrive, amazon cloud drive) and not on bucket based
local, google drive, onedrive, amazon drive) and not on bucket based
remotes (eg s3, swift, google compute storage, b2).
@ -1172,6 +1254,40 @@ Prepare a file like this files-from.txt
Then use as --files-from files-from.txt. This will only transfer
file1.jpg and file2.jpg providing they exist.
For example, let's say you had a few files you want to back up regularly
with these absolute paths:
/home/user1/important
/home/user1/dir/file
/home/user2/stuff
To copy these you'd find a common subdirectory - in this case /home and
put the remaining files in files-from.txt with or without leading /, eg
user1/important
user1/dir/file
user2/stuff
You could then copy these to a remote like this
rclone copy --files-from files-from.txt /home remote:backup
The 3 files will arrive in remote:backup with the paths as in the
files-from.txt.
You could of course choose / as the root too in which case your
files-from.txt might look like this.
/home/user1/important
/home/user1/dir/file
/home/user2/stuff
And you would transfer it like this
rclone copy --files-from files-from.txt / remote:backup
In this case there will be an extra home directory on the remote.
--min-size - Don't transfer any file smaller than this
This option controls the minimum size file which will be transferred.
@ -1862,6 +1978,59 @@ removed).
Because this is a json dump, it is encoding the / as \/, so if you use
the secret key as xxxxxx/xxxx it will work fine.
Minio
Minio is an object storage server built for cloud application developers
and devops.
It is very easy to install and provides an S3 compatible server which
can be used by rclone.
To use it, install Minio following the instructions from the web site.
When it configures itself Minio will print something like this
AccessKey: WLGDGYAQYIGI833EV05A SecretKey: BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF Region: us-east-1
Minio Object Storage:
http://127.0.0.1:9000
http://10.0.0.3:9000
Minio Browser:
http://127.0.0.1:9000
http://10.0.0.3:9000
These details need to go into rclone config like this. Note that it is
important to put the region in as stated above.
env_auth> 1
access_key_id> WLGDGYAQYIGI833EV05A
secret_access_key> BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF
region> us-east-1
endpoint> http://10.0.0.3:9000
location_constraint>
server_side_encryption>
Which makes the config file look like this
[minio]
env_auth = false
access_key_id = WLGDGYAQYIGI833EV05A
secret_access_key = BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF
region = us-east-1
endpoint = http://10.0.0.3:9000
location_constraint =
server_side_encryption =
Minio doesn't support all the features of S3 yet. In particular it
doesn't support MD5 checksums (ETags) or metadata. This means rclone
can't check MD5SUMs or store the modified date. However you can work
around this with the --size-only flag of rclone.
So once set up, for example to copy files into a bucket
rclone --size-only copy /path/to/files minio:bucket
Swift
@ -1934,6 +2103,8 @@ This will guide you through an interactive setup process.
domain> Default
Tenant name - optional
tenant>
Tenant domain - optional (v3 auth)
tenant_domain>
Region name - optional
region>
Storage URL - optional
@ -2302,9 +2473,9 @@ Paths are specified as remote:path
Paths may be as deep as required, eg remote:directory/subdirectory.
The initial setup for Amazon cloud drive involves getting a token from
Amazon which you need to do in your browser. rclone config walks you
through it.
The initial setup for Amazon Drive involves getting a token from Amazon
which you need to do in your browser. rclone config walks you through
it.
Here is an example of how to make a remote called remote. First run:
@ -2373,23 +2544,22 @@ temporarily if you are running a host firewall.
Once configured you can then use rclone like this,
List directories in top level of your Amazon cloud drive
List directories in top level of your Amazon Drive
rclone lsd remote:
List all the files in your Amazon cloud drive
List all the files in your Amazon Drive
rclone ls remote:
To copy a local directory to an Amazon cloud drive directory called
backup
To copy a local directory to an Amazon Drive directory called backup
rclone copy /home/source remote:backup
Modified time and MD5SUMs
Amazon cloud drive doesn't allow modification times to be changed via
the API so these won't be accurate or used for syncing.
Amazon Drive doesn't allow modification times to be changed via the API
so these won't be accurate or used for syncing.
It does store MD5SUMs so for a more accurate sync, you can use the
--checksum flag.
@ -2399,7 +2569,7 @@ Deleting files
Any files you delete with rclone will end up in the trash. Amazon don't
provide an API to permanently delete files, nor to empty the trash, so
you will have to do that with one of Amazon's apps or via the Amazon
cloud drive website.
Drive website.
Specific options
@ -2408,8 +2578,8 @@ Here are the command line options specific to this cloud storage system.
--acd-templink-threshold=SIZE
Files this size or more will be downloaded via their tempLink. This is
to work around a problem with Amazon Drive which blocks downloads
of files bigger than about 10GB. The default for this is 9GB which
to work around a problem with Amazon Drive which blocks downloads of
files bigger than about 10GB. The default for this is 9GB which
shouldn't need to be changed.
To download files above this threshold, rclone requests a tempLink which
@ -2418,17 +2588,17 @@ S3 storage.
Limitations
Note that Amazon cloud drive is case insensitive so you can't have a
file called "Hello.doc" and one called "hello.doc".
Note that Amazon Drive is case insensitive so you can't have a file
called "Hello.doc" and one called "hello.doc".
Amazon cloud drive has rate limiting so you may notice errors in the
sync (429 errors). rclone will automatically retry the sync up to 3
times by default (see --retries flag) which should hopefully work around
this problem.
Amazon Drive has rate limiting so you may notice errors in the sync (429
errors). rclone will automatically retry the sync up to 3 times by
default (see --retries flag) which should hopefully work around this
problem.
Amazon cloud drive has an internal limit of file sizes that can be
uploaded to the service. This limit is not officially published, but all
files larger than this will fail.
Amazon Drive has an internal limit of file sizes that can be uploaded to
the service. This limit is not officially published, but all files
larger than this will fail.
At the time of writing (Jan 2016) is in the area of 50GB per file. This
means that larger files are likely to fail.
@ -2802,19 +2972,6 @@ will be used in the syncing process. You can use the --checksum flag.
Large files which are uploaded in chunks will store their SHA1 on the
object as X-Bz-Info-large_file_sha1 as recommended by Backblaze.
Versions
When rclone uploads a new version of a file it creates a new version of
it. Likewise when you delete a file, the old version will still be
available.
The old versions of files are visible in the B2 web interface, but not
via rclone yet.
Rclone doesn't provide any way of managing old versions (downloading
them or deleting them) at the moment. When you purge a bucket, all the
old versions will be deleted.
Transfers
Backblaze recommends that you do lots of transfers simultaneously for
@ -2825,6 +2982,57 @@ hardware, how big the files are, how much you want to load your
computer, etc. The default of --transfers 4 is definitely too low for
Backblaze B2 though.
Versions
When rclone uploads a new version of a file it creates a new version of
it. Likewise when you delete a file, the old version will still be
available.
Old versions of files are visible using the --b2-versions flag.
If you wish to remove all the old versions then you can use the
rclone cleanup remote:bucket command which will delete all the old
versions of files, leaving the current ones intact. You can also supply
a path and only old versions under that path will be deleted, eg
rclone cleanup remote:bucket/path/to/stuff.
When you purge a bucket, the current and the old versions will be
deleted then the bucket will be deleted.
However delete will cause the current versions of the files to become
hidden old versions.
Here is a session showing the listing and and retreival of an old
version followed by a cleanup of the old versions.
Show current version and all the versions with --b2-versions flag.
$ rclone -q ls b2:cleanup-test
9 one.txt
$ rclone -q --b2-versions ls b2:cleanup-test
9 one.txt
8 one-v2016-07-04-141032-000.txt
16 one-v2016-07-04-141003-000.txt
15 one-v2016-07-02-155621-000.txt
Retreive an old verson
$ rclone -q --b2-versions copy b2:cleanup-test/one-v2016-07-04-141003-000.txt /tmp
$ ls -l /tmp/one-v2016-07-04-141003-000.txt
-rw-rw-r-- 1 ncw ncw 16 Jul 2 17:46 /tmp/one-v2016-07-04-141003-000.txt
Clean up all the old versions and show that they've gone.
$ rclone -q cleanup b2:cleanup-test
$ rclone -q ls b2:cleanup-test
9 one.txt
$ rclone -q --b2-versions ls b2:cleanup-test
9 one.txt
Specific options
Here are the command line options specific to this cloud storage system.
@ -2841,10 +3049,44 @@ Cutoff for switching to chunked upload (default 4.657GiB == 5GB). Files
above this size will be uploaded in chunks of --b2-chunk-size. The
default value is the largest file which can be uploaded without chunks.
API
--b2-test-mode=FLAG
Here are some notes I made on the backblaze API while integrating it
with rclone.
This is for debugging purposes only.
Setting FLAG to one of the strings below will cause b2 to return
specific errors for debugging purposes.
- fail_some_uploads
- expire_some_account_authorization_tokens
- force_cap_exceeded
These will be set in the X-Bz-Test-Mode header which is documented in
the b2 integrations checklist.
--b2-versions
When set rclone will show and act on older versions of files. For
example
Listing without --b2-versions
$ rclone -q ls b2:cleanup-test
9 one.txt
And with
$ rclone -q --b2-versions ls b2:cleanup-test
9 one.txt
8 one-v2016-07-04-141032-000.txt
16 one-v2016-07-04-141003-000.txt
15 one-v2016-07-02-155621-000.txt
Showing that the current version is unchanged but older versions can be
seen. These have the UTC date that they were uploaded to the server to
the nearest millisecond appended to them.
Note that when using --b2-versions no file write operations are
permitted, so you can't upload files or delete them.
Yandex Disk
@ -3023,6 +3265,52 @@ characters on z, so only use this option if you have to.
Changelog
- v1.31 - 2016-07-13
- New Features
- Reduce memory on sync by about 50%
- Implement --no-traverse flag to stop copy traversing the
destination remote.
- This can be used to reduce memory usage down to the
smallest possible.
- Useful to copy a small number of files into a large
destination folder.
- Implement cleanup command for emptying trash / removing old
versions of files
- Currently B2 only
- Single file handling improved
- Now copied with --files-from
- Automatically sets --no-traverse when copying a single file
- Info on using installing with ansible - thanks Stefan Weichinger
- Implement --no-update-modtime flag to stop rclone fixing the
remote modified times.
- Bug Fixes
- Fix move command - stop it running for overlapping Fses - this
was causing data loss.
- Local
- Fix incomplete hashes - this was causing problems for B2.
- Amazon Drive
- Rename Amazon Cloud Drive to Amazon Drive - no changes to config
file needed.
- Swift
- Add support for non-default project domain - thanks
Antonio Messina.
- S3
- Add instructions on how to use rclone with minio.
- Add ap-northeast-2 (Seoul) and ap-south-1 (Mumbai) regions.
- Skip setting the modified time for objects > 5GB as it
isn't possible.
- Backblaze B2
- Add --b2-versions flag so old versions can be listed
and retreived.
- Treat 403 errors (eg cap exceeded) as fatal.
- Implement cleanup command for deleting old file versions.
- Make error handling compliant with B2 integrations notes.
- Fix handling of token expiry.
- Implement --b2-test-mode to set X-Bz-Test-Mode header.
- Set cutoff for chunked upload to 200MB as per B2 guidelines.
- Make upload multi-threaded.
- Dropbox
- Don't retry 461 errors.
- v1.30 - 2016-06-18
- New Features
- Directory listing code reworked for more features and better
@ -3639,6 +3927,8 @@ Contributors
- Leigh Klotz klotz@quixey.com
- Romain Lapray lapray.romain@gmail.com
- Justin R. Wilson jrw972@gmail.com
- Antonio Messina antonio.s.messina@gmail.com
- Stefan G. Weichinger office@oops.co.at
Contact the rclone project

View File

@ -1,12 +1,48 @@
---
title: "Documentation"
description: "Rclone Changelog"
date: "2016-06-18"
date: "2016-07-13"
---
Changelog
---------
* v1.31 - 2016-07-13
* New Features
* Reduce memory on sync by about 50%
* Implement --no-traverse flag to stop copy traversing the destination remote.
* This can be used to reduce memory usage down to the smallest possible.
* Useful to copy a small number of files into a large destination folder.
* Implement cleanup command for emptying trash / removing old versions of files
* Currently B2 only
* Single file handling improved
* Now copied with --files-from
* Automatically sets --no-traverse when copying a single file
* Info on using installing with ansible - thanks Stefan Weichinger
* Implement --no-update-modtime flag to stop rclone fixing the remote modified times.
* Bug Fixes
* Fix move command - stop it running for overlapping Fses - this was causing data loss.
* Local
* Fix incomplete hashes - this was causing problems for B2.
* Amazon Drive
* Rename Amazon Cloud Drive to Amazon Drive - no changes to config file needed.
* Swift
* Add support for non-default project domain - thanks Antonio Messina.
* S3
* Add instructions on how to use rclone with minio.
* Add ap-northeast-2 (Seoul) and ap-south-1 (Mumbai) regions.
* Skip setting the modified time for objects > 5GB as it isn't possible.
* Backblaze B2
* Add --b2-versions flag so old versions can be listed and retreived.
* Treat 403 errors (eg cap exceeded) as fatal.
* Implement cleanup command for deleting old file versions.
* Make error handling compliant with B2 integrations notes.
* Fix handling of token expiry.
* Implement --b2-test-mode to set `X-Bz-Test-Mode` header.
* Set cutoff for chunked upload to 200MB as per B2 guidelines.
* Make upload multi-threaded.
* Dropbox
* Don't retry 461 errors.
* v1.30 - 2016-06-18
* New Features
* Directory listing code reworked for more features and better error reporting (thanks to Klaus Post for help). This enables

View File

@ -2,40 +2,40 @@
title: "Rclone downloads"
description: "Download rclone binaries for your OS."
type: page
date: "2016-06-18"
date: "2016-07-13"
---
Rclone Download v1.30
Rclone Download v1.31
=====================
* Windows
* [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.30-windows-386.zip)
* [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.30-windows-amd64.zip)
* [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.31-windows-386.zip)
* [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.31-windows-amd64.zip)
* OSX
* [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.30-osx-386.zip)
* [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.30-osx-amd64.zip)
* [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.31-osx-386.zip)
* [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.31-osx-amd64.zip)
* Linux
* [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.30-linux-386.zip)
* [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.30-linux-amd64.zip)
* [ARM - 32 Bit](http://downloads.rclone.org/rclone-v1.30-linux-arm.zip)
* [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.31-linux-386.zip)
* [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.31-linux-amd64.zip)
* [ARM - 32 Bit](http://downloads.rclone.org/rclone-v1.31-linux-arm.zip)
* FreeBSD
* [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.30-freebsd-386.zip)
* [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.30-freebsd-amd64.zip)
* [ARM - 32 Bit](http://downloads.rclone.org/rclone-v1.30-freebsd-arm.zip)
* [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.31-freebsd-386.zip)
* [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.31-freebsd-amd64.zip)
* [ARM - 32 Bit](http://downloads.rclone.org/rclone-v1.31-freebsd-arm.zip)
* NetBSD
* [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.30-netbsd-386.zip)
* [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.30-netbsd-amd64.zip)
* [ARM - 32 Bit](http://downloads.rclone.org/rclone-v1.30-netbsd-arm.zip)
* [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.31-netbsd-386.zip)
* [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.31-netbsd-amd64.zip)
* [ARM - 32 Bit](http://downloads.rclone.org/rclone-v1.31-netbsd-arm.zip)
* OpenBSD
* [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.30-openbsd-386.zip)
* [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.30-openbsd-amd64.zip)
* [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.31-openbsd-386.zip)
* [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.31-openbsd-amd64.zip)
* Plan 9
* [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.30-plan9-386.zip)
* [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.30-plan9-amd64.zip)
* [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.31-plan9-386.zip)
* [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.31-plan9-amd64.zip)
* Solaris
* [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.30-solaris-amd64.zip)
* [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.31-solaris-amd64.zip)
You can also find a [mirror of the downloads on github](https://github.com/ncw/rclone/releases/tag/v1.30).
You can also find a [mirror of the downloads on github](https://github.com/ncw/rclone/releases/tag/v1.31).
Downloads for scripting
=======================

View File

@ -1,4 +1,4 @@
package fs
// Version of rclone
var Version = "v1.30"
var Version = "v1.31"

521
rclone.1
View File

@ -1,7 +1,7 @@
.\"t
.\" Automatically generated by Pandoc 1.16.0.2
.\"
.TH "rclone" "1" "Jun 18, 2016" "User Manual" ""
.TH "rclone" "1" "Jul 13, 2016" "User Manual" ""
.hy
.SH Rclone
.PP
@ -99,7 +99,26 @@ sudo\ chmod\ 755\ /usr/sbin/rclone
#install\ manpage
sudo\ mkdir\ \-p\ /usr/local/share/man/man1
sudo\ cp\ rclone.1\ /usr/local/share/man/man1/
sudo\ mandb
sudo\ mandb\
\f[]
.fi
.SS Installation with Ansible
.PP
This can be done with Stefan Weichinger\[aq]s ansible
role (https://github.com/stefangweichinger/ansible-rclone).
.PP
Instructions
.IP "1." 3
\f[C]git\ clone\ https://github.com/stefangweichinger/ansible\-rclone.git\f[]
into your local roles\-directory
.IP "2." 3
add the role to the hosts you want rclone installed to:
.IP
.nf
\f[C]
\ \ \ \ \-\ hosts:\ rclone\-hosts
\ \ \ \ \ \ roles:
\ \ \ \ \ \ \ \ \ \ \-\ rclone
\f[]
.fi
.SS Configure
@ -216,6 +235,9 @@ had written a trailing / \- meaning "copy the contents of this
directory".
This applies to all commands and whether you are talking about the
source or destination.
.PP
See the \f[C]\-\-no\-traverse\f[] option for controlling whether rclone
lists the destination directory or not.
.SS rclone sync source:path dest:path
.PP
Sync the source to the destination, changing the destination only.
@ -240,16 +262,18 @@ If dest:path doesn\[aq]t exist, it is created and the source:path
contents go there.
.SS move source:path dest:path
.PP
Moves the source to the destination.
Moves the contents of the source directory to the destination directory.
Rclone will error if the source and destination overlap.
.PP
If there are no filters in use this is equivalent to a copy followed by
a purge, but may use server side operations to speed it up if possible.
If no filters are in use and if possible this will server side move
\f[C]source:path\f[] into \f[C]dest:path\f[].
After this \f[C]source:path\f[] will no longer longer exist.
.PP
If filters are in use then it is equivalent to a copy followed by
delete, followed by an rmdir (which only removes the directory if
empty).
The individual file moves will be moved with server side operations if
possible.
Otherwise for each file in \f[C]source:path\f[] selected by the filters
(if any) this will move it into \f[C]dest:path\f[].
If possible a server side move will be used, otherwise it will copy it
(server side if possible) into \f[C]dest:path\f[] then delete the
original (if no errors on copy) in \f[C]source:path\f[].
.PP
\f[B]Important\f[]: Since this can cause data loss, test first with the
\-\-dry\-run flag.
@ -325,6 +349,11 @@ It doesn\[aq]t alter the source or destination.
.PP
\f[C]\-\-size\-only\f[] may be used to only compare the sizes, not the
MD5SUMs.
.SS rclone cleanup remote:path
.PP
Clean up the remote if possible.
Empty the trash or delete old file versions.
Not supported by all remotes.
.SS rclone dedupe remote:path
.PP
By default \f[C]dedup\f[] interactively finds duplicate files and offers
@ -433,6 +462,47 @@ Enter an interactive configuration session.
.SS rclone help
.PP
Prints help on rclone commands and options.
.SS Copying single files
.PP
rclone normally syncs or copies directories.
However if the source remote points to a file, rclone will just copy
that file.
The destination remote must point to a directory \- rclone will give the
error
\f[C]Failed\ to\ create\ file\ system\ for\ "remote:file":\ is\ a\ file\ not\ a\ directory\f[]
if it isn\[aq]t.
.PP
For example, suppose you have a remote with a file in called
\f[C]test.jpg\f[], then you could copy just that file like this
.IP
.nf
\f[C]
rclone\ copy\ remote:test.jpg\ /tmp/download
\f[]
.fi
.PP
The file \f[C]test.jpg\f[] will be placed inside \f[C]/tmp/download\f[].
.PP
This is equivalent to specifying
.IP
.nf
\f[C]
rclone\ copy\ \-\-no\-traverse\ \-\-files\-from\ /tmp/files\ remote:\ /tmp/download
\f[]
.fi
.PP
Where \f[C]/tmp/files\f[] contains the single line
.IP
.nf
\f[C]
test.jpg
\f[]
.fi
.PP
It is recommended to use \f[C]copy\f[] when copying single files not
\f[C]sync\f[].
They have pretty much the same effect but \f[C]copy\f[] will use a lot
less memory.
.SS Quoting and the shell
.PP
When you are typing commands to your computer you are using something
@ -699,6 +769,13 @@ Useful if you\[aq]ve set the server to return files with
.PP
There is no need to set this in normal operation, and doing so will
decrease the network transfer efficiency of rclone.
.SS \-\-no\-update\-modtime
.PP
When using this flag, rclone won\[aq]t update modification times of
remote files if they are incorrect as it would normally.
.PP
This can be used if the remote is being synced with another tool also
(eg the Google Drive client).
.SS \-q, \-\-quiet
.PP
Normally rclone outputs stats and a completion message.
@ -736,10 +813,14 @@ deleted when you sync folders.
Specifying the value \f[C]\-\-delete\-before\f[] will delete all files
present on the destination, but not on the source \f[I]before\f[]
starting the transfer of any new or updated files.
This uses extra memory as it has to store the source listing before
proceeding.
.PP
Specifying \f[C]\-\-delete\-during\f[] (default value) will delete files
while checking and uploading files.
This is usually the fastest option.
Currently this works the same as \f[C]\-\-delete\-after\f[] but it may
change in the future.
.PP
Specifying \f[C]\-\-delete\-after\f[] will delay deletion of files until
all new/updated files have been successfully transfered.
@ -909,6 +990,25 @@ In this mode, TLS is susceptible to man\-in\-the\-middle attacks.
This option defaults to \f[C]false\f[].
.PP
\f[B]This should be used only for testing.\f[]
.SS \-\-no\-traverse
.PP
The \f[C]\-\-no\-traverse\f[] flag controls whether the destination file
system is traversed when using the \f[C]copy\f[] or \f[C]move\f[]
commands.
.PP
If you are only copying a small number of files and/or have a large
number of files on the destination then \f[C]\-\-no\-traverse\f[] will
stop rclone listing the destination and save time.
.PP
However if you are copying a large number of files, escpecially if you
are doing a copy where lots of the files haven\[aq]t changed and
won\[aq]t need copying then you shouldn\[aq]t use
\f[C]\-\-no\-traverse\f[].
.PP
It can also be used to reduce the memory usage of rclone when copying \-
\f[C]rclone\ \-\-no\-traverse\ copy\ src\ dst\f[] won\[aq]t load either
the source or destination listings into memory so will use the minimum
amount of memory.
.SS Filtering
.PP
For the filtering options
@ -1256,8 +1356,8 @@ non \f[C]*.jpg\f[] and \f[C]*.png\f[]
A similar process is done on directory entries before recursing into
them.
This only works on remotes which have a concept of directory (Eg local,
drive, onedrive, amazon cloud drive) and not on bucket based remotes (eg
s3, swift, google compute storage, b2).
google drive, onedrive, amazon drive) and not on bucket based remotes
(eg s3, swift, google compute storage, b2).
.SS Adding filtering rules
.PP
Filtering rules are added with the following command line flags.
@ -1383,6 +1483,62 @@ file2.jpg
Then use as \f[C]\-\-files\-from\ files\-from.txt\f[].
This will only transfer \f[C]file1.jpg\f[] and \f[C]file2.jpg\f[]
providing they exist.
.PP
For example, let\[aq]s say you had a few files you want to back up
regularly with these absolute paths:
.IP
.nf
\f[C]
/home/user1/important
/home/user1/dir/file
/home/user2/stuff
\f[]
.fi
.PP
To copy these you\[aq]d find a common subdirectory \- in this case
\f[C]/home\f[] and put the remaining files in \f[C]files\-from.txt\f[]
with or without leading \f[C]/\f[], eg
.IP
.nf
\f[C]
user1/important
user1/dir/file
user2/stuff
\f[]
.fi
.PP
You could then copy these to a remote like this
.IP
.nf
\f[C]
rclone\ copy\ \-\-files\-from\ files\-from.txt\ /home\ remote:backup
\f[]
.fi
.PP
The 3 files will arrive in \f[C]remote:backup\f[] with the paths as in
the \f[C]files\-from.txt\f[].
.PP
You could of course choose \f[C]/\f[] as the root too in which case your
\f[C]files\-from.txt\f[] might look like this.
.IP
.nf
\f[C]
/home/user1/important
/home/user1/dir/file
/home/user2/stuff
\f[]
.fi
.PP
And you would transfer it like this
.IP
.nf
\f[C]
rclone\ copy\ \-\-files\-from\ files\-from.txt\ /\ remote:backup
\f[]
.fi
.PP
In this case there will be an extra \f[C]home\f[] directory on the
remote.
.SS \f[C]\-\-min\-size\f[] \- Don\[aq]t transfer any file smaller than
this
.PP
@ -1718,7 +1874,7 @@ e/n/d/q>\ n
name>\ remote
Type\ of\ storage\ to\ configure.
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
\ 1\ /\ Amazon\ Cloud\ Drive
\ 1\ /\ Amazon\ Drive
\ \ \ \\\ "amazon\ cloud\ drive"
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph)
\ \ \ \\\ "s3"
@ -2038,7 +2194,7 @@ n/s>\ n
name>\ remote
Type\ of\ storage\ to\ configure.
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
\ 1\ /\ Amazon\ Cloud\ Drive
\ 1\ /\ Amazon\ Drive
\ \ \ \\\ "amazon\ cloud\ drive"
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph)
\ \ \ \\\ "s3"
@ -2336,6 +2492,75 @@ removed).
Because this is a json dump, it is encoding the \f[C]/\f[] as
\f[C]\\/\f[], so if you use the secret key as \f[C]xxxxxx/xxxx\f[] it
will work fine.
.SS Minio
.PP
Minio (https://minio.io/) is an object storage server built for cloud
application developers and devops.
.PP
It is very easy to install and provides an S3 compatible server which
can be used by rclone.
.PP
To use it, install Minio following the instructions from the web site.
.PP
When it configures itself Minio will print something like this
.IP
.nf
\f[C]
AccessKey:\ WLGDGYAQYIGI833EV05A\ \ SecretKey:\ BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF\ Region:\ us\-east\-1
Minio\ Object\ Storage:
\ \ \ \ \ http://127.0.0.1:9000
\ \ \ \ \ http://10.0.0.3:9000
Minio\ Browser:
\ \ \ \ \ http://127.0.0.1:9000
\ \ \ \ \ http://10.0.0.3:9000
\f[]
.fi
.PP
These details need to go into \f[C]rclone\ config\f[] like this.
Note that it is important to put the region in as stated above.
.IP
.nf
\f[C]
env_auth>\ 1
access_key_id>\ WLGDGYAQYIGI833EV05A
secret_access_key>\ BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF\ \ \
region>\ us\-east\-1
endpoint>\ http://10.0.0.3:9000
location_constraint>\
server_side_encryption>
\f[]
.fi
.PP
Which makes the config file look like this
.IP
.nf
\f[C]
[minio]
env_auth\ =\ false
access_key_id\ =\ WLGDGYAQYIGI833EV05A
secret_access_key\ =\ BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF
region\ =\ us\-east\-1
endpoint\ =\ http://10.0.0.3:9000
location_constraint\ =\
server_side_encryption\ =\
\f[]
.fi
.PP
Minio doesn\[aq]t support all the features of S3 yet.
In particular it doesn\[aq]t support MD5 checksums (ETags) or metadata.
This means rclone can\[aq]t check MD5SUMs or store the modified date.
However you can work around this with the \f[C]\-\-size\-only\f[] flag
of rclone.
.PP
So once set up, for example to copy files into a bucket
.IP
.nf
\f[C]
rclone\ \-\-size\-only\ copy\ /path/to/files\ minio:bucket
\f[]
.fi
.SS Swift
.PP
Swift refers to Openstack Object
@ -2370,7 +2595,7 @@ n/s>\ n
name>\ remote
Type\ of\ storage\ to\ configure.
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
\ 1\ /\ Amazon\ Cloud\ Drive
\ 1\ /\ Amazon\ Drive
\ \ \ \\\ "amazon\ cloud\ drive"
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph)
\ \ \ \\\ "s3"
@ -2416,6 +2641,8 @@ User\ domain\ \-\ optional\ (v3\ auth)
domain>\ Default
Tenant\ name\ \-\ optional
tenant>\
Tenant\ domain\ \-\ optional\ (v3\ auth)
tenant_domain>
Region\ name\ \-\ optional
region>\
Storage\ URL\ \-\ optional
@ -2538,7 +2765,7 @@ e/n/d/q>\ n
name>\ remote
Type\ of\ storage\ to\ configure.
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
\ 1\ /\ Amazon\ Cloud\ Drive
\ 1\ /\ Amazon\ Drive
\ \ \ \\\ "amazon\ cloud\ drive"
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph)
\ \ \ \\\ "s3"
@ -2680,7 +2907,7 @@ e/n/d/q>\ n
name>\ remote
Type\ of\ storage\ to\ configure.
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
\ 1\ /\ Amazon\ Cloud\ Drive
\ 1\ /\ Amazon\ Drive
\ \ \ \\\ "amazon\ cloud\ drive"
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph)
\ \ \ \\\ "s3"
@ -2847,8 +3074,8 @@ Paths are specified as \f[C]remote:path\f[]
Paths may be as deep as required, eg
\f[C]remote:directory/subdirectory\f[].
.PP
The initial setup for Amazon cloud drive involves getting a token from
Amazon which you need to do in your browser.
The initial setup for Amazon Drive involves getting a token from Amazon
which you need to do in your browser.
\f[C]rclone\ config\f[] walks you through it.
.PP
Here is an example of how to make a remote called \f[C]remote\f[].
@ -2871,7 +3098,7 @@ e/n/d/q>\ n
name>\ remote
Type\ of\ storage\ to\ configure.
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
\ 1\ /\ Amazon\ Cloud\ Drive
\ 1\ /\ Amazon\ Drive
\ \ \ \\\ "amazon\ cloud\ drive"
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph)
\ \ \ \\\ "s3"
@ -2928,7 +3155,7 @@ to unblock it temporarily if you are running a host firewall.
.PP
Once configured you can then use \f[C]rclone\f[] like this,
.PP
List directories in top level of your Amazon cloud drive
List directories in top level of your Amazon Drive
.IP
.nf
\f[C]
@ -2936,7 +3163,7 @@ rclone\ lsd\ remote:
\f[]
.fi
.PP
List all the files in your Amazon cloud drive
List all the files in your Amazon Drive
.IP
.nf
\f[C]
@ -2944,8 +3171,7 @@ rclone\ ls\ remote:
\f[]
.fi
.PP
To copy a local directory to an Amazon cloud drive directory called
backup
To copy a local directory to an Amazon Drive directory called backup
.IP
.nf
\f[C]
@ -2954,8 +3180,8 @@ rclone\ copy\ /home/source\ remote:backup
.fi
.SS Modified time and MD5SUMs
.PP
Amazon cloud drive doesn\[aq]t allow modification times to be changed
via the API so these won\[aq]t be accurate or used for syncing.
Amazon Drive doesn\[aq]t allow modification times to be changed via the
API so these won\[aq]t be accurate or used for syncing.
.PP
It does store MD5SUMs so for a more accurate sync, you can use the
\f[C]\-\-checksum\f[] flag.
@ -2964,7 +3190,7 @@ It does store MD5SUMs so for a more accurate sync, you can use the
Any files you delete with rclone will end up in the trash.
Amazon don\[aq]t provide an API to permanently delete files, nor to
empty the trash, so you will have to do that with one of Amazon\[aq]s
apps or via the Amazon cloud drive website.
apps or via the Amazon Drive website.
.SS Specific options
.PP
Here are the command line options specific to this cloud storage system.
@ -2980,17 +3206,17 @@ To download files above this threshold, rclone requests a
directly from the underlying S3 storage.
.SS Limitations
.PP
Note that Amazon cloud drive is case insensitive so you can\[aq]t have a
file called "Hello.doc" and one called "hello.doc".
Note that Amazon Drive is case insensitive so you can\[aq]t have a file
called "Hello.doc" and one called "hello.doc".
.PP
Amazon cloud drive has rate limiting so you may notice errors in the
sync (429 errors).
Amazon Drive has rate limiting so you may notice errors in the sync (429
errors).
rclone will automatically retry the sync up to 3 times by default (see
\f[C]\-\-retries\f[] flag) which should hopefully work around this
problem.
.PP
Amazon cloud drive has an internal limit of file sizes that can be
uploaded to the service.
Amazon Drive has an internal limit of file sizes that can be uploaded to
the service.
This limit is not officially published, but all files larger than this
will fail.
.PP
@ -3033,7 +3259,7 @@ n/s>\ n
name>\ remote
Type\ of\ storage\ to\ configure.
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
\ 1\ /\ Amazon\ Cloud\ Drive
\ 1\ /\ Amazon\ Drive
\ \ \ \\\ "amazon\ cloud\ drive"
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph)
\ \ \ \\\ "s3"
@ -3193,7 +3419,7 @@ n/s>\ n
name>\ remote
Type\ of\ storage\ to\ configure.
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
\ 1\ /\ Amazon\ Cloud\ Drive
\ 1\ /\ Amazon\ Drive
\ \ \ \\\ "amazon\ cloud\ drive"
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph)
\ \ \ \\\ "s3"
@ -3339,7 +3565,7 @@ n/q>\ n
name>\ remote
Type\ of\ storage\ to\ configure.
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
\ 1\ /\ Amazon\ Cloud\ Drive
\ 1\ /\ Amazon\ Drive
\ \ \ \\\ "amazon\ cloud\ drive"
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph)
\ \ \ \\\ "s3"
@ -3436,19 +3662,6 @@ You can use the \f[C]\-\-checksum\f[] flag.
Large files which are uploaded in chunks will store their SHA1 on the
object as \f[C]X\-Bz\-Info\-large_file_sha1\f[] as recommended by
Backblaze.
.SS Versions
.PP
When rclone uploads a new version of a file it creates a new version of
it (https://www.backblaze.com/b2/docs/file_versions.html).
Likewise when you delete a file, the old version will still be
available.
.PP
The old versions of files are visible in the B2 web interface, but not
via rclone yet.
.PP
Rclone doesn\[aq]t provide any way of managing old versions (downloading
them or deleting them) at the moment.
When you \f[C]purge\f[] a bucket, all the old versions will be deleted.
.SS Transfers
.PP
Backblaze recommends that you do lots of transfers simultaneously for
@ -3460,6 +3673,71 @@ The optimum number for you may vary depending on your hardware, how big
the files are, how much you want to load your computer, etc.
The default of \f[C]\-\-transfers\ 4\f[] is definitely too low for
Backblaze B2 though.
.SS Versions
.PP
When rclone uploads a new version of a file it creates a new version of
it (https://www.backblaze.com/b2/docs/file_versions.html).
Likewise when you delete a file, the old version will still be
available.
.PP
Old versions of files are visible using the \f[C]\-\-b2\-versions\f[]
flag.
.PP
If you wish to remove all the old versions then you can use the
\f[C]rclone\ cleanup\ remote:bucket\f[] command which will delete all
the old versions of files, leaving the current ones intact.
You can also supply a path and only old versions under that path will be
deleted, eg \f[C]rclone\ cleanup\ remote:bucket/path/to/stuff\f[].
.PP
When you \f[C]purge\f[] a bucket, the current and the old versions will
be deleted then the bucket will be deleted.
.PP
However \f[C]delete\f[] will cause the current versions of the files to
become hidden old versions.
.PP
Here is a session showing the listing and and retreival of an old
version followed by a \f[C]cleanup\f[] of the old versions.
.PP
Show current version and all the versions with \f[C]\-\-b2\-versions\f[]
flag.
.IP
.nf
\f[C]
$\ rclone\ \-q\ ls\ b2:cleanup\-test
\ \ \ \ \ \ \ \ 9\ one.txt
$\ rclone\ \-q\ \-\-b2\-versions\ ls\ b2:cleanup\-test
\ \ \ \ \ \ \ \ 9\ one.txt
\ \ \ \ \ \ \ \ 8\ one\-v2016\-07\-04\-141032\-000.txt
\ \ \ \ \ \ \ 16\ one\-v2016\-07\-04\-141003\-000.txt
\ \ \ \ \ \ \ 15\ one\-v2016\-07\-02\-155621\-000.txt
\f[]
.fi
.PP
Retreive an old verson
.IP
.nf
\f[C]
$\ rclone\ \-q\ \-\-b2\-versions\ copy\ b2:cleanup\-test/one\-v2016\-07\-04\-141003\-000.txt\ /tmp
$\ ls\ \-l\ /tmp/one\-v2016\-07\-04\-141003\-000.txt
\-rw\-rw\-r\-\-\ 1\ ncw\ ncw\ 16\ Jul\ \ 2\ 17:46\ /tmp/one\-v2016\-07\-04\-141003\-000.txt
\f[]
.fi
.PP
Clean up all the old versions and show that they\[aq]ve gone.
.IP
.nf
\f[C]
$\ rclone\ \-q\ cleanup\ b2:cleanup\-test
$\ rclone\ \-q\ ls\ b2:cleanup\-test
\ \ \ \ \ \ \ \ 9\ one.txt
$\ rclone\ \-q\ \-\-b2\-versions\ ls\ b2:cleanup\-test
\ \ \ \ \ \ \ \ 9\ one.txt
\f[]
.fi
.SS Specific options
.PP
Here are the command line options specific to this cloud storage system.
@ -3475,11 +3753,55 @@ Files above this size will be uploaded in chunks of
\f[C]\-\-b2\-chunk\-size\f[].
The default value is the largest file which can be uploaded without
chunks.
.SS API
.SS \-\-b2\-test\-mode=FLAG
.PP
Here are some notes I made on the backblaze
API (https://gist.github.com/ncw/166dabf352b399f1cc1c) while integrating
it with rclone.
This is for debugging purposes only.
.PP
Setting FLAG to one of the strings below will cause b2 to return
specific errors for debugging purposes.
.IP \[bu] 2
\f[C]fail_some_uploads\f[]
.IP \[bu] 2
\f[C]expire_some_account_authorization_tokens\f[]
.IP \[bu] 2
\f[C]force_cap_exceeded\f[]
.PP
These will be set in the \f[C]X\-Bz\-Test\-Mode\f[] header which is
documented in the b2 integrations
checklist (https://www.backblaze.com/b2/docs/integration_checklist.html).
.SS \-\-b2\-versions
.PP
When set rclone will show and act on older versions of files.
For example
.PP
Listing without \f[C]\-\-b2\-versions\f[]
.IP
.nf
\f[C]
$\ rclone\ \-q\ ls\ b2:cleanup\-test
\ \ \ \ \ \ \ \ 9\ one.txt
\f[]
.fi
.PP
And with
.IP
.nf
\f[C]
$\ rclone\ \-q\ \-\-b2\-versions\ ls\ b2:cleanup\-test
\ \ \ \ \ \ \ \ 9\ one.txt
\ \ \ \ \ \ \ \ 8\ one\-v2016\-07\-04\-141032\-000.txt
\ \ \ \ \ \ \ 16\ one\-v2016\-07\-04\-141003\-000.txt
\ \ \ \ \ \ \ 15\ one\-v2016\-07\-02\-155621\-000.txt
\f[]
.fi
.PP
Showing that the current version is unchanged but older versions can be
seen.
These have the UTC date that they were uploaded to the server to the
nearest millisecond appended to them.
.PP
Note that when using \f[C]\-\-b2\-versions\f[] no file write operations
are permitted, so you can\[aq]t upload files or delete them.
.SS Yandex Disk
.PP
Yandex Disk (https://disk.yandex.com) is a cloud storage solution
@ -3508,7 +3830,7 @@ n/s>\ n
name>\ remote
Type\ of\ storage\ to\ configure.
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
\ 1\ /\ Amazon\ Cloud\ Drive
\ 1\ /\ Amazon\ Drive
\ \ \ \\\ "amazon\ cloud\ drive"
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph)
\ \ \ \\\ "s3"
@ -3697,6 +4019,92 @@ Of course this will cause problems if the absolute path length of a file
exceeds 258 characters on z, so only use this option if you have to.
.SS Changelog
.IP \[bu] 2
v1.31 \- 2016\-07\-13
.RS 2
.IP \[bu] 2
New Features
.IP \[bu] 2
Reduce memory on sync by about 50%
.IP \[bu] 2
Implement \-\-no\-traverse flag to stop copy traversing the destination
remote.
.RS 2
.IP \[bu] 2
This can be used to reduce memory usage down to the smallest possible.
.IP \[bu] 2
Useful to copy a small number of files into a large destination folder.
.RE
.IP \[bu] 2
Implement cleanup command for emptying trash / removing old versions of
files
.RS 2
.IP \[bu] 2
Currently B2 only
.RE
.IP \[bu] 2
Single file handling improved
.RS 2
.IP \[bu] 2
Now copied with \-\-files\-from
.IP \[bu] 2
Automatically sets \-\-no\-traverse when copying a single file
.RE
.IP \[bu] 2
Info on using installing with ansible \- thanks Stefan Weichinger
.IP \[bu] 2
Implement \-\-no\-update\-modtime flag to stop rclone fixing the remote
modified times.
.IP \[bu] 2
Bug Fixes
.IP \[bu] 2
Fix move command \- stop it running for overlapping Fses \- this was
causing data loss.
.IP \[bu] 2
Local
.IP \[bu] 2
Fix incomplete hashes \- this was causing problems for B2.
.IP \[bu] 2
Amazon Drive
.IP \[bu] 2
Rename Amazon Cloud Drive to Amazon Drive \- no changes to config file
needed.
.IP \[bu] 2
Swift
.IP \[bu] 2
Add support for non\-default project domain \- thanks Antonio Messina.
.IP \[bu] 2
S3
.IP \[bu] 2
Add instructions on how to use rclone with minio.
.IP \[bu] 2
Add ap\-northeast\-2 (Seoul) and ap\-south\-1 (Mumbai) regions.
.IP \[bu] 2
Skip setting the modified time for objects > 5GB as it isn\[aq]t
possible.
.IP \[bu] 2
Backblaze B2
.IP \[bu] 2
Add \-\-b2\-versions flag so old versions can be listed and retreived.
.IP \[bu] 2
Treat 403 errors (eg cap exceeded) as fatal.
.IP \[bu] 2
Implement cleanup command for deleting old file versions.
.IP \[bu] 2
Make error handling compliant with B2 integrations notes.
.IP \[bu] 2
Fix handling of token expiry.
.IP \[bu] 2
Implement \-\-b2\-test\-mode to set \f[C]X\-Bz\-Test\-Mode\f[] header.
.IP \[bu] 2
Set cutoff for chunked upload to 200MB as per B2 guidelines.
.IP \[bu] 2
Make upload multi\-threaded.
.IP \[bu] 2
Dropbox
.IP \[bu] 2
Don\[aq]t retry 461 errors.
.RE
.IP \[bu] 2
v1.30 \- 2016\-06\-18
.RS 2
.IP \[bu] 2
@ -4806,6 +5214,11 @@ Romain Lapray <lapray.romain@gmail.com>
.IP \[bu] 2
Justin R.
Wilson <jrw972@gmail.com>
.IP \[bu] 2
Antonio Messina <antonio.s.messina@gmail.com>
.IP \[bu] 2
Stefan G.
Weichinger <office@oops.co.at>
.SS Contact the rclone project
.PP
The project website is at: