Compare commits

...

10 Commits

Author SHA1 Message Date
Nick Craig-Wood ffcaa6cd92 Version v1.65.2 2024-01-24 15:25:51 +00:00
Nick Craig-Wood 98e81a6c2b azureblob: fix data corruption bug #7590
It was reported that rclone copy occasionally uploaded corrupted data
to azure blob.

This turned out to be a race condition updating the block count which
caused blocks to be duplicated.

This bug was introduced in this commit in v1.64.0 and will be fixed in v1.65.2

0427177857 azureblob: implement OpenChunkWriter and multi-thread uploads #7056

This race only seems to happen if `--checksum` is used but can happen otherwise.

Unfortunately Azure blob does not check the MD5 that we send them so
despite sending incorrect data this corruption is not detected. The
corruption is detected when rclone tries to download the file, so
attempting to copy the files back to local disk will result in errors
such as:

    ERROR : file.pokosuf5.partial: corrupted on transfer: md5 hash differ "XXX" vs "YYY"

This adds a check to test the blocklist we upload is as we expected
which would have caught the problem had it been in place earlier.
2024-01-24 11:46:47 +00:00
Nick Craig-Wood 9e60a065b4 build: add -race flag to integration tester test_all 2024-01-24 11:46:47 +00:00
nielash 95e18bdc6f docs: add bisync to index 2024-01-24 11:46:47 +00:00
dependabot[bot] a4c7b3da15 build(deps): bump actions/cache from 3 to 4
Bumps [actions/cache](https://github.com/actions/cache) from 3 to 4.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-01-24 11:46:47 +00:00
Tera be10debded add missing backtick 2024-01-24 11:46:47 +00:00
kapitainsky a7cb8b71f0 docs: add rclone OS requirements
Adds rclone OS requirements list and latest rclone versions known to be working with specific historical OS versions.

Discussed on the forum:
https://forum.rclone.org/t/rclone-1-65-1-runtime-exception-error-crash-immediately-after-running-the-command/44051

Fixes: #7571
2024-01-24 11:46:47 +00:00
Harshit Budhraja 322d683997 imagekit: updated overview - supported operations 2024-01-24 11:46:47 +00:00
Harshit Budhraja 964753b2d5 imagekit: Updated docs and web content 2024-01-24 11:46:47 +00:00
Nick Craig-Wood 8b3bd74565 docs: add warp.dev sponsorship to github home page 2024-01-24 11:46:47 +00:00
19 changed files with 266 additions and 90 deletions

View File

@ -168,7 +168,7 @@ jobs:
env
- name: Go module cache
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: ~/go/pkg/mod
key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
@ -271,7 +271,7 @@ jobs:
go-version: '1.21'
- name: Go module cache
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: ~/go/pkg/mod
key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}

103
MANUAL.html generated
View File

@ -81,7 +81,7 @@
<header id="title-block-header">
<h1 class="title">rclone(1) User Manual</h1>
<p class="author">Nick Craig-Wood</p>
<p class="date">Jan 08, 2024</p>
<p class="date">Jan 24, 2024</p>
</header>
<h1 id="rclone-syncs-your-files-to-cloud-storage">Rclone syncs your files to cloud storage</h1>
<p><img width="50%" src="https://rclone.org/img/logo_on_light__horizontal_color.svg" alt="rclone logo" style="float:right; padding: 5px;" ></p>
@ -127,6 +127,7 @@
</ul></li>
<li><a href="https://rclone.org/commands/rclone_copy/">Copy</a> new or changed files to cloud storage</li>
<li><a href="https://rclone.org/commands/rclone_sync/">Sync</a> (one way) to make a directory identical</li>
<li><a href="https://rclone.org/bisync/">Bisync</a> (two way) to keep two directories in sync bidirectionally</li>
<li><a href="https://rclone.org/commands/rclone_move/">Move</a> files to cloud storage deleting the local after verification</li>
<li><a href="https://rclone.org/commands/rclone_check/">Check</a> hashes and for missing/extra files</li>
<li><a href="https://rclone.org/commands/rclone_mount/">Mount</a> your cloud storage as a network disk</li>
@ -162,6 +163,7 @@
<li>Hetzner Storage Box</li>
<li>HiDrive</li>
<li>HTTP</li>
<li>ImageKit</li>
<li>Internet Archive</li>
<li>Jottacloud</li>
<li>IBM COS S3</li>
@ -2501,7 +2503,11 @@ Supported hashes are:
* sha1
* whirlpool
* crc32
* sha256</code></pre>
* sha256
* dropbox
* hidrive
* mailru
* quickxor</code></pre>
<p>Then</p>
<pre><code>$ rclone hashsum MD5 remote:path</code></pre>
<p>Note that hash names are case insensitive and values are output in lower case.</p>
@ -7860,7 +7866,7 @@ file2.avi</code></pre>
+ *.png
+ file2.avi
- *</code></pre>
<p>Files <code>file1.jpg</code>, <code>file3.png</code> and <code>file2.avi</code> are listed whilst <code>secret17.jpg</code> and files without the suffix .jpg<code>or</code>.png` are excluded.</p>
<p>Files <code>file1.jpg</code>, <code>file3.png</code> and <code>file2.avi</code> are listed whilst <code>secret17.jpg</code> and files without the suffix <code>.jpg</code> or <code>.png</code> are excluded.</p>
<p>E.g. for an alternative <code>filter-file.txt</code>:</p>
<pre><code>+ *.jpg
+ *.gif
@ -10563,6 +10569,20 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">Yes</td>
</tr>
<tr class="odd">
<td>ImageKit</td>
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">No</td>
<td style="text-align: center;">No</td>
<td style="text-align: center;">No</td>
<td style="text-align: center;">No</td>
<td style="text-align: left;">No</td>
<td style="text-align: center;">No</td>
<td style="text-align: center;">No</td>
<td style="text-align: center;">Yes</td>
</tr>
<tr class="even">
<td>Internet Archive</td>
<td style="text-align: center;">No</td>
<td style="text-align: center;">Yes</td>
@ -10576,7 +10596,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">No</td>
</tr>
<tr class="even">
<tr class="odd">
<td>Jottacloud</td>
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
@ -10590,7 +10610,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
</tr>
<tr class="odd">
<tr class="even">
<td>Koofr</td>
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
@ -10604,7 +10624,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
</tr>
<tr class="even">
<tr class="odd">
<td>Mail.ru Cloud</td>
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
@ -10618,7 +10638,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
</tr>
<tr class="odd">
<tr class="even">
<td>Mega</td>
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">No</td>
@ -10632,7 +10652,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
</tr>
<tr class="even">
<tr class="odd">
<td>Memory</td>
<td style="text-align: center;">No</td>
<td style="text-align: center;">Yes</td>
@ -10646,7 +10666,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">No</td>
<td style="text-align: center;">No</td>
</tr>
<tr class="odd">
<tr class="even">
<td>Microsoft Azure Blob Storage</td>
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
@ -10660,7 +10680,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">No</td>
<td style="text-align: center;">No</td>
</tr>
<tr class="even">
<tr class="odd">
<td>Microsoft Azure Files Storage</td>
<td style="text-align: center;">No</td>
<td style="text-align: center;">Yes</td>
@ -10674,7 +10694,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
</tr>
<tr class="odd">
<tr class="even">
<td>Microsoft OneDrive</td>
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
@ -10688,7 +10708,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
</tr>
<tr class="even">
<tr class="odd">
<td>OpenDrive</td>
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
@ -10702,7 +10722,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">No</td>
<td style="text-align: center;">Yes</td>
</tr>
<tr class="odd">
<tr class="even">
<td>OpenStack Swift</td>
<td style="text-align: center;">Yes ¹</td>
<td style="text-align: center;">Yes</td>
@ -10716,7 +10736,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">No</td>
</tr>
<tr class="even">
<tr class="odd">
<td>Oracle Object Storage</td>
<td style="text-align: center;">No</td>
<td style="text-align: center;">Yes</td>
@ -10730,7 +10750,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">No</td>
<td style="text-align: center;">No</td>
</tr>
<tr class="odd">
<tr class="even">
<td>pCloud</td>
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
@ -10744,7 +10764,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
</tr>
<tr class="even">
<tr class="odd">
<td>PikPak</td>
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
@ -10758,7 +10778,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
</tr>
<tr class="odd">
<tr class="even">
<td>premiumize.me</td>
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">No</td>
@ -10772,7 +10792,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
</tr>
<tr class="even">
<tr class="odd">
<td>put.io</td>
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">No</td>
@ -10786,7 +10806,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
</tr>
<tr class="odd">
<tr class="even">
<td>Proton Drive</td>
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">No</td>
@ -10800,7 +10820,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
</tr>
<tr class="even">
<tr class="odd">
<td>QingStor</td>
<td style="text-align: center;">No</td>
<td style="text-align: center;">Yes</td>
@ -10814,7 +10834,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">No</td>
<td style="text-align: center;">No</td>
</tr>
<tr class="odd">
<tr class="even">
<td>Quatrix by Maytech</td>
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
@ -10828,7 +10848,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
</tr>
<tr class="even">
<tr class="odd">
<td>Seafile</td>
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
@ -10842,7 +10862,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
</tr>
<tr class="odd">
<tr class="even">
<td>SFTP</td>
<td style="text-align: center;">No</td>
<td style="text-align: center;">Yes ⁴</td>
@ -10856,7 +10876,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
</tr>
<tr class="even">
<tr class="odd">
<td>Sia</td>
<td style="text-align: center;">No</td>
<td style="text-align: center;">No</td>
@ -10870,7 +10890,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">No</td>
<td style="text-align: center;">Yes</td>
</tr>
<tr class="odd">
<tr class="even">
<td>SMB</td>
<td style="text-align: center;">No</td>
<td style="text-align: center;">No</td>
@ -10884,7 +10904,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">No</td>
<td style="text-align: center;">Yes</td>
</tr>
<tr class="even">
<tr class="odd">
<td>SugarSync</td>
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
@ -10898,7 +10918,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">No</td>
<td style="text-align: center;">Yes</td>
</tr>
<tr class="odd">
<tr class="even">
<td>Storj</td>
<td style="text-align: center;">Yes ²</td>
<td style="text-align: center;">Yes</td>
@ -10912,7 +10932,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">No</td>
<td style="text-align: center;">No</td>
</tr>
<tr class="even">
<tr class="odd">
<td>Uptobox</td>
<td style="text-align: center;">No</td>
<td style="text-align: center;">Yes</td>
@ -10926,7 +10946,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">No</td>
<td style="text-align: center;">No</td>
</tr>
<tr class="odd">
<tr class="even">
<td>WebDAV</td>
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
@ -10940,7 +10960,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
</tr>
<tr class="even">
<tr class="odd">
<td>Yandex Disk</td>
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
@ -10954,7 +10974,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
</tr>
<tr class="odd">
<tr class="even">
<td>Zoho WorkDrive</td>
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
@ -10968,7 +10988,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">Yes</td>
</tr>
<tr class="even">
<tr class="odd">
<td>The local filesystem</td>
<td style="text-align: center;">Yes</td>
<td style="text-align: center;">No</td>
@ -11096,7 +11116,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
--tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default &quot;rclone/v1.65.1&quot;)</code></pre>
--user-agent string Set the user-agent to a specified string (default &quot;rclone/v1.65.2&quot;)</code></pre>
<h2 id="performance">Performance</h2>
<p>Flags helpful for increasing performance.</p>
<pre><code> --buffer-size SizeSuffix In memory buffer size when reading files for each --transfer (default 16Mi)
@ -36687,6 +36707,23 @@ $ tree /tmp/b
<li>"error": return an error based on option value</li>
</ul>
<h1 id="changelog-1">Changelog</h1>
<h2 id="v1.65.2---2024-01-24">v1.65.2 - 2024-01-24</h2>
<p><a href="https://github.com/rclone/rclone/compare/v1.65.1...v1.65.2">See commits</a></p>
<ul>
<li>Bug Fixes
<ul>
<li>build: bump github.com/cloudflare/circl from 1.3.6 to 1.3.7 (dependabot)</li>
<li>docs updates (Nick Craig-Wood, kapitainsky, nielash, Tera, Harshit Budhraja)</li>
</ul></li>
<li>VFS
<ul>
<li>Fix stale data when using <code>--vfs-cache-mode</code> full (Nick Craig-Wood)</li>
</ul></li>
<li>Azure Blob
<ul>
<li><strong>IMPORTANT</strong> Fix data corruption bug - see <a href="https://github.com/rclone/rclone/issues/7590">#7590</a> (Nick Craig-Wood)</li>
</ul></li>
</ul>
<h2 id="v1.65.1---2024-01-08">v1.65.1 - 2024-01-08</h2>
<p><a href="https://github.com/rclone/rclone/compare/v1.65.0...v1.65.1">See commits</a></p>
<ul>

25
MANUAL.md generated
View File

@ -1,6 +1,6 @@
% rclone(1) User Manual
% Nick Craig-Wood
% Jan 08, 2024
% Jan 24, 2024
# Rclone syncs your files to cloud storage
@ -89,6 +89,7 @@ Rclone helps you:
- Can use multi-threaded downloads to local disk
- [Copy](https://rclone.org/commands/rclone_copy/) new or changed files to cloud storage
- [Sync](https://rclone.org/commands/rclone_sync/) (one way) to make a directory identical
- [Bisync](https://rclone.org/bisync/) (two way) to keep two directories in sync bidirectionally
- [Move](https://rclone.org/commands/rclone_move/) files to cloud storage deleting the local after verification
- [Check](https://rclone.org/commands/rclone_check/) hashes and for missing/extra files
- [Mount](https://rclone.org/commands/rclone_mount/) your cloud storage as a network disk
@ -127,6 +128,7 @@ WebDAV or S3, that work out of the box.)
- Hetzner Storage Box
- HiDrive
- HTTP
- ImageKit
- Internet Archive
- Jottacloud
- IBM COS S3
@ -4503,6 +4505,10 @@ Run without a hash to see the list of all supported hashes, e.g.
* whirlpool
* crc32
* sha256
* dropbox
* hidrive
* mailru
* quickxor
Then
@ -15918,7 +15924,7 @@ E.g. for an alternative `filter-file.txt`:
- *
Files `file1.jpg`, `file3.png` and `file2.avi` are listed whilst
`secret17.jpg` and files without the suffix .jpg` or `.png` are excluded.
`secret17.jpg` and files without the suffix `.jpg` or `.png` are excluded.
E.g. for an alternative `filter-file.txt`:
@ -18913,6 +18919,7 @@ upon backend-specific capabilities.
| HDFS | Yes | No | Yes | Yes | No | No | Yes | No | No | Yes | Yes |
| HiDrive | Yes | Yes | Yes | Yes | No | No | Yes | No | No | No | Yes |
| HTTP | No | No | No | No | No | No | No | No | No | No | Yes |
| ImageKit | Yes | Yes | Yes | No | No | No | No | No | No | No | Yes |
| Internet Archive | No | Yes | No | No | Yes | Yes | No | No | Yes | Yes | No |
| Jottacloud | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes |
| Koofr | Yes | Yes | Yes | Yes | No | No | Yes | No | Yes | Yes | Yes |
@ -19150,7 +19157,7 @@ General networking and HTTP stuff.
--tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default "rclone/v1.65.1")
--user-agent string Set the user-agent to a specified string (default "rclone/v1.65.2")
```
@ -50412,6 +50419,18 @@ Options:
# Changelog
## v1.65.2 - 2024-01-24
[See commits](https://github.com/rclone/rclone/compare/v1.65.1...v1.65.2)
* Bug Fixes
* build: bump github.com/cloudflare/circl from 1.3.6 to 1.3.7 (dependabot)
* docs updates (Nick Craig-Wood, kapitainsky, nielash, Tera, Harshit Budhraja)
* VFS
* Fix stale data when using `--vfs-cache-mode` full (Nick Craig-Wood)
* Azure Blob
* **IMPORTANT** Fix data corruption bug - see [#7590](https://github.com/rclone/rclone/issues/7590) (Nick Craig-Wood)
## v1.65.1 - 2024-01-08
[See commits](https://github.com/rclone/rclone/compare/v1.65.0...v1.65.1)

29
MANUAL.txt generated
View File

@ -1,6 +1,6 @@
rclone(1) User Manual
Nick Craig-Wood
Jan 08, 2024
Jan 24, 2024
Rclone syncs your files to cloud storage
@ -79,6 +79,7 @@ Features
- Can use multi-threaded downloads to local disk
- Copy new or changed files to cloud storage
- Sync (one way) to make a directory identical
- Bisync (two way) to keep two directories in sync bidirectionally
- Move files to cloud storage deleting the local after verification
- Check hashes and for missing/extra files
- Mount your cloud storage as a network disk
@ -116,6 +117,7 @@ S3, that work out of the box.)
- Hetzner Storage Box
- HiDrive
- HTTP
- ImageKit
- Internet Archive
- Jottacloud
- IBM COS S3
@ -4030,6 +4032,10 @@ Run without a hash to see the list of all supported hashes, e.g.
* whirlpool
* crc32
* sha256
* dropbox
* hidrive
* mailru
* quickxor
Then
@ -15202,7 +15208,7 @@ E.g. for an alternative filter-file.txt:
- *
Files file1.jpg, file3.png and file2.avi are listed whilst secret17.jpg
and files without the suffix .jpgor.png` are excluded.
and files without the suffix .jpg or .png are excluded.
E.g. for an alternative filter-file.txt:
@ -18234,6 +18240,8 @@ upon backend-specific capabilities.
HTTP No No No No No No No No No No Yes
ImageKit Yes Yes Yes No No No No No No No Yes
Internet No Yes No No Yes Yes No No Yes Yes No
Archive
@ -18492,7 +18500,7 @@ General networking and HTTP stuff.
--tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default "rclone/v1.65.1")
--user-agent string Set the user-agent to a specified string (default "rclone/v1.65.2")
Performance
@ -47925,6 +47933,21 @@ Options:
Changelog
v1.65.2 - 2024-01-24
See commits
- Bug Fixes
- build: bump github.com/cloudflare/circl from 1.3.6 to 1.3.7
(dependabot)
- docs updates (Nick Craig-Wood, kapitainsky, nielash, Tera,
Harshit Budhraja)
- VFS
- Fix stale data when using --vfs-cache-mode full (Nick
Craig-Wood)
- Azure Blob
- IMPORTANT Fix data corruption bug - see #7590 (Nick Craig-Wood)
v1.65.1 - 2024-01-08
See commits

View File

@ -1,5 +1,7 @@
[<img src="https://rclone.org/img/logo_on_light__horizontal_color.svg" width="50%" alt="rclone logo">](https://rclone.org/#gh-light-mode-only)
[<img src="https://rclone.org/img/logo_on_dark__horizontal_color.svg" width="50%" alt="rclone logo">](https://rclone.org/#gh-dark-mode-only)
[<img src="https://rclone.org/img/logos/warp-github-light.svg" title="Visit warp.dev to learn more." align="right">](https://www.warp.dev/?utm_source=github&utm_medium=referral&utm_campaign=rclone_20231103#gh-light-mode-only)
[<img src="https://rclone.org/img/logos/warp-github-dark.svg" title="Visit warp.dev to learn more." align="right">](https://www.warp.dev/?utm_source=github&utm_medium=referral&utm_campaign=rclone_20231103#gh-dark-mode-only)
[Website](https://rclone.org) |
[Documentation](https://rclone.org/docs/) |
@ -46,6 +48,7 @@ Rclone *("rsync for cloud storage")* is a command-line program to sync files and
* HiDrive [:page_facing_up:](https://rclone.org/hidrive/)
* HTTP [:page_facing_up:](https://rclone.org/http/)
* Huawei Cloud Object Storage Service(OBS) [:page_facing_up:](https://rclone.org/s3/#huawei-obs)
* ImageKit [:page_facing_up:](https://rclone.org/imagekit/)
* Internet Archive [:page_facing_up:](https://rclone.org/internetarchive/)
* Jottacloud [:page_facing_up:](https://rclone.org/jottacloud/)
* IBM COS S3 [:page_facing_up:](https://rclone.org/s3/#ibm-cos-s3)
@ -120,6 +123,7 @@ These backends adapt or modify other storage providers
* Partial syncs supported on a whole file basis
* [Copy](https://rclone.org/commands/rclone_copy/) mode to just copy new/changed files
* [Sync](https://rclone.org/commands/rclone_sync/) (one way) mode to make a directory identical
* [Bisync](https://rclone.org/bisync/) (two way) to keep two directories in sync bidirectionally
* [Check](https://rclone.org/commands/rclone_check/) mode to check for file hash equality
* Can sync to and from network, e.g. two different cloud accounts
* Optional large file chunking ([Chunker](https://rclone.org/chunker/))

View File

@ -8,6 +8,7 @@ import (
"context"
"crypto/md5"
"encoding/base64"
"encoding/binary"
"encoding/hex"
"encoding/json"
"errors"
@ -1966,34 +1967,21 @@ func (rs *readSeekCloser) Close() error {
return nil
}
// increment the array as LSB binary
func increment(xs *[8]byte) {
for i, digit := range xs {
newDigit := digit + 1
xs[i] = newDigit
if newDigit >= digit {
// exit if no carry
break
}
}
}
// record chunk number and id for Close
type azBlock struct {
chunkNumber int
chunkNumber uint64
id string
}
// Implements the fs.ChunkWriter interface
type azChunkWriter struct {
chunkSize int64
size int64
f *Fs
ui uploadInfo
blocksMu sync.Mutex // protects the below
blocks []azBlock // list of blocks for finalize
binaryBlockID [8]byte // block counter as LSB first 8 bytes
o *Object
chunkSize int64
size int64
f *Fs
ui uploadInfo
blocksMu sync.Mutex // protects the below
blocks []azBlock // list of blocks for finalize
o *Object
}
// OpenChunkWriter returns the chunk size and a ChunkWriter
@ -2081,13 +2069,14 @@ func (w *azChunkWriter) WriteChunk(ctx context.Context, chunkNumber int, reader
transactionalMD5 := md5sum[:]
// increment the blockID and save the blocks for finalize
increment(&w.binaryBlockID)
blockID := base64.StdEncoding.EncodeToString(w.binaryBlockID[:])
var binaryBlockID [8]byte // block counter as LSB first 8 bytes
binary.LittleEndian.PutUint64(binaryBlockID[:], uint64(chunkNumber))
blockID := base64.StdEncoding.EncodeToString(binaryBlockID[:])
// Save the blockID for the commit
w.blocksMu.Lock()
w.blocks = append(w.blocks, azBlock{
chunkNumber: chunkNumber,
chunkNumber: uint64(chunkNumber),
id: blockID,
})
w.blocksMu.Unlock()
@ -2152,9 +2141,20 @@ func (w *azChunkWriter) Close(ctx context.Context) (err error) {
return w.blocks[i].chunkNumber < w.blocks[j].chunkNumber
})
// Create a list of block IDs
// Create and check a list of block IDs
blockIDs := make([]string, len(w.blocks))
for i := range w.blocks {
if w.blocks[i].chunkNumber != uint64(i) {
return fmt.Errorf("internal error: expecting chunkNumber %d but got %d", i, w.blocks[i].chunkNumber)
}
chunkBytes, err := base64.StdEncoding.DecodeString(w.blocks[i].id)
if err != nil {
return fmt.Errorf("internal error: bad block ID: %w", err)
}
chunkNumber := binary.LittleEndian.Uint64(chunkBytes)
if w.blocks[i].chunkNumber != chunkNumber {
return fmt.Errorf("internal error: expecting decoded chunkNumber %d but got %d", w.blocks[i].chunkNumber, chunkNumber)
}
blockIDs[i] = w.blocks[i].id
}

View File

@ -17,21 +17,3 @@ func (f *Fs) InternalTest(t *testing.T) {
enabled = f.Features().GetTier
assert.True(t, enabled)
}
func TestIncrement(t *testing.T) {
for _, test := range []struct {
in [8]byte
want [8]byte
}{
{[8]byte{0, 0, 0, 0}, [8]byte{1, 0, 0, 0}},
{[8]byte{0xFE, 0, 0, 0}, [8]byte{0xFF, 0, 0, 0}},
{[8]byte{0xFF, 0, 0, 0}, [8]byte{0, 1, 0, 0}},
{[8]byte{0, 1, 0, 0}, [8]byte{1, 1, 0, 0}},
{[8]byte{0xFF, 0xFF, 0xFF, 0xFE}, [8]byte{0, 0, 0, 0xFF}},
{[8]byte{0xFF, 0xFF, 0xFF, 0xFF}, [8]byte{0, 0, 0, 0, 1}},
{[8]byte{0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF}, [8]byte{0, 0, 0, 0, 0, 0, 0}},
} {
increment(&test.in)
assert.Equal(t, test.want, test.in)
}
}

View File

@ -92,6 +92,7 @@ Rclone helps you:
- Can use multi-threaded downloads to local disk
- [Copy](/commands/rclone_copy/) new or changed files to cloud storage
- [Sync](/commands/rclone_sync/) (one way) to make a directory identical
- [Bisync](/bisync/) (two way) to keep two directories in sync bidirectionally
- [Move](/commands/rclone_move/) files to cloud storage deleting the local after verification
- [Check](/commands/rclone_check/) hashes and for missing/extra files
- [Mount](/commands/rclone_mount/) your cloud storage as a network disk
@ -130,6 +131,7 @@ WebDAV or S3, that work out of the box.)
{{< provider name="Hetzner Storage Box" home="https://www.hetzner.com/storage/storage-box" config="/sftp/#hetzner-storage-box" >}}
{{< provider name="HiDrive" home="https://www.strato.de/cloud-speicher/" config="/hidrive/" >}}
{{< provider name="HTTP" home="https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol" config="/http/" >}}
{{< provider name="ImageKit" home="https://imagekit.io" config="/imagekit/" >}}
{{< provider name="Internet Archive" home="https://archive.org/" config="/internetarchive/" >}}
{{< provider name="Jottacloud" home="https://www.jottacloud.com/en/" config="/jottacloud/" >}}
{{< provider name="IBM COS S3" home="http://www.ibm.com/cloud/object-storage" config="/s3/#ibm-cos-s3" >}}

View File

@ -5,6 +5,18 @@ description: "Rclone Changelog"
# Changelog
## v1.65.2 - 2024-01-24
[See commits](https://github.com/rclone/rclone/compare/v1.65.1...v1.65.2)
* Bug Fixes
* build: bump github.com/cloudflare/circl from 1.3.6 to 1.3.7 (dependabot)
* docs updates (Nick Craig-Wood, kapitainsky, nielash, Tera, Harshit Budhraja)
* VFS
* Fix stale data when using `--vfs-cache-mode` full (Nick Craig-Wood)
* Azure Blob
* **IMPORTANT** Fix data corruption bug - see [#7590](https://github.com/rclone/rclone/issues/7590) (Nick Craig-Wood)
## v1.65.1 - 2024-01-08
[See commits](https://github.com/rclone/rclone/compare/v1.65.0...v1.65.1)

View File

@ -831,7 +831,7 @@ rclone [flags]
--use-json-log Use json log format
--use-mmap Use mmap allocator (see docs)
--use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string (default "rclone/v1.65.1")
--user-agent string Set the user-agent to a specified string (default "rclone/v1.65.2")
-v, --verbose count Print lots more stuff (repeat for more)
-V, --version Print the version number
--webdav-bearer-token string Bearer token instead of user/pass (e.g. a Macaroon)

View File

@ -40,6 +40,10 @@ Run without a hash to see the list of all supported hashes, e.g.
* whirlpool
* crc32
* sha256
* dropbox
* hidrive
* mailru
* quickxor
Then

View File

@ -10,6 +10,17 @@ Rclone is single executable (`rclone`, or `rclone.exe` on Windows) that you can
simply download as a zip archive and extract into a location of your choosing.
See the [install](https://rclone.org/install/) documentation for more details.
## Release {{% version %}} OS requirements {#osrequirements}
| OS | Minimum Version |
|:-------:|:-------:|
| Linux | kernel 2.6.32 |
| macOS (Intel) | 10.15 (Catalina) |
| macOS (ARM64) | 11 (Big Sur) |
| Windows | 10, Server 2016 |
| FreeBSD | 12.2 |
| OpenBSD | 6.9 |
## Release {{% version %}} {#release}
| Arch-OS | Windows | macOS | Linux | .deb | .rpm | FreeBSD | NetBSD | OpenBSD | Plan9 | Solaris |
@ -101,3 +112,19 @@ script) from a URL which doesn't change then you can use these links.
## Older Downloads
Older downloads can be found [here](https://downloads.rclone.org/).
The latest `rclone` version working for:
| OS | Maximum rclone version |
|:-------:|:-------:|
| Windows 7 | v1.63.1 |
| Windows Server 2008 | v1.63.1 |
| Windows Server 2012 | v1.63.1 |
| Windows XP | v1.42 |
| Windows Vista | v1.42 |
| macOS 10.14 (Mojave) | v1.63.1 |
| macOS 10.13 (High Sierra) | v1.63.1 |
| macOS 10.12 (Sierra) | v1.56.0 |
| macOS 10.11 (El Capitan) | v1.52.0 |
| macOS 10.10 (Yosemite) | v1.49.0 |
| OS X 10.9 (Mavericks) | v1.42 |
| OS X 10.8 (Mountain Lion) | v1.42 |

View File

@ -533,7 +533,7 @@ E.g. for an alternative `filter-file.txt`:
- *
Files `file1.jpg`, `file3.png` and `file2.avi` are listed whilst
`secret17.jpg` and files without the suffix .jpg` or `.png` are excluded.
`secret17.jpg` and files without the suffix `.jpg` or `.png` are excluded.
E.g. for an alternative `filter-file.txt`:

View File

@ -112,7 +112,7 @@ General networking and HTTP stuff.
--tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default "rclone/v1.65.1")
--user-agent string Set the user-agent to a specified string (default "rclone/v1.65.2")
```

View File

@ -485,6 +485,7 @@ upon backend-specific capabilities.
| HDFS | Yes | No | Yes | Yes | No | No | Yes | No | No | Yes | Yes |
| HiDrive | Yes | Yes | Yes | Yes | No | No | Yes | No | No | No | Yes |
| HTTP | No | No | No | No | No | No | No | No | No | No | Yes |
| ImageKit | Yes | Yes | Yes | No | No | No | No | No | No | No | Yes |
| Internet Archive | No | Yes | No | No | Yes | Yes | No | No | Yes | Yes | No |
| Jottacloud | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes |
| Koofr | Yes | Yes | Yes | Yes | No | No | Yes | No | Yes | Yes | Yes |

View File

@ -74,6 +74,7 @@
<a class="dropdown-item" href="/hdfs/"><i class="fa fa-globe fa-fw"></i> HDFS (Hadoop Distributed Filesystem)</a>
<a class="dropdown-item" href="/hidrive/"><i class="fa fa-cloud fa-fw"></i> HiDrive</a>
<a class="dropdown-item" href="/http/"><i class="fa fa-globe fa-fw"></i> HTTP</a>
<a class="dropdown-item" href="/imagekit/"><i class="fa fa-cloud fa-fw"></i> ImageKit</a>
<a class="dropdown-item" href="/internetarchive/"><i class="fa fa-archive fa-fw"></i> Internet Archive</a>
<a class="dropdown-item" href="/jottacloud/"><i class="fa fa-cloud fa-fw"></i> Jottacloud</a>
<a class="dropdown-item" href="/koofr/"><i class="fa fa-suitcase fa-fw"></i> Koofr</a>

View File

@ -286,6 +286,9 @@ func (r *Run) MakeTestBinary() {
binaryName := r.BinaryName()
log.Printf("%s: Making test binary %q", r.Path, binaryName)
CmdLine := []string{"go", "test", "-c"}
if *race {
CmdLine = append(CmdLine, "-race")
}
if *dryRun {
log.Printf("Not executing: %v", CmdLine)
return

View File

@ -35,6 +35,7 @@ var (
clean = flag.Bool("clean", false, "Instead of testing, clean all left over test directories")
runOnly = flag.String("run", "", "Run only those tests matching the regexp supplied")
timeout = flag.Duration("timeout", 60*time.Minute, "Maximum time to run each test for before giving up")
race = flag.Bool("race", false, "If set run the tests under the race detector")
configFile = flag.String("config", "fstest/test_all/config.yaml", "Path to config file")
outputDir = flag.String("output", path.Join(os.TempDir(), "rclone-integration-tests"), "Place to store results")
emailReport = flag.String("email", "", "Set to email the report to the address supplied")

66
rclone.1 generated
View File

@ -1,7 +1,7 @@
.\"t
.\" Automatically generated by Pandoc 2.9.2.1
.\"
.TH "rclone" "1" "Jan 08, 2024" "User Manual" ""
.TH "rclone" "1" "Jan 24, 2024" "User Manual" ""
.hy
.SH Rclone syncs your files to cloud storage
.PP
@ -122,6 +122,9 @@ cloud storage
Sync (https://rclone.org/commands/rclone_sync/) (one way) to make a
directory identical
.IP \[bu] 2
Bisync (https://rclone.org/bisync/) (two way) to keep two directories in
sync bidirectionally
.IP \[bu] 2
Move (https://rclone.org/commands/rclone_move/) files to cloud storage
deleting the local after verification
.IP \[bu] 2
@ -193,6 +196,8 @@ HiDrive
.IP \[bu] 2
HTTP
.IP \[bu] 2
ImageKit
.IP \[bu] 2
Internet Archive
.IP \[bu] 2
Jottacloud
@ -5410,6 +5415,10 @@ Supported hashes are:
* whirlpool
* crc32
* sha256
* dropbox
* hidrive
* mailru
* quickxor
\f[R]
.fi
.PP
@ -19552,7 +19561,7 @@ for an alternative \f[C]filter-file.txt\f[R]:
.PP
Files \f[C]file1.jpg\f[R], \f[C]file3.png\f[R] and \f[C]file2.avi\f[R]
are listed whilst \f[C]secret17.jpg\f[R] and files without the suffix
\&.jpg\f[C]or\f[R].png\[ga] are excluded.
\f[C].jpg\f[R] or \f[C].png\f[R] are excluded.
.PP
E.g.
for an alternative \f[C]filter-file.txt\f[R]:
@ -24673,6 +24682,31 @@ T}@T{
Yes
T}
T{
ImageKit
T}@T{
Yes
T}@T{
Yes
T}@T{
Yes
T}@T{
No
T}@T{
No
T}@T{
No
T}@T{
No
T}@T{
No
T}@T{
No
T}@T{
No
T}@T{
Yes
T}
T{
Internet Archive
T}@T{
No
@ -25630,7 +25664,7 @@ General networking and HTTP stuff.
--tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default \[dq]rclone/v1.65.1\[dq])
--user-agent string Set the user-agent to a specified string (default \[dq]rclone/v1.65.2\[dq])
\f[R]
.fi
.SS Performance
@ -60043,6 +60077,32 @@ Options:
.IP \[bu] 2
\[dq]error\[dq]: return an error based on option value
.SH Changelog
.SS v1.65.2 - 2024-01-24
.PP
See commits (https://github.com/rclone/rclone/compare/v1.65.1...v1.65.2)
.IP \[bu] 2
Bug Fixes
.RS 2
.IP \[bu] 2
build: bump github.com/cloudflare/circl from 1.3.6 to 1.3.7 (dependabot)
.IP \[bu] 2
docs updates (Nick Craig-Wood, kapitainsky, nielash, Tera, Harshit
Budhraja)
.RE
.IP \[bu] 2
VFS
.RS 2
.IP \[bu] 2
Fix stale data when using \f[C]--vfs-cache-mode\f[R] full (Nick
Craig-Wood)
.RE
.IP \[bu] 2
Azure Blob
.RS 2
.IP \[bu] 2
\f[B]IMPORTANT\f[R] Fix data corruption bug - see
#7590 (https://github.com/rclone/rclone/issues/7590) (Nick Craig-Wood)
.RE
.SS v1.65.1 - 2024-01-08
.PP
See commits (https://github.com/rclone/rclone/compare/v1.65.0...v1.65.1)