community
/
rclone
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
5,431 Commits 2 Branches 109 Tags 198 MiB
Go 98.5%
HTML 0.4%
Shell 0.4%
Python 0.3%
JavaScript 0.2%
Other 0.1%
Go to file
Nick Craig-Wood 0537791d14 sftp: Fix performance regression by re-enabling concurrent writes #5197 
Betweeen rclone v1.54 and v1.55 there was an approx 3x performance
regression when transferring to distant SFTP servers (in particular
rsync.net).

This turned out to be due to the library github.com/pkg/sftp rclone
uses. Concurrent writes used to be enabled in this library by default
(for v1.12.0 as used in rclone v1.54) but they are no longer enabled
(for v1.13.0 as used in rclone v1.55) for safety reasons and it is
necessary to enable them specifically.

The safety concerns are due to the uncertainty as to whether writes
come in order and whether a half completed file might have holes in
it. This isn't a problem for rclone since a) it doesn't restart
uploads and b) it has a post-transfer checksum test.

This change introduces a new flag `--sftp-disable-concurrent-writes`
to control the feature which defaults to false, meaning that
concurrent writes are enabled as in v1.54.

However this isn't quite enough to fix the problem as the sftp library
needs to be able to sniff the size of the stream from the reader
passed in, so this also adds a `Size` interface to the reader to
enable this. This involved a patch to the library.

The library was reverted to v1.12.0 for v1.55.1 - this patch installs
v1.13.0+master to fix the Size interface problem.

See: https://github.com/pkg/sftp/issues/426
 		2021-04-26 09:24:28 +01:00
.github build: fix version numbers in android branch builds 2021-04-20 17:40:06 +01:00
backend sftp: Fix performance regression by re-enabling concurrent writes #5197 2021-04-26 09:24:28 +01:00
bin make_manual.py: fix missing comma for doc build after uptobox merge 2021-04-19 16:18:18 +01:00
cmd test changenotify: add command to help debugging changenotify 2021-04-21 10:52:05 +01:00
contrib/docker Contrib: Add sample WebDAV server Docker Compose manifest. 2019-09-15 16:06:54 +01:00
docs Changelog updates from Version v1.55.1 2021-04-26 09:22:49 +01:00
fs config: treat any config file paths with filename notfound as memory-only config (#5235) 2021-04-18 00:09:03 +02:00
fstest uptobox: integration tests 2021-04-13 17:46:07 +02:00
graphics build: Add file properties and icon to Windows executable (fixes #4304) 2020-06-11 09:26:14 +01:00
lib oauthutil: fix #5265 old authorize result not recognised 2021-04-23 01:20:52 +08:00
vfs vfs: fix a code path which allows dirty data to be removed causing data loss 2021-04-20 16:36:38 +01:00
.gitattributes build: add azure pipelines build 2019-08-06 10:31:32 +01:00
.gitignore cmd: rclone selfupdate (#5080) 2021-03-11 22:39:30 +03:00
.golangci.yml build: move linter build tags into Makefile to fix golangci-lint 2019-04-12 15:48:36 +01:00
CONTRIBUTING.md contributing.md: update setup instructions for go1.16 2021-04-04 09:10:43 +01:00
COPYING Initial commit - some small parts working 2012-11-18 17:32:31 +00:00
Dockerfile build: add an rclone user to the Docker image but don't use it by default 2021-02-11 17:45:44 +00:00
MAINTAINERS.md compress: finish docs 2020-12-02 16:30:02 +01:00
MANUAL.html Version v1.55.0 2021-03-31 19:12:08 +01:00
MANUAL.md Version v1.55.0 2021-03-31 19:12:08 +01:00
MANUAL.txt Version v1.55.0 2021-03-31 19:12:08 +01:00
Makefile build: add missing BUILD_FLAGS to compile_only to speed up other_os build 2021-03-15 19:22:07 +00:00
README.md HDFS (Hadoop Distributed File System) implementation - #42 2021-01-07 09:48:51 +00:00
RELEASE.md build: update notes on how to build the release manually with docker 2021-03-24 14:22:27 +00:00
VERSION Start v1.56.0-DEV development 2021-03-31 19:51:43 +01:00
go.mod sftp: Fix performance regression by re-enabling concurrent writes #5197 2021-04-26 09:24:28 +01:00
go.sum sftp: Fix performance regression by re-enabling concurrent writes #5197 2021-04-26 09:24:28 +01:00
notes.txt docs: spelling: etc. 2020-10-28 18:16:23 +00:00
rclone.1 Version v1.55.0 2021-03-31 19:12:08 +01:00
rclone.go lib: add plugin support 2019-09-29 11:05:10 +01:00

README.md

rclone logo

Website | Documentation | Download | Contributing | Changelog | Installation | Forum

Build Status Go Report Card GoDoc Docker Pulls

Rclone

Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers.

Storage providers

Please see the full list of all storage providers and their features

Features

  • MD5/SHA-1 hashes checked at all times for file integrity
  • Timestamps preserved on files
  • Partial syncs supported on a whole file basis
  • Copy mode to just copy new/changed files
  • Sync (one way) mode to make a directory identical
  • Check mode to check for file hash equality
  • Can sync to and from network, e.g. two different cloud accounts
  • Optional large file chunking (Chunker)
  • Optional transparent compression (Compress)
  • Optional encryption (Crypt)
  • Optional cache (Cache)
  • Optional FUSE mount (rclone mount)
  • Multi-threaded downloads to local disk
  • Can serve local or remote files over HTTP/WebDav/FTP/SFTP/dlna

Installation & documentation

Please see the rclone website for:

Downloads

License

This is free software under the terms of MIT the license (check the COPYING file included in this package).