community
/
rclone
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7,127 Commits 2 Branches 109 Tags 198 MiB
Go 98.5%
HTML 0.4%
Shell 0.4%
Python 0.3%
JavaScript 0.2%
Other 0.1%
Go to file
Nick Craig-Wood 07133b892d dirtree: fix performance with large directories of directories and --fast-list 
Before this change if using --fast-list on a directory with more than
a few thousand directories in it DirTree.CheckParents became very slow
taking up to 24 hours for a directory with 1,000,000 directories in
it.

This is because it becomes an O(N²) operation as DirTree.Find has to
search each directory in a linear fashion as it is stored as a slice.

This patch fixes the problem by scanning the DirTree for directories
before starting the CheckParents process so it never has to call
DirTree.Find.

After the fix calling DirTree.CheckParents on a directory with
1,000,000 directories in it will take about 1 second.

Anything which calls DirTree.Find can potentially have bad performance
so in the future we should redesign the DirTree to use a different
underlying datastructure or have an index.

https://forum.rclone.org/t/almost-24-hours-cpu-compute-time-during-sync-between-two-large-s3-buckets/39375/
 		2023-07-03 14:09:21 +01:00
.github Revert sponsors back to organization 2023-06-30 10:10:05 +01:00
backend jottacloud: add Onlime provider 2023-07-02 11:16:07 +01:00
bin pikpak: new backend 2023-04-04 16:33:48 +01:00
cmd docs: listremotes also includes remotes from env vars 2023-06-24 15:46:23 +01:00
cmdtest cmdtest: fix typos found by codespell 2023-03-24 20:44:25 +00:00
contrib docker volume plugin: add missing fuse3 dependency #6844 2023-03-15 15:57:53 +00:00
docs Add Fjodor42 to contributors 2023-07-03 14:09:21 +01:00
fs dirtree: fix performance with large directories of directories and --fast-list 2023-07-03 14:09:21 +01:00
fstest fstests: allow ObjectUpdate test to retry upload 2023-06-27 16:02:33 +01:00
graphics build: Add file properties and icon to Windows executable (fixes #4304) 2020-06-11 09:26:14 +01:00
lib http: add client certificate user auth middleware 2023-06-26 14:33:53 +01:00
librclone librclone:Added example on using librclone with golang. 2023-03-17 12:00:27 +00:00
vfs vfs: fix backends being Shutdown too early when startup takes a long time 2023-05-18 16:16:12 +01:00
.gitattributes build: add azure pipelines build 2019-08-06 10:31:32 +01:00
.gitignore chunker: fix md5all test for no-meta test remotes 2021-10-18 18:04:07 +03:00
.golangci.yml Use jwt-go (golang-jwt) instead of deprecated jws (x/oauth2/jws) 2023-03-26 19:20:50 +02:00
CONTRIBUTING.md all: fix spelling across the project 2022-08-30 11:16:26 +02:00
COPYING Initial commit - some small parts working 2012-11-18 17:32:31 +00:00
Dockerfile docker: add missing fuse3 dependency - fixes #6844 2023-03-15 10:54:30 +00:00
MAINTAINERS.md Add albertony to maintainers 2023-05-17 15:31:07 +02:00
MANUAL.html Version v1.63.0 2023-06-30 14:11:17 +01:00
MANUAL.md Version v1.63.0 2023-06-30 14:11:17 +01:00
MANUAL.txt Version v1.63.0 2023-06-30 14:11:17 +01:00
Makefile build: fix build failure installing nfpm 2023-06-07 15:41:52 +01:00
README.md docs: add blomp cloud storage guide 2023-06-26 17:49:27 +01:00
RELEASE.md build: update release docs to be more careful with the tag 2023-03-15 14:53:21 +00:00
VERSION Start v1.64.0-DEV development 2023-06-30 15:39:03 +01:00
go.mod build: fix macos builds for versions < 12 2023-07-01 18:03:50 +01:00
go.sum build: fix macos builds for versions < 12 2023-07-01 18:03:50 +01:00
notes.txt docs: spelling: etc. 2020-10-28 18:16:23 +00:00
rclone.1 Version v1.63.0 2023-06-30 14:11:17 +01:00
rclone.go lib: add plugin support 2019-09-29 11:05:10 +01:00

README.md

rclone logo rclone logo

Website | Documentation | Download | Contributing | Changelog | Installation | Forum

Build Status Go Report Card GoDoc Docker Pulls

Rclone

Rclone ("rsync for cloud storage") is a command-line program to sync files and directories to and from different cloud storage providers.

Storage providers

Please see the full list of all storage providers and their features

Virtual storage providers

These backends adapt or modify other storage providers

  • Alias: rename existing remotes 📄
  • Cache: cache remotes (DEPRECATED) 📄
  • Chunker: split large files 📄
  • Combine: combine multiple remotes into a directory tree 📄
  • Compress: compress files 📄
  • Crypt: encrypt files 📄
  • Hasher: hash files 📄
  • Union: join multiple remotes to work together 📄

Features

  • MD5/SHA-1 hashes checked at all times for file integrity
  • Timestamps preserved on files
  • Partial syncs supported on a whole file basis
  • Copy mode to just copy new/changed files
  • Sync (one way) mode to make a directory identical
  • Check mode to check for file hash equality
  • Can sync to and from network, e.g. two different cloud accounts
  • Optional large file chunking (Chunker)
  • Optional transparent compression (Compress)
  • Optional encryption (Crypt)
  • Optional FUSE mount (rclone mount)
  • Multi-threaded downloads to local disk
  • Can serve local or remote files over HTTP/WebDAV/FTP/SFTP/DLNA

Installation & documentation

Please see the rclone website for:

Downloads

License

This is free software under the terms of the MIT license (check the COPYING file included in this package).