community
/
rclone
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,703 Commits 2 Branches 109 Tags 198 MiB
Go 98.5%
HTML 0.4%
Shell 0.4%
Python 0.3%
JavaScript 0.2%
Other 0.1%
Go to file
Leo Luan c665201b85 vfs: support synchronous cache space recovery upon ENOSPC 
This patch provides the support of synchronous cache space recovery
to allow read threads to recover from ENOSPC errors when cache space
can be recovered from cache items that are not in use or safe to be
reset/emptied .

The patch complements the existing cache cleaning process in two ways.

Firstly, the existing cache cleaning process is time-driven that runs
periodically. The cache space can run out while the cache cleaner
thread is still waiting for its next scheduled run. The io threads
encountering ENOSPC return an internal error to the applications
in this case even when cache space can be recovered to avoid this
error. This patch addresses this problem by having the read threads
kick the cache cleaner thread in this condition to recover cache
space preventing unnecessary ENOSPC errors from being seen by the
applications.

Secondly, this patch enhances the cache cleaner to support cache
item reset. Currently the cache purge process removes cache
items that are not in use. This may not be sufficient when the
total size of the working set exceeds the cache directory's
capacity. Like in the current code, this patch starts the purge
process by removing cache files that are not in use. Cache items
whose access times are older than vfs-cache-max-age are removed first.
After that, other not-in-use items are removed in LRU order until
vfs-cache-max-size is reached. If the vfs-cache-max-size (the quota)
is still not reached at this time, this patch adds a cache reset
step to reset/empty cache files that are still in use but not
dirtied.  This enables application processes to continue without
seeing an error even when the working set depletes the cache space
as long as there is not a large write working set hoarding the
entire cache space.

By design this patch does not add ENOSPC error recovery for write
IOs. Rclone does not empty a write cache item until the file data
is written back to the backend upon close. Allowing more cache
space to be consumed by dirty cache items when the cache space is
already running low would increase the risk of exhausting the cache
space in a way that the vfs mount becomes unreadable.
 		2020-08-25 21:12:06 +01:00
.github build: build with go1.15.x by default now that it is released 2020-08-12 09:51:22 +01:00
backend fichier: Detect Flood detected: IP Locked error and sleep for 30s 2020-08-23 18:01:22 +01:00
bin vfs: set the modtime of the cache file immediately 2020-08-20 16:24:04 +01:00
cmd build: make rclone build with wasm 2020-08-10 17:32:21 +01:00
contrib/docker Contrib: Add sample WebDAV server Docker Compose manifest. 2019-09-15 16:06:54 +01:00
docs Add Jay McEntire to contributors 2020-08-23 18:01:22 +01:00
fs vfs: support synchronous cache space recovery upon ENOSPC 2020-08-25 21:12:06 +01:00
fstest test_all: disable chunker + mailru tests while mailru is broken #4376 2020-08-20 12:50:20 +01:00
graphics build: Add file properties and icon to Windows executable (fixes #4304) 2020-06-11 09:26:14 +01:00
lib lib: Add file name compression 2020-08-13 16:14:11 +01:00
vfs vfs: support synchronous cache space recovery upon ENOSPC 2020-08-25 21:12:06 +01:00
.gitattributes build: add azure pipelines build 2019-08-06 10:31:32 +01:00
.gitignore lib: Add file name compression 2020-08-13 16:14:11 +01:00
.golangci.yml build: move linter build tags into Makefile to fix golangci-lint 2019-04-12 15:48:36 +01:00
CONTRIBUTING.md Stop doing vendoring - fixes #4032 2020-07-21 16:09:53 +01:00
COPYING Initial commit - some small parts working 2012-11-18 17:32:31 +00:00
Dockerfile build: fix custom timezone in Docker image 2020-06-17 10:43:03 +01:00
MAINTAINERS.md doc: fix typos throughout docs and code 2020-05-25 11:23:58 +01:00
MANUAL.html Version v1.52.0 2020-05-27 17:31:10 +01:00
MANUAL.md Version v1.52.0 2020-05-27 17:31:10 +01:00
MANUAL.txt Version v1.52.0 2020-05-27 17:31:10 +01:00
Makefile build: actions update, cache, go1.15-rc1 build 2020-07-25 18:52:33 +01:00
README.md Tardigrade Backend 2020-05-12 15:56:50 +00:00
RELEASE.md Stop doing vendoring - fixes #4032 2020-07-21 16:09:53 +01:00
VERSION Start v1.52.3-DEV development 2020-08-08 10:35:06 +01:00
go.mod vfs: support synchronous cache space recovery upon ENOSPC 2020-08-25 21:12:06 +01:00
go.sum vfs: support synchronous cache space recovery upon ENOSPC 2020-08-25 21:12:06 +01:00
notes.txt Replace test_all.sh with test_all.go which is cross platform and parallel 2015-12-30 09:26:34 +00:00
rclone.1 Version v1.52.0 2020-05-27 17:31:10 +01:00
rclone.go lib: add plugin support 2019-09-29 11:05:10 +01:00

README.md

rclone logo

Website | Documentation | Download | Contributing | Changelog | Installation | Forum

Build Status Go Report Card GoDoc Docker Pulls

Rclone

Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers.

Storage providers

Please see the full list of all storage providers and their features

Features

  • MD5/SHA-1 hashes checked at all times for file integrity
  • Timestamps preserved on files
  • Partial syncs supported on a whole file basis
  • Copy mode to just copy new/changed files
  • Sync (one way) mode to make a directory identical
  • Check mode to check for file hash equality
  • Can sync to and from network, e.g. two different cloud accounts
  • Optional large file chunking (Chunker)
  • Optional encryption (Crypt)
  • Optional cache (Cache)
  • Optional FUSE mount (rclone mount)
  • Multi-threaded downloads to local disk
  • Can serve local or remote files over HTTP/WebDav/FTP/SFTP/dlna

Installation & documentation

Please see the rclone website for:

Downloads

License

This is free software under the terms of MIT the license (check the COPYING file included in this package).