From 63708d73beb86da422f925fc99b28f1375db6f14 Mon Sep 17 00:00:00 2001 From: albertony <12441419+albertony@users.noreply.github.com> Date: Thu, 19 Aug 2021 19:29:41 +0200 Subject: [PATCH] docs/vfs: Merge duplicate chunked reading documentation from mount docs --- cmd/mountlib/help.go | 16 ---------- vfs/help.go | 71 +++++++++++++++++++++++++++----------------- 2 files changed, 43 insertions(+), 44 deletions(-) diff --git a/cmd/mountlib/help.go b/cmd/mountlib/help.go index 973840863..32d160088 100644 --- a/cmd/mountlib/help.go +++ b/cmd/mountlib/help.go @@ -283,20 +283,4 @@ to use Type=notify. In this case the service will enter the started state after the mountpoint has been successfully set up. Units having the rclone @ service specified as a requirement will see all files and folders immediately in this mode. - -### chunked reading - -|--vfs-read-chunk-size| will enable reading the source objects in parts. -This can reduce the used download quota for some remotes by requesting only chunks -from the remote that are actually read at the cost of an increased number of requests. - -When |--vfs-read-chunk-size-limit| is also specified and greater than -|--vfs-read-chunk-size|, the chunk size for each open file will get doubled -for each chunk read, until the specified value is reached. A value of |-1| will disable -the limit and the chunk size will grow indefinitely. - -With |--vfs-read-chunk-size 100M| and |--vfs-read-chunk-size-limit 0| -the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. -When |--vfs-read-chunk-size-limit 500M| is specified, the result would be -0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on. ` diff --git a/vfs/help.go b/vfs/help.go index 39b41c47d..1cff104fb 100644 --- a/vfs/help.go +++ b/vfs/help.go @@ -100,8 +100,8 @@ The higher the cache mode the more compatible rclone becomes at the cost of using disk space. Note that files are written back to the remote only when they are -closed and if they haven't been accessed for --vfs-write-back -second. If rclone is quit or dies with files that haven't been +closed and if they haven't been accessed for !--vfs-write-back! +seconds. If rclone is quit or dies with files that haven't been uploaded, these will be uploaded next time rclone is run with the same flags. @@ -170,27 +170,55 @@ their full size in the cache, but they will be sparse files with only the data that has been downloaded present in them. This mode should support all normal file system operations and is -otherwise identical to --vfs-cache-mode writes. +otherwise identical to !--vfs-cache-mode! writes. -When reading a file rclone will read --buffer-size plus ---vfs-read-ahead bytes ahead. The --buffer-size is buffered in memory -whereas the --vfs-read-ahead is buffered on disk. +When reading a file rclone will read !--buffer-size! plus +!--vfs-read-ahead! bytes ahead. The !--buffer-size! is buffered in memory +whereas the !--vfs-read-ahead! is buffered on disk. -When using this mode it is recommended that --buffer-size is not set -too big and --vfs-read-ahead is set large if required. +When using this mode it is recommended that !--buffer-size! is not set +too large and !--vfs-read-ahead! is set large if required. **IMPORTANT** not all file systems support sparse files. In particular FAT/exFAT do not. Rclone will perform very badly if the cache directory is on a filesystem which doesn't support sparse files and it will log an ERROR message if one is detected. +### VFS Chunked Reading + +When rclone reads files from a remote it reads them in chunks. This +means that rather than requesting the whole file rclone reads the +chunk specified. This can reduce the used download quota for some +remotes by requesting only chunks from the remote that are actually +read, at the cost of an increased number of requests. + +These flags control the chunking: + + --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) + --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + +Rclone will start reading a chunk of size !--vfs-read-chunk-size!, +and then double the size for each read. When !--vfs-read-chunk-size-limit! is +specified, and greater than !--vfs-read-chunk-size!, the chunk size for each +open file will get doubled only until the specified value is reached. If the +value is "off", which is the default, the limit is disabled and the chunk size +will grow indefinitely. + +With !--vfs-read-chunk-size 100M! and !--vfs-read-chunk-size-limit 0! +the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. +When !--vfs-read-chunk-size-limit 500M! is specified, the result would be +0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on. + +Setting !--vfs-read-chunk-size! to !0! or "off" disables chunked reading. + ### VFS Performance These flags may be used to enable/disable features of the VFS for -performance or other reasons. +performance or other reasons. See also the [chunked reading](#vfs-chunked-reading) +feature. -In particular S3 and Swift benefit hugely from the --no-modtime flag -(or use --use-server-modtime for a slightly different effect) as each +In particular S3 and Swift benefit hugely from the !--no-modtime! flag +(or use !--use-server-modtime! for a slightly different effect) as each read of the modification time takes a transaction. --no-checksum Don't compare checksums on up/download. @@ -198,19 +226,6 @@ read of the modification time takes a transaction. --no-seek Don't allow seeking in files. --read-only Mount read-only. -When rclone reads files from a remote it reads them in chunks. This -means that rather than requesting the whole file rclone reads the -chunk specified. This is advantageous because some cloud providers -account for reads being all the data requested, not all the data -delivered. - -Rclone will keep doubling the chunk size requested starting at ---vfs-read-chunk-size with a maximum of --vfs-read-chunk-size-limit -unless it is set to "off" in which case there will be no limit. - - --vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M) - --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default "off") - Sometimes rclone is delivered reads or writes out of order. Rather than seeking rclone will wait a short time for the in sequence read or write to come in. These flags only come into effect when not using an @@ -219,9 +234,9 @@ on disk cache file. --vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms) --vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s) -When using VFS write caching (--vfs-cache-mode with value writes or full), -the global flag --transfers can be set to adjust the number of parallel uploads of -modified files from cache (the related global flag --checkers have no effect on mount). +When using VFS write caching (!--vfs-cache-mode! with value writes or full), +the global flag !--transfers! can be set to adjust the number of parallel uploads of +modified files from cache (the related global flag !--checkers! have no effect on mount). --transfers int Number of file transfers to run in parallel. (default 4) @@ -236,7 +251,7 @@ to create the file is preserved and available for programs to query. It is not allowed for two files in the same directory to differ only by case. Usually file systems on macOS are case-insensitive. It is possible to make macOS -file systems case-sensitive but that is not the default +file systems case-sensitive but that is not the default. The !--vfs-case-insensitive! mount flag controls how rclone handles these two cases. If its value is "false", rclone passes file names to the mounted