community
/
rclone
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

chunker: Update documentation to mention issue with small files 

See: https://forum.rclone.org/t/chunker-not-deactivating-for-small-files-and-wasting-api-calls/40122
This commit is contained in:
Ricardo D'O. Albanus 2023-07-22 18:40:50 -05:00 committed by GitHub
parent bb0cd76a5f
commit 0957c8fb74
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 9 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
docs/content/chunker.md
View File

@ -97,7 +97,7 @@ will put files in a directory called `name` in the current directory.
When rclone starts a file upload, chunker checks the file size. If it
doesn't exceed the configured chunk size, chunker will just pass the file
to the wrapped remote. If a file is large, chunker will transparently cut
to the wrapped remote (however, see caveat below). If a file is large, chunker will transparently cut
data in pieces with temporary names and stream them one by one, on the fly.
Each data chunk will contain the specified number of bytes, except for the
last one which may have less data. If file size is unknown in advance
@ -129,6 +129,14 @@ proceed with current command.
You can set the `--chunker-fail-hard` flag to have commands abort with
error message in such cases.
**Caveat**: As it is now, chunker will always create a temporary file in the
backend and then rename it, even if the file is below the chunk threshold.
This will result in unnecessary API calls and can severely restrict throughput
when handling transfers primarily composed of small files on some backends (e.g. Box).
A workaround to this issue is to use chunker only for files above the chunk threshold
via `--min-size` and then perform a separate call without chunker on the remaining
files.
#### Chunk names