diff --git a/MANUAL.html b/MANUAL.html
index d16296fa0..efa203d92 100644
--- a/MANUAL.html
+++ b/MANUAL.html
@@ -81,7 +81,7 @@
Nov 26, 2023 Jan 08, 2024rclone(1) User Manual
-
Mount the remote as file system on a mountpoint.
+rclone nfsmount allows Linux, FreeBSD, macOS and Windows to mount any of Rclone's cloud storage systems as a file system with FUSE.
+First set up your remote using rclone config
. Check it works with rclone ls
etc.
On Linux and macOS, you can run mount in either foreground or background (aka daemon) mode. Mount runs in foreground mode by default. Use the --daemon
flag to force background mode. On Windows you can run mount in foreground only, the flag is ignored.
In background mode rclone acts as a generic Unix mount program: the main program starts, spawns background rclone process to setup and maintain the mount, waits until success or timeout and exits with appropriate code (killing the child process if it fails).
+On Linux/macOS/FreeBSD start the mount like this, where /path/to/local/mount
is an empty existing directory:
rclone nfsmount remote:path/to/files /path/to/local/mount
+On Windows you can start a mount in different ways. See below for details. If foreground mount is used interactively from a console window, rclone will serve the mount and occupy the console so another window should be used to work with the mount until rclone is interrupted e.g. by pressing Ctrl-C.
+The following examples will mount to an automatically assigned drive, to specific drive letter X:
, to path C:\path\parent\mount
(where parent directory or drive must exist, and mount must not exist, and is not supported when mounting as a network drive), and the last example will mount as network share \\cloud\remote
and map it to an automatically assigned drive:
rclone nfsmount remote:path/to/files *
+rclone nfsmount remote:path/to/files X:
+rclone nfsmount remote:path/to/files C:\path\parent\mount
+rclone nfsmount remote:path/to/files \\cloud\remote
+When the program ends while in foreground mode, either via Ctrl+C or receiving a SIGINT or SIGTERM signal, the mount should be automatically stopped.
+When running in background mode the user will have to stop the mount manually:
+# Linux
+fusermount -u /path/to/local/mount
+# OS X
+umount /path/to/local/mount
+The umount operation can fail, for example when the mountpoint is busy. When that happens, it is the user's responsibility to stop the mount manually.
+The size of the mounted file system will be set according to information retrieved from the remote, the same as returned by the rclone about command. Remotes with unlimited storage may report the used size only, then an additional 1 PiB of free space is assumed. If the remote does not support the about feature at all, then 1 PiB is set as both the total and the free size.
+To run rclone nfsmount on Windows, you will need to download and install WinFsp.
+WinFsp is an open-source Windows File System Proxy which makes it easy to write user space file systems for Windows. It provides a FUSE emulation layer which rclone uses combination with cgofuse. Both of these packages are by Bill Zissimopoulos who was very helpful during the implementation of rclone nfsmount for Windows.
+Unlike other operating systems, Microsoft Windows provides a different filesystem type for network and fixed drives. It optimises access on the assumption fixed disk drives are fast and reliable, while network drives have relatively high latency and less reliability. Some settings can also be differentiated between the two types, for example that Windows Explorer should just display icons and not create preview thumbnails for image and video files on network drives.
+In most cases, rclone will mount the remote as a normal, fixed disk drive by default. However, you can also choose to mount it as a remote network drive, often described as a network share. If you mount an rclone remote using the default, fixed drive mode and experience unexpected program errors, freezes or other issues, consider mounting as a network drive instead.
+When mounting as a fixed disk drive you can either mount to an unused drive letter, or to a path representing a nonexistent subdirectory of an existing parent directory or drive. Using the special value *
will tell rclone to automatically assign the next available drive letter, starting with Z: and moving backward. Examples:
rclone nfsmount remote:path/to/files *
+rclone nfsmount remote:path/to/files X:
+rclone nfsmount remote:path/to/files C:\path\parent\mount
+rclone nfsmount remote:path/to/files X:
+Option --volname
can be used to set a custom volume name for the mounted file system. The default is to use the remote name and path.
To mount as network drive, you can add option --network-mode
to your nfsmount command. Mounting to a directory path is not supported in this mode, it is a limitation Windows imposes on junctions, so the remote must always be mounted to a drive letter.
rclone nfsmount remote:path/to/files X: --network-mode
+A volume name specified with --volname
will be used to create the network share path. A complete UNC path, such as \\cloud\remote
, optionally with path \\cloud\remote\madeup\path
, will be used as is. Any other string will be used as the share part, after a default prefix \\server\
. If no volume name is specified then \\server\share
will be used. You must make sure the volume name is unique when you are mounting more than one drive, or else the mount command will fail. The share name will treated as the volume label for the mapped drive, shown in Windows Explorer etc, while the complete \\server\share
will be reported as the remote UNC path by net use
etc, just like a normal network drive mapping.
If you specify a full network share UNC path with --volname
, this will implicitly set the --network-mode
option, so the following two examples have same result:
rclone nfsmount remote:path/to/files X: --network-mode
+rclone nfsmount remote:path/to/files X: --volname \\server\share
+You may also specify the network share UNC path as the mountpoint itself. Then rclone will automatically assign a drive letter, same as with *
and use that as mountpoint, and instead use the UNC path specified as the volume name, as if it were specified with the --volname
option. This will also implicitly set the --network-mode
option. This means the following two examples have same result:
rclone nfsmount remote:path/to/files \\cloud\remote
+rclone nfsmount remote:path/to/files * --volname \\cloud\remote
+There is yet another way to enable network mode, and to set the share path, and that is to pass the "native" libfuse/WinFsp option directly: --fuse-flag --VolumePrefix=\server\share
. Note that the path must be with just a single backslash prefix in this case.
Note: In previous versions of rclone this was the only supported method.
+ +See also Limitations section below.
+The FUSE emulation layer on Windows must convert between the POSIX-based permission model used in FUSE, and the permission model used in Windows, based on access-control lists (ACL).
+The mounted filesystem will normally get three entries in its access-control list (ACL), representing permissions for the POSIX permission scopes: Owner, group and others. By default, the owner and group will be taken from the current user, and the built-in group "Everyone" will be used to represent others. The user/group can be customized with FUSE options "UserName" and "GroupName", e.g. -o UserName=user123 -o GroupName="Authenticated Users"
. The permissions on each entry will be set according to options --dir-perms
and --file-perms
, which takes a value in traditional Unix numeric notation.
The default permissions corresponds to --file-perms 0666 --dir-perms 0777
, i.e. read and write permissions to everyone. This means you will not be able to start any programs from the mount. To be able to do that you must add execute permissions, e.g. --file-perms 0777 --dir-perms 0777
to add it to everyone. If the program needs to write files, chances are you will have to enable VFS File Caching as well (see also limitations). Note that the default write permission have some restrictions for accounts other than the owner, specifically it lacks the "write extended attributes", as explained next.
The mapping of permissions is not always trivial, and the result you see in Windows Explorer may not be exactly like you expected. For example, when setting a value that includes write access for the group or others scope, this will be mapped to individual permissions "write attributes", "write data" and "append data", but not "write extended attributes". Windows will then show this as basic permission "Special" instead of "Write", because "Write" also covers the "write extended attributes" permission. When setting digit 0 for group or others, to indicate no permissions, they will still get individual permissions "read attributes", "read extended attributes" and "read permissions". This is done for compatibility reasons, e.g. to allow users without additional permissions to be able to read basic metadata about files like in Unix.
+WinFsp 2021 (version 1.9) introduced a new FUSE option "FileSecurity", that allows the complete specification of file security descriptors using SDDL. With this you get detailed control of the resulting permissions, compared to use of the POSIX permissions described above, and no additional permissions will be added automatically for compatibility with Unix. Some example use cases will following.
+If you set POSIX permissions for only allowing access to the owner, using --file-perms 0600 --dir-perms 0700
, the user group and the built-in "Everyone" group will still be given some special permissions, as described above. Some programs may then (incorrectly) interpret this as the file being accessible by everyone, for example an SSH client may warn about "unprotected private key file". You can work around this by specifying -o FileSecurity="D:P(A;;FA;;;OW)"
, which sets file all access (FA) to the owner (OW), and nothing else.
When setting write permissions then, except for the owner, this does not include the "write extended attributes" permission, as mentioned above. This may prevent applications from writing to files, giving permission denied error instead. To set working write permissions for the built-in "Everyone" group, similar to what it gets by default but with the addition of the "write extended attributes", you can specify -o FileSecurity="D:P(A;;FRFW;;;WD)"
, which sets file read (FR) and file write (FW) to everyone (WD). If file execute (FX) is also needed, then change to -o FileSecurity="D:P(A;;FRFWFX;;;WD)"
, or set file all access (FA) to get full access permissions, including delete, with -o FileSecurity="D:P(A;;FA;;;WD)"
.
Drives created as Administrator are not visible to other accounts, not even an account that was elevated to Administrator with the User Account Control (UAC) feature. A result of this is that if you mount to a drive letter from a Command Prompt run as Administrator, and then try to access the same drive from Windows Explorer (which does not run as Administrator), you will not be able to see the mounted drive.
+If you don't need to access the drive from applications running with administrative privileges, the easiest way around this is to always create the mount from a non-elevated command prompt.
+To make mapped drives available to the user account that created them regardless if elevated or not, there is a special Windows setting called linked connections that can be enabled.
+It is also possible to make a drive mount available to everyone on the system, by running the process creating it as the built-in SYSTEM account. There are several ways to do this: One is to use the command-line utility PsExec, from Microsoft's Sysinternals suite, which has option -s
to start processes as the SYSTEM account. Another alternative is to run the mount command from a Windows Scheduled Task, or a Windows Service, configured to run as the SYSTEM account. A third alternative is to use the WinFsp.Launcher infrastructure). Read more in the install documentation. Note that when running rclone as another user, it will not use the configuration file from your profile unless you tell it to with the --config
option. Note also that it is now the SYSTEM account that will have the owner permissions, and other accounts will have permissions according to the group or others scopes. As mentioned above, these will then not get the "write extended attributes" permission, and this may prevent writing to files. You can work around this with the FileSecurity option, see example above.
Note that mapping to a directory path, instead of a drive letter, does not suffer from the same limitations.
+Mounting on macOS can be done either via built-in NFS server, macFUSE (also known as osxfuse) or FUSE-T. macFUSE is a traditional FUSE driver utilizing a macOS kernel extension (kext). FUSE-T is an alternative FUSE system which "mounts" via an NFSv4 local server.
+This method spins up an NFS server using serve nfs command and mounts it to the specified mountpoint. If you run this in background mode using |--daemon|, you will need to send SIGTERM signal to the rclone process using |kill| command to stop the mount.
+If installing macFUSE using dmg packages from the website, rclone will locate the macFUSE libraries without any further intervention. If however, macFUSE is installed using the macports package manager, the following addition steps are required.
+sudo mkdir /usr/local/lib
+cd /usr/local/lib
+sudo ln -s /opt/local/lib/libfuse.2.dylib
+There are some limitations, caveats, and notes about how it works. These are current as of FUSE-T version 1.0.14.
+As per the FUSE-T wiki:
+++File access and modification times cannot be set separately as it seems to be an issue with the NFS client which always modifies both. Can be reproduced with 'touch -m' and 'touch -a' commands
+
This means that viewing files with various tools, notably macOS Finder, will cause rlcone to update the modification time of the file. This may make rclone upload a full new copy of the file.
+Rclone includes flags for unicode normalization with macFUSE that should be updated for FUSE-T. See this forum post and FUSE-T issue #16. The following flag should be added to the rclone mount
command.
-o modules=iconv,from_code=UTF-8,to_code=UTF-8
+When mounting with --read-only
, attempts to write to files will fail silently as opposed to with a clear warning as in macFUSE.
Without the use of --vfs-cache-mode
this can only write files sequentially, it can only seek when reading. This means that many applications won't work with their files on an rclone mount without --vfs-cache-mode writes
or --vfs-cache-mode full
. See the VFS File Caching section for more info. When using NFS mount on macOS, if you don't specify |--vfs-cache-mode| the mount point will be read-only.
The bucket-based remotes (e.g. Swift, S3, Google Compute Storage, B2) do not support the concept of empty directories, so empty directories will have a tendency to disappear once they fall out of the directory cache.
+When rclone mount
is invoked on Unix with --daemon
flag, the main rclone program will wait for the background mount to become ready or until the timeout specified by the --daemon-wait
flag. On Linux it can check mount status using ProcFS so the flag in fact sets maximum time to wait, while the real wait can be less. On macOS / BSD the time to wait is constant and the check is performed only at the end. We advise you to set wait time on macOS reasonably.
Only supported on Linux, FreeBSD, OS X and Windows at the moment.
+File systems expect things to be 100% reliable, whereas cloud storage systems are a long way from 100% reliable. The rclone sync/copy commands cope with this with lots of retries. However rclone nfsmount can't use retries in the same way without making local copies of the uploads. Look at the VFS File Caching for solutions to make nfsmount more reliable.
+You can use the flag --attr-timeout
to set the time the kernel caches the attributes (size, modification time, etc.) for directory entries.
The default is 1s
which caches files just long enough to avoid too many callbacks to rclone from the kernel.
In theory 0s should be the correct value for filesystems which can change outside the control of the kernel. However this causes quite a few problems such as rclone using too much memory, rclone not serving files to samba and excessive time listing directories.
+The kernel can cache the info about a file for the time given by --attr-timeout
. You may see corruption if the remote file changes length during this window. It will show up as either a truncated file or a file with garbage on the end. With --attr-timeout 1s
this is very unlikely but not impossible. The higher you set --attr-timeout
the more likely it is. The default setting of "1s" is the lowest setting which mitigates the problems above.
If you set it higher (10s
or 1m
say) then the kernel will call back to rclone less often making it more efficient, however there is more chance of the corruption issue above.
If files don't change on the remote outside of the control of rclone then there is no chance of corruption.
+This is the same as setting the attr_timeout option in mount.fuse.
+Note that all the rclone filters can be used to select a subset of the files to be visible in the mount.
+When running rclone nfsmount as a systemd service, it is possible to use Type=notify. In this case the service will enter the started state after the mountpoint has been successfully set up. Units having the rclone nfsmount service specified as a requirement will see all files and folders immediately in this mode.
+Note that systemd runs mount units without any environment variables including PATH
or HOME
. This means that tilde (~
) expansion will not work and you should provide --config
and --cache-dir
explicitly as absolute paths via rclone arguments. Since mounting requires the fusermount
program, rclone will use the fallback PATH of /bin:/usr/bin
in this scenario. Please ensure that fusermount
is present on this PATH.
The core Unix program /bin/mount
normally takes the -t FSTYPE
argument then runs the /sbin/mount.FSTYPE
helper program passing it mount options as -o key=val,...
or --opt=...
. Automount (classic or systemd) behaves in a similar way.
rclone by default expects GNU-style flags --key val
. To run it as a mount helper you should symlink rclone binary to /sbin/mount.rclone
and optionally /usr/bin/rclonefs
, e.g. ln -s /usr/bin/rclone /sbin/mount.rclone
. rclone will detect it and translate command-line arguments appropriately.
Now you can run classic mounts like this:
+mount sftp1:subdir /mnt/data -t rclone -o vfs_cache_mode=writes,sftp_key_file=/path/to/pem
+or create systemd mount units:
+# /etc/systemd/system/mnt-data.mount
+[Unit]
+Description=Mount for /mnt/data
+[Mount]
+Type=rclone
+What=sftp1:subdir
+Where=/mnt/data
+Options=rw,_netdev,allow_other,args2env,vfs-cache-mode=writes,config=/etc/rclone.conf,cache-dir=/var/rclone
+optionally accompanied by systemd automount unit
+# /etc/systemd/system/mnt-data.automount
+[Unit]
+Description=AutoMount for /mnt/data
+[Automount]
+Where=/mnt/data
+TimeoutIdleSec=600
+[Install]
+WantedBy=multi-user.target
+or add in /etc/fstab
a line like
sftp1:subdir /mnt/data rclone rw,noauto,nofail,_netdev,x-systemd.automount,args2env,vfs_cache_mode=writes,config=/etc/rclone.conf,cache_dir=/var/cache/rclone 0 0
+or use classic Automountd. Remember to provide explicit config=...,cache-dir=...
as a workaround for mount units being run without HOME
.
Rclone in the mount helper mode will split -o
argument(s) by comma, replace _
by -
and prepend --
to get the command-line flags. Options containing commas or spaces can be wrapped in single or double quotes. Any inner quotes inside outer quotes of the same type should be doubled.
Mount option syntax includes a few extra options treated specially:
+env.NAME=VALUE
will set an environment variable for the mount process. This helps with Automountd and Systemd.mount which don't allow setting custom environment for mount helpers. Typically you will use env.HTTPS_PROXY=proxy.host:3128
or env.HOME=/root
command=cmount
can be used to run cmount
or any other rclone command rather than the default mount
.args2env
will pass mount options to the mount helper running in background via environment variables instead of command line arguments. This allows to hide secrets from such commands as ps
or pgrep
.vv...
will be transformed into appropriate --verbose=N
x-systemd.automount
, _netdev
, nosuid
and alike are intended only for Automountd and ignored by rclone. ## VFS - Virtual File SystemThis command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.
+Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.
+The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.
+Using the --dir-cache-time
flag, you can control how long a directory should be considered up to date and not refreshed from the backend. Changes made through the VFS will appear immediately or invalidate the cache.
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
+--poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
+However, changes made directly on the cloud storage by the web interface or a different copy of rclone will only be picked up once the directory cache expires if the backend configured does not support polling for changes. If the backend supports polling, changes will be picked up within the polling interval.
+You can send a SIGHUP
signal to rclone for it to flush all directory caches, regardless of how old they are. Assuming only one rclone instance is running, you can reset the cache like this:
kill -SIGHUP $(pidof rclone)
+If you configure rclone with a remote control then you can use rclone rc to flush the whole directory cache:
+rclone rc vfs/forget
+Or individual files or directories:
+rclone rc vfs/forget file=path/to/file dir=path/to/dir
+The --buffer-size
flag determines the amount of memory, that will be used to buffer data in advance.
Each open file will try to keep the specified amount of data in memory at all times. The buffered data is bound to one open file and won't be shared.
+This flag is a upper limit for the used memory per open file. The buffer will only use memory for data that is downloaded but not not yet read. If the buffer is empty, only a small amount of memory will be used.
+The maximum memory used by rclone for buffering can be up to --buffer-size * open files
.
These flags control the VFS file caching options. File caching is necessary to make the VFS layer appear compatible with a normal file system. It can be disabled at the cost of some compatibility.
+For example you'll need to enable VFS caching if you want to read and write simultaneously to a file. See below for more details.
+Note that the VFS cache is separate from the cache backend and you may find that you need one or the other or both.
+--cache-dir string Directory rclone will use for caching.
+--vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
+--vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s)
+--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
+--vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
+--vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
+--vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
+If run with -vv
rclone will print the location of the file cache. The files are stored in the user cache file area which is OS dependent but can be controlled with --cache-dir
or setting the appropriate environment variable.
The cache has 4 different modes selected by --vfs-cache-mode
. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.
Note that files are written back to the remote only when they are closed and if they haven't been accessed for --vfs-write-back
seconds. If rclone is quit or dies with files that haven't been uploaded, these will be uploaded next time rclone is run with the same flags.
If using --vfs-cache-max-size
or --vfs-cache-min-free-size
note that the cache may exceed these quotas for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval
. Secondly because open files cannot be evicted from the cache. When --vfs-cache-max-size
or --vfs-cache-min-free-size
is exceeded, rclone will attempt to evict the least accessed files from the cache first. rclone will start with files that haven't been accessed for the longest. This cache flushing strategy is efficient and more relevant files are likely to remain cached.
The --vfs-cache-max-age
will evict files from the cache after the set time since last access has passed. The default value of 1 hour will start evicting files from cache that haven't been accessed for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0 and will wait for 1 more hour before evicting. Specify the time with standard notation, s, m, h, d, w .
You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off
. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir
. You don't need to worry about this if the remotes in use don't overlap.
In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.
+This will mean some operations are not possible
+This is very similar to "off" except that files opened for read AND write will be buffered to disk. This means that files opened for write will be a lot more compatible, but uses the minimal disk space.
+These operations are not possible
+In this mode files opened for read only are still read directly from the remote, write only and read/write files are buffered to disk first.
+This mode should support all normal file system operations.
+If an upload fails it will be retried at exponentially increasing intervals up to 1 minute.
+In this mode all reads and writes are buffered to and from disk. When data is read from the remote this is buffered to disk as well.
+In this mode the files in the cache will be sparse files and rclone will keep track of which bits of the files it has downloaded.
+So if an application only reads the starts of each file, then rclone will only buffer the start of the file. These files will appear to be their full size in the cache, but they will be sparse files with only the data that has been downloaded present in them.
+This mode should support all normal file system operations and is otherwise identical to --vfs-cache-mode
writes.
When reading a file rclone will read --buffer-size
plus --vfs-read-ahead
bytes ahead. The --buffer-size
is buffered in memory whereas the --vfs-read-ahead
is buffered on disk.
When using this mode it is recommended that --buffer-size
is not set too large and --vfs-read-ahead
is set large if required.
IMPORTANT not all file systems support sparse files. In particular FAT/exFAT do not. Rclone will perform very badly if the cache directory is on a filesystem which doesn't support sparse files and it will log an ERROR message if one is detected.
+Various parts of the VFS use fingerprinting to see if a local file copy has changed relative to a remote file. Fingerprints are made from:
+where available on an object.
+On some backends some of these attributes are slow to read (they take an extra API call per object, or extra work per object).
+For example hash
is slow with the local
and sftp
backends as they have to read the entire file and hash it, and modtime
is slow with the s3
, swift
, ftp
and qinqstor
backends because they need to do an extra API call to fetch it.
If you use the --vfs-fast-fingerprint
flag then rclone will not include the slow operations in the fingerprint. This makes the fingerprinting less accurate but much faster and will improve the opening time of cached files.
If you are running a vfs cache over local
, s3
or swift
backends then using this flag is recommended.
Note that if you change the value of this flag, the fingerprints of the files in the cache may be invalidated and the files will need to be downloaded again.
+When rclone reads files from a remote it reads them in chunks. This means that rather than requesting the whole file rclone reads the chunk specified. This can reduce the used download quota for some remotes by requesting only chunks from the remote that are actually read, at the cost of an increased number of requests.
+These flags control the chunking:
+--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M)
+--vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off)
+Rclone will start reading a chunk of size --vfs-read-chunk-size
, and then double the size for each read. When --vfs-read-chunk-size-limit
is specified, and greater than --vfs-read-chunk-size
, the chunk size for each open file will get doubled only until the specified value is reached. If the value is "off", which is the default, the limit is disabled and the chunk size will grow indefinitely.
With --vfs-read-chunk-size 100M
and --vfs-read-chunk-size-limit 0
the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. When --vfs-read-chunk-size-limit 500M
is specified, the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.
Setting --vfs-read-chunk-size
to 0
or "off" disables chunked reading.
These flags may be used to enable/disable features of the VFS for performance or other reasons. See also the chunked reading feature.
+In particular S3 and Swift benefit hugely from the --no-modtime
flag (or use --use-server-modtime
for a slightly different effect) as each read of the modification time takes a transaction.
--no-checksum Don't compare checksums on up/download.
+--no-modtime Don't read/write the modification time (can speed things up).
+--no-seek Don't allow seeking in files.
+--read-only Only allow read-only access.
+Sometimes rclone is delivered reads or writes out of order. Rather than seeking rclone will wait a short time for the in sequence read or write to come in. These flags only come into effect when not using an on disk cache file.
+--vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
+--vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
+When using VFS write caching (--vfs-cache-mode
with value writes or full), the global flag --transfers
can be set to adjust the number of parallel uploads of modified files from the cache (the related global flag --checkers
has no effect on the VFS).
--transfers int Number of file transfers to run in parallel (default 4)
+Linux file systems are case-sensitive: two files can differ only by case, and the exact case must be used when opening a file.
+File systems in modern Windows are case-insensitive but case-preserving: although existing files can be opened using any case, the exact case used to create the file is preserved and available for programs to query. It is not allowed for two files in the same directory to differ only by case.
+Usually file systems on macOS are case-insensitive. It is possible to make macOS file systems case-sensitive but that is not the default.
+The --vfs-case-insensitive
VFS flag controls how rclone handles these two cases. If its value is "false", rclone passes file names to the remote as-is. If the flag is "true" (or appears without a value on the command line), rclone may perform a "fixup" as explained below.
The user may specify a file name to open/delete/rename/etc with a case different than what is stored on the remote. If an argument refers to an existing file with exactly the same name, then the case of the existing file on the disk will be used. However, if a file name with exactly the same name is not found but a name differing only by case exists, rclone will transparently fixup the name. This fixup happens only when an existing file is requested. Case sensitivity of file names created anew by rclone is controlled by the underlying remote.
+Note that case sensitivity of the operating system running rclone (the target) may differ from case sensitivity of a file system presented by rclone (the source). The flag controls whether "fixup" is performed to satisfy the target.
+If the flag is not provided on the command line, then its default value depends on the operating system where rclone runs: "true" on Windows and macOS, "false" otherwise. If the flag is provided without a value, then it is "true".
+This flag allows you to manually set the statistics about the filing system. It can be useful when those statistics cannot be read correctly automatically.
+--vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1)
+Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df
on the filesystem, then pass the flag --vfs-used-is-size
to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size
and compute the total used space itself.
WARNING. Contrary to rclone size
, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.
rclone nfsmount remote:path /path/to/mountpoint [flags]
+ --allow-non-empty Allow mounting over a non-empty directory (not supported on Windows)
+ --allow-other Allow access to other users (not supported on Windows)
+ --allow-root Allow access to root user (not supported on Windows)
+ --async-read Use asynchronous reads (not supported on Windows) (default true)
+ --attr-timeout Duration Time for which file/directory attributes are cached (default 1s)
+ --daemon Run mount in background and exit parent process (as background output is suppressed, use --log-file with --log-format=pid,... to monitor) (not supported on Windows)
+ --daemon-timeout Duration Time limit for rclone to respond to kernel (not supported on Windows) (default 0s)
+ --daemon-wait Duration Time to wait for ready mount from daemon (maximum time on Linux, constant sleep time on OSX/BSD) (not supported on Windows) (default 1m0s)
+ --debug-fuse Debug the FUSE internals - needs -v
+ --default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows)
+ --devname string Set the device name - default is remote:path
+ --dir-cache-time Duration Time to cache directory entries for (default 5m0s)
+ --dir-perms FileMode Directory permissions (default 0777)
+ --file-perms FileMode File permissions (default 0666)
+ --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required)
+ --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
+ -h, --help help for nfsmount
+ --max-read-ahead SizeSuffix The number of bytes that can be prefetched for sequential reads (not supported on Windows) (default 128Ki)
+ --mount-case-insensitive Tristate Tell the OS the mount is case insensitive (true) or sensitive (false) regardless of the backend (auto) (default unset)
+ --network-mode Mount as remote network drive, instead of fixed disk drive (supported on Windows only)
+ --no-checksum Don't compare checksums on up/download
+ --no-modtime Don't read/write the modification time (can speed things up)
+ --no-seek Don't allow seeking in files
+ --noappledouble Ignore Apple Double (._) and .DS_Store files (supported on OSX only) (default true)
+ --noapplexattr Ignore all "com.apple.*" extended attributes (supported on OSX only)
+ -o, --option stringArray Option for libfuse/WinFsp (repeat if required)
+ --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
+ --read-only Only allow read-only access
+ --sudo Use sudo to run the mount command as root.
+ --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
+ --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
+ --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s)
+ --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
+ --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
+ --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
+ --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s)
+ --vfs-case-insensitive If a file name not found, find a case insensitive match
+ --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off)
+ --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection
+ --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full
+ --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
+ --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
+ --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively in the background on start
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size
+ --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
+ --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
+ --volname string Set the volume name (supported on Windows and OSX only)
+ --write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows)
+Flags for filtering directory listings.
+ --delete-excluded Delete files on dest excluded from sync
+ --exclude stringArray Exclude files matching pattern
+ --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin)
+ --exclude-if-present stringArray Exclude directories if filename is present
+ --files-from stringArray Read list of source-file names from file (use - to read from stdin)
+ --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin)
+ -f, --filter stringArray Add a file filtering rule
+ --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin)
+ --ignore-case Ignore case in filters (case insensitive)
+ --include stringArray Include files matching pattern
+ --include-from stringArray Read file include patterns from file (use - to read from stdin)
+ --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --max-depth int If set limits the recursion depth to this (default -1)
+ --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
+ --metadata-exclude stringArray Exclude metadatas matching pattern
+ --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin)
+ --metadata-filter stringArray Add a metadata filtering rule
+ --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin)
+ --metadata-include stringArray Include metadatas matching pattern
+ --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin)
+ --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
+See the global flags page for global options not listed here.
+Obscure password for use in the rclone config file.
-In the rclone config file, human-readable passwords are obscured. Obscuring them is done by encrypting them and writing them out in base64. This is not a secure way of encrypting these passwords as rclone can decrypt them - it is to prevent "eyedropping" - namely someone seeing a password in the rclone config file by accident.
Many equally important things (like access tokens) are not obscured in the config file. However it is very hard to shoulder surf a 64 character hex token.
This command can also accept a password through STDIN instead of an argument by passing a hyphen as an argument. This will use the first line of STDIN as the password not including the trailing newline.
@@ -3303,16 +3638,16 @@ if src is directoryIf there is no data on STDIN to read, rclone obscure will default to obfuscating the hyphen itself.
If you want to encrypt the config file then please use config file encryption - see rclone config for more info.
rclone obscure password [flags]
- -h, --help help for obscure
See the global flags page for global options not listed here.
-Run a command against a running rclone.
-This runs a command against a running rclone. Use the --url
flag to specify an non default URL to connect on. This can be either a ":port" which is taken to mean "http://localhost:port" or a "host:port" which is taken to mean "http://host:port"
A username and password can be passed in with --user
and --pass
.
Note that --rc-addr
, --rc-user
, --rc-pass
will be read also for --url
, --user
, --pass
.
rclone rc --loopback operations/about fs=/
Use rclone rc
to see a list of all possible commands.
rclone rc commands parameter [flags]
- -a, --arg stringArray Argument placed in the "arg" array
-h, --help help for rc
--json string Input JSON - use instead of key=value args
@@ -3342,13 +3677,13 @@ if src is directory
--url string URL to connect to rclone remote control (default "http://localhost:5572/")
--user string Username to use to rclone remote control
See the global flags page for global options not listed here.
-Copies standard input to file on remote.
-rclone rcat reads from standard input (stdin) and copies it to a single remote file.
echo "hello world" | rclone rcat remote:path/to/file
ffmpeg - | rclone rcat remote:path/to/file
@@ -3358,7 +3693,7 @@ ffmpeg - | rclone rcat remote:path/to/file
--size
should be the exact size of the input stream in bytes. If the size of the stream is different in length to the --size
passed in then the transfer will likely fail.
Note that the upload cannot be retried because the data is not stored. If the backend supports multipart uploading then individual chunks can be retried. If you need to transfer a lot of data, you may be better off caching it locally and then rclone move
it to the destination which can use retries.
rclone rcat remote:path [flags]
- -h, --help help for rcat
--size int File size hint to preallocate (default -1)
See the global flags page for global options not listed here.
-Run rclone listening to remote control commands only.
-This runs rclone so that it only listens to remote control commands.
This is useful if you are controlling rclone via the rc API.
If you pass in a path to a directory, rclone will serve that directory for GET requests on the URL passed in. It will also open the URL in the browser when rclone is run.
@@ -3514,7 +3849,7 @@ htpasswd -B htpasswd anotherUserUse --rc-realm
to set the authentication realm.
Use --rc-salt
to change the password hashing salt from the default.
rclone rcd <path to files to serve>* [flags]
- -h, --help help for rcd
Flags to control the Remote Control API.
@@ -3547,20 +3882,20 @@ htpasswd -B htpasswd anotherUser --rc-web-gui-no-open-browser Don't open the browser automatically --rc-web-gui-update Check and update to latest version of web guiSee the global flags page for global options not listed here.
-Remove empty directories under the path.
-This recursively removes any empty directories (including directories that only contain empty directories), that it finds under the path. The root path itself will also be removed if it is empty, unless you supply the --leave-root
flag.
Use command rmdir to delete just the empty directory given by path, not recurse.
This is useful for tidying up remotes that rclone has left a lot of empty directories in. For example the delete command will delete files but leave the directory structure (unless used with option --rmdirs
).
This will delete --checkers
directories concurrently so if you have thousands of empty directories consider increasing this number.
To delete a path and any objects in it, use the purge command.
rclone rmdirs remote:path [flags]
- -h, --help help for rmdirs
--leave-root Do not remove root directory if empty
See the global flags page for global options not listed here.
-Update the rclone binary.
-This command downloads the latest release of rclone and replaces the currently running binary. The download is verified with a hashsum and cryptographically signed signature; see the release signing docs for details.
If used without flags (or with implied --stable
flag), this command will install the latest stable release. However, some issues may be fixed (or features added) only in the latest beta release. In such cases you should run the command with the --beta
flag, i.e. rclone selfupdate --beta
. You can check in advance what version would be installed by adding the --check
flag, then repeat the command without it when you are satisfied.
Sometimes the rclone team may recommend you a concrete beta or stable rclone release to troubleshoot your issue or add a bleeding edge feature. The --version VER
flag, if given, will update to the concrete version instead of the latest one. If you omit micro version from VER
(for example 1.53
), the latest matching micro version will be used.
Note: Windows forbids deletion of a currently running executable so this command will rename the old executable to 'rclone.old.exe' upon success.
Please note that this command was not available before rclone version 1.55. If it fails for you with the message unknown command "selfupdate"
then you will need to update manually following the install instructions located at https://rclone.org/install/
rclone selfupdate [flags]
- --beta Install beta release
--check Check for latest release, do not download
-h, --help help for selfupdate
@@ -3594,21 +3929,21 @@ htpasswd -B htpasswd anotherUser
--stable Install stable release (this is the default)
--version string Install the given rclone version (default: latest)
See the global flags page for global options not listed here.
-Serve a remote over a protocol.
-Serve a remote over a given protocol. Requires the use of a subcommand to specify the protocol, e.g.
rclone serve http remote:
Each subcommand has its own options which you can see in their help.
rclone serve <protocol> [opts] <remote> [flags]
- -h, --help help for serve
See the global flags page for global options not listed here.
-Serve remote:path over DLNA
-Run a DLNA media server for media stored in an rclone remote. Many devices, such as the Xbox and PlayStation, can automatically discover this server in the LAN and play audio/video from it. VLC is also supported. Service discovery uses UDP multicast packets (SSDP) and will thus only work on LANs.
Rclone will list all files present in the remote, without filtering based on media formats or file extensions. Additionally, there is no media transcoding support. This means that some players might show files that they are not able to play back correctly.
This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.
Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.
The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.
-Using the --dir-cache-time
flag, you can control how long a directory should be considered up to date and not refreshed from the backend. Changes made through the VFS will appear immediately or invalidate the cache.
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
---poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
-However, changes made directly on the cloud storage by the web interface or a different copy of rclone will only be picked up once the directory cache expires if the backend configured does not support polling for changes. If the backend supports polling, changes will be picked up within the polling interval.
-You can send a SIGHUP
signal to rclone for it to flush all directory caches, regardless of how old they are. Assuming only one rclone instance is running, you can reset the cache like this:
kill -SIGHUP $(pidof rclone)
-If you configure rclone with a remote control then you can use rclone rc to flush the whole directory cache:
-rclone rc vfs/forget
-Or individual files or directories:
-rclone rc vfs/forget file=path/to/file dir=path/to/dir
-The --buffer-size
flag determines the amount of memory, that will be used to buffer data in advance.
Each open file will try to keep the specified amount of data in memory at all times. The buffered data is bound to one open file and won't be shared.
-This flag is a upper limit for the used memory per open file. The buffer will only use memory for data that is downloaded but not not yet read. If the buffer is empty, only a small amount of memory will be used.
-The maximum memory used by rclone for buffering can be up to --buffer-size * open files
.
These flags control the VFS file caching options. File caching is necessary to make the VFS layer appear compatible with a normal file system. It can be disabled at the cost of some compatibility.
-For example you'll need to enable VFS caching if you want to read and write simultaneously to a file. See below for more details.
-Note that the VFS cache is separate from the cache backend and you may find that you need one or the other or both.
---cache-dir string Directory rclone will use for caching.
---vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
---vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s)
---vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
---vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
---vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
---vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
-If run with -vv
rclone will print the location of the file cache. The files are stored in the user cache file area which is OS dependent but can be controlled with --cache-dir
or setting the appropriate environment variable.
The cache has 4 different modes selected by --vfs-cache-mode
. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.
Note that files are written back to the remote only when they are closed and if they haven't been accessed for --vfs-write-back
seconds. If rclone is quit or dies with files that haven't been uploaded, these will be uploaded next time rclone is run with the same flags.
If using --vfs-cache-max-size
or --vfs-cache-min-free-size
note that the cache may exceed these quotas for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval
. Secondly because open files cannot be evicted from the cache. When --vfs-cache-max-size
or --vfs-cache-min-free-size
is exceeded, rclone will attempt to evict the least accessed files from the cache first. rclone will start with files that haven't been accessed for the longest. This cache flushing strategy is efficient and more relevant files are likely to remain cached.
The --vfs-cache-max-age
will evict files from the cache after the set time since last access has passed. The default value of 1 hour will start evicting files from cache that haven't been accessed for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0 and will wait for 1 more hour before evicting. Specify the time with standard notation, s, m, h, d, w .
You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off
. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir
. You don't need to worry about this if the remotes in use don't overlap.
In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.
-This will mean some operations are not possible
-This is very similar to "off" except that files opened for read AND write will be buffered to disk. This means that files opened for write will be a lot more compatible, but uses the minimal disk space.
-These operations are not possible
-In this mode files opened for read only are still read directly from the remote, write only and read/write files are buffered to disk first.
-This mode should support all normal file system operations.
-If an upload fails it will be retried at exponentially increasing intervals up to 1 minute.
-In this mode all reads and writes are buffered to and from disk. When data is read from the remote this is buffered to disk as well.
-In this mode the files in the cache will be sparse files and rclone will keep track of which bits of the files it has downloaded.
-So if an application only reads the starts of each file, then rclone will only buffer the start of the file. These files will appear to be their full size in the cache, but they will be sparse files with only the data that has been downloaded present in them.
-This mode should support all normal file system operations and is otherwise identical to --vfs-cache-mode
writes.
When reading a file rclone will read --buffer-size
plus --vfs-read-ahead
bytes ahead. The --buffer-size
is buffered in memory whereas the --vfs-read-ahead
is buffered on disk.
When using this mode it is recommended that --buffer-size
is not set too large and --vfs-read-ahead
is set large if required.
IMPORTANT not all file systems support sparse files. In particular FAT/exFAT do not. Rclone will perform very badly if the cache directory is on a filesystem which doesn't support sparse files and it will log an ERROR message if one is detected.
-Various parts of the VFS use fingerprinting to see if a local file copy has changed relative to a remote file. Fingerprints are made from:
-where available on an object.
-On some backends some of these attributes are slow to read (they take an extra API call per object, or extra work per object).
-For example hash
is slow with the local
and sftp
backends as they have to read the entire file and hash it, and modtime
is slow with the s3
, swift
, ftp
and qinqstor
backends because they need to do an extra API call to fetch it.
If you use the --vfs-fast-fingerprint
flag then rclone will not include the slow operations in the fingerprint. This makes the fingerprinting less accurate but much faster and will improve the opening time of cached files.
If you are running a vfs cache over local
, s3
or swift
backends then using this flag is recommended.
Note that if you change the value of this flag, the fingerprints of the files in the cache may be invalidated and the files will need to be downloaded again.
-When rclone reads files from a remote it reads them in chunks. This means that rather than requesting the whole file rclone reads the chunk specified. This can reduce the used download quota for some remotes by requesting only chunks from the remote that are actually read, at the cost of an increased number of requests.
-These flags control the chunking:
---vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M)
---vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off)
-Rclone will start reading a chunk of size --vfs-read-chunk-size
, and then double the size for each read. When --vfs-read-chunk-size-limit
is specified, and greater than --vfs-read-chunk-size
, the chunk size for each open file will get doubled only until the specified value is reached. If the value is "off", which is the default, the limit is disabled and the chunk size will grow indefinitely.
With --vfs-read-chunk-size 100M
and --vfs-read-chunk-size-limit 0
the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. When --vfs-read-chunk-size-limit 500M
is specified, the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.
Setting --vfs-read-chunk-size
to 0
or "off" disables chunked reading.
These flags may be used to enable/disable features of the VFS for performance or other reasons. See also the chunked reading feature.
-In particular S3 and Swift benefit hugely from the --no-modtime
flag (or use --use-server-modtime
for a slightly different effect) as each read of the modification time takes a transaction.
--no-checksum Don't compare checksums on up/download.
---no-modtime Don't read/write the modification time (can speed things up).
---no-seek Don't allow seeking in files.
---read-only Only allow read-only access.
-Sometimes rclone is delivered reads or writes out of order. Rather than seeking rclone will wait a short time for the in sequence read or write to come in. These flags only come into effect when not using an on disk cache file.
---vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
---vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
-When using VFS write caching (--vfs-cache-mode
with value writes or full), the global flag --transfers
can be set to adjust the number of parallel uploads of modified files from the cache (the related global flag --checkers
has no effect on the VFS).
--transfers int Number of file transfers to run in parallel (default 4)
-Linux file systems are case-sensitive: two files can differ only by case, and the exact case must be used when opening a file.
-File systems in modern Windows are case-insensitive but case-preserving: although existing files can be opened using any case, the exact case used to create the file is preserved and available for programs to query. It is not allowed for two files in the same directory to differ only by case.
-Usually file systems on macOS are case-insensitive. It is possible to make macOS file systems case-sensitive but that is not the default.
-The --vfs-case-insensitive
VFS flag controls how rclone handles these two cases. If its value is "false", rclone passes file names to the remote as-is. If the flag is "true" (or appears without a value on the command line), rclone may perform a "fixup" as explained below.
The user may specify a file name to open/delete/rename/etc with a case different than what is stored on the remote. If an argument refers to an existing file with exactly the same name, then the case of the existing file on the disk will be used. However, if a file name with exactly the same name is not found but a name differing only by case exists, rclone will transparently fixup the name. This fixup happens only when an existing file is requested. Case sensitivity of file names created anew by rclone is controlled by the underlying remote.
-Note that case sensitivity of the operating system running rclone (the target) may differ from case sensitivity of a file system presented by rclone (the source). The flag controls whether "fixup" is performed to satisfy the target.
-If the flag is not provided on the command line, then its default value depends on the operating system where rclone runs: "true" on Windows and macOS, "false" otherwise. If the flag is provided without a value, then it is "true".
-This flag allows you to manually set the statistics about the filing system. It can be useful when those statistics cannot be read correctly automatically.
---vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1)
-Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df
on the filesystem, then pass the flag --vfs-used-is-size
to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size
and compute the total used space itself.
WARNING. Contrary to rclone size
, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.
rclone serve dlna remote:path [flags]
- --addr string The ip:port or :port to bind the DLNA http server to (default ":7879")
- --announce-interval Duration The interval between SSDP announcements (default 12m0s)
- --dir-cache-time Duration Time to cache directory entries for (default 5m0s)
- --dir-perms FileMode Directory permissions (default 0777)
- --file-perms FileMode File permissions (default 0666)
- --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
- -h, --help help for dlna
- --interface stringArray The interface to use for SSDP (repeat as necessary)
- --log-trace Enable trace logging of SOAP traffic
- --name string Name of DLNA server
- --no-checksum Don't compare checksums on up/download
- --no-modtime Don't read/write the modification time (can speed things up)
- --no-seek Don't allow seeking in files
- --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
- --read-only Only allow read-only access
- --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
- --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
- --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s)
- --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
- --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
- --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
- --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s)
- --vfs-case-insensitive If a file name not found, find a case insensitive match
- --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off)
- --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection
- --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full
- --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
- --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
- --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
- --vfs-refresh Refreshes the directory cache recursively on start
- --vfs-used-is-size rclone size Use the rclone size algorithm for Used size
- --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
- --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
-Flags for filtering directory listings.
- --delete-excluded Delete files on dest excluded from sync
- --exclude stringArray Exclude files matching pattern
- --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin)
- --exclude-if-present stringArray Exclude directories if filename is present
- --files-from stringArray Read list of source-file names from file (use - to read from stdin)
- --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin)
- -f, --filter stringArray Add a file filtering rule
- --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin)
- --ignore-case Ignore case in filters (case insensitive)
- --include stringArray Include files matching pattern
- --include-from stringArray Read file include patterns from file (use - to read from stdin)
- --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
- --max-depth int If set limits the recursion depth to this (default -1)
- --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
- --metadata-exclude stringArray Exclude metadatas matching pattern
- --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin)
- --metadata-filter stringArray Add a metadata filtering rule
- --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin)
- --metadata-include stringArray Include metadatas matching pattern
- --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin)
- --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
- --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
-See the global flags page for global options not listed here.
-Serve any remote on docker's volume plugin API.
-This command implements the Docker volume plugin API allowing docker to use rclone as a data storage mechanism for various cloud providers. rclone provides docker volume plugin based on it.
-To create a docker plugin, one must create a Unix or TCP socket that Docker will look for when you use the plugin and then it listens for commands from docker daemon and runs the corresponding code when necessary. Docker plugins can run as a managed plugin under control of the docker daemon or as an independent native service. For testing, you can just run it directly from the command line, for example:
-sudo rclone serve docker --base-dir /tmp/rclone-volumes --socket-addr localhost:8787 -vv
-Running rclone serve docker
will create the said socket, listening for commands from Docker to create the necessary Volumes. Normally you need not give the --socket-addr
flag. The API will listen on the unix domain socket at /run/docker/plugins/rclone.sock
. In the example above rclone will create a TCP socket and a small file /etc/docker/plugins/rclone.spec
containing the socket address. We use sudo
because both paths are writeable only by the root user.
If you later decide to change listening socket, the docker daemon must be restarted to reconnect to /run/docker/plugins/rclone.sock
or parse new /etc/docker/plugins/rclone.spec
. Until you restart, any volume related docker commands will timeout trying to access the old socket. Running directly is supported on Linux only, not on Windows or MacOS. This is not a problem with managed plugin mode described in details in the full documentation.
The command will create volume mounts under the path given by --base-dir
(by default /var/lib/docker-volumes/rclone
available only to root) and maintain the JSON formatted file docker-plugin.state
in the rclone cache directory with book-keeping records of created and mounted volumes.
All mount and VFS options are submitted by the docker daemon via API, but you can also provide defaults on the command line as well as set path to the config file and cache directory or adjust logging verbosity. ## VFS - Virtual File System
-This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.
-Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.
-The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.
Using the --dir-cache-time
flag, you can control how long a directory should be considered up to date and not refreshed from the backend. Changes made through the VFS will appear immediately or invalidate the cache.
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
@@ -3936,41 +4081,23 @@ htpasswd -B htpasswd anotherUser
Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df
on the filesystem, then pass the flag --vfs-used-is-size
to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size
and compute the total used space itself.
WARNING. Contrary to rclone size
, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.
rclone serve docker [flags]
+rclone serve dlna remote:path [flags]
--allow-non-empty Allow mounting over a non-empty directory (not supported on Windows)
- --allow-other Allow access to other users (not supported on Windows)
- --allow-root Allow access to root user (not supported on Windows)
- --async-read Use asynchronous reads (not supported on Windows) (default true)
- --attr-timeout Duration Time for which file/directory attributes are cached (default 1s)
- --base-dir string Base directory for volumes (default "/var/lib/docker-volumes/rclone")
- --daemon Run mount in background and exit parent process (as background output is suppressed, use --log-file with --log-format=pid,... to monitor) (not supported on Windows)
- --daemon-timeout Duration Time limit for rclone to respond to kernel (not supported on Windows) (default 0s)
- --daemon-wait Duration Time to wait for ready mount from daemon (maximum time on Linux, constant sleep time on OSX/BSD) (not supported on Windows) (default 1m0s)
- --debug-fuse Debug the FUSE internals - needs -v
- --default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows)
- --devname string Set the device name - default is remote:path
+ --addr string The ip:port or :port to bind the DLNA http server to (default ":7879")
+ --announce-interval Duration The interval between SSDP announcements (default 12m0s)
--dir-cache-time Duration Time to cache directory entries for (default 5m0s)
--dir-perms FileMode Directory permissions (default 0777)
--file-perms FileMode File permissions (default 0666)
- --forget-state Skip restoring previous state
- --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required)
--gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
- -h, --help help for docker
- --max-read-ahead SizeSuffix The number of bytes that can be prefetched for sequential reads (not supported on Windows) (default 128Ki)
- --mount-case-insensitive Tristate Tell the OS the mount is case insensitive (true) or sensitive (false) regardless of the backend (auto) (default unset)
- --network-mode Mount as remote network drive, instead of fixed disk drive (supported on Windows only)
+ -h, --help help for dlna
+ --interface stringArray The interface to use for SSDP (repeat as necessary)
+ --log-trace Enable trace logging of SOAP traffic
+ --name string Name of DLNA server
--no-checksum Don't compare checksums on up/download
--no-modtime Don't read/write the modification time (can speed things up)
--no-seek Don't allow seeking in files
- --no-spec Do not write spec file
- --noappledouble Ignore Apple Double (._) and .DS_Store files (supported on OSX only) (default true)
- --noapplexattr Ignore all "com.apple.*" extended attributes (supported on OSX only)
- -o, --option stringArray Option for libfuse/WinFsp (repeat if required)
--poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
--read-only Only allow read-only access
- --socket-addr string Address <host:port> or absolute path (default: /run/docker/plugins/rclone.sock)
- --socket-gid int GID for unix socket (default: current process GID) (default 1000)
--uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
--umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
--vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s)
@@ -3985,12 +4112,10 @@ htpasswd -B htpasswd anotherUser
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
- --vfs-refresh Refreshes the directory cache recursively on start
+ --vfs-refresh Refreshes the directory cache recursively in the background on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
- --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
- --volname string Set the volume name (supported on Windows and OSX only)
- --write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows)
+ --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
Flags for filtering directory listings.
--delete-excluded Delete files on dest excluded from sync
@@ -4020,16 +4145,16 @@ htpasswd -B htpasswd anotherUser
Serve remote:path over FTP.
+Serve any remote on docker's volume plugin API.
Run a basic FTP server to serve a remote over FTP protocol. This can be viewed with a FTP client or you can make a remote of type FTP to read and write it.
-Use --addr to specify which IP address and port the server should listen on, e.g. --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. By default it only listens on localhost. You can use port :0 to let the OS choose an available port.
-If you set --addr to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.
-By default this will serve files without needing a login.
-You can set a single username and password with the --user and --pass flags. ## VFS - Virtual File System
+This command implements the Docker volume plugin API allowing docker to use rclone as a data storage mechanism for various cloud providers. rclone provides docker volume plugin based on it.
+To create a docker plugin, one must create a Unix or TCP socket that Docker will look for when you use the plugin and then it listens for commands from docker daemon and runs the corresponding code when necessary. Docker plugins can run as a managed plugin under control of the docker daemon or as an independent native service. For testing, you can just run it directly from the command line, for example:
+sudo rclone serve docker --base-dir /tmp/rclone-volumes --socket-addr localhost:8787 -vv
+Running rclone serve docker
will create the said socket, listening for commands from Docker to create the necessary Volumes. Normally you need not give the --socket-addr
flag. The API will listen on the unix domain socket at /run/docker/plugins/rclone.sock
. In the example above rclone will create a TCP socket and a small file /etc/docker/plugins/rclone.spec
containing the socket address. We use sudo
because both paths are writeable only by the root user.
If you later decide to change listening socket, the docker daemon must be restarted to reconnect to /run/docker/plugins/rclone.sock
or parse new /etc/docker/plugins/rclone.spec
. Until you restart, any volume related docker commands will timeout trying to access the old socket. Running directly is supported on Linux only, not on Windows or MacOS. This is not a problem with managed plugin mode described in details in the full documentation.
The command will create volume mounts under the path given by --base-dir
(by default /var/lib/docker-volumes/rclone
available only to root) and maintain the JSON formatted file docker-plugin.state
in the rclone cache directory with book-keeping records of created and mounted volumes.
All mount and VFS options are submitted by the docker daemon via API, but you can also provide defaults on the command line as well as set path to the config file and cache directory or adjust logging verbosity. ## VFS - Virtual File System
This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.
Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.
The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.
@@ -4146,10 +4271,220 @@ htpasswd -B htpasswd anotherUserSome backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df
on the filesystem, then pass the flag --vfs-used-is-size
to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size
and compute the total used space itself.
WARNING. Contrary to rclone size
, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.
rclone serve docker [flags]
+ --allow-non-empty Allow mounting over a non-empty directory (not supported on Windows)
+ --allow-other Allow access to other users (not supported on Windows)
+ --allow-root Allow access to root user (not supported on Windows)
+ --async-read Use asynchronous reads (not supported on Windows) (default true)
+ --attr-timeout Duration Time for which file/directory attributes are cached (default 1s)
+ --base-dir string Base directory for volumes (default "/var/lib/docker-volumes/rclone")
+ --daemon Run mount in background and exit parent process (as background output is suppressed, use --log-file with --log-format=pid,... to monitor) (not supported on Windows)
+ --daemon-timeout Duration Time limit for rclone to respond to kernel (not supported on Windows) (default 0s)
+ --daemon-wait Duration Time to wait for ready mount from daemon (maximum time on Linux, constant sleep time on OSX/BSD) (not supported on Windows) (default 1m0s)
+ --debug-fuse Debug the FUSE internals - needs -v
+ --default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows)
+ --devname string Set the device name - default is remote:path
+ --dir-cache-time Duration Time to cache directory entries for (default 5m0s)
+ --dir-perms FileMode Directory permissions (default 0777)
+ --file-perms FileMode File permissions (default 0666)
+ --forget-state Skip restoring previous state
+ --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required)
+ --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
+ -h, --help help for docker
+ --max-read-ahead SizeSuffix The number of bytes that can be prefetched for sequential reads (not supported on Windows) (default 128Ki)
+ --mount-case-insensitive Tristate Tell the OS the mount is case insensitive (true) or sensitive (false) regardless of the backend (auto) (default unset)
+ --network-mode Mount as remote network drive, instead of fixed disk drive (supported on Windows only)
+ --no-checksum Don't compare checksums on up/download
+ --no-modtime Don't read/write the modification time (can speed things up)
+ --no-seek Don't allow seeking in files
+ --no-spec Do not write spec file
+ --noappledouble Ignore Apple Double (._) and .DS_Store files (supported on OSX only) (default true)
+ --noapplexattr Ignore all "com.apple.*" extended attributes (supported on OSX only)
+ -o, --option stringArray Option for libfuse/WinFsp (repeat if required)
+ --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
+ --read-only Only allow read-only access
+ --socket-addr string Address <host:port> or absolute path (default: /run/docker/plugins/rclone.sock)
+ --socket-gid int GID for unix socket (default: current process GID) (default 1000)
+ --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
+ --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
+ --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s)
+ --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
+ --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
+ --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
+ --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s)
+ --vfs-case-insensitive If a file name not found, find a case insensitive match
+ --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off)
+ --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection
+ --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full
+ --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
+ --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
+ --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively in the background on start
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size
+ --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
+ --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
+ --volname string Set the volume name (supported on Windows and OSX only)
+ --write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows)
+Flags for filtering directory listings.
+ --delete-excluded Delete files on dest excluded from sync
+ --exclude stringArray Exclude files matching pattern
+ --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin)
+ --exclude-if-present stringArray Exclude directories if filename is present
+ --files-from stringArray Read list of source-file names from file (use - to read from stdin)
+ --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin)
+ -f, --filter stringArray Add a file filtering rule
+ --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin)
+ --ignore-case Ignore case in filters (case insensitive)
+ --include stringArray Include files matching pattern
+ --include-from stringArray Read file include patterns from file (use - to read from stdin)
+ --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --max-depth int If set limits the recursion depth to this (default -1)
+ --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
+ --metadata-exclude stringArray Exclude metadatas matching pattern
+ --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin)
+ --metadata-filter stringArray Add a metadata filtering rule
+ --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin)
+ --metadata-include stringArray Include metadatas matching pattern
+ --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin)
+ --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
+See the global flags page for global options not listed here.
+Serve remote:path over FTP.
+Run a basic FTP server to serve a remote over FTP protocol. This can be viewed with a FTP client or you can make a remote of type FTP to read and write it.
+Use --addr to specify which IP address and port the server should listen on, e.g. --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. By default it only listens on localhost. You can use port :0 to let the OS choose an available port.
+If you set --addr to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.
+By default this will serve files without needing a login.
+You can set a single username and password with the --user and --pass flags. ## VFS - Virtual File System
+This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.
+Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.
+The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.
+Using the --dir-cache-time
flag, you can control how long a directory should be considered up to date and not refreshed from the backend. Changes made through the VFS will appear immediately or invalidate the cache.
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
+--poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
+However, changes made directly on the cloud storage by the web interface or a different copy of rclone will only be picked up once the directory cache expires if the backend configured does not support polling for changes. If the backend supports polling, changes will be picked up within the polling interval.
+You can send a SIGHUP
signal to rclone for it to flush all directory caches, regardless of how old they are. Assuming only one rclone instance is running, you can reset the cache like this:
kill -SIGHUP $(pidof rclone)
+If you configure rclone with a remote control then you can use rclone rc to flush the whole directory cache:
+rclone rc vfs/forget
+Or individual files or directories:
+rclone rc vfs/forget file=path/to/file dir=path/to/dir
+The --buffer-size
flag determines the amount of memory, that will be used to buffer data in advance.
Each open file will try to keep the specified amount of data in memory at all times. The buffered data is bound to one open file and won't be shared.
+This flag is a upper limit for the used memory per open file. The buffer will only use memory for data that is downloaded but not not yet read. If the buffer is empty, only a small amount of memory will be used.
+The maximum memory used by rclone for buffering can be up to --buffer-size * open files
.
These flags control the VFS file caching options. File caching is necessary to make the VFS layer appear compatible with a normal file system. It can be disabled at the cost of some compatibility.
+For example you'll need to enable VFS caching if you want to read and write simultaneously to a file. See below for more details.
+Note that the VFS cache is separate from the cache backend and you may find that you need one or the other or both.
+--cache-dir string Directory rclone will use for caching.
+--vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
+--vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s)
+--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
+--vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
+--vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
+--vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
+If run with -vv
rclone will print the location of the file cache. The files are stored in the user cache file area which is OS dependent but can be controlled with --cache-dir
or setting the appropriate environment variable.
The cache has 4 different modes selected by --vfs-cache-mode
. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.
Note that files are written back to the remote only when they are closed and if they haven't been accessed for --vfs-write-back
seconds. If rclone is quit or dies with files that haven't been uploaded, these will be uploaded next time rclone is run with the same flags.
If using --vfs-cache-max-size
or --vfs-cache-min-free-size
note that the cache may exceed these quotas for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval
. Secondly because open files cannot be evicted from the cache. When --vfs-cache-max-size
or --vfs-cache-min-free-size
is exceeded, rclone will attempt to evict the least accessed files from the cache first. rclone will start with files that haven't been accessed for the longest. This cache flushing strategy is efficient and more relevant files are likely to remain cached.
The --vfs-cache-max-age
will evict files from the cache after the set time since last access has passed. The default value of 1 hour will start evicting files from cache that haven't been accessed for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0 and will wait for 1 more hour before evicting. Specify the time with standard notation, s, m, h, d, w .
You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off
. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir
. You don't need to worry about this if the remotes in use don't overlap.
In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.
+This will mean some operations are not possible
+This is very similar to "off" except that files opened for read AND write will be buffered to disk. This means that files opened for write will be a lot more compatible, but uses the minimal disk space.
+These operations are not possible
+In this mode files opened for read only are still read directly from the remote, write only and read/write files are buffered to disk first.
+This mode should support all normal file system operations.
+If an upload fails it will be retried at exponentially increasing intervals up to 1 minute.
+In this mode all reads and writes are buffered to and from disk. When data is read from the remote this is buffered to disk as well.
+In this mode the files in the cache will be sparse files and rclone will keep track of which bits of the files it has downloaded.
+So if an application only reads the starts of each file, then rclone will only buffer the start of the file. These files will appear to be their full size in the cache, but they will be sparse files with only the data that has been downloaded present in them.
+This mode should support all normal file system operations and is otherwise identical to --vfs-cache-mode
writes.
When reading a file rclone will read --buffer-size
plus --vfs-read-ahead
bytes ahead. The --buffer-size
is buffered in memory whereas the --vfs-read-ahead
is buffered on disk.
When using this mode it is recommended that --buffer-size
is not set too large and --vfs-read-ahead
is set large if required.
IMPORTANT not all file systems support sparse files. In particular FAT/exFAT do not. Rclone will perform very badly if the cache directory is on a filesystem which doesn't support sparse files and it will log an ERROR message if one is detected.
+Various parts of the VFS use fingerprinting to see if a local file copy has changed relative to a remote file. Fingerprints are made from:
+where available on an object.
+On some backends some of these attributes are slow to read (they take an extra API call per object, or extra work per object).
+For example hash
is slow with the local
and sftp
backends as they have to read the entire file and hash it, and modtime
is slow with the s3
, swift
, ftp
and qinqstor
backends because they need to do an extra API call to fetch it.
If you use the --vfs-fast-fingerprint
flag then rclone will not include the slow operations in the fingerprint. This makes the fingerprinting less accurate but much faster and will improve the opening time of cached files.
If you are running a vfs cache over local
, s3
or swift
backends then using this flag is recommended.
Note that if you change the value of this flag, the fingerprints of the files in the cache may be invalidated and the files will need to be downloaded again.
+When rclone reads files from a remote it reads them in chunks. This means that rather than requesting the whole file rclone reads the chunk specified. This can reduce the used download quota for some remotes by requesting only chunks from the remote that are actually read, at the cost of an increased number of requests.
+These flags control the chunking:
+--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M)
+--vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off)
+Rclone will start reading a chunk of size --vfs-read-chunk-size
, and then double the size for each read. When --vfs-read-chunk-size-limit
is specified, and greater than --vfs-read-chunk-size
, the chunk size for each open file will get doubled only until the specified value is reached. If the value is "off", which is the default, the limit is disabled and the chunk size will grow indefinitely.
With --vfs-read-chunk-size 100M
and --vfs-read-chunk-size-limit 0
the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. When --vfs-read-chunk-size-limit 500M
is specified, the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.
Setting --vfs-read-chunk-size
to 0
or "off" disables chunked reading.
These flags may be used to enable/disable features of the VFS for performance or other reasons. See also the chunked reading feature.
+In particular S3 and Swift benefit hugely from the --no-modtime
flag (or use --use-server-modtime
for a slightly different effect) as each read of the modification time takes a transaction.
--no-checksum Don't compare checksums on up/download.
+--no-modtime Don't read/write the modification time (can speed things up).
+--no-seek Don't allow seeking in files.
+--read-only Only allow read-only access.
+Sometimes rclone is delivered reads or writes out of order. Rather than seeking rclone will wait a short time for the in sequence read or write to come in. These flags only come into effect when not using an on disk cache file.
+--vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
+--vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
+When using VFS write caching (--vfs-cache-mode
with value writes or full), the global flag --transfers
can be set to adjust the number of parallel uploads of modified files from the cache (the related global flag --checkers
has no effect on the VFS).
--transfers int Number of file transfers to run in parallel (default 4)
+Linux file systems are case-sensitive: two files can differ only by case, and the exact case must be used when opening a file.
+File systems in modern Windows are case-insensitive but case-preserving: although existing files can be opened using any case, the exact case used to create the file is preserved and available for programs to query. It is not allowed for two files in the same directory to differ only by case.
+Usually file systems on macOS are case-insensitive. It is possible to make macOS file systems case-sensitive but that is not the default.
+The --vfs-case-insensitive
VFS flag controls how rclone handles these two cases. If its value is "false", rclone passes file names to the remote as-is. If the flag is "true" (or appears without a value on the command line), rclone may perform a "fixup" as explained below.
The user may specify a file name to open/delete/rename/etc with a case different than what is stored on the remote. If an argument refers to an existing file with exactly the same name, then the case of the existing file on the disk will be used. However, if a file name with exactly the same name is not found but a name differing only by case exists, rclone will transparently fixup the name. This fixup happens only when an existing file is requested. Case sensitivity of file names created anew by rclone is controlled by the underlying remote.
+Note that case sensitivity of the operating system running rclone (the target) may differ from case sensitivity of a file system presented by rclone (the source). The flag controls whether "fixup" is performed to satisfy the target.
+If the flag is not provided on the command line, then its default value depends on the operating system where rclone runs: "true" on Windows and macOS, "false" otherwise. If the flag is provided without a value, then it is "true".
+This flag allows you to manually set the statistics about the filing system. It can be useful when those statistics cannot be read correctly automatically.
+--vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1)
+Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df
on the filesystem, then pass the flag --vfs-used-is-size
to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size
and compute the total used space itself.
WARNING. Contrary to rclone size
, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.
If you supply the parameter --auth-proxy /path/to/program
then rclone will use that program to generate backends on the fly which then are used to authenticate incoming requests. This uses a simple JSON based protocol with input on STDIN and output on STDOUT.
PLEASE NOTE: --auth-proxy
and --authorized-keys
cannot be used together, if --auth-proxy
is set the authorized keys option will be ignored.
There is an example program bin/test_proxy.py in the rclone source code.
+There is an example program bin/test_proxy.py in the rclone source code.
The program's job is to take a user
and pass
on the input and turn those into the config for a backend on STDOUT in JSON format. This config will have any default parameters for the backend added, but it won't use configuration from environment variables or command line options - it is the job of the proxy program to make a complete config.
This config generated must have this extra parameter - _root
- root to use for the backend
And it may have this parameter - _obscure
- comma separated strings for parameters to obscure
Note that an internal cache is keyed on user
so only use that for configuration, don't use pass
or public_key
. This also means that if a user's password or public-key is changed the cache will need to expire (which takes 5 mins) before it takes effect.
This can be used to build general purpose proxies to any kind of backend that rclone supports.
rclone serve ftp remote:path [flags]
- --addr string IPaddress:Port or :Port to bind server to (default "localhost:2121")
--auth-proxy string A program to use to create the backend from the auth
--cert string TLS PEM key (concatenation of certificate and CA certificate)
@@ -4210,11 +4545,11 @@ htpasswd -B htpasswd anotherUser
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
- --vfs-refresh Refreshes the directory cache recursively on start
+ --vfs-refresh Refreshes the directory cache recursively in the background on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
-Flags for filtering directory listings.
--delete-excluded Delete files on dest excluded from sync
--exclude stringArray Exclude files matching pattern
@@ -4239,13 +4574,13 @@ htpasswd -B htpasswd anotherUser
--min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
--min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
See the global flags page for global options not listed here.
-Serve the remote over HTTP.
-Run a basic web server to serve a remote over HTTP. This can be viewed in a web browser or you can make a remote of type http read from it.
You can use the filter flags (e.g. --include
, --exclude
) to control what is served.
The server will log errors. Use -v
to see access logs.
This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.
Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.
The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.
-Using the --dir-cache-time
flag, you can control how long a directory should be considered up to date and not refreshed from the backend. Changes made through the VFS will appear immediately or invalidate the cache.
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
---poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
-However, changes made directly on the cloud storage by the web interface or a different copy of rclone will only be picked up once the directory cache expires if the backend configured does not support polling for changes. If the backend supports polling, changes will be picked up within the polling interval.
-You can send a SIGHUP
signal to rclone for it to flush all directory caches, regardless of how old they are. Assuming only one rclone instance is running, you can reset the cache like this:
kill -SIGHUP $(pidof rclone)
-If you configure rclone with a remote control then you can use rclone rc to flush the whole directory cache:
-rclone rc vfs/forget
-Or individual files or directories:
-rclone rc vfs/forget file=path/to/file dir=path/to/dir
-The --buffer-size
flag determines the amount of memory, that will be used to buffer data in advance.
Each open file will try to keep the specified amount of data in memory at all times. The buffered data is bound to one open file and won't be shared.
-This flag is a upper limit for the used memory per open file. The buffer will only use memory for data that is downloaded but not not yet read. If the buffer is empty, only a small amount of memory will be used.
-The maximum memory used by rclone for buffering can be up to --buffer-size * open files
.
These flags control the VFS file caching options. File caching is necessary to make the VFS layer appear compatible with a normal file system. It can be disabled at the cost of some compatibility.
-For example you'll need to enable VFS caching if you want to read and write simultaneously to a file. See below for more details.
-Note that the VFS cache is separate from the cache backend and you may find that you need one or the other or both.
---cache-dir string Directory rclone will use for caching.
---vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
---vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s)
---vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
---vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
---vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
---vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
-If run with -vv
rclone will print the location of the file cache. The files are stored in the user cache file area which is OS dependent but can be controlled with --cache-dir
or setting the appropriate environment variable.
The cache has 4 different modes selected by --vfs-cache-mode
. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.
Note that files are written back to the remote only when they are closed and if they haven't been accessed for --vfs-write-back
seconds. If rclone is quit or dies with files that haven't been uploaded, these will be uploaded next time rclone is run with the same flags.
If using --vfs-cache-max-size
or --vfs-cache-min-free-size
note that the cache may exceed these quotas for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval
. Secondly because open files cannot be evicted from the cache. When --vfs-cache-max-size
or --vfs-cache-min-free-size
is exceeded, rclone will attempt to evict the least accessed files from the cache first. rclone will start with files that haven't been accessed for the longest. This cache flushing strategy is efficient and more relevant files are likely to remain cached.
The --vfs-cache-max-age
will evict files from the cache after the set time since last access has passed. The default value of 1 hour will start evicting files from cache that haven't been accessed for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0 and will wait for 1 more hour before evicting. Specify the time with standard notation, s, m, h, d, w .
You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off
. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir
. You don't need to worry about this if the remotes in use don't overlap.
In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.
-This will mean some operations are not possible
-This is very similar to "off" except that files opened for read AND write will be buffered to disk. This means that files opened for write will be a lot more compatible, but uses the minimal disk space.
-These operations are not possible
-In this mode files opened for read only are still read directly from the remote, write only and read/write files are buffered to disk first.
-This mode should support all normal file system operations.
-If an upload fails it will be retried at exponentially increasing intervals up to 1 minute.
-In this mode all reads and writes are buffered to and from disk. When data is read from the remote this is buffered to disk as well.
-In this mode the files in the cache will be sparse files and rclone will keep track of which bits of the files it has downloaded.
-So if an application only reads the starts of each file, then rclone will only buffer the start of the file. These files will appear to be their full size in the cache, but they will be sparse files with only the data that has been downloaded present in them.
-This mode should support all normal file system operations and is otherwise identical to --vfs-cache-mode
writes.
When reading a file rclone will read --buffer-size
plus --vfs-read-ahead
bytes ahead. The --buffer-size
is buffered in memory whereas the --vfs-read-ahead
is buffered on disk.
When using this mode it is recommended that --buffer-size
is not set too large and --vfs-read-ahead
is set large if required.
IMPORTANT not all file systems support sparse files. In particular FAT/exFAT do not. Rclone will perform very badly if the cache directory is on a filesystem which doesn't support sparse files and it will log an ERROR message if one is detected.
-Various parts of the VFS use fingerprinting to see if a local file copy has changed relative to a remote file. Fingerprints are made from:
-where available on an object.
-On some backends some of these attributes are slow to read (they take an extra API call per object, or extra work per object).
-For example hash
is slow with the local
and sftp
backends as they have to read the entire file and hash it, and modtime
is slow with the s3
, swift
, ftp
and qinqstor
backends because they need to do an extra API call to fetch it.
If you use the --vfs-fast-fingerprint
flag then rclone will not include the slow operations in the fingerprint. This makes the fingerprinting less accurate but much faster and will improve the opening time of cached files.
If you are running a vfs cache over local
, s3
or swift
backends then using this flag is recommended.
Note that if you change the value of this flag, the fingerprints of the files in the cache may be invalidated and the files will need to be downloaded again.
-When rclone reads files from a remote it reads them in chunks. This means that rather than requesting the whole file rclone reads the chunk specified. This can reduce the used download quota for some remotes by requesting only chunks from the remote that are actually read, at the cost of an increased number of requests.
-These flags control the chunking:
---vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M)
---vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off)
-Rclone will start reading a chunk of size --vfs-read-chunk-size
, and then double the size for each read. When --vfs-read-chunk-size-limit
is specified, and greater than --vfs-read-chunk-size
, the chunk size for each open file will get doubled only until the specified value is reached. If the value is "off", which is the default, the limit is disabled and the chunk size will grow indefinitely.
With --vfs-read-chunk-size 100M
and --vfs-read-chunk-size-limit 0
the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. When --vfs-read-chunk-size-limit 500M
is specified, the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.
Setting --vfs-read-chunk-size
to 0
or "off" disables chunked reading.
These flags may be used to enable/disable features of the VFS for performance or other reasons. See also the chunked reading feature.
-In particular S3 and Swift benefit hugely from the --no-modtime
flag (or use --use-server-modtime
for a slightly different effect) as each read of the modification time takes a transaction.
--no-checksum Don't compare checksums on up/download.
---no-modtime Don't read/write the modification time (can speed things up).
---no-seek Don't allow seeking in files.
---read-only Only allow read-only access.
-Sometimes rclone is delivered reads or writes out of order. Rather than seeking rclone will wait a short time for the in sequence read or write to come in. These flags only come into effect when not using an on disk cache file.
---vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
---vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
-When using VFS write caching (--vfs-cache-mode
with value writes or full), the global flag --transfers
can be set to adjust the number of parallel uploads of modified files from the cache (the related global flag --checkers
has no effect on the VFS).
--transfers int Number of file transfers to run in parallel (default 4)
-Linux file systems are case-sensitive: two files can differ only by case, and the exact case must be used when opening a file.
-File systems in modern Windows are case-insensitive but case-preserving: although existing files can be opened using any case, the exact case used to create the file is preserved and available for programs to query. It is not allowed for two files in the same directory to differ only by case.
-Usually file systems on macOS are case-insensitive. It is possible to make macOS file systems case-sensitive but that is not the default.
-The --vfs-case-insensitive
VFS flag controls how rclone handles these two cases. If its value is "false", rclone passes file names to the remote as-is. If the flag is "true" (or appears without a value on the command line), rclone may perform a "fixup" as explained below.
The user may specify a file name to open/delete/rename/etc with a case different than what is stored on the remote. If an argument refers to an existing file with exactly the same name, then the case of the existing file on the disk will be used. However, if a file name with exactly the same name is not found but a name differing only by case exists, rclone will transparently fixup the name. This fixup happens only when an existing file is requested. Case sensitivity of file names created anew by rclone is controlled by the underlying remote.
-Note that case sensitivity of the operating system running rclone (the target) may differ from case sensitivity of a file system presented by rclone (the source). The flag controls whether "fixup" is performed to satisfy the target.
-If the flag is not provided on the command line, then its default value depends on the operating system where rclone runs: "true" on Windows and macOS, "false" otherwise. If the flag is provided without a value, then it is "true".
-This flag allows you to manually set the statistics about the filing system. It can be useful when those statistics cannot be read correctly automatically.
---vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1)
-Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df
on the filesystem, then pass the flag --vfs-used-is-size
to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size
and compute the total used space itself.
WARNING. Contrary to rclone size
, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.
If you supply the parameter --auth-proxy /path/to/program
then rclone will use that program to generate backends on the fly which then are used to authenticate incoming requests. This uses a simple JSON based protocol with input on STDIN and output on STDOUT.
PLEASE NOTE: --auth-proxy
and --authorized-keys
cannot be used together, if --auth-proxy
is set the authorized keys option will be ignored.
There is an example program bin/test_proxy.py in the rclone source code.
-The program's job is to take a user
and pass
on the input and turn those into the config for a backend on STDOUT in JSON format. This config will have any default parameters for the backend added, but it won't use configuration from environment variables or command line options - it is the job of the proxy program to make a complete config.
This config generated must have this extra parameter - _root
- root to use for the backend
And it may have this parameter - _obscure
- comma separated strings for parameters to obscure
If password authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:
-{
- "user": "me",
- "pass": "mypassword"
-}
-If public-key authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:
-{
- "user": "me",
- "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
-}
-And as an example return this on STDOUT
-{
- "type": "sftp",
- "_root": "",
- "_obscure": "pass",
- "user": "me",
- "pass": "mypassword",
- "host": "sftp.example.com"
-}
-This would mean that an SFTP backend would be created on the fly for the user
and pass
/public_key
returned in the output to the host given. Note that since _obscure
is set to pass
, rclone will obscure the pass
parameter before creating the backend (which is required for sftp backends).
The program can manipulate the supplied user
in any way, for example to make proxy to many different sftp backends, you could make the user
be user@example.com
and then set the host
to example.com
in the output and the user to user
. For security you'd probably want to restrict the host
to a limited list.
Note that an internal cache is keyed on user
so only use that for configuration, don't use pass
or public_key
. This also means that if a user's password or public-key is changed the cache will need to expire (which takes 5 mins) before it takes effect.
This can be used to build general purpose proxies to any kind of backend that rclone supports.
-rclone serve http remote:path [flags]
- --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
- --allow-origin string Origin which cross-domain request (CORS) can be executed from
- --auth-proxy string A program to use to create the backend from the auth
- --baseurl string Prefix for URLs - leave blank for root
- --cert string TLS PEM key (concatenation of certificate and CA certificate)
- --client-ca string Client certificate authority to verify clients with
- --dir-cache-time Duration Time to cache directory entries for (default 5m0s)
- --dir-perms FileMode Directory permissions (default 0777)
- --file-perms FileMode File permissions (default 0666)
- --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
- -h, --help help for http
- --htpasswd string A htpasswd file - if not provided no authentication is done
- --key string TLS PEM Private key
- --max-header-bytes int Maximum size of request header (default 4096)
- --min-tls-version string Minimum TLS version that is acceptable (default "tls1.0")
- --no-checksum Don't compare checksums on up/download
- --no-modtime Don't read/write the modification time (can speed things up)
- --no-seek Don't allow seeking in files
- --pass string Password for authentication
- --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
- --read-only Only allow read-only access
- --realm string Realm for authentication
- --salt string Password hashing salt (default "dlPL2MqE")
- --server-read-timeout Duration Timeout for server reading data (default 1h0m0s)
- --server-write-timeout Duration Timeout for server writing data (default 1h0m0s)
- --template string User-specified template
- --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
- --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
- --user string User name for authentication
- --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s)
- --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
- --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
- --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
- --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s)
- --vfs-case-insensitive If a file name not found, find a case insensitive match
- --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off)
- --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection
- --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full
- --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
- --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
- --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
- --vfs-refresh Refreshes the directory cache recursively on start
- --vfs-used-is-size rclone size Use the rclone size algorithm for Used size
- --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
- --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
-Flags for filtering directory listings.
- --delete-excluded Delete files on dest excluded from sync
- --exclude stringArray Exclude files matching pattern
- --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin)
- --exclude-if-present stringArray Exclude directories if filename is present
- --files-from stringArray Read list of source-file names from file (use - to read from stdin)
- --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin)
- -f, --filter stringArray Add a file filtering rule
- --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin)
- --ignore-case Ignore case in filters (case insensitive)
- --include stringArray Include files matching pattern
- --include-from stringArray Read file include patterns from file (use - to read from stdin)
- --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
- --max-depth int If set limits the recursion depth to this (default -1)
- --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
- --metadata-exclude stringArray Exclude metadatas matching pattern
- --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin)
- --metadata-filter stringArray Add a metadata filtering rule
- --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin)
- --metadata-include stringArray Include metadatas matching pattern
- --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin)
- --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
- --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
-See the global flags page for global options not listed here.
-Serve the remote as an NFS mount
-Create an NFS server that serves the given remote over the network.
-The primary purpose for this command is to enable mount command on recent macOS versions where installing FUSE is very cumbersome.
-Since this is running on NFSv3, no authentication method is available. Any client will be able to access the data. To limit access, you can use serve NFS on loopback address and rely on secure tunnels (such as SSH). For this reason, by default, a random TCP port is chosen and loopback interface is used for the listening address; meaning that it is only available to the local machine. If you want other machines to access the NFS mount over local network, you need to specify the listening address and port using --addr
flag.
Modifying files through NFS protocol requires VFS caching. Usually you will need to specify --vfs-cache-mode
in order to be able to write to the mountpoint (full is recommended). If you don't specify VFS cache mode, the mount will be read-only.
To serve NFS over the network use following command:
-rclone serve nfs remote: --addr 0.0.0.0:$PORT --vfs-cache-mode=full
-We specify a specific port that we can use in the mount command:
-To mount the server under Linux/macOS, use the following command:
-mount -oport=$PORT,mountport=$PORT $HOSTNAME: path/to/mountpoint
-Where $PORT
is the same port number we used in the serve nfs command.
This feature is only available on Unix platforms.
-This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.
-Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.
-The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.
Using the --dir-cache-time
flag, you can control how long a directory should be considered up to date and not refreshed from the backend. Changes made through the VFS will appear immediately or invalidate the cache.
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
@@ -4738,21 +4836,67 @@ htpasswd -B htpasswd anotherUser
Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df
on the filesystem, then pass the flag --vfs-used-is-size
to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size
and compute the total used space itself.
WARNING. Contrary to rclone size
, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.
rclone serve nfs remote:path [flags]
+If you supply the parameter --auth-proxy /path/to/program
then rclone will use that program to generate backends on the fly which then are used to authenticate incoming requests. This uses a simple JSON based protocol with input on STDIN and output on STDOUT.
PLEASE NOTE: --auth-proxy
and --authorized-keys
cannot be used together, if --auth-proxy
is set the authorized keys option will be ignored.
There is an example program bin/test_proxy.py in the rclone source code.
+The program's job is to take a user
and pass
on the input and turn those into the config for a backend on STDOUT in JSON format. This config will have any default parameters for the backend added, but it won't use configuration from environment variables or command line options - it is the job of the proxy program to make a complete config.
This config generated must have this extra parameter - _root
- root to use for the backend
And it may have this parameter - _obscure
- comma separated strings for parameters to obscure
If password authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:
+{
+ "user": "me",
+ "pass": "mypassword"
+}
+If public-key authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:
+{
+ "user": "me",
+ "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
+}
+And as an example return this on STDOUT
+{
+ "type": "sftp",
+ "_root": "",
+ "_obscure": "pass",
+ "user": "me",
+ "pass": "mypassword",
+ "host": "sftp.example.com"
+}
+This would mean that an SFTP backend would be created on the fly for the user
and pass
/public_key
returned in the output to the host given. Note that since _obscure
is set to pass
, rclone will obscure the pass
parameter before creating the backend (which is required for sftp backends).
The program can manipulate the supplied user
in any way, for example to make proxy to many different sftp backends, you could make the user
be user@example.com
and then set the host
to example.com
in the output and the user to user
. For security you'd probably want to restrict the host
to a limited list.
Note that an internal cache is keyed on user
so only use that for configuration, don't use pass
or public_key
. This also means that if a user's password or public-key is changed the cache will need to expire (which takes 5 mins) before it takes effect.
This can be used to build general purpose proxies to any kind of backend that rclone supports.
+rclone serve http remote:path [flags]
--addr string IPaddress:Port or :Port to bind server to
+ --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
+ --allow-origin string Origin which cross-domain request (CORS) can be executed from
+ --auth-proxy string A program to use to create the backend from the auth
+ --baseurl string Prefix for URLs - leave blank for root
+ --cert string TLS PEM key (concatenation of certificate and CA certificate)
+ --client-ca string Client certificate authority to verify clients with
--dir-cache-time Duration Time to cache directory entries for (default 5m0s)
--dir-perms FileMode Directory permissions (default 0777)
--file-perms FileMode File permissions (default 0666)
--gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
- -h, --help help for nfs
+ -h, --help help for http
+ --htpasswd string A htpasswd file - if not provided no authentication is done
+ --key string TLS PEM Private key
+ --max-header-bytes int Maximum size of request header (default 4096)
+ --min-tls-version string Minimum TLS version that is acceptable (default "tls1.0")
--no-checksum Don't compare checksums on up/download
--no-modtime Don't read/write the modification time (can speed things up)
--no-seek Don't allow seeking in files
+ --pass string Password for authentication
--poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
--read-only Only allow read-only access
+ --realm string Realm for authentication
+ --salt string Password hashing salt (default "dlPL2MqE")
+ --server-read-timeout Duration Timeout for server reading data (default 1h0m0s)
+ --server-write-timeout Duration Timeout for server writing data (default 1h0m0s)
+ --template string User-specified template
--uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
--umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
+ --user string User name for authentication
--vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s)
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
--vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
@@ -4765,7 +4909,7 @@ htpasswd -B htpasswd anotherUser
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
- --vfs-refresh Refreshes the directory cache recursively on start
+ --vfs-refresh Refreshes the directory cache recursively in the background on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
@@ -4798,170 +4942,21 @@ htpasswd -B htpasswd anotherUser
Serve the remote for restic's REST API.
+Serve the remote as an NFS mount
Run a basic web server to serve a remote over restic's REST backend API over HTTP. This allows restic to use rclone as a data storage mechanism for cloud providers that restic does not support directly.
-Restic is a command-line program for doing backups.
-The server will log errors. Use -v to see access logs.
---bwlimit
will be respected for file transfers. Use --stats
to control the stats printing.
First set up a remote for your chosen cloud provider.
-Once you have set up the remote, check it is working with, for example "rclone lsd remote:". You may have called the remote something other than "remote:" - just substitute whatever you called it in the following instructions.
-Now start the rclone restic server
-rclone serve restic -v remote:backup
-Where you can replace "backup" in the above by whatever path in the remote you wish to use.
-By default this will serve on "localhost:8080" you can change this with use of the --addr
flag.
You might wish to start this server on boot.
-Adding --cache-objects=false
will cause rclone to stop caching objects returned from the List call. Caching is normally desirable as it speeds up downloading objects, saves transactions and uses very little memory.
Now you can follow the restic instructions on setting up restic.
-Note that you will need restic 0.8.2 or later to interoperate with rclone.
-For the example above you will want to use "http://localhost:8080/" as the URL for the REST server.
-For example:
-$ export RESTIC_REPOSITORY=rest:http://localhost:8080/
-$ export RESTIC_PASSWORD=yourpassword
-$ restic init
-created restic backend 8b1a4b56ae at rest:http://localhost:8080/
-
-Please note that knowledge of your password is required to access
-the repository. Losing your password means that your data is
-irrecoverably lost.
-$ restic backup /path/to/files/to/backup
-scan [/path/to/files/to/backup]
-scanned 189 directories, 312 files in 0:00
-[0:00] 100.00% 38.128 MiB / 38.128 MiB 501 / 501 items 0 errors ETA 0:00
-duration: 0:00
-snapshot 45c8fdd8 saved
-Note that you can use the endpoint to host multiple repositories. Do this by adding a directory name or path after the URL. Note that these must end with /. Eg
-$ export RESTIC_REPOSITORY=rest:http://localhost:8080/user1repo/
-# backup user1 stuff
-$ export RESTIC_REPOSITORY=rest:http://localhost:8080/user2repo/
-# backup user2 stuff
-The--private-repos
flag can be used to limit users to repositories starting with a path of /<username>/
.
Use --addr
to specify which IP address and port the server should listen on, eg --addr 1.2.3.4:8000
or --addr :8080
to listen to all IPs. By default it only listens on localhost. You can use port :0 to let the OS choose an available port.
If you set --addr
to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.
You can use a unix socket by setting the url to unix:///path/to/socket
or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions.
--addr
may be repeated to listen on multiple IPs/ports/sockets.
--server-read-timeout
and --server-write-timeout
can be used to control the timeouts on the server. Note that this is the total time for a transfer.
--max-header-bytes
controls the maximum number of bytes the server will accept in the HTTP header.
--baseurl
controls the URL prefix that rclone serves from. By default rclone will serve from the root. If you used --baseurl "/rclone"
then rclone would serve from a URL starting with "/rclone/". This is useful if you wish to proxy rclone serve. Rclone automatically inserts leading and trailing "/" on --baseurl
, so --baseurl "rclone"
, --baseurl "/rclone"
and --baseurl "/rclone/"
are all treated identically.
By default this will serve over http. If you want you can serve over https. You will need to supply the --cert
and --key
flags. If you wish to do client side certificate validation then you will need to supply --client-ca
also.
--cert
should be a either a PEM encoded certificate or a concatenation of that with the CA certificate. --key
should be the PEM encoded private key and --client-ca
should be the PEM encoded client certificate authority certificate.
--min-tls-version is minimum TLS version that is acceptable. Valid values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0").
-By default this will serve files without needing a login.
-You can either use an htpasswd file which can take lots of users, or set a single username and password with the --user
and --pass
flags.
If no static users are configured by either of the above methods, and client certificates are required by the --client-ca
flag passed to the server, the client certificate common name will be considered as the username.
Use --htpasswd /path/to/htpasswd
to provide an htpasswd file. This is in standard apache format and supports MD5, SHA1 and BCrypt for basic authentication. Bcrypt is recommended.
To create an htpasswd file:
-touch htpasswd
-htpasswd -B htpasswd user
-htpasswd -B htpasswd anotherUser
-The password file can be updated while rclone is running.
-Use --realm
to set the authentication realm.
Use --salt
to change the password hashing salt from the default.
rclone serve restic remote:path [flags]
- --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
- --allow-origin string Origin which cross-domain request (CORS) can be executed from
- --append-only Disallow deletion of repository data
- --baseurl string Prefix for URLs - leave blank for root
- --cache-objects Cache listed objects (default true)
- --cert string TLS PEM key (concatenation of certificate and CA certificate)
- --client-ca string Client certificate authority to verify clients with
- -h, --help help for restic
- --htpasswd string A htpasswd file - if not provided no authentication is done
- --key string TLS PEM Private key
- --max-header-bytes int Maximum size of request header (default 4096)
- --min-tls-version string Minimum TLS version that is acceptable (default "tls1.0")
- --pass string Password for authentication
- --private-repos Users can only access their private repo
- --realm string Realm for authentication
- --salt string Password hashing salt (default "dlPL2MqE")
- --server-read-timeout Duration Timeout for server reading data (default 1h0m0s)
- --server-write-timeout Duration Timeout for server writing data (default 1h0m0s)
- --stdio Run an HTTP2 server on stdin/stdout
- --user string User name for authentication
-See the global flags page for global options not listed here.
-Serve remote:path over s3.
-serve s3
implements a basic s3 server that serves a remote via s3. This can be viewed with an s3 client, or you can make an s3 type remote to read and write to it with rclone.
serve s3
is considered Experimental so use with care.
S3 server supports Signature Version 4 authentication. Just use --auth-key accessKey,secretKey
and set the Authorization
header correctly in the request. (See the AWS docs).
--auth-key
can be repeated for multiple auth pairs. If --auth-key
is not provided then serve s3
will allow anonymous access.
Please note that some clients may require HTTPS endpoints. See the SSL docs for more information.
-This command uses the VFS directory cache. All the functionality will work with --vfs-cache-mode off
. Using --vfs-cache-mode full
(or writes
) can be used to cache objects locally to improve performance.
Use --force-path-style=false
if you want to use the bucket name as a part of the hostname (such as mybucket.local)
Use --etag-hash
if you want to change the hash uses for the ETag
. Note that using anything other than MD5
(the default) is likely to cause problems for S3 clients which rely on the Etag being the MD5.
For a simple set up, to serve remote:path
over s3, run the server like this:
rclone serve s3 --auth-key ACCESS_KEY_ID,SECRET_ACCESS_KEY remote:path
-This will be compatible with an rclone remote which is defined like this:
-[serves3]
-type = s3
-provider = Rclone
-endpoint = http://127.0.0.1:8080/
-access_key_id = ACCESS_KEY_ID
-secret_access_key = SECRET_ACCESS_KEY
-use_multipart_uploads = false
-Note that setting disable_multipart_uploads = true
is to work around a bug which will be fixed in due course.
When uploading multipart files serve s3
holds all the parts in memory (see #7453). This is a limitaton of the library rclone uses for serving S3 and will hopefully be fixed at some point.
Multipart server side copies do not work (see #7454). These take a very long time and eventually fail. The default threshold for multipart server side copies is 5G which is the maximum it can be, so files above this side will fail to be server side copied.
-For a current list of serve s3
bugs see the serve s3 bug category on GitHub.
serve s3
will treat all directories in the root as buckets and ignore all files in the root. You can use CreateBucket
to create folders under the root, but you can't create empty folders under other folders not in the root.
When using PutObject
or DeleteObject
, rclone will automatically create or clean up empty folders. If you don't want to clean up empty folders automatically, use --no-cleanup
.
When using ListObjects
, rclone will use /
when the delimiter is empty. This reduces backend requests with no effect on most operations, but if the delimiter is something other than /
and empty, rclone will do a full recursive search of the backend, which can take some time.
Versioning is not currently supported.
-Metadata will only be saved in memory other than the rclone mtime
metadata which will be set as the modification time of the file.
serve s3
currently supports the following operations.
ListBuckets
CreateBucket
DeleteBucket
HeadObject
ListObjects
GetObject
PutObject
DeleteObject
DeleteObjects
CreateMultipartUpload
CompleteMultipartUpload
AbortMultipartUpload
CopyObject
UploadPart
Other operations will return error Unimplemented
.
Use --addr
to specify which IP address and port the server should listen on, eg --addr 1.2.3.4:8000
or --addr :8080
to listen to all IPs. By default it only listens on localhost. You can use port :0 to let the OS choose an available port.
If you set --addr
to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.
You can use a unix socket by setting the url to unix:///path/to/socket
or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions.
--addr
may be repeated to listen on multiple IPs/ports/sockets.
--server-read-timeout
and --server-write-timeout
can be used to control the timeouts on the server. Note that this is the total time for a transfer.
--max-header-bytes
controls the maximum number of bytes the server will accept in the HTTP header.
--baseurl
controls the URL prefix that rclone serves from. By default rclone will serve from the root. If you used --baseurl "/rclone"
then rclone would serve from a URL starting with "/rclone/". This is useful if you wish to proxy rclone serve. Rclone automatically inserts leading and trailing "/" on --baseurl
, so --baseurl "rclone"
, --baseurl "/rclone"
and --baseurl "/rclone/"
are all treated identically.
By default this will serve over http. If you want you can serve over https. You will need to supply the --cert
and --key
flags. If you wish to do client side certificate validation then you will need to supply --client-ca
also.
--cert
should be a either a PEM encoded certificate or a concatenation of that with the CA certificate. --key
should be the PEM encoded private key and --client-ca
should be the PEM encoded client certificate authority certificate.
--min-tls-version is minimum TLS version that is acceptable. Valid values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). ## VFS - Virtual File System
+Create an NFS server that serves the given remote over the network.
+The primary purpose for this command is to enable mount command on recent macOS versions where installing FUSE is very cumbersome.
+Since this is running on NFSv3, no authentication method is available. Any client will be able to access the data. To limit access, you can use serve NFS on loopback address and rely on secure tunnels (such as SSH). For this reason, by default, a random TCP port is chosen and loopback interface is used for the listening address; meaning that it is only available to the local machine. If you want other machines to access the NFS mount over local network, you need to specify the listening address and port using --addr
flag.
Modifying files through NFS protocol requires VFS caching. Usually you will need to specify --vfs-cache-mode
in order to be able to write to the mountpoint (full is recommended). If you don't specify VFS cache mode, the mount will be read-only.
To serve NFS over the network use following command:
+rclone serve nfs remote: --addr 0.0.0.0:$PORT --vfs-cache-mode=full
+We specify a specific port that we can use in the mount command:
+To mount the server under Linux/macOS, use the following command:
+mount -oport=$PORT,mountport=$PORT $HOSTNAME: path/to/mountpoint
+Where $PORT
is the same port number we used in the serve nfs command.
This feature is only available on Unix platforms.
+This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.
Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.
The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.
@@ -5078,32 +5073,19 @@ use_multipart_uploads = falseSome backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df
on the filesystem, then pass the flag --vfs-used-is-size
to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size
and compute the total used space itself.
WARNING. Contrary to rclone size
, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.
rclone serve s3 remote:path [flags]
- --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
- --allow-origin string Origin which cross-domain request (CORS) can be executed from
- --auth-key stringArray Set key pair for v4 authorization: access_key_id,secret_access_key
- --baseurl string Prefix for URLs - leave blank for root
- --cert string TLS PEM key (concatenation of certificate and CA certificate)
- --client-ca string Client certificate authority to verify clients with
+rclone serve nfs remote:path [flags]
+Options
+ --addr string IPaddress:Port or :Port to bind server to
--dir-cache-time Duration Time to cache directory entries for (default 5m0s)
--dir-perms FileMode Directory permissions (default 0777)
- --etag-hash string Which hash to use for the ETag, or auto or blank for off (default "MD5")
--file-perms FileMode File permissions (default 0666)
- --force-path-style If true use path style access if false use virtual hosted style (default true) (default true)
--gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
- -h, --help help for s3
- --key string TLS PEM Private key
- --max-header-bytes int Maximum size of request header (default 4096)
- --min-tls-version string Minimum TLS version that is acceptable (default "tls1.0")
+ -h, --help help for nfs
--no-checksum Don't compare checksums on up/download
- --no-cleanup Not to cleanup empty folder after object is deleted
--no-modtime Don't read/write the modification time (can speed things up)
--no-seek Don't allow seeking in files
--poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
--read-only Only allow read-only access
- --server-read-timeout Duration Timeout for server reading data (default 1h0m0s)
- --server-write-timeout Duration Timeout for server writing data (default 1h0m0s)
--uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
--umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
--vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s)
@@ -5118,7 +5100,7 @@ use_multipart_uploads = false
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
- --vfs-refresh Refreshes the directory cache recursively on start
+ --vfs-refresh Refreshes the directory cache recursively in the background on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
@@ -5147,28 +5129,174 @@ use_multipart_uploads = false
--min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
--min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
See the global flags page for global options not listed here.
+Serve the remote for restic's REST API.
+Run a basic web server to serve a remote over restic's REST backend API over HTTP. This allows restic to use rclone as a data storage mechanism for cloud providers that restic does not support directly.
+Restic is a command-line program for doing backups.
+The server will log errors. Use -v to see access logs.
+--bwlimit
will be respected for file transfers. Use --stats
to control the stats printing.
First set up a remote for your chosen cloud provider.
+Once you have set up the remote, check it is working with, for example "rclone lsd remote:". You may have called the remote something other than "remote:" - just substitute whatever you called it in the following instructions.
+Now start the rclone restic server
+rclone serve restic -v remote:backup
+Where you can replace "backup" in the above by whatever path in the remote you wish to use.
+By default this will serve on "localhost:8080" you can change this with use of the --addr
flag.
You might wish to start this server on boot.
+Adding --cache-objects=false
will cause rclone to stop caching objects returned from the List call. Caching is normally desirable as it speeds up downloading objects, saves transactions and uses very little memory.
Now you can follow the restic instructions on setting up restic.
+Note that you will need restic 0.8.2 or later to interoperate with rclone.
+For the example above you will want to use "http://localhost:8080/" as the URL for the REST server.
+For example:
+$ export RESTIC_REPOSITORY=rest:http://localhost:8080/
+$ export RESTIC_PASSWORD=yourpassword
+$ restic init
+created restic backend 8b1a4b56ae at rest:http://localhost:8080/
+
+Please note that knowledge of your password is required to access
+the repository. Losing your password means that your data is
+irrecoverably lost.
+$ restic backup /path/to/files/to/backup
+scan [/path/to/files/to/backup]
+scanned 189 directories, 312 files in 0:00
+[0:00] 100.00% 38.128 MiB / 38.128 MiB 501 / 501 items 0 errors ETA 0:00
+duration: 0:00
+snapshot 45c8fdd8 saved
+Note that you can use the endpoint to host multiple repositories. Do this by adding a directory name or path after the URL. Note that these must end with /. Eg
+$ export RESTIC_REPOSITORY=rest:http://localhost:8080/user1repo/
+# backup user1 stuff
+$ export RESTIC_REPOSITORY=rest:http://localhost:8080/user2repo/
+# backup user2 stuff
+The--private-repos
flag can be used to limit users to repositories starting with a path of /<username>/
.
Use --addr
to specify which IP address and port the server should listen on, eg --addr 1.2.3.4:8000
or --addr :8080
to listen to all IPs. By default it only listens on localhost. You can use port :0 to let the OS choose an available port.
If you set --addr
to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.
You can use a unix socket by setting the url to unix:///path/to/socket
or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions.
--addr
may be repeated to listen on multiple IPs/ports/sockets.
--server-read-timeout
and --server-write-timeout
can be used to control the timeouts on the server. Note that this is the total time for a transfer.
--max-header-bytes
controls the maximum number of bytes the server will accept in the HTTP header.
--baseurl
controls the URL prefix that rclone serves from. By default rclone will serve from the root. If you used --baseurl "/rclone"
then rclone would serve from a URL starting with "/rclone/". This is useful if you wish to proxy rclone serve. Rclone automatically inserts leading and trailing "/" on --baseurl
, so --baseurl "rclone"
, --baseurl "/rclone"
and --baseurl "/rclone/"
are all treated identically.
By default this will serve over http. If you want you can serve over https. You will need to supply the --cert
and --key
flags. If you wish to do client side certificate validation then you will need to supply --client-ca
also.
--cert
should be a either a PEM encoded certificate or a concatenation of that with the CA certificate. --key
should be the PEM encoded private key and --client-ca
should be the PEM encoded client certificate authority certificate.
--min-tls-version is minimum TLS version that is acceptable. Valid values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0").
+By default this will serve files without needing a login.
+You can either use an htpasswd file which can take lots of users, or set a single username and password with the --user
and --pass
flags.
If no static users are configured by either of the above methods, and client certificates are required by the --client-ca
flag passed to the server, the client certificate common name will be considered as the username.
Use --htpasswd /path/to/htpasswd
to provide an htpasswd file. This is in standard apache format and supports MD5, SHA1 and BCrypt for basic authentication. Bcrypt is recommended.
To create an htpasswd file:
+touch htpasswd
+htpasswd -B htpasswd user
+htpasswd -B htpasswd anotherUser
+The password file can be updated while rclone is running.
+Use --realm
to set the authentication realm.
Use --salt
to change the password hashing salt from the default.
rclone serve restic remote:path [flags]
+ --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
+ --allow-origin string Origin which cross-domain request (CORS) can be executed from
+ --append-only Disallow deletion of repository data
+ --baseurl string Prefix for URLs - leave blank for root
+ --cache-objects Cache listed objects (default true)
+ --cert string TLS PEM key (concatenation of certificate and CA certificate)
+ --client-ca string Client certificate authority to verify clients with
+ -h, --help help for restic
+ --htpasswd string A htpasswd file - if not provided no authentication is done
+ --key string TLS PEM Private key
+ --max-header-bytes int Maximum size of request header (default 4096)
+ --min-tls-version string Minimum TLS version that is acceptable (default "tls1.0")
+ --pass string Password for authentication
+ --private-repos Users can only access their private repo
+ --realm string Realm for authentication
+ --salt string Password hashing salt (default "dlPL2MqE")
+ --server-read-timeout Duration Timeout for server reading data (default 1h0m0s)
+ --server-write-timeout Duration Timeout for server writing data (default 1h0m0s)
+ --stdio Run an HTTP2 server on stdin/stdout
+ --user string User name for authentication
+See the global flags page for global options not listed here.
Serve the remote over SFTP.
+Serve remote:path over s3.
Run an SFTP server to serve a remote over SFTP. This can be used with an SFTP client or you can make a remote of type sftp to use with it.
-You can use the filter flags (e.g. --include
, --exclude
) to control what is served.
The server will respond to a small number of shell commands, mainly md5sum, sha1sum and df, which enable it to provide support for checksums and the about feature when accessed from an sftp remote.
-Note that this server uses standard 32 KiB packet payload size, which means you must not configure the client to expect anything else, e.g. with the chunk_size option on an sftp remote.
-The server will log errors. Use -v
to see access logs.
--bwlimit
will be respected for file transfers. Use --stats
to control the stats printing.
You must provide some means of authentication, either with --user
/--pass
, an authorized keys file (specify location with --authorized-keys
- the default is the same as ssh), an --auth-proxy
, or set the --no-auth
flag for no authentication when logging in.
If you don't supply a host --key
then rclone will generate rsa, ecdsa and ed25519 variants, and cache them for later use in rclone's cache directory (see rclone help flags cache-dir
) in the "serve-sftp" directory.
By default the server binds to localhost:2022 - if you want it to be reachable externally then supply --addr :2022
for example.
Note that the default of --vfs-cache-mode off
is fine for the rclone sftp backend, but it may not be with other SFTP clients.
If --stdio
is specified, rclone will serve SFTP over stdio, which can be used with sshd via ~/.ssh/authorized_keys, for example:
restrict,command="rclone serve sftp --stdio ./photos" ssh-rsa ...
-On the client you need to set --transfers 1
when using --stdio
. Otherwise multiple instances of the rclone server are started by OpenSSH which can lead to "corrupted on transfer" errors. This is the case because the client chooses indiscriminately which server to send commands to while the servers all have different views of the state of the filing system.
The "restrict" in authorized_keys prevents SHA1SUMs and MD5SUMs from being used. Omitting "restrict" and using --sftp-path-override
to enable checksumming is possible but less secure and you could use the SFTP server provided by OpenSSH in this case.
serve s3
implements a basic s3 server that serves a remote via s3. This can be viewed with an s3 client, or you can make an s3 type remote to read and write to it with rclone.
serve s3
is considered Experimental so use with care.
S3 server supports Signature Version 4 authentication. Just use --auth-key accessKey,secretKey
and set the Authorization
header correctly in the request. (See the AWS docs).
--auth-key
can be repeated for multiple auth pairs. If --auth-key
is not provided then serve s3
will allow anonymous access.
Please note that some clients may require HTTPS endpoints. See the SSL docs for more information.
+This command uses the VFS directory cache. All the functionality will work with --vfs-cache-mode off
. Using --vfs-cache-mode full
(or writes
) can be used to cache objects locally to improve performance.
Use --force-path-style=false
if you want to use the bucket name as a part of the hostname (such as mybucket.local)
Use --etag-hash
if you want to change the hash uses for the ETag
. Note that using anything other than MD5
(the default) is likely to cause problems for S3 clients which rely on the Etag being the MD5.
For a simple set up, to serve remote:path
over s3, run the server like this:
rclone serve s3 --auth-key ACCESS_KEY_ID,SECRET_ACCESS_KEY remote:path
+This will be compatible with an rclone remote which is defined like this:
+[serves3]
+type = s3
+provider = Rclone
+endpoint = http://127.0.0.1:8080/
+access_key_id = ACCESS_KEY_ID
+secret_access_key = SECRET_ACCESS_KEY
+use_multipart_uploads = false
+Note that setting disable_multipart_uploads = true
is to work around a bug which will be fixed in due course.
When uploading multipart files serve s3
holds all the parts in memory (see #7453). This is a limitaton of the library rclone uses for serving S3 and will hopefully be fixed at some point.
Multipart server side copies do not work (see #7454). These take a very long time and eventually fail. The default threshold for multipart server side copies is 5G which is the maximum it can be, so files above this side will fail to be server side copied.
+For a current list of serve s3
bugs see the serve s3 bug category on GitHub.
serve s3
will treat all directories in the root as buckets and ignore all files in the root. You can use CreateBucket
to create folders under the root, but you can't create empty folders under other folders not in the root.
When using PutObject
or DeleteObject
, rclone will automatically create or clean up empty folders. If you don't want to clean up empty folders automatically, use --no-cleanup
.
When using ListObjects
, rclone will use /
when the delimiter is empty. This reduces backend requests with no effect on most operations, but if the delimiter is something other than /
and empty, rclone will do a full recursive search of the backend, which can take some time.
Versioning is not currently supported.
+Metadata will only be saved in memory other than the rclone mtime
metadata which will be set as the modification time of the file.
serve s3
currently supports the following operations.
ListBuckets
CreateBucket
DeleteBucket
HeadObject
ListObjects
GetObject
PutObject
DeleteObject
DeleteObjects
CreateMultipartUpload
CompleteMultipartUpload
AbortMultipartUpload
CopyObject
UploadPart
Other operations will return error Unimplemented
.
Use --addr
to specify which IP address and port the server should listen on, eg --addr 1.2.3.4:8000
or --addr :8080
to listen to all IPs. By default it only listens on localhost. You can use port :0 to let the OS choose an available port.
If you set --addr
to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.
You can use a unix socket by setting the url to unix:///path/to/socket
or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions.
--addr
may be repeated to listen on multiple IPs/ports/sockets.
--server-read-timeout
and --server-write-timeout
can be used to control the timeouts on the server. Note that this is the total time for a transfer.
--max-header-bytes
controls the maximum number of bytes the server will accept in the HTTP header.
--baseurl
controls the URL prefix that rclone serves from. By default rclone will serve from the root. If you used --baseurl "/rclone"
then rclone would serve from a URL starting with "/rclone/". This is useful if you wish to proxy rclone serve. Rclone automatically inserts leading and trailing "/" on --baseurl
, so --baseurl "rclone"
, --baseurl "/rclone"
and --baseurl "/rclone/"
are all treated identically.
By default this will serve over http. If you want you can serve over https. You will need to supply the --cert
and --key
flags. If you wish to do client side certificate validation then you will need to supply --client-ca
also.
--cert
should be a either a PEM encoded certificate or a concatenation of that with the CA certificate. --key
should be the PEM encoded private key and --client-ca
should be the PEM encoded client certificate authority certificate.
--min-tls-version is minimum TLS version that is acceptable. Valid values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). ## VFS - Virtual File System
This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.
Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.
The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.
@@ -5285,58 +5413,34 @@ use_multipart_uploads = falseSome backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df
on the filesystem, then pass the flag --vfs-used-is-size
to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size
and compute the total used space itself.
WARNING. Contrary to rclone size
, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.
If you supply the parameter --auth-proxy /path/to/program
then rclone will use that program to generate backends on the fly which then are used to authenticate incoming requests. This uses a simple JSON based protocol with input on STDIN and output on STDOUT.
PLEASE NOTE: --auth-proxy
and --authorized-keys
cannot be used together, if --auth-proxy
is set the authorized keys option will be ignored.
There is an example program bin/test_proxy.py in the rclone source code.
-The program's job is to take a user
and pass
on the input and turn those into the config for a backend on STDOUT in JSON format. This config will have any default parameters for the backend added, but it won't use configuration from environment variables or command line options - it is the job of the proxy program to make a complete config.
This config generated must have this extra parameter - _root
- root to use for the backend
And it may have this parameter - _obscure
- comma separated strings for parameters to obscure
If password authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:
-{
- "user": "me",
- "pass": "mypassword"
-}
-If public-key authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:
-{
- "user": "me",
- "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
-}
-And as an example return this on STDOUT
-{
- "type": "sftp",
- "_root": "",
- "_obscure": "pass",
- "user": "me",
- "pass": "mypassword",
- "host": "sftp.example.com"
-}
-This would mean that an SFTP backend would be created on the fly for the user
and pass
/public_key
returned in the output to the host given. Note that since _obscure
is set to pass
, rclone will obscure the pass
parameter before creating the backend (which is required for sftp backends).
The program can manipulate the supplied user
in any way, for example to make proxy to many different sftp backends, you could make the user
be user@example.com
and then set the host
to example.com
in the output and the user to user
. For security you'd probably want to restrict the host
to a limited list.
Note that an internal cache is keyed on user
so only use that for configuration, don't use pass
or public_key
. This also means that if a user's password or public-key is changed the cache will need to expire (which takes 5 mins) before it takes effect.
This can be used to build general purpose proxies to any kind of backend that rclone supports.
-rclone serve sftp remote:path [flags]
+rclone serve s3 remote:path [flags]
--addr string IPaddress:Port or :Port to bind server to (default "localhost:2022")
- --auth-proxy string A program to use to create the backend from the auth
- --authorized-keys string Authorized keys file (default "~/.ssh/authorized_keys")
+ --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
+ --allow-origin string Origin which cross-domain request (CORS) can be executed from
+ --auth-key stringArray Set key pair for v4 authorization: access_key_id,secret_access_key
+ --baseurl string Prefix for URLs - leave blank for root
+ --cert string TLS PEM key (concatenation of certificate and CA certificate)
+ --client-ca string Client certificate authority to verify clients with
--dir-cache-time Duration Time to cache directory entries for (default 5m0s)
--dir-perms FileMode Directory permissions (default 0777)
+ --etag-hash string Which hash to use for the ETag, or auto or blank for off (default "MD5")
--file-perms FileMode File permissions (default 0666)
+ --force-path-style If true use path style access if false use virtual hosted style (default true) (default true)
--gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
- -h, --help help for sftp
- --key stringArray SSH private host key file (Can be multi-valued, leave blank to auto generate)
- --no-auth Allow connections with no authentication if set
+ -h, --help help for s3
+ --key string TLS PEM Private key
+ --max-header-bytes int Maximum size of request header (default 4096)
+ --min-tls-version string Minimum TLS version that is acceptable (default "tls1.0")
--no-checksum Don't compare checksums on up/download
+ --no-cleanup Not to cleanup empty folder after object is deleted
--no-modtime Don't read/write the modification time (can speed things up)
--no-seek Don't allow seeking in files
- --pass string Password for authentication
--poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
--read-only Only allow read-only access
- --stdio Run an sftp server on stdin/stdout
+ --server-read-timeout Duration Timeout for server reading data (default 1h0m0s)
+ --server-write-timeout Duration Timeout for server writing data (default 1h0m0s)
--uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
--umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
- --user string User name for authentication
--vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s)
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
--vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
@@ -5349,7 +5453,7 @@ use_multipart_uploads = false
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
- --vfs-refresh Refreshes the directory cache recursively on start
+ --vfs-refresh Refreshes the directory cache recursively in the background on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
@@ -5382,9 +5486,240 @@ use_multipart_uploads = false
Serve the remote over SFTP.
+Run an SFTP server to serve a remote over SFTP. This can be used with an SFTP client or you can make a remote of type sftp to use with it.
+You can use the filter flags (e.g. --include
, --exclude
) to control what is served.
The server will respond to a small number of shell commands, mainly md5sum, sha1sum and df, which enable it to provide support for checksums and the about feature when accessed from an sftp remote.
+Note that this server uses standard 32 KiB packet payload size, which means you must not configure the client to expect anything else, e.g. with the chunk_size option on an sftp remote.
+The server will log errors. Use -v
to see access logs.
--bwlimit
will be respected for file transfers. Use --stats
to control the stats printing.
You must provide some means of authentication, either with --user
/--pass
, an authorized keys file (specify location with --authorized-keys
- the default is the same as ssh), an --auth-proxy
, or set the --no-auth
flag for no authentication when logging in.
If you don't supply a host --key
then rclone will generate rsa, ecdsa and ed25519 variants, and cache them for later use in rclone's cache directory (see rclone help flags cache-dir
) in the "serve-sftp" directory.
By default the server binds to localhost:2022 - if you want it to be reachable externally then supply --addr :2022
for example.
Note that the default of --vfs-cache-mode off
is fine for the rclone sftp backend, but it may not be with other SFTP clients.
If --stdio
is specified, rclone will serve SFTP over stdio, which can be used with sshd via ~/.ssh/authorized_keys, for example:
restrict,command="rclone serve sftp --stdio ./photos" ssh-rsa ...
+On the client you need to set --transfers 1
when using --stdio
. Otherwise multiple instances of the rclone server are started by OpenSSH which can lead to "corrupted on transfer" errors. This is the case because the client chooses indiscriminately which server to send commands to while the servers all have different views of the state of the filing system.
The "restrict" in authorized_keys prevents SHA1SUMs and MD5SUMs from being used. Omitting "restrict" and using --sftp-path-override
to enable checksumming is possible but less secure and you could use the SFTP server provided by OpenSSH in this case.
This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.
+Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.
+The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.
+Using the --dir-cache-time
flag, you can control how long a directory should be considered up to date and not refreshed from the backend. Changes made through the VFS will appear immediately or invalidate the cache.
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
+--poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
+However, changes made directly on the cloud storage by the web interface or a different copy of rclone will only be picked up once the directory cache expires if the backend configured does not support polling for changes. If the backend supports polling, changes will be picked up within the polling interval.
+You can send a SIGHUP
signal to rclone for it to flush all directory caches, regardless of how old they are. Assuming only one rclone instance is running, you can reset the cache like this:
kill -SIGHUP $(pidof rclone)
+If you configure rclone with a remote control then you can use rclone rc to flush the whole directory cache:
+rclone rc vfs/forget
+Or individual files or directories:
+rclone rc vfs/forget file=path/to/file dir=path/to/dir
+The --buffer-size
flag determines the amount of memory, that will be used to buffer data in advance.
Each open file will try to keep the specified amount of data in memory at all times. The buffered data is bound to one open file and won't be shared.
+This flag is a upper limit for the used memory per open file. The buffer will only use memory for data that is downloaded but not not yet read. If the buffer is empty, only a small amount of memory will be used.
+The maximum memory used by rclone for buffering can be up to --buffer-size * open files
.
These flags control the VFS file caching options. File caching is necessary to make the VFS layer appear compatible with a normal file system. It can be disabled at the cost of some compatibility.
+For example you'll need to enable VFS caching if you want to read and write simultaneously to a file. See below for more details.
+Note that the VFS cache is separate from the cache backend and you may find that you need one or the other or both.
+--cache-dir string Directory rclone will use for caching.
+--vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
+--vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s)
+--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
+--vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
+--vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
+--vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
+If run with -vv
rclone will print the location of the file cache. The files are stored in the user cache file area which is OS dependent but can be controlled with --cache-dir
or setting the appropriate environment variable.
The cache has 4 different modes selected by --vfs-cache-mode
. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.
Note that files are written back to the remote only when they are closed and if they haven't been accessed for --vfs-write-back
seconds. If rclone is quit or dies with files that haven't been uploaded, these will be uploaded next time rclone is run with the same flags.
If using --vfs-cache-max-size
or --vfs-cache-min-free-size
note that the cache may exceed these quotas for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval
. Secondly because open files cannot be evicted from the cache. When --vfs-cache-max-size
or --vfs-cache-min-free-size
is exceeded, rclone will attempt to evict the least accessed files from the cache first. rclone will start with files that haven't been accessed for the longest. This cache flushing strategy is efficient and more relevant files are likely to remain cached.
The --vfs-cache-max-age
will evict files from the cache after the set time since last access has passed. The default value of 1 hour will start evicting files from cache that haven't been accessed for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0 and will wait for 1 more hour before evicting. Specify the time with standard notation, s, m, h, d, w .
You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off
. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir
. You don't need to worry about this if the remotes in use don't overlap.
In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.
+This will mean some operations are not possible
+This is very similar to "off" except that files opened for read AND write will be buffered to disk. This means that files opened for write will be a lot more compatible, but uses the minimal disk space.
+These operations are not possible
+In this mode files opened for read only are still read directly from the remote, write only and read/write files are buffered to disk first.
+This mode should support all normal file system operations.
+If an upload fails it will be retried at exponentially increasing intervals up to 1 minute.
+In this mode all reads and writes are buffered to and from disk. When data is read from the remote this is buffered to disk as well.
+In this mode the files in the cache will be sparse files and rclone will keep track of which bits of the files it has downloaded.
+So if an application only reads the starts of each file, then rclone will only buffer the start of the file. These files will appear to be their full size in the cache, but they will be sparse files with only the data that has been downloaded present in them.
+This mode should support all normal file system operations and is otherwise identical to --vfs-cache-mode
writes.
When reading a file rclone will read --buffer-size
plus --vfs-read-ahead
bytes ahead. The --buffer-size
is buffered in memory whereas the --vfs-read-ahead
is buffered on disk.
When using this mode it is recommended that --buffer-size
is not set too large and --vfs-read-ahead
is set large if required.
IMPORTANT not all file systems support sparse files. In particular FAT/exFAT do not. Rclone will perform very badly if the cache directory is on a filesystem which doesn't support sparse files and it will log an ERROR message if one is detected.
+Various parts of the VFS use fingerprinting to see if a local file copy has changed relative to a remote file. Fingerprints are made from:
+where available on an object.
+On some backends some of these attributes are slow to read (they take an extra API call per object, or extra work per object).
+For example hash
is slow with the local
and sftp
backends as they have to read the entire file and hash it, and modtime
is slow with the s3
, swift
, ftp
and qinqstor
backends because they need to do an extra API call to fetch it.
If you use the --vfs-fast-fingerprint
flag then rclone will not include the slow operations in the fingerprint. This makes the fingerprinting less accurate but much faster and will improve the opening time of cached files.
If you are running a vfs cache over local
, s3
or swift
backends then using this flag is recommended.
Note that if you change the value of this flag, the fingerprints of the files in the cache may be invalidated and the files will need to be downloaded again.
+When rclone reads files from a remote it reads them in chunks. This means that rather than requesting the whole file rclone reads the chunk specified. This can reduce the used download quota for some remotes by requesting only chunks from the remote that are actually read, at the cost of an increased number of requests.
+These flags control the chunking:
+--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M)
+--vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off)
+Rclone will start reading a chunk of size --vfs-read-chunk-size
, and then double the size for each read. When --vfs-read-chunk-size-limit
is specified, and greater than --vfs-read-chunk-size
, the chunk size for each open file will get doubled only until the specified value is reached. If the value is "off", which is the default, the limit is disabled and the chunk size will grow indefinitely.
With --vfs-read-chunk-size 100M
and --vfs-read-chunk-size-limit 0
the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. When --vfs-read-chunk-size-limit 500M
is specified, the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.
Setting --vfs-read-chunk-size
to 0
or "off" disables chunked reading.
These flags may be used to enable/disable features of the VFS for performance or other reasons. See also the chunked reading feature.
+In particular S3 and Swift benefit hugely from the --no-modtime
flag (or use --use-server-modtime
for a slightly different effect) as each read of the modification time takes a transaction.
--no-checksum Don't compare checksums on up/download.
+--no-modtime Don't read/write the modification time (can speed things up).
+--no-seek Don't allow seeking in files.
+--read-only Only allow read-only access.
+Sometimes rclone is delivered reads or writes out of order. Rather than seeking rclone will wait a short time for the in sequence read or write to come in. These flags only come into effect when not using an on disk cache file.
+--vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
+--vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
+When using VFS write caching (--vfs-cache-mode
with value writes or full), the global flag --transfers
can be set to adjust the number of parallel uploads of modified files from the cache (the related global flag --checkers
has no effect on the VFS).
--transfers int Number of file transfers to run in parallel (default 4)
+Linux file systems are case-sensitive: two files can differ only by case, and the exact case must be used when opening a file.
+File systems in modern Windows are case-insensitive but case-preserving: although existing files can be opened using any case, the exact case used to create the file is preserved and available for programs to query. It is not allowed for two files in the same directory to differ only by case.
+Usually file systems on macOS are case-insensitive. It is possible to make macOS file systems case-sensitive but that is not the default.
+The --vfs-case-insensitive
VFS flag controls how rclone handles these two cases. If its value is "false", rclone passes file names to the remote as-is. If the flag is "true" (or appears without a value on the command line), rclone may perform a "fixup" as explained below.
The user may specify a file name to open/delete/rename/etc with a case different than what is stored on the remote. If an argument refers to an existing file with exactly the same name, then the case of the existing file on the disk will be used. However, if a file name with exactly the same name is not found but a name differing only by case exists, rclone will transparently fixup the name. This fixup happens only when an existing file is requested. Case sensitivity of file names created anew by rclone is controlled by the underlying remote.
+Note that case sensitivity of the operating system running rclone (the target) may differ from case sensitivity of a file system presented by rclone (the source). The flag controls whether "fixup" is performed to satisfy the target.
+If the flag is not provided on the command line, then its default value depends on the operating system where rclone runs: "true" on Windows and macOS, "false" otherwise. If the flag is provided without a value, then it is "true".
+This flag allows you to manually set the statistics about the filing system. It can be useful when those statistics cannot be read correctly automatically.
+--vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1)
+Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df
on the filesystem, then pass the flag --vfs-used-is-size
to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size
and compute the total used space itself.
WARNING. Contrary to rclone size
, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.
If you supply the parameter --auth-proxy /path/to/program
then rclone will use that program to generate backends on the fly which then are used to authenticate incoming requests. This uses a simple JSON based protocol with input on STDIN and output on STDOUT.
PLEASE NOTE: --auth-proxy
and --authorized-keys
cannot be used together, if --auth-proxy
is set the authorized keys option will be ignored.
There is an example program bin/test_proxy.py in the rclone source code.
+The program's job is to take a user
and pass
on the input and turn those into the config for a backend on STDOUT in JSON format. This config will have any default parameters for the backend added, but it won't use configuration from environment variables or command line options - it is the job of the proxy program to make a complete config.
This config generated must have this extra parameter - _root
- root to use for the backend
And it may have this parameter - _obscure
- comma separated strings for parameters to obscure
If password authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:
+{
+ "user": "me",
+ "pass": "mypassword"
+}
+If public-key authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:
+{
+ "user": "me",
+ "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
+}
+And as an example return this on STDOUT
+{
+ "type": "sftp",
+ "_root": "",
+ "_obscure": "pass",
+ "user": "me",
+ "pass": "mypassword",
+ "host": "sftp.example.com"
+}
+This would mean that an SFTP backend would be created on the fly for the user
and pass
/public_key
returned in the output to the host given. Note that since _obscure
is set to pass
, rclone will obscure the pass
parameter before creating the backend (which is required for sftp backends).
The program can manipulate the supplied user
in any way, for example to make proxy to many different sftp backends, you could make the user
be user@example.com
and then set the host
to example.com
in the output and the user to user
. For security you'd probably want to restrict the host
to a limited list.
Note that an internal cache is keyed on user
so only use that for configuration, don't use pass
or public_key
. This also means that if a user's password or public-key is changed the cache will need to expire (which takes 5 mins) before it takes effect.
This can be used to build general purpose proxies to any kind of backend that rclone supports.
+rclone serve sftp remote:path [flags]
+ --addr string IPaddress:Port or :Port to bind server to (default "localhost:2022")
+ --auth-proxy string A program to use to create the backend from the auth
+ --authorized-keys string Authorized keys file (default "~/.ssh/authorized_keys")
+ --dir-cache-time Duration Time to cache directory entries for (default 5m0s)
+ --dir-perms FileMode Directory permissions (default 0777)
+ --file-perms FileMode File permissions (default 0666)
+ --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
+ -h, --help help for sftp
+ --key stringArray SSH private host key file (Can be multi-valued, leave blank to auto generate)
+ --no-auth Allow connections with no authentication if set
+ --no-checksum Don't compare checksums on up/download
+ --no-modtime Don't read/write the modification time (can speed things up)
+ --no-seek Don't allow seeking in files
+ --pass string Password for authentication
+ --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
+ --read-only Only allow read-only access
+ --stdio Run an sftp server on stdin/stdout
+ --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
+ --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
+ --user string User name for authentication
+ --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s)
+ --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
+ --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
+ --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
+ --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s)
+ --vfs-case-insensitive If a file name not found, find a case insensitive match
+ --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off)
+ --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection
+ --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full
+ --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
+ --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
+ --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively in the background on start
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size
+ --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
+ --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
+Flags for filtering directory listings.
+ --delete-excluded Delete files on dest excluded from sync
+ --exclude stringArray Exclude files matching pattern
+ --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin)
+ --exclude-if-present stringArray Exclude directories if filename is present
+ --files-from stringArray Read list of source-file names from file (use - to read from stdin)
+ --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin)
+ -f, --filter stringArray Add a file filtering rule
+ --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin)
+ --ignore-case Ignore case in filters (case insensitive)
+ --include stringArray Include files matching pattern
+ --include-from stringArray Read file include patterns from file (use - to read from stdin)
+ --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --max-depth int If set limits the recursion depth to this (default -1)
+ --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
+ --metadata-exclude stringArray Exclude metadatas matching pattern
+ --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin)
+ --metadata-filter stringArray Add a metadata filtering rule
+ --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin)
+ --metadata-include stringArray Include metadatas matching pattern
+ --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin)
+ --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
+See the global flags page for global options not listed here.
+Serve remote:path over WebDAV.
-Run a basic WebDAV server to serve a remote over HTTP via the WebDAV protocol. This can be viewed with a WebDAV client, through a web browser, or you can make a remote of type WebDAV to read and write it.
This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.
Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.
The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.
-Using the --dir-cache-time
flag, you can control how long a directory should be considered up to date and not refreshed from the backend. Changes made through the VFS will appear immediately or invalidate the cache.
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
--poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
@@ -5544,12 +5879,12 @@ htpasswd -B htpasswd anotherUser
rclone rc vfs/forget
Or individual files or directories:
rclone rc vfs/forget file=path/to/file dir=path/to/dir
-The --buffer-size
flag determines the amount of memory, that will be used to buffer data in advance.
Each open file will try to keep the specified amount of data in memory at all times. The buffered data is bound to one open file and won't be shared.
This flag is a upper limit for the used memory per open file. The buffer will only use memory for data that is downloaded but not not yet read. If the buffer is empty, only a small amount of memory will be used.
The maximum memory used by rclone for buffering can be up to --buffer-size * open files
.
These flags control the VFS file caching options. File caching is necessary to make the VFS layer appear compatible with a normal file system. It can be disabled at the cost of some compatibility.
For example you'll need to enable VFS caching if you want to read and write simultaneously to a file. See below for more details.
Note that the VFS cache is separate from the cache backend and you may find that you need one or the other or both.
@@ -5566,7 +5901,7 @@ htpasswd -B htpasswd anotherUserIf using --vfs-cache-max-size
or --vfs-cache-min-free-size
note that the cache may exceed these quotas for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval
. Secondly because open files cannot be evicted from the cache. When --vfs-cache-max-size
or --vfs-cache-min-free-size
is exceeded, rclone will attempt to evict the least accessed files from the cache first. rclone will start with files that haven't been accessed for the longest. This cache flushing strategy is efficient and more relevant files are likely to remain cached.
The --vfs-cache-max-age
will evict files from the cache after the set time since last access has passed. The default value of 1 hour will start evicting files from cache that haven't been accessed for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0 and will wait for 1 more hour before evicting. Specify the time with standard notation, s, m, h, d, w .
You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off
. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir
. You don't need to worry about this if the remotes in use don't overlap.
In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.
This will mean some operations are not possible
This is very similar to "off" except that files opened for read AND write will be buffered to disk. This means that files opened for write will be a lot more compatible, but uses the minimal disk space.
These operations are not possible
In this mode files opened for read only are still read directly from the remote, write only and read/write files are buffered to disk first.
This mode should support all normal file system operations.
If an upload fails it will be retried at exponentially increasing intervals up to 1 minute.
-In this mode all reads and writes are buffered to and from disk. When data is read from the remote this is buffered to disk as well.
In this mode the files in the cache will be sparse files and rclone will keep track of which bits of the files it has downloaded.
So if an application only reads the starts of each file, then rclone will only buffer the start of the file. These files will appear to be their full size in the cache, but they will be sparse files with only the data that has been downloaded present in them.
@@ -5599,7 +5934,7 @@ htpasswd -B htpasswd anotherUserWhen reading a file rclone will read --buffer-size
plus --vfs-read-ahead
bytes ahead. The --buffer-size
is buffered in memory whereas the --vfs-read-ahead
is buffered on disk.
When using this mode it is recommended that --buffer-size
is not set too large and --vfs-read-ahead
is set large if required.
IMPORTANT not all file systems support sparse files. In particular FAT/exFAT do not. Rclone will perform very badly if the cache directory is on a filesystem which doesn't support sparse files and it will log an ERROR message if one is detected.
-Various parts of the VFS use fingerprinting to see if a local file copy has changed relative to a remote file. Fingerprints are made from:
If you use the --vfs-fast-fingerprint
flag then rclone will not include the slow operations in the fingerprint. This makes the fingerprinting less accurate but much faster and will improve the opening time of cached files.
If you are running a vfs cache over local
, s3
or swift
backends then using this flag is recommended.
Note that if you change the value of this flag, the fingerprints of the files in the cache may be invalidated and the files will need to be downloaded again.
-When rclone reads files from a remote it reads them in chunks. This means that rather than requesting the whole file rclone reads the chunk specified. This can reduce the used download quota for some remotes by requesting only chunks from the remote that are actually read, at the cost of an increased number of requests.
These flags control the chunking:
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M)
@@ -5620,7 +5955,7 @@ htpasswd -B htpasswd anotherUser
Rclone will start reading a chunk of size --vfs-read-chunk-size
, and then double the size for each read. When --vfs-read-chunk-size-limit
is specified, and greater than --vfs-read-chunk-size
, the chunk size for each open file will get doubled only until the specified value is reached. If the value is "off", which is the default, the limit is disabled and the chunk size will grow indefinitely.
With --vfs-read-chunk-size 100M
and --vfs-read-chunk-size-limit 0
the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. When --vfs-read-chunk-size-limit 500M
is specified, the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.
Setting --vfs-read-chunk-size
to 0
or "off" disables chunked reading.
These flags may be used to enable/disable features of the VFS for performance or other reasons. See also the chunked reading feature.
In particular S3 and Swift benefit hugely from the --no-modtime
flag (or use --use-server-modtime
for a slightly different effect) as each read of the modification time takes a transaction.
--no-checksum Don't compare checksums on up/download.
@@ -5632,7 +5967,7 @@ htpasswd -B htpasswd anotherUser
--vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
When using VFS write caching (--vfs-cache-mode
with value writes or full), the global flag --transfers
can be set to adjust the number of parallel uploads of modified files from the cache (the related global flag --checkers
has no effect on the VFS).
--transfers int Number of file transfers to run in parallel (default 4)
-Linux file systems are case-sensitive: two files can differ only by case, and the exact case must be used when opening a file.
File systems in modern Windows are case-insensitive but case-preserving: although existing files can be opened using any case, the exact case used to create the file is preserved and available for programs to query. It is not allowed for two files in the same directory to differ only by case.
Usually file systems on macOS are case-insensitive. It is possible to make macOS file systems case-sensitive but that is not the default.
@@ -5640,16 +5975,16 @@ htpasswd -B htpasswd anotherUserThe user may specify a file name to open/delete/rename/etc with a case different than what is stored on the remote. If an argument refers to an existing file with exactly the same name, then the case of the existing file on the disk will be used. However, if a file name with exactly the same name is not found but a name differing only by case exists, rclone will transparently fixup the name. This fixup happens only when an existing file is requested. Case sensitivity of file names created anew by rclone is controlled by the underlying remote.
Note that case sensitivity of the operating system running rclone (the target) may differ from case sensitivity of a file system presented by rclone (the source). The flag controls whether "fixup" is performed to satisfy the target.
If the flag is not provided on the command line, then its default value depends on the operating system where rclone runs: "true" on Windows and macOS, "false" otherwise. If the flag is provided without a value, then it is "true".
-This flag allows you to manually set the statistics about the filing system. It can be useful when those statistics cannot be read correctly automatically.
--vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1)
-Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df
on the filesystem, then pass the flag --vfs-used-is-size
to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size
and compute the total used space itself.
WARNING. Contrary to rclone size
, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.
If you supply the parameter --auth-proxy /path/to/program
then rclone will use that program to generate backends on the fly which then are used to authenticate incoming requests. This uses a simple JSON based protocol with input on STDIN and output on STDOUT.
PLEASE NOTE: --auth-proxy
and --authorized-keys
cannot be used together, if --auth-proxy
is set the authorized keys option will be ignored.
There is an example program bin/test_proxy.py in the rclone source code.
+There is an example program bin/test_proxy.py in the rclone source code.
The program's job is to take a user
and pass
on the input and turn those into the config for a backend on STDOUT in JSON format. This config will have any default parameters for the backend added, but it won't use configuration from environment variables or command line options - it is the job of the proxy program to make a complete config.
This config generated must have this extra parameter - _root
- root to use for the backend
And it may have this parameter - _obscure
- comma separated strings for parameters to obscure
Note that an internal cache is keyed on user
so only use that for configuration, don't use pass
or public_key
. This also means that if a user's password or public-key is changed the cache will need to expire (which takes 5 mins) before it takes effect.
This can be used to build general purpose proxies to any kind of backend that rclone supports.
rclone serve webdav remote:path [flags]
- --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
--allow-origin string Origin which cross-domain request (CORS) can be executed from
--auth-proxy string A program to use to create the backend from the auth
@@ -5721,11 +6056,11 @@ htpasswd -B htpasswd anotherUser
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
- --vfs-refresh Refreshes the directory cache recursively on start
+ --vfs-refresh Refreshes the directory cache recursively in the background on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
-Flags for filtering directory listings.
--delete-excluded Delete files on dest excluded from sync
--exclude stringArray Exclude files matching pattern
@@ -5750,13 +6085,13 @@ htpasswd -B htpasswd anotherUser
--min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
--min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
See the global flags page for global options not listed here.
-Changes storage class/tier of objects in remote.
-rclone settier changes storage tier or class at remote if supported. Few cloud storage services provides different storage classes on objects, for example AWS S3 and Glacier, Azure Blob storage - Hot, Cool and Archive, Google Cloud Storage, Regional Storage, Nearline, Coldline etc.
Note that, certain tier changes make objects not available to access immediately. For example tiering to archive in azure blob storage makes objects in frozen state, user can restore by setting tier to Hot/Cool, similarly S3 to Glacier makes object inaccessible.true
You can use it to tier single object
@@ -5766,25 +6101,25 @@ htpasswd -B htpasswd anotherUserOr just provide remote directory and all files in directory will be tiered
rclone settier tier remote:path/dir
rclone settier tier remote:path [flags]
- -h, --help help for settier
See the global flags page for global options not listed here.
-Run a test command
-Rclone test is used to run test commands.
Select which test command you want with the subcommand, eg
rclone test memory remote:
Each subcommand has its own options which you can see in their help.
NB Be careful running these commands, they may do strange things so reading their documentation first is recommended.
- -h, --help help for test
See the global flags page for global options not listed here.
-Log any change notify requests for the remote passed in.
rclone test changenotify remote: [flags]
- -h, --help help for changenotify
--poll-interval Duration Time to wait between polling for changes (default 10s)
See the global flags page for global options not listed here.
-Makes a histogram of file name characters.
-This command outputs JSON which shows the histogram of characters used in filenames in the remote:path specified.
-The data doesn't contain any identifying information but is useful for the rclone developers when developing filename compression.
-rclone test histogram [remote:path] [flags]
- -h, --help help for histogram
-See the global flags page for global options not listed here.
Makes a histogram of file name characters.
+This command outputs JSON which shows the histogram of characters used in filenames in the remote:path specified.
+The data doesn't contain any identifying information but is useful for the rclone developers when developing filename compression.
+rclone test histogram [remote:path] [flags]
+ -h, --help help for histogram
+See the global flags page for global options not listed here.
+Discovers file name or other limitations for paths.
-rclone info discovers what filenames and upload methods are possible to write to the paths passed in and how long they can be. It can take some time. It will write test files into the remote:path passed in. It outputs a bit of go code for each one.
NB this can create undeletable files and other hazards - use with care
rclone test info [remote:path]+ [flags]
- --all Run all tests
--check-base32768 Check can store all possible base32768 characters
--check-control Check control characters
@@ -5835,14 +6170,14 @@ htpasswd -B htpasswd anotherUser
--upload-wait Duration Wait after writing a file (default 0s)
--write-json string Write results to file
See the global flags page for global options not listed here.
-Make files with random contents of the size given
rclone test makefile <size> [<file>]+ [flags]
- --ascii Fill files with random ASCII printable bytes only
--chargen Fill files with a ASCII chargen pattern
-h, --help help for makefile
@@ -5851,14 +6186,14 @@ htpasswd -B htpasswd anotherUser
--sparse Make the files sparse (appear to be filled with ASCII 0x00)
--zero Fill files with ASCII 0x00
See the global flags page for global options not listed here.
-Make a random file hierarchy in a directory
rclone test makefiles <dir> [flags]
- --ascii Fill files with random ASCII printable bytes only
--chargen Fill files with a ASCII chargen pattern
--files int Number of files to create (default 1000)
@@ -5874,23 +6209,23 @@ htpasswd -B htpasswd anotherUser
--sparse Make the files sparse (appear to be filled with ASCII 0x00)
--zero Fill files with ASCII 0x00
See the global flags page for global options not listed here.
-Load all the objects at remote:path into memory and report memory stats.
rclone test memory remote:path [flags]
- -h, --help help for memory
See the global flags page for global options not listed here.
-Create new file or change file modification time.
-Set the modification time on file(s) as specified by remote:path to have the current time.
If remote:path does not exist then a zero sized file will be created, unless --no-create
or --recursive
is provided.
If --recursive
is used then recursively sets the modification time on all existing files that is found under the path. Filters are supported, and you can test with the --dry-run
or the --interactive
/-i
flag.
Note that value of --timestamp
is in UTC. If you want local time then add the --localtime
flag.
rclone touch remote:path [flags]
- -h, --help help for touch
--localtime Use localtime for timestamp, not UTC
-C, --no-create Do not create the file if it does not exist (implied with --recursive)
@@ -5913,7 +6248,7 @@ htpasswd -B htpasswd anotherUser
-n, --dry-run Do a trial run with no permanent changes
-i, --interactive Enable interactive mode
-v, --verbose count Print lots more stuff (repeat for more)
-Flags for filtering directory listings.
--delete-excluded Delete files on dest excluded from sync
--exclude stringArray Exclude files matching pattern
@@ -5942,13 +6277,13 @@ htpasswd -B htpasswd anotherUser
--default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
--fast-list Use recursive list if available; uses more memory but fewer transactions
See the global flags page for global options not listed here.
-List the contents of the remote in a tree like fashion.
-rclone tree lists the contents of a remote in a similar way to the unix tree command.
For example
$ rclone tree remote:path
@@ -5965,7 +6300,7 @@ htpasswd -B htpasswd anotherUser
The tree command has many options for controlling the listing which are compatible with the tree command, for example you can include file sizes with --size
. Note that not all of them have short options as they conflict with rclone's short options.
For a more interactive navigation of the remote see the ncdu command.
rclone tree remote:path [flags]
- -a, --all All files are listed (list . files too)
-d, --dirs-only List directories only
--dirsfirst List directories before files (-U disables)
@@ -5985,7 +6320,7 @@ htpasswd -B htpasswd anotherUser
-r, --sort-reverse Reverse the order of the sort
-U, --unsorted Leave files unsorted
--version Sort files alphanumerically by version
-Flags for filtering directory listings.
--delete-excluded Delete files on dest excluded from sync
--exclude stringArray Exclude files matching pattern
@@ -6014,7 +6349,7 @@ htpasswd -B htpasswd anotherUser
--default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
--fast-list Use recursive list if available; uses more memory but fewer transactions
See the global flags page for global options not listed here.
-The metadata keys mtime
and content-type
will take precedence if supplied in the metadata over reading the Content-Type
or modification time of the source object.
Hashes are not included in system metadata as there is a well defined way of reading those already.
-Rclone has a number of options to control its behaviour.
Options that take parameters can have the values passed in two ways, --option=value
or --option value
. However boolean (true/false) options behave slightly differently to the other options in that --boolean
sets the option to true
and the absence of the flag sets it to false
. It is also possible to specify --boolean=false
or --boolean=true
. Note that --boolean false
is not valid - this is parsed as --boolean
and the false
is parsed as an extra command line argument for rclone.
ID
is the source ID
of the file if known.Metadata
is the backend specific metadata as described in the backend docs.{
-"SrcFs": "gdrive:",
- "SrcFsType": "drive",
- "DstFs": "newdrive:user",
- "DstFsType": "onedrive",
- "Remote": "test.txt",
- "Size": 6,
- "MimeType": "text/plain; charset=utf-8",
- "ModTime": "2022-10-11T17:53:10.286745272+01:00",
- "IsDir": false,
- "ID": "xyz",
- "Metadata": {
- "btime": "2022-10-11T16:53:11Z",
- "content-type": "text/plain; charset=utf-8",
- "mtime": "2022-10-11T17:53:10.286745272+01:00",
- "owner": "user1@domain1.com",
- "permissions": "...",
- "description": "my nice file",
- "starred": "false"
- }
- }
{
+"SrcFs": "gdrive:",
+ "SrcFsType": "drive",
+ "DstFs": "newdrive:user",
+ "DstFsType": "onedrive",
+ "Remote": "test.txt",
+ "Size": 6,
+ "MimeType": "text/plain; charset=utf-8",
+ "ModTime": "2022-10-11T17:53:10.286745272+01:00",
+ "IsDir": false,
+ "ID": "xyz",
+ "Metadata": {
+ "btime": "2022-10-11T16:53:11Z",
+ "content-type": "text/plain; charset=utf-8",
+ "mtime": "2022-10-11T17:53:10.286745272+01:00",
+ "owner": "user1@domain1.com",
+ "permissions": "...",
+ "description": "my nice file",
+ "starred": "false"
+ }
+ }
The program should then modify the input as desired and send it to STDOUT. The returned Metadata
field will be used in its entirety for the destination object. Any other fields will be ignored. Note in this example we translate user names and permissions and add something to the description:
{
-"Metadata": {
- "btime": "2022-10-11T16:53:11Z",
- "content-type": "text/plain; charset=utf-8",
- "mtime": "2022-10-11T17:53:10.286745272+01:00",
- "owner": "user1@domain2.com",
- "permissions": "...",
- "description": "my nice file [migrated from domain1]",
- "starred": "false"
- }
- }
{
+"Metadata": {
+ "btime": "2022-10-11T16:53:11Z",
+ "content-type": "text/plain; charset=utf-8",
+ "mtime": "2022-10-11T17:53:10.286745272+01:00",
+ "owner": "user1@domain2.com",
+ "permissions": "...",
+ "description": "my nice file [migrated from domain1]",
+ "starred": "false"
+ }
+ }
Metadata can be removed here too.
An example python program might look something like this to implement the above transformations.
-import sys, json
-
-= json.load(sys.stdin)
- i = i["Metadata"]
- metadata # Add tag to description
-if "description" in metadata:
-"description"] += " [migrated from domain1]"
- metadata[else:
-"description"] = "[migrated from domain1]"
- metadata[# Modify owner
-if "owner" in metadata:
-"owner"] = metadata["owner"].replace("domain1.com", "domain2.com")
- metadata[= { "Metadata": metadata }
- o ="\t") json.dump(o, sys.stdout, indent
You can find this example (slightly expanded) in the rclone source code at bin/test_metadata_mapper.py.
+import sys, json
+
+= json.load(sys.stdin)
+ i = i["Metadata"]
+ metadata # Add tag to description
+if "description" in metadata:
+"description"] += " [migrated from domain1]"
+ metadata[else:
+"description"] = "[migrated from domain1]"
+ metadata[# Modify owner
+if "owner" in metadata:
+"owner"] = metadata["owner"].replace("domain1.com", "domain2.com")
+ metadata[= { "Metadata": metadata }
+ o ="\t") json.dump(o, sys.stdout, indent
You can find this example (slightly expanded) in the rclone source code at bin/test_metadata_mapper.py.
If you want to see the input to the metadata mapper and the output returned from it in the log you can use -vv --dump mapper
.
See the metadata section for more info.
In this case the value of this option is used (default 64Mi).
When transferring files above SIZE to capable backends, rclone will use multiple threads to transfer the file (default 256M).
-Capable backends are marked in the overview as MultithreadUpload
. (They need to implement either the OpenWriterAt
or OpenChunkedWriter
internal interfaces). These include include, local
, s3
, azureblob
, b2
, oracleobjectstorage
and smb
at the time of writing.
Capable backends are marked in the overview as MultithreadUpload
. (They need to implement either the OpenWriterAt
or OpenChunkWriter
internal interfaces). These include include, local
, s3
, azureblob
, b2
, oracleobjectstorage
and smb
at the time of writing.
On the local disk, rclone preallocates the file (using fallocate(FALLOC_FL_KEEP_SIZE)
on unix or NTSetInformationFile
on Windows both of which takes no time) then each thread writes directly into the file at the correct place. This means that rclone won't create fragmented or sparse files and there won't be any assembly time at the end of the transfer.
The number of threads used to transfer is controlled by --multi-thread-streams
.
Use -vv
if you wish to see info about the threads.
--order-by name
- send the files with alphabetically by path firstIf the --order-by
flag is not supplied or it is supplied with an empty string then the default ordering will be used which is as scanned. With --checkers 1
this is mostly alphabetical, however with the default --checkers 8
it is somewhat random.
The --order-by
flag does not do a separate pass over the data. This means that it may transfer some files out of the order specified if
Rclone can be configured entirely using environment variables. These can be used to set defaults for options or config file entries.
-Every option in rclone can have its default set by environment variable.
To find the name of the environment variable, first, take the long option name, strip the leading --
, change -
to _
, make upper case and prepend RCLONE_
.
For example, to always set --stats 5s
, set the environment variable RCLONE_STATS=5s
. If you set stats on the command line this will override the environment variable setting.
Flags helpful for increasing performance.
--buffer-size SizeSuffix In memory buffer size when reading files for each --transfer (default 16Mi)
@@ -12034,7 +12369,7 @@ rclone copy Path2 Path1 --create-empty-src-dirs
Note that while concurrent bisync runs are allowed, be very cautious that there is no overlap in the trees being synched between concurrent runs, lest there be replicated files, deleted files and general mayhem.
rclone bisync
returns the following codes to calling program: - 0
on a successful run, - 1
for a non-critical failing run (a rerun may be successful), - 2
for a critically aborted run (requires a --resync
to recover).
Bisync is considered BETA and has been tested with the following backends: - Local filesystem - Google Drive - Dropbox - OneDrive - S3 - SFTP - Yandex Disk
It has not been fully tested with other services yet. If it works, or sorta works, please let us know and we'll update the list. Run the test suite to check for proper operation as described below.
@@ -12724,7 +13059,7 @@ y/e/d> yrclone about
is not supported by the 1Fichier backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs
(most free space) as a member of an rclone union remote.
See List of backends that do not support rclone about and rclone about
Note that Amazon Drive is case insensitive so you can't have a file called "Hello.doc" and one called "hello.doc".
Amazon Drive has rate limiting so you may notice errors in the sync (429 errors). rclone will automatically retry the sync up to 3 times by default (see --retries
flag) which should hopefully work around this problem.
Amazon Drive has an internal limit of file sizes that can be uploaded to the service. This limit is not officially published, but all files larger than this will fail.
@@ -13434,6 +13769,7 @@ $ rclone -q --s3-versions ls s3:cleanup-testGetObject
PutObject
PutObjectACL
CreateBucket
(unless using s3-no-check-bucket)When using the lsd
subcommand, the ListAllMyBuckets
permission is required.
Example policy:
@@ -13468,6 +13804,7 @@ $ rclone -q --s3-versions ls s3:cleanup-testUSER_NAME
has been created."arn:aws:s3:::BUCKET_NAME"
doesn't have to be included.For reference, here's an Ansible script that will generate one or more buckets that will work with rclone sync
.
For Netease NOS configure as per the configurator rclone config
setting the provider Netease
. This will automatically set force_path_style = false
which is necessary for it to run properly.
Here is an example of making a Petabox configuration. First run:
-rclone config
rclone config
This will guide you through an interactive setup process.
No remotes found, make a new one?
n) New remote
@@ -17139,7 +17476,7 @@ y/n> n
Use the the native protocol to take advantage of client-side encryption as well as to achieve the best possible download performance. Uploads will be erasure-coded locally, thus a 1gb upload will result in 2.68gb of data being uploaded to storage nodes across the network.
Use this backend and the S3 compatible Hosted Gateway to increase upload performance and reduce the load on your systems and network. Uploads will be encrypted and erasure-coded server-side, thus a 1GB upload will result in only in 1GB of data being uploaded to storage nodes across the network.
For more detailed comparison please check the documentation of the storj backend.
-rclone about
is not supported by the S3 backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs
(most free space) as a member of an rclone union remote.
See List of backends that do not support rclone about and rclone about
On some SFTP servers (e.g. Synology) the paths are different for SSH and SFTP so the hashes can't be calculated properly. For them using disable_hashcheck
is a good idea.
The only ssh agent supported under Windows is Putty's pageant.
The Go SSH library disables the use of the aes128-cbc cipher by default, due to security concerns. This can be re-enabled on a per-connection basis by setting the use_insecure_cipher
setting in the configuration file to true
. Further details on the insecurity of this cipher can be found in this paper.
rclone sync --interactive --progress remote-us:bucket/path/to/dir/ remote-europe:bucket/path/to/dir/
Or even between another cloud storage and Storj.
rclone sync --interactive --progress s3:bucket/path/to/dir/ storj:bucket/path/to/dir/
-rclone about
is not supported by the rclone Storj backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs
(most free space) as a member of an rclone union remote.
See List of backends that do not support rclone about and rclone about
rclone about
is not supported by the SugarSync backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs
(most free space) as a member of an rclone union remote.
See List of backends that do not support rclone about and rclone about
Uptobox will delete inactive files that have not been accessed in 60 days.
rclone about
is not supported by this backend an overview of used space can however been seen in the uptobox web interface.
To check if your upstream supports the field, run rclone about remote: [flags]
and see if the required field exists.
Policies basically search upstream remotes and create a list of files / paths for functions to work on. The policy is responsible for filtering and sorting. The policy type defines the sorting but filtering is mostly uniform as described below.
When uploading very large files (bigger than about 5 GiB) you will need to increase the --timeout
parameter. This is because Yandex pauses (perhaps to calculate the MD5SUM for the entire file) before returning confirmation that the file has been uploaded. The default handling of timeouts in rclone is to assume a 5 minute pause is an error and close the connection - you'll see net/http: timeout awaiting response headers
errors in the logs if this is happening. Setting the timeout to twice the max size of file in GiB should be enough, so if you want to upload a 30 GiB file set a timeout of 2 * 30 = 60m
, that is --timeout 60m
.
Having a Yandex Mail account is mandatory to use the Yandex.Disk subscription. Token generation will work without a mail account, but Rclone won't be able to complete any actions.
[403 - DiskUnsupportedUserAccountTypeError] User account type is not supported.
@@ -36350,6 +36687,89 @@ $ tree /tmp/b
--sudo
and fix error/option handling (Nick Craig-Wood)*token
and *ts.token
are the same (rkonfj)--vfs-refresh
runs in the background (Nick Craig-Wood)<nil>
(Nick Craig-Wood)Rclone doesn't currently preserve the timestamps of directories. This is because rclone only really considers objects when syncing.