From 863b4125c3ef994094b1821e87518a0619f4b484 Mon Sep 17 00:00:00 2001 From: Nick Craig-Wood Date: Mon, 8 Jan 2024 10:51:38 +0000 Subject: [PATCH] Version v1.65.1 --- MANUAL.html | 2138 +++++++++++------- MANUAL.md | 982 +++++++- MANUAL.txt | 976 +++++++- docs/content/changelog.md | 50 + docs/content/commands/rclone.md | 3 +- docs/content/commands/rclone_mount.md | 2 +- docs/content/commands/rclone_nfsmount.md | 904 ++++++++ docs/content/commands/rclone_serve_dlna.md | 2 +- docs/content/commands/rclone_serve_docker.md | 2 +- docs/content/commands/rclone_serve_ftp.md | 4 +- docs/content/commands/rclone_serve_http.md | 4 +- docs/content/commands/rclone_serve_nfs.md | 3 +- docs/content/commands/rclone_serve_s3.md | 2 +- docs/content/commands/rclone_serve_sftp.md | 4 +- docs/content/commands/rclone_serve_webdav.md | 2 +- docs/content/flags.md | 2 +- rclone.1 | 1282 ++++++++++- 17 files changed, 5440 insertions(+), 922 deletions(-) create mode 100644 docs/content/commands/rclone_nfsmount.md diff --git a/MANUAL.html b/MANUAL.html index d16296fa0..efa203d92 100644 --- a/MANUAL.html +++ b/MANUAL.html @@ -81,7 +81,7 @@

rclone(1) User Manual

Nick Craig-Wood

-

Nov 26, 2023

+

Jan 08, 2024

Rclone syncs your files to cloud storage

rclone logo

@@ -3092,7 +3092,7 @@ WantedBy=multi-user.target --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -3293,9 +3293,344 @@ if src is directory +

rclone nfsmount

+

Mount the remote as file system on a mountpoint.

+

Synopsis

+

rclone nfsmount allows Linux, FreeBSD, macOS and Windows to mount any of Rclone's cloud storage systems as a file system with FUSE.

+

First set up your remote using rclone config. Check it works with rclone ls etc.

+

On Linux and macOS, you can run mount in either foreground or background (aka daemon) mode. Mount runs in foreground mode by default. Use the --daemon flag to force background mode. On Windows you can run mount in foreground only, the flag is ignored.

+

In background mode rclone acts as a generic Unix mount program: the main program starts, spawns background rclone process to setup and maintain the mount, waits until success or timeout and exits with appropriate code (killing the child process if it fails).

+

On Linux/macOS/FreeBSD start the mount like this, where /path/to/local/mount is an empty existing directory:

+
rclone nfsmount remote:path/to/files /path/to/local/mount
+

On Windows you can start a mount in different ways. See below for details. If foreground mount is used interactively from a console window, rclone will serve the mount and occupy the console so another window should be used to work with the mount until rclone is interrupted e.g. by pressing Ctrl-C.

+

The following examples will mount to an automatically assigned drive, to specific drive letter X:, to path C:\path\parent\mount (where parent directory or drive must exist, and mount must not exist, and is not supported when mounting as a network drive), and the last example will mount as network share \\cloud\remote and map it to an automatically assigned drive:

+
rclone nfsmount remote:path/to/files *
+rclone nfsmount remote:path/to/files X:
+rclone nfsmount remote:path/to/files C:\path\parent\mount
+rclone nfsmount remote:path/to/files \\cloud\remote
+

When the program ends while in foreground mode, either via Ctrl+C or receiving a SIGINT or SIGTERM signal, the mount should be automatically stopped.

+

When running in background mode the user will have to stop the mount manually:

+
# Linux
+fusermount -u /path/to/local/mount
+# OS X
+umount /path/to/local/mount
+

The umount operation can fail, for example when the mountpoint is busy. When that happens, it is the user's responsibility to stop the mount manually.

+

The size of the mounted file system will be set according to information retrieved from the remote, the same as returned by the rclone about command. Remotes with unlimited storage may report the used size only, then an additional 1 PiB of free space is assumed. If the remote does not support the about feature at all, then 1 PiB is set as both the total and the free size.

+

Installing on Windows

+

To run rclone nfsmount on Windows, you will need to download and install WinFsp.

+

WinFsp is an open-source Windows File System Proxy which makes it easy to write user space file systems for Windows. It provides a FUSE emulation layer which rclone uses combination with cgofuse. Both of these packages are by Bill Zissimopoulos who was very helpful during the implementation of rclone nfsmount for Windows.

+

Mounting modes on windows

+

Unlike other operating systems, Microsoft Windows provides a different filesystem type for network and fixed drives. It optimises access on the assumption fixed disk drives are fast and reliable, while network drives have relatively high latency and less reliability. Some settings can also be differentiated between the two types, for example that Windows Explorer should just display icons and not create preview thumbnails for image and video files on network drives.

+

In most cases, rclone will mount the remote as a normal, fixed disk drive by default. However, you can also choose to mount it as a remote network drive, often described as a network share. If you mount an rclone remote using the default, fixed drive mode and experience unexpected program errors, freezes or other issues, consider mounting as a network drive instead.

+

When mounting as a fixed disk drive you can either mount to an unused drive letter, or to a path representing a nonexistent subdirectory of an existing parent directory or drive. Using the special value * will tell rclone to automatically assign the next available drive letter, starting with Z: and moving backward. Examples:

+
rclone nfsmount remote:path/to/files *
+rclone nfsmount remote:path/to/files X:
+rclone nfsmount remote:path/to/files C:\path\parent\mount
+rclone nfsmount remote:path/to/files X:
+

Option --volname can be used to set a custom volume name for the mounted file system. The default is to use the remote name and path.

+

To mount as network drive, you can add option --network-mode to your nfsmount command. Mounting to a directory path is not supported in this mode, it is a limitation Windows imposes on junctions, so the remote must always be mounted to a drive letter.

+
rclone nfsmount remote:path/to/files X: --network-mode
+

A volume name specified with --volname will be used to create the network share path. A complete UNC path, such as \\cloud\remote, optionally with path \\cloud\remote\madeup\path, will be used as is. Any other string will be used as the share part, after a default prefix \\server\. If no volume name is specified then \\server\share will be used. You must make sure the volume name is unique when you are mounting more than one drive, or else the mount command will fail. The share name will treated as the volume label for the mapped drive, shown in Windows Explorer etc, while the complete \\server\share will be reported as the remote UNC path by net use etc, just like a normal network drive mapping.

+

If you specify a full network share UNC path with --volname, this will implicitly set the --network-mode option, so the following two examples have same result:

+
rclone nfsmount remote:path/to/files X: --network-mode
+rclone nfsmount remote:path/to/files X: --volname \\server\share
+

You may also specify the network share UNC path as the mountpoint itself. Then rclone will automatically assign a drive letter, same as with * and use that as mountpoint, and instead use the UNC path specified as the volume name, as if it were specified with the --volname option. This will also implicitly set the --network-mode option. This means the following two examples have same result:

+
rclone nfsmount remote:path/to/files \\cloud\remote
+rclone nfsmount remote:path/to/files * --volname \\cloud\remote
+

There is yet another way to enable network mode, and to set the share path, and that is to pass the "native" libfuse/WinFsp option directly: --fuse-flag --VolumePrefix=\server\share. Note that the path must be with just a single backslash prefix in this case.

+

Note: In previous versions of rclone this was the only supported method.

+

Read more about drive mapping

+

See also Limitations section below.

+

Windows filesystem permissions

+

The FUSE emulation layer on Windows must convert between the POSIX-based permission model used in FUSE, and the permission model used in Windows, based on access-control lists (ACL).

+

The mounted filesystem will normally get three entries in its access-control list (ACL), representing permissions for the POSIX permission scopes: Owner, group and others. By default, the owner and group will be taken from the current user, and the built-in group "Everyone" will be used to represent others. The user/group can be customized with FUSE options "UserName" and "GroupName", e.g. -o UserName=user123 -o GroupName="Authenticated Users". The permissions on each entry will be set according to options --dir-perms and --file-perms, which takes a value in traditional Unix numeric notation.

+

The default permissions corresponds to --file-perms 0666 --dir-perms 0777, i.e. read and write permissions to everyone. This means you will not be able to start any programs from the mount. To be able to do that you must add execute permissions, e.g. --file-perms 0777 --dir-perms 0777 to add it to everyone. If the program needs to write files, chances are you will have to enable VFS File Caching as well (see also limitations). Note that the default write permission have some restrictions for accounts other than the owner, specifically it lacks the "write extended attributes", as explained next.

+

The mapping of permissions is not always trivial, and the result you see in Windows Explorer may not be exactly like you expected. For example, when setting a value that includes write access for the group or others scope, this will be mapped to individual permissions "write attributes", "write data" and "append data", but not "write extended attributes". Windows will then show this as basic permission "Special" instead of "Write", because "Write" also covers the "write extended attributes" permission. When setting digit 0 for group or others, to indicate no permissions, they will still get individual permissions "read attributes", "read extended attributes" and "read permissions". This is done for compatibility reasons, e.g. to allow users without additional permissions to be able to read basic metadata about files like in Unix.

+

WinFsp 2021 (version 1.9) introduced a new FUSE option "FileSecurity", that allows the complete specification of file security descriptors using SDDL. With this you get detailed control of the resulting permissions, compared to use of the POSIX permissions described above, and no additional permissions will be added automatically for compatibility with Unix. Some example use cases will following.

+

If you set POSIX permissions for only allowing access to the owner, using --file-perms 0600 --dir-perms 0700, the user group and the built-in "Everyone" group will still be given some special permissions, as described above. Some programs may then (incorrectly) interpret this as the file being accessible by everyone, for example an SSH client may warn about "unprotected private key file". You can work around this by specifying -o FileSecurity="D:P(A;;FA;;;OW)", which sets file all access (FA) to the owner (OW), and nothing else.

+

When setting write permissions then, except for the owner, this does not include the "write extended attributes" permission, as mentioned above. This may prevent applications from writing to files, giving permission denied error instead. To set working write permissions for the built-in "Everyone" group, similar to what it gets by default but with the addition of the "write extended attributes", you can specify -o FileSecurity="D:P(A;;FRFW;;;WD)", which sets file read (FR) and file write (FW) to everyone (WD). If file execute (FX) is also needed, then change to -o FileSecurity="D:P(A;;FRFWFX;;;WD)", or set file all access (FA) to get full access permissions, including delete, with -o FileSecurity="D:P(A;;FA;;;WD)".

+

Windows caveats

+

Drives created as Administrator are not visible to other accounts, not even an account that was elevated to Administrator with the User Account Control (UAC) feature. A result of this is that if you mount to a drive letter from a Command Prompt run as Administrator, and then try to access the same drive from Windows Explorer (which does not run as Administrator), you will not be able to see the mounted drive.

+

If you don't need to access the drive from applications running with administrative privileges, the easiest way around this is to always create the mount from a non-elevated command prompt.

+

To make mapped drives available to the user account that created them regardless if elevated or not, there is a special Windows setting called linked connections that can be enabled.

+

It is also possible to make a drive mount available to everyone on the system, by running the process creating it as the built-in SYSTEM account. There are several ways to do this: One is to use the command-line utility PsExec, from Microsoft's Sysinternals suite, which has option -s to start processes as the SYSTEM account. Another alternative is to run the mount command from a Windows Scheduled Task, or a Windows Service, configured to run as the SYSTEM account. A third alternative is to use the WinFsp.Launcher infrastructure). Read more in the install documentation. Note that when running rclone as another user, it will not use the configuration file from your profile unless you tell it to with the --config option. Note also that it is now the SYSTEM account that will have the owner permissions, and other accounts will have permissions according to the group or others scopes. As mentioned above, these will then not get the "write extended attributes" permission, and this may prevent writing to files. You can work around this with the FileSecurity option, see example above.

+

Note that mapping to a directory path, instead of a drive letter, does not suffer from the same limitations.

+

Mounting on macOS

+

Mounting on macOS can be done either via built-in NFS server, macFUSE (also known as osxfuse) or FUSE-T. macFUSE is a traditional FUSE driver utilizing a macOS kernel extension (kext). FUSE-T is an alternative FUSE system which "mounts" via an NFSv4 local server.

+

NFS mount

+

This method spins up an NFS server using serve nfs command and mounts it to the specified mountpoint. If you run this in background mode using |--daemon|, you will need to send SIGTERM signal to the rclone process using |kill| command to stop the mount.

+

macFUSE Notes

+

If installing macFUSE using dmg packages from the website, rclone will locate the macFUSE libraries without any further intervention. If however, macFUSE is installed using the macports package manager, the following addition steps are required.

+
sudo mkdir /usr/local/lib
+cd /usr/local/lib
+sudo ln -s /opt/local/lib/libfuse.2.dylib
+

FUSE-T Limitations, Caveats, and Notes

+

There are some limitations, caveats, and notes about how it works. These are current as of FUSE-T version 1.0.14.

+

ModTime update on read

+

As per the FUSE-T wiki:

+
+

File access and modification times cannot be set separately as it seems to be an issue with the NFS client which always modifies both. Can be reproduced with 'touch -m' and 'touch -a' commands

+
+

This means that viewing files with various tools, notably macOS Finder, will cause rlcone to update the modification time of the file. This may make rclone upload a full new copy of the file.

+

Unicode Normalization

+

Rclone includes flags for unicode normalization with macFUSE that should be updated for FUSE-T. See this forum post and FUSE-T issue #16. The following flag should be added to the rclone mount command.

+
-o modules=iconv,from_code=UTF-8,to_code=UTF-8
+

Read Only mounts

+

When mounting with --read-only, attempts to write to files will fail silently as opposed to with a clear warning as in macFUSE.

+

Limitations

+

Without the use of --vfs-cache-mode this can only write files sequentially, it can only seek when reading. This means that many applications won't work with their files on an rclone mount without --vfs-cache-mode writes or --vfs-cache-mode full. See the VFS File Caching section for more info. When using NFS mount on macOS, if you don't specify |--vfs-cache-mode| the mount point will be read-only.

+

The bucket-based remotes (e.g. Swift, S3, Google Compute Storage, B2) do not support the concept of empty directories, so empty directories will have a tendency to disappear once they fall out of the directory cache.

+

When rclone mount is invoked on Unix with --daemon flag, the main rclone program will wait for the background mount to become ready or until the timeout specified by the --daemon-wait flag. On Linux it can check mount status using ProcFS so the flag in fact sets maximum time to wait, while the real wait can be less. On macOS / BSD the time to wait is constant and the check is performed only at the end. We advise you to set wait time on macOS reasonably.

+

Only supported on Linux, FreeBSD, OS X and Windows at the moment.

+

rclone nfsmount vs rclone sync/copy

+

File systems expect things to be 100% reliable, whereas cloud storage systems are a long way from 100% reliable. The rclone sync/copy commands cope with this with lots of retries. However rclone nfsmount can't use retries in the same way without making local copies of the uploads. Look at the VFS File Caching for solutions to make nfsmount more reliable.

+

Attribute caching

+

You can use the flag --attr-timeout to set the time the kernel caches the attributes (size, modification time, etc.) for directory entries.

+

The default is 1s which caches files just long enough to avoid too many callbacks to rclone from the kernel.

+

In theory 0s should be the correct value for filesystems which can change outside the control of the kernel. However this causes quite a few problems such as rclone using too much memory, rclone not serving files to samba and excessive time listing directories.

+

The kernel can cache the info about a file for the time given by --attr-timeout. You may see corruption if the remote file changes length during this window. It will show up as either a truncated file or a file with garbage on the end. With --attr-timeout 1s this is very unlikely but not impossible. The higher you set --attr-timeout the more likely it is. The default setting of "1s" is the lowest setting which mitigates the problems above.

+

If you set it higher (10s or 1m say) then the kernel will call back to rclone less often making it more efficient, however there is more chance of the corruption issue above.

+

If files don't change on the remote outside of the control of rclone then there is no chance of corruption.

+

This is the same as setting the attr_timeout option in mount.fuse.

+

Filters

+

Note that all the rclone filters can be used to select a subset of the files to be visible in the mount.

+

systemd

+

When running rclone nfsmount as a systemd service, it is possible to use Type=notify. In this case the service will enter the started state after the mountpoint has been successfully set up. Units having the rclone nfsmount service specified as a requirement will see all files and folders immediately in this mode.

+

Note that systemd runs mount units without any environment variables including PATH or HOME. This means that tilde (~) expansion will not work and you should provide --config and --cache-dir explicitly as absolute paths via rclone arguments. Since mounting requires the fusermount program, rclone will use the fallback PATH of /bin:/usr/bin in this scenario. Please ensure that fusermount is present on this PATH.

+

Rclone as Unix mount helper

+

The core Unix program /bin/mount normally takes the -t FSTYPE argument then runs the /sbin/mount.FSTYPE helper program passing it mount options as -o key=val,... or --opt=.... Automount (classic or systemd) behaves in a similar way.

+

rclone by default expects GNU-style flags --key val. To run it as a mount helper you should symlink rclone binary to /sbin/mount.rclone and optionally /usr/bin/rclonefs, e.g. ln -s /usr/bin/rclone /sbin/mount.rclone. rclone will detect it and translate command-line arguments appropriately.

+

Now you can run classic mounts like this:

+
mount sftp1:subdir /mnt/data -t rclone -o vfs_cache_mode=writes,sftp_key_file=/path/to/pem
+

or create systemd mount units:

+
# /etc/systemd/system/mnt-data.mount
+[Unit]
+Description=Mount for /mnt/data
+[Mount]
+Type=rclone
+What=sftp1:subdir
+Where=/mnt/data
+Options=rw,_netdev,allow_other,args2env,vfs-cache-mode=writes,config=/etc/rclone.conf,cache-dir=/var/rclone
+

optionally accompanied by systemd automount unit

+
# /etc/systemd/system/mnt-data.automount
+[Unit]
+Description=AutoMount for /mnt/data
+[Automount]
+Where=/mnt/data
+TimeoutIdleSec=600
+[Install]
+WantedBy=multi-user.target
+

or add in /etc/fstab a line like

+
sftp1:subdir /mnt/data rclone rw,noauto,nofail,_netdev,x-systemd.automount,args2env,vfs_cache_mode=writes,config=/etc/rclone.conf,cache_dir=/var/cache/rclone 0 0
+

or use classic Automountd. Remember to provide explicit config=...,cache-dir=... as a workaround for mount units being run without HOME.

+

Rclone in the mount helper mode will split -o argument(s) by comma, replace _ by - and prepend -- to get the command-line flags. Options containing commas or spaces can be wrapped in single or double quotes. Any inner quotes inside outer quotes of the same type should be doubled.

+

Mount option syntax includes a few extra options treated specially:

+ +

This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.

+

Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.

+

The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.

+

VFS Directory Cache

+

Using the --dir-cache-time flag, you can control how long a directory should be considered up to date and not refreshed from the backend. Changes made through the VFS will appear immediately or invalidate the cache.

+
--dir-cache-time duration   Time to cache directory entries for (default 5m0s)
+--poll-interval duration    Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
+

However, changes made directly on the cloud storage by the web interface or a different copy of rclone will only be picked up once the directory cache expires if the backend configured does not support polling for changes. If the backend supports polling, changes will be picked up within the polling interval.

+

You can send a SIGHUP signal to rclone for it to flush all directory caches, regardless of how old they are. Assuming only one rclone instance is running, you can reset the cache like this:

+
kill -SIGHUP $(pidof rclone)
+

If you configure rclone with a remote control then you can use rclone rc to flush the whole directory cache:

+
rclone rc vfs/forget
+

Or individual files or directories:

+
rclone rc vfs/forget file=path/to/file dir=path/to/dir
+

VFS File Buffering

+

The --buffer-size flag determines the amount of memory, that will be used to buffer data in advance.

+

Each open file will try to keep the specified amount of data in memory at all times. The buffered data is bound to one open file and won't be shared.

+

This flag is a upper limit for the used memory per open file. The buffer will only use memory for data that is downloaded but not not yet read. If the buffer is empty, only a small amount of memory will be used.

+

The maximum memory used by rclone for buffering can be up to --buffer-size * open files.

+

VFS File Caching

+

These flags control the VFS file caching options. File caching is necessary to make the VFS layer appear compatible with a normal file system. It can be disabled at the cost of some compatibility.

+

For example you'll need to enable VFS caching if you want to read and write simultaneously to a file. See below for more details.

+

Note that the VFS cache is separate from the cache backend and you may find that you need one or the other or both.

+
--cache-dir string                     Directory rclone will use for caching.
+--vfs-cache-mode CacheMode             Cache mode off|minimal|writes|full (default off)
+--vfs-cache-max-age duration           Max time since last access of objects in the cache (default 1h0m0s)
+--vfs-cache-max-size SizeSuffix        Max total size of objects in the cache (default off)
+--vfs-cache-min-free-space SizeSuffix  Target minimum free space on the disk containing the cache (default off)
+--vfs-cache-poll-interval duration     Interval to poll the cache for stale objects (default 1m0s)
+--vfs-write-back duration              Time to writeback files after last use when using cache (default 5s)
+

If run with -vv rclone will print the location of the file cache. The files are stored in the user cache file area which is OS dependent but can be controlled with --cache-dir or setting the appropriate environment variable.

+

The cache has 4 different modes selected by --vfs-cache-mode. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.

+

Note that files are written back to the remote only when they are closed and if they haven't been accessed for --vfs-write-back seconds. If rclone is quit or dies with files that haven't been uploaded, these will be uploaded next time rclone is run with the same flags.

+

If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the cache may exceed these quotas for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval. Secondly because open files cannot be evicted from the cache. When --vfs-cache-max-size or --vfs-cache-min-free-size is exceeded, rclone will attempt to evict the least accessed files from the cache first. rclone will start with files that haven't been accessed for the longest. This cache flushing strategy is efficient and more relevant files are likely to remain cached.

+

The --vfs-cache-max-age will evict files from the cache after the set time since last access has passed. The default value of 1 hour will start evicting files from cache that haven't been accessed for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0 and will wait for 1 more hour before evicting. Specify the time with standard notation, s, m, h, d, w .

+

You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir. You don't need to worry about this if the remotes in use don't overlap.

+

--vfs-cache-mode off

+

In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.

+

This will mean some operations are not possible

+ +

--vfs-cache-mode minimal

+

This is very similar to "off" except that files opened for read AND write will be buffered to disk. This means that files opened for write will be a lot more compatible, but uses the minimal disk space.

+

These operations are not possible

+ +

--vfs-cache-mode writes

+

In this mode files opened for read only are still read directly from the remote, write only and read/write files are buffered to disk first.

+

This mode should support all normal file system operations.

+

If an upload fails it will be retried at exponentially increasing intervals up to 1 minute.

+

--vfs-cache-mode full

+

In this mode all reads and writes are buffered to and from disk. When data is read from the remote this is buffered to disk as well.

+

In this mode the files in the cache will be sparse files and rclone will keep track of which bits of the files it has downloaded.

+

So if an application only reads the starts of each file, then rclone will only buffer the start of the file. These files will appear to be their full size in the cache, but they will be sparse files with only the data that has been downloaded present in them.

+

This mode should support all normal file system operations and is otherwise identical to --vfs-cache-mode writes.

+

When reading a file rclone will read --buffer-size plus --vfs-read-ahead bytes ahead. The --buffer-size is buffered in memory whereas the --vfs-read-ahead is buffered on disk.

+

When using this mode it is recommended that --buffer-size is not set too large and --vfs-read-ahead is set large if required.

+

IMPORTANT not all file systems support sparse files. In particular FAT/exFAT do not. Rclone will perform very badly if the cache directory is on a filesystem which doesn't support sparse files and it will log an ERROR message if one is detected.

+

Fingerprinting

+

Various parts of the VFS use fingerprinting to see if a local file copy has changed relative to a remote file. Fingerprints are made from:

+ +

where available on an object.

+

On some backends some of these attributes are slow to read (they take an extra API call per object, or extra work per object).

+

For example hash is slow with the local and sftp backends as they have to read the entire file and hash it, and modtime is slow with the s3, swift, ftp and qinqstor backends because they need to do an extra API call to fetch it.

+

If you use the --vfs-fast-fingerprint flag then rclone will not include the slow operations in the fingerprint. This makes the fingerprinting less accurate but much faster and will improve the opening time of cached files.

+

If you are running a vfs cache over local, s3 or swift backends then using this flag is recommended.

+

Note that if you change the value of this flag, the fingerprints of the files in the cache may be invalidated and the files will need to be downloaded again.

+

VFS Chunked Reading

+

When rclone reads files from a remote it reads them in chunks. This means that rather than requesting the whole file rclone reads the chunk specified. This can reduce the used download quota for some remotes by requesting only chunks from the remote that are actually read, at the cost of an increased number of requests.

+

These flags control the chunking:

+
--vfs-read-chunk-size SizeSuffix        Read the source objects in chunks (default 128M)
+--vfs-read-chunk-size-limit SizeSuffix  Max chunk doubling size (default off)
+

Rclone will start reading a chunk of size --vfs-read-chunk-size, and then double the size for each read. When --vfs-read-chunk-size-limit is specified, and greater than --vfs-read-chunk-size, the chunk size for each open file will get doubled only until the specified value is reached. If the value is "off", which is the default, the limit is disabled and the chunk size will grow indefinitely.

+

With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.

+

Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading.

+

VFS Performance

+

These flags may be used to enable/disable features of the VFS for performance or other reasons. See also the chunked reading feature.

+

In particular S3 and Swift benefit hugely from the --no-modtime flag (or use --use-server-modtime for a slightly different effect) as each read of the modification time takes a transaction.

+
--no-checksum     Don't compare checksums on up/download.
+--no-modtime      Don't read/write the modification time (can speed things up).
+--no-seek         Don't allow seeking in files.
+--read-only       Only allow read-only access.
+

Sometimes rclone is delivered reads or writes out of order. Rather than seeking rclone will wait a short time for the in sequence read or write to come in. These flags only come into effect when not using an on disk cache file.

+
--vfs-read-wait duration   Time to wait for in-sequence read before seeking (default 20ms)
+--vfs-write-wait duration  Time to wait for in-sequence write before giving error (default 1s)
+

When using VFS write caching (--vfs-cache-mode with value writes or full), the global flag --transfers can be set to adjust the number of parallel uploads of modified files from the cache (the related global flag --checkers has no effect on the VFS).

+
--transfers int  Number of file transfers to run in parallel (default 4)
+

VFS Case Sensitivity

+

Linux file systems are case-sensitive: two files can differ only by case, and the exact case must be used when opening a file.

+

File systems in modern Windows are case-insensitive but case-preserving: although existing files can be opened using any case, the exact case used to create the file is preserved and available for programs to query. It is not allowed for two files in the same directory to differ only by case.

+

Usually file systems on macOS are case-insensitive. It is possible to make macOS file systems case-sensitive but that is not the default.

+

The --vfs-case-insensitive VFS flag controls how rclone handles these two cases. If its value is "false", rclone passes file names to the remote as-is. If the flag is "true" (or appears without a value on the command line), rclone may perform a "fixup" as explained below.

+

The user may specify a file name to open/delete/rename/etc with a case different than what is stored on the remote. If an argument refers to an existing file with exactly the same name, then the case of the existing file on the disk will be used. However, if a file name with exactly the same name is not found but a name differing only by case exists, rclone will transparently fixup the name. This fixup happens only when an existing file is requested. Case sensitivity of file names created anew by rclone is controlled by the underlying remote.

+

Note that case sensitivity of the operating system running rclone (the target) may differ from case sensitivity of a file system presented by rclone (the source). The flag controls whether "fixup" is performed to satisfy the target.

+

If the flag is not provided on the command line, then its default value depends on the operating system where rclone runs: "true" on Windows and macOS, "false" otherwise. If the flag is provided without a value, then it is "true".

+

VFS Disk Options

+

This flag allows you to manually set the statistics about the filing system. It can be useful when those statistics cannot be read correctly automatically.

+
--vfs-disk-space-total-size    Manually set the total disk space size (example: 256G, default: -1)
+

Alternate report of used bytes

+

Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df on the filesystem, then pass the flag --vfs-used-is-size to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size and compute the total used space itself.

+

WARNING. Contrary to rclone size, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.

+
rclone nfsmount remote:path /path/to/mountpoint [flags]
+

Options

+
      --allow-non-empty                        Allow mounting over a non-empty directory (not supported on Windows)
+      --allow-other                            Allow access to other users (not supported on Windows)
+      --allow-root                             Allow access to root user (not supported on Windows)
+      --async-read                             Use asynchronous reads (not supported on Windows) (default true)
+      --attr-timeout Duration                  Time for which file/directory attributes are cached (default 1s)
+      --daemon                                 Run mount in background and exit parent process (as background output is suppressed, use --log-file with --log-format=pid,... to monitor) (not supported on Windows)
+      --daemon-timeout Duration                Time limit for rclone to respond to kernel (not supported on Windows) (default 0s)
+      --daemon-wait Duration                   Time to wait for ready mount from daemon (maximum time on Linux, constant sleep time on OSX/BSD) (not supported on Windows) (default 1m0s)
+      --debug-fuse                             Debug the FUSE internals - needs -v
+      --default-permissions                    Makes kernel enforce access control based on the file mode (not supported on Windows)
+      --devname string                         Set the device name - default is remote:path
+      --dir-cache-time Duration                Time to cache directory entries for (default 5m0s)
+      --dir-perms FileMode                     Directory permissions (default 0777)
+      --file-perms FileMode                    File permissions (default 0666)
+      --fuse-flag stringArray                  Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required)
+      --gid uint32                             Override the gid field set by the filesystem (not supported on Windows) (default 1000)
+  -h, --help                                   help for nfsmount
+      --max-read-ahead SizeSuffix              The number of bytes that can be prefetched for sequential reads (not supported on Windows) (default 128Ki)
+      --mount-case-insensitive Tristate        Tell the OS the mount is case insensitive (true) or sensitive (false) regardless of the backend (auto) (default unset)
+      --network-mode                           Mount as remote network drive, instead of fixed disk drive (supported on Windows only)
+      --no-checksum                            Don't compare checksums on up/download
+      --no-modtime                             Don't read/write the modification time (can speed things up)
+      --no-seek                                Don't allow seeking in files
+      --noappledouble                          Ignore Apple Double (._) and .DS_Store files (supported on OSX only) (default true)
+      --noapplexattr                           Ignore all "com.apple.*" extended attributes (supported on OSX only)
+  -o, --option stringArray                     Option for libfuse/WinFsp (repeat if required)
+      --poll-interval Duration                 Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
+      --read-only                              Only allow read-only access
+      --sudo                                   Use sudo to run the mount command as root.
+      --uid uint32                             Override the uid field set by the filesystem (not supported on Windows) (default 1000)
+      --umask int                              Override the permission bits set by the filesystem (not supported on Windows) (default 2)
+      --vfs-cache-max-age Duration             Max time since last access of objects in the cache (default 1h0m0s)
+      --vfs-cache-max-size SizeSuffix          Max total size of objects in the cache (default off)
+      --vfs-cache-min-free-space SizeSuffix    Target minimum free space on the disk containing the cache (default off)
+      --vfs-cache-mode CacheMode               Cache mode off|minimal|writes|full (default off)
+      --vfs-cache-poll-interval Duration       Interval to poll the cache for stale objects (default 1m0s)
+      --vfs-case-insensitive                   If a file name not found, find a case insensitive match
+      --vfs-disk-space-total-size SizeSuffix   Specify the total space of disk (default off)
+      --vfs-fast-fingerprint                   Use fast (less accurate) fingerprints for change detection
+      --vfs-read-ahead SizeSuffix              Extra read ahead over --buffer-size when using cache-mode full
+      --vfs-read-chunk-size SizeSuffix         Read the source objects in chunks (default 128Mi)
+      --vfs-read-chunk-size-limit SizeSuffix   If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
+      --vfs-read-wait Duration                 Time to wait for in-sequence read before seeking (default 20ms)
+      --vfs-refresh                            Refreshes the directory cache recursively in the background on start
+      --vfs-used-is-size rclone size           Use the rclone size algorithm for Used size
+      --vfs-write-back Duration                Time to writeback files after last use when using cache (default 5s)
+      --vfs-write-wait Duration                Time to wait for in-sequence write before giving error (default 1s)
+      --volname string                         Set the volume name (supported on Windows and OSX only)
+      --write-back-cache                       Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows)
+

Filter Options

+

Flags for filtering directory listings.

+
      --delete-excluded                     Delete files on dest excluded from sync
+      --exclude stringArray                 Exclude files matching pattern
+      --exclude-from stringArray            Read file exclude patterns from file (use - to read from stdin)
+      --exclude-if-present stringArray      Exclude directories if filename is present
+      --files-from stringArray              Read list of source-file names from file (use - to read from stdin)
+      --files-from-raw stringArray          Read list of source-file names from file without any processing of lines (use - to read from stdin)
+  -f, --filter stringArray                  Add a file filtering rule
+      --filter-from stringArray             Read file filtering patterns from a file (use - to read from stdin)
+      --ignore-case                         Ignore case in filters (case insensitive)
+      --include stringArray                 Include files matching pattern
+      --include-from stringArray            Read file include patterns from file (use - to read from stdin)
+      --max-age Duration                    Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+      --max-depth int                       If set limits the recursion depth to this (default -1)
+      --max-size SizeSuffix                 Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
+      --metadata-exclude stringArray        Exclude metadatas matching pattern
+      --metadata-exclude-from stringArray   Read metadata exclude patterns from file (use - to read from stdin)
+      --metadata-filter stringArray         Add a metadata filtering rule
+      --metadata-filter-from stringArray    Read metadata filtering patterns from a file (use - to read from stdin)
+      --metadata-include stringArray        Include metadatas matching pattern
+      --metadata-include-from stringArray   Read metadata include patterns from file (use - to read from stdin)
+      --min-age Duration                    Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+      --min-size SizeSuffix                 Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
+

See the global flags page for global options not listed here.

+

SEE ALSO

+

rclone obscure

Obscure password for use in the rclone config file.

-

Synopsis

+

Synopsis

In the rclone config file, human-readable passwords are obscured. Obscuring them is done by encrypting them and writing them out in base64. This is not a secure way of encrypting these passwords as rclone can decrypt them - it is to prevent "eyedropping" - namely someone seeing a password in the rclone config file by accident.

Many equally important things (like access tokens) are not obscured in the config file. However it is very hard to shoulder surf a 64 character hex token.

This command can also accept a password through STDIN instead of an argument by passing a hyphen as an argument. This will use the first line of STDIN as the password not including the trailing newline.

@@ -3303,16 +3638,16 @@ if src is directory

If there is no data on STDIN to read, rclone obscure will default to obfuscating the hyphen itself.

If you want to encrypt the config file then please use config file encryption - see rclone config for more info.

rclone obscure password [flags]
-

Options

+

Options

  -h, --help   help for obscure

See the global flags page for global options not listed here.

-

SEE ALSO

+

SEE ALSO

rclone rc

Run a command against a running rclone.

-

Synopsis

+

Synopsis

This runs a command against a running rclone. Use the --url flag to specify an non default URL to connect on. This can be either a ":port" which is taken to mean "http://localhost:port" or a "host:port" which is taken to mean "http://host:port"

A username and password can be passed in with --user and --pass.

Note that --rc-addr, --rc-user, --rc-pass will be read also for --url, --user, --pass.

@@ -3331,7 +3666,7 @@ if src is directory 
rclone rc --loopback operations/about fs=/

Use rclone rc to see a list of all possible commands.

rclone rc commands parameter [flags]
-

Options

+

Options

  -a, --arg stringArray   Argument placed in the "arg" array
   -h, --help              help for rc
       --json string       Input JSON - use instead of key=value args
@@ -3342,13 +3677,13 @@ if src is directory
       --url string        URL to connect to rclone remote control (default "http://localhost:5572/")
       --user string       Username to use to rclone remote control

See the global flags page for global options not listed here.

-

SEE ALSO

+

SEE ALSO

rclone rcat

Copies standard input to file on remote.

-

Synopsis

+

Synopsis

rclone rcat reads from standard input (stdin) and copies it to a single remote file.

echo "hello world" | rclone rcat remote:path/to/file
 ffmpeg - | rclone rcat remote:path/to/file
@@ -3358,7 +3693,7 @@ ffmpeg - | rclone rcat remote:path/to/file

--size should be the exact size of the input stream in bytes. If the size of the stream is different in length to the --size passed in then the transfer will likely fail.

Note that the upload cannot be retried because the data is not stored. If the backend supports multipart uploading then individual chunks can be retried. If you need to transfer a lot of data, you may be better off caching it locally and then rclone move it to the destination which can use retries.

rclone rcat remote:path [flags]
-

Options

+

Options

  -h, --help       help for rcat
       --size int   File size hint to preallocate (default -1)

Important Options

@@ -3367,13 +3702,13 @@ ffmpeg - | rclone rcat remote:path/to/file -i, --interactive Enable interactive mode -v, --verbose count Print lots more stuff (repeat for more)

See the global flags page for global options not listed here.

-

SEE ALSO

+

SEE ALSO

rclone rcd

Run rclone listening to remote control commands only.

-

Synopsis

+

Synopsis

This runs rclone so that it only listens to remote control commands.

This is useful if you are controlling rclone via the rc API.

If you pass in a path to a directory, rclone will serve that directory for GET requests on the URL passed in. It will also open the URL in the browser when rclone is run.

@@ -3514,7 +3849,7 @@ htpasswd -B htpasswd anotherUser

Use --rc-realm to set the authentication realm.

Use --rc-salt to change the password hashing salt from the default.

rclone rcd <path to files to serve>* [flags]
-

Options

+

Options

  -h, --help   help for rcd

RC Options

Flags to control the Remote Control API.

@@ -3547,20 +3882,20 @@ htpasswd -B htpasswd anotherUser --rc-web-gui-no-open-browser Don't open the browser automatically --rc-web-gui-update Check and update to latest version of web gui

See the global flags page for global options not listed here.

-

SEE ALSO

+

SEE ALSO

rclone rmdirs

Remove empty directories under the path.

-

Synopsis

+

Synopsis

This recursively removes any empty directories (including directories that only contain empty directories), that it finds under the path. The root path itself will also be removed if it is empty, unless you supply the --leave-root flag.

Use command rmdir to delete just the empty directory given by path, not recurse.

This is useful for tidying up remotes that rclone has left a lot of empty directories in. For example the delete command will delete files but leave the directory structure (unless used with option --rmdirs).

This will delete --checkers directories concurrently so if you have thousands of empty directories consider increasing this number.

To delete a path and any objects in it, use the purge command.

rclone rmdirs remote:path [flags]
-

Options

+

Options

  -h, --help         help for rmdirs
       --leave-root   Do not remove root directory if empty

Important Options

@@ -3569,13 +3904,13 @@ htpasswd -B htpasswd anotherUser -i, --interactive Enable interactive mode -v, --verbose count Print lots more stuff (repeat for more)

See the global flags page for global options not listed here.

-

SEE ALSO

+

SEE ALSO

rclone selfupdate

Update the rclone binary.

-

Synopsis

+

Synopsis

This command downloads the latest release of rclone and replaces the currently running binary. The download is verified with a hashsum and cryptographically signed signature; see the release signing docs for details.

If used without flags (or with implied --stable flag), this command will install the latest stable release. However, some issues may be fixed (or features added) only in the latest beta release. In such cases you should run the command with the --beta flag, i.e. rclone selfupdate --beta. You can check in advance what version would be installed by adding the --check flag, then repeat the command without it when you are satisfied.

Sometimes the rclone team may recommend you a concrete beta or stable rclone release to troubleshoot your issue or add a bleeding edge feature. The --version VER flag, if given, will update to the concrete version instead of the latest one. If you omit micro version from VER (for example 1.53), the latest matching micro version will be used.

@@ -3585,7 +3920,7 @@ htpasswd -B htpasswd anotherUser

Note: Windows forbids deletion of a currently running executable so this command will rename the old executable to 'rclone.old.exe' upon success.

Please note that this command was not available before rclone version 1.55. If it fails for you with the message unknown command "selfupdate" then you will need to update manually following the install instructions located at https://rclone.org/install/

rclone selfupdate [flags]
-

Options

+

Options

      --beta             Install beta release
       --check            Check for latest release, do not download
   -h, --help             help for selfupdate
@@ -3594,21 +3929,21 @@ htpasswd -B htpasswd anotherUser
--stable Install stable release (this is the default) --version string Install the given rclone version (default: latest)

See the global flags page for global options not listed here.

-

SEE ALSO

+

SEE ALSO

rclone serve

Serve a remote over a protocol.

-

Synopsis

+

Synopsis

Serve a remote over a given protocol. Requires the use of a subcommand to specify the protocol, e.g.

rclone serve http remote:

Each subcommand has its own options which you can see in their help.

rclone serve <protocol> [opts] <remote> [flags]
-

Options

+

Options

  -h, --help   help for serve

See the global flags page for global options not listed here.

-

SEE ALSO

+

SEE ALSO

rclone serve dlna

Serve remote:path over DLNA

-

Synopsis

+

Synopsis

Run a DLNA media server for media stored in an rclone remote. Many devices, such as the Xbox and PlayStation, can automatically discover this server in the LAN and play audio/video from it. VLC is also supported. Service discovery uses UDP multicast packets (SSDP) and will thus only work on LANs.

Rclone will list all files present in the remote, without filtering based on media formats or file extensions. Additionally, there is no media transcoding support. This means that some players might show files that they are not able to play back correctly.

Server options

@@ -3633,196 +3968,6 @@ htpasswd -B htpasswd anotherUser

This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.

Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.

The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.

-

VFS Directory Cache

-

Using the --dir-cache-time flag, you can control how long a directory should be considered up to date and not refreshed from the backend. Changes made through the VFS will appear immediately or invalidate the cache.

-
--dir-cache-time duration   Time to cache directory entries for (default 5m0s)
---poll-interval duration    Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
-

However, changes made directly on the cloud storage by the web interface or a different copy of rclone will only be picked up once the directory cache expires if the backend configured does not support polling for changes. If the backend supports polling, changes will be picked up within the polling interval.

-

You can send a SIGHUP signal to rclone for it to flush all directory caches, regardless of how old they are. Assuming only one rclone instance is running, you can reset the cache like this:

-
kill -SIGHUP $(pidof rclone)
-

If you configure rclone with a remote control then you can use rclone rc to flush the whole directory cache:

-
rclone rc vfs/forget
-

Or individual files or directories:

-
rclone rc vfs/forget file=path/to/file dir=path/to/dir
-

VFS File Buffering

-

The --buffer-size flag determines the amount of memory, that will be used to buffer data in advance.

-

Each open file will try to keep the specified amount of data in memory at all times. The buffered data is bound to one open file and won't be shared.

-

This flag is a upper limit for the used memory per open file. The buffer will only use memory for data that is downloaded but not not yet read. If the buffer is empty, only a small amount of memory will be used.

-

The maximum memory used by rclone for buffering can be up to --buffer-size * open files.

-

VFS File Caching

-

These flags control the VFS file caching options. File caching is necessary to make the VFS layer appear compatible with a normal file system. It can be disabled at the cost of some compatibility.

-

For example you'll need to enable VFS caching if you want to read and write simultaneously to a file. See below for more details.

-

Note that the VFS cache is separate from the cache backend and you may find that you need one or the other or both.

-
--cache-dir string                     Directory rclone will use for caching.
---vfs-cache-mode CacheMode             Cache mode off|minimal|writes|full (default off)
---vfs-cache-max-age duration           Max time since last access of objects in the cache (default 1h0m0s)
---vfs-cache-max-size SizeSuffix        Max total size of objects in the cache (default off)
---vfs-cache-min-free-space SizeSuffix  Target minimum free space on the disk containing the cache (default off)
---vfs-cache-poll-interval duration     Interval to poll the cache for stale objects (default 1m0s)
---vfs-write-back duration              Time to writeback files after last use when using cache (default 5s)
-

If run with -vv rclone will print the location of the file cache. The files are stored in the user cache file area which is OS dependent but can be controlled with --cache-dir or setting the appropriate environment variable.

-

The cache has 4 different modes selected by --vfs-cache-mode. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.

-

Note that files are written back to the remote only when they are closed and if they haven't been accessed for --vfs-write-back seconds. If rclone is quit or dies with files that haven't been uploaded, these will be uploaded next time rclone is run with the same flags.

-

If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the cache may exceed these quotas for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval. Secondly because open files cannot be evicted from the cache. When --vfs-cache-max-size or --vfs-cache-min-free-size is exceeded, rclone will attempt to evict the least accessed files from the cache first. rclone will start with files that haven't been accessed for the longest. This cache flushing strategy is efficient and more relevant files are likely to remain cached.

-

The --vfs-cache-max-age will evict files from the cache after the set time since last access has passed. The default value of 1 hour will start evicting files from cache that haven't been accessed for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0 and will wait for 1 more hour before evicting. Specify the time with standard notation, s, m, h, d, w .

-

You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir. You don't need to worry about this if the remotes in use don't overlap.

-

--vfs-cache-mode off

-

In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.

-

This will mean some operations are not possible

- -

--vfs-cache-mode minimal

-

This is very similar to "off" except that files opened for read AND write will be buffered to disk. This means that files opened for write will be a lot more compatible, but uses the minimal disk space.

-

These operations are not possible

- -

--vfs-cache-mode writes

-

In this mode files opened for read only are still read directly from the remote, write only and read/write files are buffered to disk first.

-

This mode should support all normal file system operations.

-

If an upload fails it will be retried at exponentially increasing intervals up to 1 minute.

-

--vfs-cache-mode full

-

In this mode all reads and writes are buffered to and from disk. When data is read from the remote this is buffered to disk as well.

-

In this mode the files in the cache will be sparse files and rclone will keep track of which bits of the files it has downloaded.

-

So if an application only reads the starts of each file, then rclone will only buffer the start of the file. These files will appear to be their full size in the cache, but they will be sparse files with only the data that has been downloaded present in them.

-

This mode should support all normal file system operations and is otherwise identical to --vfs-cache-mode writes.

-

When reading a file rclone will read --buffer-size plus --vfs-read-ahead bytes ahead. The --buffer-size is buffered in memory whereas the --vfs-read-ahead is buffered on disk.

-

When using this mode it is recommended that --buffer-size is not set too large and --vfs-read-ahead is set large if required.

-

IMPORTANT not all file systems support sparse files. In particular FAT/exFAT do not. Rclone will perform very badly if the cache directory is on a filesystem which doesn't support sparse files and it will log an ERROR message if one is detected.

-

Fingerprinting

-

Various parts of the VFS use fingerprinting to see if a local file copy has changed relative to a remote file. Fingerprints are made from:

- -

where available on an object.

-

On some backends some of these attributes are slow to read (they take an extra API call per object, or extra work per object).

-

For example hash is slow with the local and sftp backends as they have to read the entire file and hash it, and modtime is slow with the s3, swift, ftp and qinqstor backends because they need to do an extra API call to fetch it.

-

If you use the --vfs-fast-fingerprint flag then rclone will not include the slow operations in the fingerprint. This makes the fingerprinting less accurate but much faster and will improve the opening time of cached files.

-

If you are running a vfs cache over local, s3 or swift backends then using this flag is recommended.

-

Note that if you change the value of this flag, the fingerprints of the files in the cache may be invalidated and the files will need to be downloaded again.

-

VFS Chunked Reading

-

When rclone reads files from a remote it reads them in chunks. This means that rather than requesting the whole file rclone reads the chunk specified. This can reduce the used download quota for some remotes by requesting only chunks from the remote that are actually read, at the cost of an increased number of requests.

-

These flags control the chunking:

-
--vfs-read-chunk-size SizeSuffix        Read the source objects in chunks (default 128M)
---vfs-read-chunk-size-limit SizeSuffix  Max chunk doubling size (default off)
-

Rclone will start reading a chunk of size --vfs-read-chunk-size, and then double the size for each read. When --vfs-read-chunk-size-limit is specified, and greater than --vfs-read-chunk-size, the chunk size for each open file will get doubled only until the specified value is reached. If the value is "off", which is the default, the limit is disabled and the chunk size will grow indefinitely.

-

With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.

-

Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading.

-

VFS Performance

-

These flags may be used to enable/disable features of the VFS for performance or other reasons. See also the chunked reading feature.

-

In particular S3 and Swift benefit hugely from the --no-modtime flag (or use --use-server-modtime for a slightly different effect) as each read of the modification time takes a transaction.

-
--no-checksum     Don't compare checksums on up/download.
---no-modtime      Don't read/write the modification time (can speed things up).
---no-seek         Don't allow seeking in files.
---read-only       Only allow read-only access.
-

Sometimes rclone is delivered reads or writes out of order. Rather than seeking rclone will wait a short time for the in sequence read or write to come in. These flags only come into effect when not using an on disk cache file.

-
--vfs-read-wait duration   Time to wait for in-sequence read before seeking (default 20ms)
---vfs-write-wait duration  Time to wait for in-sequence write before giving error (default 1s)
-

When using VFS write caching (--vfs-cache-mode with value writes or full), the global flag --transfers can be set to adjust the number of parallel uploads of modified files from the cache (the related global flag --checkers has no effect on the VFS).

-
--transfers int  Number of file transfers to run in parallel (default 4)
-

VFS Case Sensitivity

-

Linux file systems are case-sensitive: two files can differ only by case, and the exact case must be used when opening a file.

-

File systems in modern Windows are case-insensitive but case-preserving: although existing files can be opened using any case, the exact case used to create the file is preserved and available for programs to query. It is not allowed for two files in the same directory to differ only by case.

-

Usually file systems on macOS are case-insensitive. It is possible to make macOS file systems case-sensitive but that is not the default.

-

The --vfs-case-insensitive VFS flag controls how rclone handles these two cases. If its value is "false", rclone passes file names to the remote as-is. If the flag is "true" (or appears without a value on the command line), rclone may perform a "fixup" as explained below.

-

The user may specify a file name to open/delete/rename/etc with a case different than what is stored on the remote. If an argument refers to an existing file with exactly the same name, then the case of the existing file on the disk will be used. However, if a file name with exactly the same name is not found but a name differing only by case exists, rclone will transparently fixup the name. This fixup happens only when an existing file is requested. Case sensitivity of file names created anew by rclone is controlled by the underlying remote.

-

Note that case sensitivity of the operating system running rclone (the target) may differ from case sensitivity of a file system presented by rclone (the source). The flag controls whether "fixup" is performed to satisfy the target.

-

If the flag is not provided on the command line, then its default value depends on the operating system where rclone runs: "true" on Windows and macOS, "false" otherwise. If the flag is provided without a value, then it is "true".

-

VFS Disk Options

-

This flag allows you to manually set the statistics about the filing system. It can be useful when those statistics cannot be read correctly automatically.

-
--vfs-disk-space-total-size    Manually set the total disk space size (example: 256G, default: -1)
-

Alternate report of used bytes

-

Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df on the filesystem, then pass the flag --vfs-used-is-size to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size and compute the total used space itself.

-

WARNING. Contrary to rclone size, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.

-
rclone serve dlna remote:path [flags]
-

Options

-
      --addr string                            The ip:port or :port to bind the DLNA http server to (default ":7879")
-      --announce-interval Duration             The interval between SSDP announcements (default 12m0s)
-      --dir-cache-time Duration                Time to cache directory entries for (default 5m0s)
-      --dir-perms FileMode                     Directory permissions (default 0777)
-      --file-perms FileMode                    File permissions (default 0666)
-      --gid uint32                             Override the gid field set by the filesystem (not supported on Windows) (default 1000)
-  -h, --help                                   help for dlna
-      --interface stringArray                  The interface to use for SSDP (repeat as necessary)
-      --log-trace                              Enable trace logging of SOAP traffic
-      --name string                            Name of DLNA server
-      --no-checksum                            Don't compare checksums on up/download
-      --no-modtime                             Don't read/write the modification time (can speed things up)
-      --no-seek                                Don't allow seeking in files
-      --poll-interval Duration                 Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
-      --read-only                              Only allow read-only access
-      --uid uint32                             Override the uid field set by the filesystem (not supported on Windows) (default 1000)
-      --umask int                              Override the permission bits set by the filesystem (not supported on Windows) (default 2)
-      --vfs-cache-max-age Duration             Max time since last access of objects in the cache (default 1h0m0s)
-      --vfs-cache-max-size SizeSuffix          Max total size of objects in the cache (default off)
-      --vfs-cache-min-free-space SizeSuffix    Target minimum free space on the disk containing the cache (default off)
-      --vfs-cache-mode CacheMode               Cache mode off|minimal|writes|full (default off)
-      --vfs-cache-poll-interval Duration       Interval to poll the cache for stale objects (default 1m0s)
-      --vfs-case-insensitive                   If a file name not found, find a case insensitive match
-      --vfs-disk-space-total-size SizeSuffix   Specify the total space of disk (default off)
-      --vfs-fast-fingerprint                   Use fast (less accurate) fingerprints for change detection
-      --vfs-read-ahead SizeSuffix              Extra read ahead over --buffer-size when using cache-mode full
-      --vfs-read-chunk-size SizeSuffix         Read the source objects in chunks (default 128Mi)
-      --vfs-read-chunk-size-limit SizeSuffix   If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
-      --vfs-read-wait Duration                 Time to wait for in-sequence read before seeking (default 20ms)
-      --vfs-refresh                            Refreshes the directory cache recursively on start
-      --vfs-used-is-size rclone size           Use the rclone size algorithm for Used size
-      --vfs-write-back Duration                Time to writeback files after last use when using cache (default 5s)
-      --vfs-write-wait Duration                Time to wait for in-sequence write before giving error (default 1s)
-

Filter Options

-

Flags for filtering directory listings.

-
      --delete-excluded                     Delete files on dest excluded from sync
-      --exclude stringArray                 Exclude files matching pattern
-      --exclude-from stringArray            Read file exclude patterns from file (use - to read from stdin)
-      --exclude-if-present stringArray      Exclude directories if filename is present
-      --files-from stringArray              Read list of source-file names from file (use - to read from stdin)
-      --files-from-raw stringArray          Read list of source-file names from file without any processing of lines (use - to read from stdin)
-  -f, --filter stringArray                  Add a file filtering rule
-      --filter-from stringArray             Read file filtering patterns from a file (use - to read from stdin)
-      --ignore-case                         Ignore case in filters (case insensitive)
-      --include stringArray                 Include files matching pattern
-      --include-from stringArray            Read file include patterns from file (use - to read from stdin)
-      --max-age Duration                    Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
-      --max-depth int                       If set limits the recursion depth to this (default -1)
-      --max-size SizeSuffix                 Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
-      --metadata-exclude stringArray        Exclude metadatas matching pattern
-      --metadata-exclude-from stringArray   Read metadata exclude patterns from file (use - to read from stdin)
-      --metadata-filter stringArray         Add a metadata filtering rule
-      --metadata-filter-from stringArray    Read metadata filtering patterns from a file (use - to read from stdin)
-      --metadata-include stringArray        Include metadatas matching pattern
-      --metadata-include-from stringArray   Read metadata include patterns from file (use - to read from stdin)
-      --min-age Duration                    Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
-      --min-size SizeSuffix                 Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
-

See the global flags page for global options not listed here.

-

SEE ALSO

- -

rclone serve docker

-

Serve any remote on docker's volume plugin API.

-

Synopsis

-

This command implements the Docker volume plugin API allowing docker to use rclone as a data storage mechanism for various cloud providers. rclone provides docker volume plugin based on it.

-

To create a docker plugin, one must create a Unix or TCP socket that Docker will look for when you use the plugin and then it listens for commands from docker daemon and runs the corresponding code when necessary. Docker plugins can run as a managed plugin under control of the docker daemon or as an independent native service. For testing, you can just run it directly from the command line, for example:

-
sudo rclone serve docker --base-dir /tmp/rclone-volumes --socket-addr localhost:8787 -vv
-

Running rclone serve docker will create the said socket, listening for commands from Docker to create the necessary Volumes. Normally you need not give the --socket-addr flag. The API will listen on the unix domain socket at /run/docker/plugins/rclone.sock. In the example above rclone will create a TCP socket and a small file /etc/docker/plugins/rclone.spec containing the socket address. We use sudo because both paths are writeable only by the root user.

-

If you later decide to change listening socket, the docker daemon must be restarted to reconnect to /run/docker/plugins/rclone.sock or parse new /etc/docker/plugins/rclone.spec. Until you restart, any volume related docker commands will timeout trying to access the old socket. Running directly is supported on Linux only, not on Windows or MacOS. This is not a problem with managed plugin mode described in details in the full documentation.

-

The command will create volume mounts under the path given by --base-dir (by default /var/lib/docker-volumes/rclone available only to root) and maintain the JSON formatted file docker-plugin.state in the rclone cache directory with book-keeping records of created and mounted volumes.

-

All mount and VFS options are submitted by the docker daemon via API, but you can also provide defaults on the command line as well as set path to the config file and cache directory or adjust logging verbosity. ## VFS - Virtual File System

-

This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.

-

Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.

-

The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.

VFS Directory Cache

Using the --dir-cache-time flag, you can control how long a directory should be considered up to date and not refreshed from the backend. Changes made through the VFS will appear immediately or invalidate the cache.

--dir-cache-time duration   Time to cache directory entries for (default 5m0s)
@@ -3936,41 +4081,23 @@ htpasswd -B htpasswd anotherUser

Alternate report of used bytes

Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df on the filesystem, then pass the flag --vfs-used-is-size to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size and compute the total used space itself.

WARNING. Contrary to rclone size, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.

-
rclone serve docker [flags]
+
rclone serve dlna remote:path [flags]

Options

-
      --allow-non-empty                        Allow mounting over a non-empty directory (not supported on Windows)
-      --allow-other                            Allow access to other users (not supported on Windows)
-      --allow-root                             Allow access to root user (not supported on Windows)
-      --async-read                             Use asynchronous reads (not supported on Windows) (default true)
-      --attr-timeout Duration                  Time for which file/directory attributes are cached (default 1s)
-      --base-dir string                        Base directory for volumes (default "/var/lib/docker-volumes/rclone")
-      --daemon                                 Run mount in background and exit parent process (as background output is suppressed, use --log-file with --log-format=pid,... to monitor) (not supported on Windows)
-      --daemon-timeout Duration                Time limit for rclone to respond to kernel (not supported on Windows) (default 0s)
-      --daemon-wait Duration                   Time to wait for ready mount from daemon (maximum time on Linux, constant sleep time on OSX/BSD) (not supported on Windows) (default 1m0s)
-      --debug-fuse                             Debug the FUSE internals - needs -v
-      --default-permissions                    Makes kernel enforce access control based on the file mode (not supported on Windows)
-      --devname string                         Set the device name - default is remote:path
+
      --addr string                            The ip:port or :port to bind the DLNA http server to (default ":7879")
+      --announce-interval Duration             The interval between SSDP announcements (default 12m0s)
       --dir-cache-time Duration                Time to cache directory entries for (default 5m0s)
       --dir-perms FileMode                     Directory permissions (default 0777)
       --file-perms FileMode                    File permissions (default 0666)
-      --forget-state                           Skip restoring previous state
-      --fuse-flag stringArray                  Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required)
       --gid uint32                             Override the gid field set by the filesystem (not supported on Windows) (default 1000)
-  -h, --help                                   help for docker
-      --max-read-ahead SizeSuffix              The number of bytes that can be prefetched for sequential reads (not supported on Windows) (default 128Ki)
-      --mount-case-insensitive Tristate        Tell the OS the mount is case insensitive (true) or sensitive (false) regardless of the backend (auto) (default unset)
-      --network-mode                           Mount as remote network drive, instead of fixed disk drive (supported on Windows only)
+  -h, --help                                   help for dlna
+      --interface stringArray                  The interface to use for SSDP (repeat as necessary)
+      --log-trace                              Enable trace logging of SOAP traffic
+      --name string                            Name of DLNA server
       --no-checksum                            Don't compare checksums on up/download
       --no-modtime                             Don't read/write the modification time (can speed things up)
       --no-seek                                Don't allow seeking in files
-      --no-spec                                Do not write spec file
-      --noappledouble                          Ignore Apple Double (._) and .DS_Store files (supported on OSX only) (default true)
-      --noapplexattr                           Ignore all "com.apple.*" extended attributes (supported on OSX only)
-  -o, --option stringArray                     Option for libfuse/WinFsp (repeat if required)
       --poll-interval Duration                 Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
       --read-only                              Only allow read-only access
-      --socket-addr string                     Address <host:port> or absolute path (default: /run/docker/plugins/rclone.sock)
-      --socket-gid int                         GID for unix socket (default: current process GID) (default 1000)
       --uid uint32                             Override the uid field set by the filesystem (not supported on Windows) (default 1000)
       --umask int                              Override the permission bits set by the filesystem (not supported on Windows) (default 2)
       --vfs-cache-max-age Duration             Max time since last access of objects in the cache (default 1h0m0s)
@@ -3985,12 +4112,10 @@ htpasswd -B htpasswd anotherUser

       --vfs-read-chunk-size SizeSuffix         Read the source objects in chunks (default 128Mi)
       --vfs-read-chunk-size-limit SizeSuffix   If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
       --vfs-read-wait Duration                 Time to wait for in-sequence read before seeking (default 20ms)
-      --vfs-refresh                            Refreshes the directory cache recursively on start
+      --vfs-refresh                            Refreshes the directory cache recursively in the background on start
       --vfs-used-is-size rclone size           Use the rclone size algorithm for Used size
       --vfs-write-back Duration                Time to writeback files after last use when using cache (default 5s)
-      --vfs-write-wait Duration                Time to wait for in-sequence write before giving error (default 1s)
-      --volname string                         Set the volume name (supported on Windows and OSX only)
-      --write-back-cache                       Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows)
+ --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)

Filter Options

Flags for filtering directory listings.

      --delete-excluded                     Delete files on dest excluded from sync
@@ -4020,16 +4145,16 @@ htpasswd -B htpasswd anotherUser
-

rclone serve ftp

-

Serve remote:path over FTP.

+

rclone serve docker

+

Serve any remote on docker's volume plugin API.

Synopsis

-

Run a basic FTP server to serve a remote over FTP protocol. This can be viewed with a FTP client or you can make a remote of type FTP to read and write it.

-

Server options

-

Use --addr to specify which IP address and port the server should listen on, e.g. --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. By default it only listens on localhost. You can use port :0 to let the OS choose an available port.

-

If you set --addr to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.

-

Authentication

-

By default this will serve files without needing a login.

-

You can set a single username and password with the --user and --pass flags. ## VFS - Virtual File System

+

This command implements the Docker volume plugin API allowing docker to use rclone as a data storage mechanism for various cloud providers. rclone provides docker volume plugin based on it.

+

To create a docker plugin, one must create a Unix or TCP socket that Docker will look for when you use the plugin and then it listens for commands from docker daemon and runs the corresponding code when necessary. Docker plugins can run as a managed plugin under control of the docker daemon or as an independent native service. For testing, you can just run it directly from the command line, for example:

+
sudo rclone serve docker --base-dir /tmp/rclone-volumes --socket-addr localhost:8787 -vv
+

Running rclone serve docker will create the said socket, listening for commands from Docker to create the necessary Volumes. Normally you need not give the --socket-addr flag. The API will listen on the unix domain socket at /run/docker/plugins/rclone.sock. In the example above rclone will create a TCP socket and a small file /etc/docker/plugins/rclone.spec containing the socket address. We use sudo because both paths are writeable only by the root user.

+

If you later decide to change listening socket, the docker daemon must be restarted to reconnect to /run/docker/plugins/rclone.sock or parse new /etc/docker/plugins/rclone.spec. Until you restart, any volume related docker commands will timeout trying to access the old socket. Running directly is supported on Linux only, not on Windows or MacOS. This is not a problem with managed plugin mode described in details in the full documentation.

+

The command will create volume mounts under the path given by --base-dir (by default /var/lib/docker-volumes/rclone available only to root) and maintain the JSON formatted file docker-plugin.state in the rclone cache directory with book-keeping records of created and mounted volumes.

+

All mount and VFS options are submitted by the docker daemon via API, but you can also provide defaults on the command line as well as set path to the config file and cache directory or adjust logging verbosity. ## VFS - Virtual File System

This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.

Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.

The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.

@@ -4146,10 +4271,220 @@ htpasswd -B htpasswd anotherUser

Alternate report of used bytes

Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df on the filesystem, then pass the flag --vfs-used-is-size to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size and compute the total used space itself.

WARNING. Contrary to rclone size, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.

+
rclone serve docker [flags]
+

Options

+
      --allow-non-empty                        Allow mounting over a non-empty directory (not supported on Windows)
+      --allow-other                            Allow access to other users (not supported on Windows)
+      --allow-root                             Allow access to root user (not supported on Windows)
+      --async-read                             Use asynchronous reads (not supported on Windows) (default true)
+      --attr-timeout Duration                  Time for which file/directory attributes are cached (default 1s)
+      --base-dir string                        Base directory for volumes (default "/var/lib/docker-volumes/rclone")
+      --daemon                                 Run mount in background and exit parent process (as background output is suppressed, use --log-file with --log-format=pid,... to monitor) (not supported on Windows)
+      --daemon-timeout Duration                Time limit for rclone to respond to kernel (not supported on Windows) (default 0s)
+      --daemon-wait Duration                   Time to wait for ready mount from daemon (maximum time on Linux, constant sleep time on OSX/BSD) (not supported on Windows) (default 1m0s)
+      --debug-fuse                             Debug the FUSE internals - needs -v
+      --default-permissions                    Makes kernel enforce access control based on the file mode (not supported on Windows)
+      --devname string                         Set the device name - default is remote:path
+      --dir-cache-time Duration                Time to cache directory entries for (default 5m0s)
+      --dir-perms FileMode                     Directory permissions (default 0777)
+      --file-perms FileMode                    File permissions (default 0666)
+      --forget-state                           Skip restoring previous state
+      --fuse-flag stringArray                  Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required)
+      --gid uint32                             Override the gid field set by the filesystem (not supported on Windows) (default 1000)
+  -h, --help                                   help for docker
+      --max-read-ahead SizeSuffix              The number of bytes that can be prefetched for sequential reads (not supported on Windows) (default 128Ki)
+      --mount-case-insensitive Tristate        Tell the OS the mount is case insensitive (true) or sensitive (false) regardless of the backend (auto) (default unset)
+      --network-mode                           Mount as remote network drive, instead of fixed disk drive (supported on Windows only)
+      --no-checksum                            Don't compare checksums on up/download
+      --no-modtime                             Don't read/write the modification time (can speed things up)
+      --no-seek                                Don't allow seeking in files
+      --no-spec                                Do not write spec file
+      --noappledouble                          Ignore Apple Double (._) and .DS_Store files (supported on OSX only) (default true)
+      --noapplexattr                           Ignore all "com.apple.*" extended attributes (supported on OSX only)
+  -o, --option stringArray                     Option for libfuse/WinFsp (repeat if required)
+      --poll-interval Duration                 Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
+      --read-only                              Only allow read-only access
+      --socket-addr string                     Address <host:port> or absolute path (default: /run/docker/plugins/rclone.sock)
+      --socket-gid int                         GID for unix socket (default: current process GID) (default 1000)
+      --uid uint32                             Override the uid field set by the filesystem (not supported on Windows) (default 1000)
+      --umask int                              Override the permission bits set by the filesystem (not supported on Windows) (default 2)
+      --vfs-cache-max-age Duration             Max time since last access of objects in the cache (default 1h0m0s)
+      --vfs-cache-max-size SizeSuffix          Max total size of objects in the cache (default off)
+      --vfs-cache-min-free-space SizeSuffix    Target minimum free space on the disk containing the cache (default off)
+      --vfs-cache-mode CacheMode               Cache mode off|minimal|writes|full (default off)
+      --vfs-cache-poll-interval Duration       Interval to poll the cache for stale objects (default 1m0s)
+      --vfs-case-insensitive                   If a file name not found, find a case insensitive match
+      --vfs-disk-space-total-size SizeSuffix   Specify the total space of disk (default off)
+      --vfs-fast-fingerprint                   Use fast (less accurate) fingerprints for change detection
+      --vfs-read-ahead SizeSuffix              Extra read ahead over --buffer-size when using cache-mode full
+      --vfs-read-chunk-size SizeSuffix         Read the source objects in chunks (default 128Mi)
+      --vfs-read-chunk-size-limit SizeSuffix   If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
+      --vfs-read-wait Duration                 Time to wait for in-sequence read before seeking (default 20ms)
+      --vfs-refresh                            Refreshes the directory cache recursively in the background on start
+      --vfs-used-is-size rclone size           Use the rclone size algorithm for Used size
+      --vfs-write-back Duration                Time to writeback files after last use when using cache (default 5s)
+      --vfs-write-wait Duration                Time to wait for in-sequence write before giving error (default 1s)
+      --volname string                         Set the volume name (supported on Windows and OSX only)
+      --write-back-cache                       Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows)
+

Filter Options

+

Flags for filtering directory listings.

+
      --delete-excluded                     Delete files on dest excluded from sync
+      --exclude stringArray                 Exclude files matching pattern
+      --exclude-from stringArray            Read file exclude patterns from file (use - to read from stdin)
+      --exclude-if-present stringArray      Exclude directories if filename is present
+      --files-from stringArray              Read list of source-file names from file (use - to read from stdin)
+      --files-from-raw stringArray          Read list of source-file names from file without any processing of lines (use - to read from stdin)
+  -f, --filter stringArray                  Add a file filtering rule
+      --filter-from stringArray             Read file filtering patterns from a file (use - to read from stdin)
+      --ignore-case                         Ignore case in filters (case insensitive)
+      --include stringArray                 Include files matching pattern
+      --include-from stringArray            Read file include patterns from file (use - to read from stdin)
+      --max-age Duration                    Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+      --max-depth int                       If set limits the recursion depth to this (default -1)
+      --max-size SizeSuffix                 Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
+      --metadata-exclude stringArray        Exclude metadatas matching pattern
+      --metadata-exclude-from stringArray   Read metadata exclude patterns from file (use - to read from stdin)
+      --metadata-filter stringArray         Add a metadata filtering rule
+      --metadata-filter-from stringArray    Read metadata filtering patterns from a file (use - to read from stdin)
+      --metadata-include stringArray        Include metadatas matching pattern
+      --metadata-include-from stringArray   Read metadata include patterns from file (use - to read from stdin)
+      --min-age Duration                    Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+      --min-size SizeSuffix                 Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
+

See the global flags page for global options not listed here.

+

SEE ALSO

+ +

rclone serve ftp

+

Serve remote:path over FTP.

+

Synopsis

+

Run a basic FTP server to serve a remote over FTP protocol. This can be viewed with a FTP client or you can make a remote of type FTP to read and write it.

+

Server options

+

Use --addr to specify which IP address and port the server should listen on, e.g. --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. By default it only listens on localhost. You can use port :0 to let the OS choose an available port.

+

If you set --addr to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.

+

Authentication

+

By default this will serve files without needing a login.

+

You can set a single username and password with the --user and --pass flags. ## VFS - Virtual File System

+

This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.

+

Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.

+

The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.

+

VFS Directory Cache

+

Using the --dir-cache-time flag, you can control how long a directory should be considered up to date and not refreshed from the backend. Changes made through the VFS will appear immediately or invalidate the cache.

+
--dir-cache-time duration   Time to cache directory entries for (default 5m0s)
+--poll-interval duration    Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
+

However, changes made directly on the cloud storage by the web interface or a different copy of rclone will only be picked up once the directory cache expires if the backend configured does not support polling for changes. If the backend supports polling, changes will be picked up within the polling interval.

+

You can send a SIGHUP signal to rclone for it to flush all directory caches, regardless of how old they are. Assuming only one rclone instance is running, you can reset the cache like this:

+
kill -SIGHUP $(pidof rclone)
+

If you configure rclone with a remote control then you can use rclone rc to flush the whole directory cache:

+
rclone rc vfs/forget
+

Or individual files or directories:

+
rclone rc vfs/forget file=path/to/file dir=path/to/dir
+

VFS File Buffering

+

The --buffer-size flag determines the amount of memory, that will be used to buffer data in advance.

+

Each open file will try to keep the specified amount of data in memory at all times. The buffered data is bound to one open file and won't be shared.

+

This flag is a upper limit for the used memory per open file. The buffer will only use memory for data that is downloaded but not not yet read. If the buffer is empty, only a small amount of memory will be used.

+

The maximum memory used by rclone for buffering can be up to --buffer-size * open files.

+

VFS File Caching

+

These flags control the VFS file caching options. File caching is necessary to make the VFS layer appear compatible with a normal file system. It can be disabled at the cost of some compatibility.

+

For example you'll need to enable VFS caching if you want to read and write simultaneously to a file. See below for more details.

+

Note that the VFS cache is separate from the cache backend and you may find that you need one or the other or both.

+
--cache-dir string                     Directory rclone will use for caching.
+--vfs-cache-mode CacheMode             Cache mode off|minimal|writes|full (default off)
+--vfs-cache-max-age duration           Max time since last access of objects in the cache (default 1h0m0s)
+--vfs-cache-max-size SizeSuffix        Max total size of objects in the cache (default off)
+--vfs-cache-min-free-space SizeSuffix  Target minimum free space on the disk containing the cache (default off)
+--vfs-cache-poll-interval duration     Interval to poll the cache for stale objects (default 1m0s)
+--vfs-write-back duration              Time to writeback files after last use when using cache (default 5s)
+

If run with -vv rclone will print the location of the file cache. The files are stored in the user cache file area which is OS dependent but can be controlled with --cache-dir or setting the appropriate environment variable.

+

The cache has 4 different modes selected by --vfs-cache-mode. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.

+

Note that files are written back to the remote only when they are closed and if they haven't been accessed for --vfs-write-back seconds. If rclone is quit or dies with files that haven't been uploaded, these will be uploaded next time rclone is run with the same flags.

+

If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the cache may exceed these quotas for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval. Secondly because open files cannot be evicted from the cache. When --vfs-cache-max-size or --vfs-cache-min-free-size is exceeded, rclone will attempt to evict the least accessed files from the cache first. rclone will start with files that haven't been accessed for the longest. This cache flushing strategy is efficient and more relevant files are likely to remain cached.

+

The --vfs-cache-max-age will evict files from the cache after the set time since last access has passed. The default value of 1 hour will start evicting files from cache that haven't been accessed for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0 and will wait for 1 more hour before evicting. Specify the time with standard notation, s, m, h, d, w .

+

You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir. You don't need to worry about this if the remotes in use don't overlap.

+

--vfs-cache-mode off

+

In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.

+

This will mean some operations are not possible

+ +

--vfs-cache-mode minimal

+

This is very similar to "off" except that files opened for read AND write will be buffered to disk. This means that files opened for write will be a lot more compatible, but uses the minimal disk space.

+

These operations are not possible

+ +

--vfs-cache-mode writes

+

In this mode files opened for read only are still read directly from the remote, write only and read/write files are buffered to disk first.

+

This mode should support all normal file system operations.

+

If an upload fails it will be retried at exponentially increasing intervals up to 1 minute.

+

--vfs-cache-mode full

+

In this mode all reads and writes are buffered to and from disk. When data is read from the remote this is buffered to disk as well.

+

In this mode the files in the cache will be sparse files and rclone will keep track of which bits of the files it has downloaded.

+

So if an application only reads the starts of each file, then rclone will only buffer the start of the file. These files will appear to be their full size in the cache, but they will be sparse files with only the data that has been downloaded present in them.

+

This mode should support all normal file system operations and is otherwise identical to --vfs-cache-mode writes.

+

When reading a file rclone will read --buffer-size plus --vfs-read-ahead bytes ahead. The --buffer-size is buffered in memory whereas the --vfs-read-ahead is buffered on disk.

+

When using this mode it is recommended that --buffer-size is not set too large and --vfs-read-ahead is set large if required.

+

IMPORTANT not all file systems support sparse files. In particular FAT/exFAT do not. Rclone will perform very badly if the cache directory is on a filesystem which doesn't support sparse files and it will log an ERROR message if one is detected.

+

Fingerprinting

+

Various parts of the VFS use fingerprinting to see if a local file copy has changed relative to a remote file. Fingerprints are made from:

+ +

where available on an object.

+

On some backends some of these attributes are slow to read (they take an extra API call per object, or extra work per object).

+

For example hash is slow with the local and sftp backends as they have to read the entire file and hash it, and modtime is slow with the s3, swift, ftp and qinqstor backends because they need to do an extra API call to fetch it.

+

If you use the --vfs-fast-fingerprint flag then rclone will not include the slow operations in the fingerprint. This makes the fingerprinting less accurate but much faster and will improve the opening time of cached files.

+

If you are running a vfs cache over local, s3 or swift backends then using this flag is recommended.

+

Note that if you change the value of this flag, the fingerprints of the files in the cache may be invalidated and the files will need to be downloaded again.

+

VFS Chunked Reading

+

When rclone reads files from a remote it reads them in chunks. This means that rather than requesting the whole file rclone reads the chunk specified. This can reduce the used download quota for some remotes by requesting only chunks from the remote that are actually read, at the cost of an increased number of requests.

+

These flags control the chunking:

+
--vfs-read-chunk-size SizeSuffix        Read the source objects in chunks (default 128M)
+--vfs-read-chunk-size-limit SizeSuffix  Max chunk doubling size (default off)
+

Rclone will start reading a chunk of size --vfs-read-chunk-size, and then double the size for each read. When --vfs-read-chunk-size-limit is specified, and greater than --vfs-read-chunk-size, the chunk size for each open file will get doubled only until the specified value is reached. If the value is "off", which is the default, the limit is disabled and the chunk size will grow indefinitely.

+

With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.

+

Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading.

+

VFS Performance

+

These flags may be used to enable/disable features of the VFS for performance or other reasons. See also the chunked reading feature.

+

In particular S3 and Swift benefit hugely from the --no-modtime flag (or use --use-server-modtime for a slightly different effect) as each read of the modification time takes a transaction.

+
--no-checksum     Don't compare checksums on up/download.
+--no-modtime      Don't read/write the modification time (can speed things up).
+--no-seek         Don't allow seeking in files.
+--read-only       Only allow read-only access.
+

Sometimes rclone is delivered reads or writes out of order. Rather than seeking rclone will wait a short time for the in sequence read or write to come in. These flags only come into effect when not using an on disk cache file.

+
--vfs-read-wait duration   Time to wait for in-sequence read before seeking (default 20ms)
+--vfs-write-wait duration  Time to wait for in-sequence write before giving error (default 1s)
+

When using VFS write caching (--vfs-cache-mode with value writes or full), the global flag --transfers can be set to adjust the number of parallel uploads of modified files from the cache (the related global flag --checkers has no effect on the VFS).

+
--transfers int  Number of file transfers to run in parallel (default 4)
+

VFS Case Sensitivity

+

Linux file systems are case-sensitive: two files can differ only by case, and the exact case must be used when opening a file.

+

File systems in modern Windows are case-insensitive but case-preserving: although existing files can be opened using any case, the exact case used to create the file is preserved and available for programs to query. It is not allowed for two files in the same directory to differ only by case.

+

Usually file systems on macOS are case-insensitive. It is possible to make macOS file systems case-sensitive but that is not the default.

+

The --vfs-case-insensitive VFS flag controls how rclone handles these two cases. If its value is "false", rclone passes file names to the remote as-is. If the flag is "true" (or appears without a value on the command line), rclone may perform a "fixup" as explained below.

+

The user may specify a file name to open/delete/rename/etc with a case different than what is stored on the remote. If an argument refers to an existing file with exactly the same name, then the case of the existing file on the disk will be used. However, if a file name with exactly the same name is not found but a name differing only by case exists, rclone will transparently fixup the name. This fixup happens only when an existing file is requested. Case sensitivity of file names created anew by rclone is controlled by the underlying remote.

+

Note that case sensitivity of the operating system running rclone (the target) may differ from case sensitivity of a file system presented by rclone (the source). The flag controls whether "fixup" is performed to satisfy the target.

+

If the flag is not provided on the command line, then its default value depends on the operating system where rclone runs: "true" on Windows and macOS, "false" otherwise. If the flag is provided without a value, then it is "true".

+

VFS Disk Options

+

This flag allows you to manually set the statistics about the filing system. It can be useful when those statistics cannot be read correctly automatically.

+
--vfs-disk-space-total-size    Manually set the total disk space size (example: 256G, default: -1)
+

Alternate report of used bytes

+

Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df on the filesystem, then pass the flag --vfs-used-is-size to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size and compute the total used space itself.

+

WARNING. Contrary to rclone size, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.

Auth Proxy

If you supply the parameter --auth-proxy /path/to/program then rclone will use that program to generate backends on the fly which then are used to authenticate incoming requests. This uses a simple JSON based protocol with input on STDIN and output on STDOUT.

PLEASE NOTE: --auth-proxy and --authorized-keys cannot be used together, if --auth-proxy is set the authorized keys option will be ignored.

-

There is an example program bin/test_proxy.py in the rclone source code.

+

There is an example program bin/test_proxy.py in the rclone source code.

The program's job is to take a user and pass on the input and turn those into the config for a backend on STDOUT in JSON format. This config will have any default parameters for the backend added, but it won't use configuration from environment variables or command line options - it is the job of the proxy program to make a complete config.

This config generated must have this extra parameter - _root - root to use for the backend

And it may have this parameter - _obscure - comma separated strings for parameters to obscure

@@ -4177,7 +4512,7 @@ htpasswd -B htpasswd anotherUser

Note that an internal cache is keyed on user so only use that for configuration, don't use pass or public_key. This also means that if a user's password or public-key is changed the cache will need to expire (which takes 5 mins) before it takes effect.

This can be used to build general purpose proxies to any kind of backend that rclone supports.

rclone serve ftp remote:path [flags]
-

Options

+

Options

      --addr string                            IPaddress:Port or :Port to bind server to (default "localhost:2121")
       --auth-proxy string                      A program to use to create the backend from the auth
       --cert string                            TLS PEM key (concatenation of certificate and CA certificate)
@@ -4210,11 +4545,11 @@ htpasswd -B htpasswd anotherUser
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) -

Filter Options

+

Filter Options

Flags for filtering directory listings.

      --delete-excluded                     Delete files on dest excluded from sync
       --exclude stringArray                 Exclude files matching pattern
@@ -4239,13 +4574,13 @@ htpasswd -B htpasswd anotherUser
--min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)

See the global flags page for global options not listed here.

-

SEE ALSO

+

SEE ALSO

rclone serve http

Serve the remote over HTTP.

-

Synopsis

+

Synopsis

Run a basic web server to serve a remote over HTTP. This can be viewed in a web browser or you can make a remote of type http read from it.

You can use the filter flags (e.g. --include, --exclude) to control what is served.

The server will log errors. Use -v to see access logs.

@@ -4388,243 +4723,6 @@ htpasswd -B htpasswd anotherUser

This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.

Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.

The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.

-

VFS Directory Cache

-

Using the --dir-cache-time flag, you can control how long a directory should be considered up to date and not refreshed from the backend. Changes made through the VFS will appear immediately or invalidate the cache.

-
--dir-cache-time duration   Time to cache directory entries for (default 5m0s)
---poll-interval duration    Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
-

However, changes made directly on the cloud storage by the web interface or a different copy of rclone will only be picked up once the directory cache expires if the backend configured does not support polling for changes. If the backend supports polling, changes will be picked up within the polling interval.

-

You can send a SIGHUP signal to rclone for it to flush all directory caches, regardless of how old they are. Assuming only one rclone instance is running, you can reset the cache like this:

-
kill -SIGHUP $(pidof rclone)
-

If you configure rclone with a remote control then you can use rclone rc to flush the whole directory cache:

-
rclone rc vfs/forget
-

Or individual files or directories:

-
rclone rc vfs/forget file=path/to/file dir=path/to/dir
-

VFS File Buffering

-

The --buffer-size flag determines the amount of memory, that will be used to buffer data in advance.

-

Each open file will try to keep the specified amount of data in memory at all times. The buffered data is bound to one open file and won't be shared.

-

This flag is a upper limit for the used memory per open file. The buffer will only use memory for data that is downloaded but not not yet read. If the buffer is empty, only a small amount of memory will be used.

-

The maximum memory used by rclone for buffering can be up to --buffer-size * open files.

-

VFS File Caching

-

These flags control the VFS file caching options. File caching is necessary to make the VFS layer appear compatible with a normal file system. It can be disabled at the cost of some compatibility.

-

For example you'll need to enable VFS caching if you want to read and write simultaneously to a file. See below for more details.

-

Note that the VFS cache is separate from the cache backend and you may find that you need one or the other or both.

-
--cache-dir string                     Directory rclone will use for caching.
---vfs-cache-mode CacheMode             Cache mode off|minimal|writes|full (default off)
---vfs-cache-max-age duration           Max time since last access of objects in the cache (default 1h0m0s)
---vfs-cache-max-size SizeSuffix        Max total size of objects in the cache (default off)
---vfs-cache-min-free-space SizeSuffix  Target minimum free space on the disk containing the cache (default off)
---vfs-cache-poll-interval duration     Interval to poll the cache for stale objects (default 1m0s)
---vfs-write-back duration              Time to writeback files after last use when using cache (default 5s)
-

If run with -vv rclone will print the location of the file cache. The files are stored in the user cache file area which is OS dependent but can be controlled with --cache-dir or setting the appropriate environment variable.

-

The cache has 4 different modes selected by --vfs-cache-mode. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.

-

Note that files are written back to the remote only when they are closed and if they haven't been accessed for --vfs-write-back seconds. If rclone is quit or dies with files that haven't been uploaded, these will be uploaded next time rclone is run with the same flags.

-

If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the cache may exceed these quotas for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval. Secondly because open files cannot be evicted from the cache. When --vfs-cache-max-size or --vfs-cache-min-free-size is exceeded, rclone will attempt to evict the least accessed files from the cache first. rclone will start with files that haven't been accessed for the longest. This cache flushing strategy is efficient and more relevant files are likely to remain cached.

-

The --vfs-cache-max-age will evict files from the cache after the set time since last access has passed. The default value of 1 hour will start evicting files from cache that haven't been accessed for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0 and will wait for 1 more hour before evicting. Specify the time with standard notation, s, m, h, d, w .

-

You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir. You don't need to worry about this if the remotes in use don't overlap.

-

--vfs-cache-mode off

-

In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.

-

This will mean some operations are not possible

- -

--vfs-cache-mode minimal

-

This is very similar to "off" except that files opened for read AND write will be buffered to disk. This means that files opened for write will be a lot more compatible, but uses the minimal disk space.

-

These operations are not possible

- -

--vfs-cache-mode writes

-

In this mode files opened for read only are still read directly from the remote, write only and read/write files are buffered to disk first.

-

This mode should support all normal file system operations.

-

If an upload fails it will be retried at exponentially increasing intervals up to 1 minute.

-

--vfs-cache-mode full

-

In this mode all reads and writes are buffered to and from disk. When data is read from the remote this is buffered to disk as well.

-

In this mode the files in the cache will be sparse files and rclone will keep track of which bits of the files it has downloaded.

-

So if an application only reads the starts of each file, then rclone will only buffer the start of the file. These files will appear to be their full size in the cache, but they will be sparse files with only the data that has been downloaded present in them.

-

This mode should support all normal file system operations and is otherwise identical to --vfs-cache-mode writes.

-

When reading a file rclone will read --buffer-size plus --vfs-read-ahead bytes ahead. The --buffer-size is buffered in memory whereas the --vfs-read-ahead is buffered on disk.

-

When using this mode it is recommended that --buffer-size is not set too large and --vfs-read-ahead is set large if required.

-

IMPORTANT not all file systems support sparse files. In particular FAT/exFAT do not. Rclone will perform very badly if the cache directory is on a filesystem which doesn't support sparse files and it will log an ERROR message if one is detected.

-

Fingerprinting

-

Various parts of the VFS use fingerprinting to see if a local file copy has changed relative to a remote file. Fingerprints are made from:

- -

where available on an object.

-

On some backends some of these attributes are slow to read (they take an extra API call per object, or extra work per object).

-

For example hash is slow with the local and sftp backends as they have to read the entire file and hash it, and modtime is slow with the s3, swift, ftp and qinqstor backends because they need to do an extra API call to fetch it.

-

If you use the --vfs-fast-fingerprint flag then rclone will not include the slow operations in the fingerprint. This makes the fingerprinting less accurate but much faster and will improve the opening time of cached files.

-

If you are running a vfs cache over local, s3 or swift backends then using this flag is recommended.

-

Note that if you change the value of this flag, the fingerprints of the files in the cache may be invalidated and the files will need to be downloaded again.

-

VFS Chunked Reading

-

When rclone reads files from a remote it reads them in chunks. This means that rather than requesting the whole file rclone reads the chunk specified. This can reduce the used download quota for some remotes by requesting only chunks from the remote that are actually read, at the cost of an increased number of requests.

-

These flags control the chunking:

-
--vfs-read-chunk-size SizeSuffix        Read the source objects in chunks (default 128M)
---vfs-read-chunk-size-limit SizeSuffix  Max chunk doubling size (default off)
-

Rclone will start reading a chunk of size --vfs-read-chunk-size, and then double the size for each read. When --vfs-read-chunk-size-limit is specified, and greater than --vfs-read-chunk-size, the chunk size for each open file will get doubled only until the specified value is reached. If the value is "off", which is the default, the limit is disabled and the chunk size will grow indefinitely.

-

With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.

-

Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading.

-

VFS Performance

-

These flags may be used to enable/disable features of the VFS for performance or other reasons. See also the chunked reading feature.

-

In particular S3 and Swift benefit hugely from the --no-modtime flag (or use --use-server-modtime for a slightly different effect) as each read of the modification time takes a transaction.

-
--no-checksum     Don't compare checksums on up/download.
---no-modtime      Don't read/write the modification time (can speed things up).
---no-seek         Don't allow seeking in files.
---read-only       Only allow read-only access.
-

Sometimes rclone is delivered reads or writes out of order. Rather than seeking rclone will wait a short time for the in sequence read or write to come in. These flags only come into effect when not using an on disk cache file.

-
--vfs-read-wait duration   Time to wait for in-sequence read before seeking (default 20ms)
---vfs-write-wait duration  Time to wait for in-sequence write before giving error (default 1s)
-

When using VFS write caching (--vfs-cache-mode with value writes or full), the global flag --transfers can be set to adjust the number of parallel uploads of modified files from the cache (the related global flag --checkers has no effect on the VFS).

-
--transfers int  Number of file transfers to run in parallel (default 4)
-

VFS Case Sensitivity

-

Linux file systems are case-sensitive: two files can differ only by case, and the exact case must be used when opening a file.

-

File systems in modern Windows are case-insensitive but case-preserving: although existing files can be opened using any case, the exact case used to create the file is preserved and available for programs to query. It is not allowed for two files in the same directory to differ only by case.

-

Usually file systems on macOS are case-insensitive. It is possible to make macOS file systems case-sensitive but that is not the default.

-

The --vfs-case-insensitive VFS flag controls how rclone handles these two cases. If its value is "false", rclone passes file names to the remote as-is. If the flag is "true" (or appears without a value on the command line), rclone may perform a "fixup" as explained below.

-

The user may specify a file name to open/delete/rename/etc with a case different than what is stored on the remote. If an argument refers to an existing file with exactly the same name, then the case of the existing file on the disk will be used. However, if a file name with exactly the same name is not found but a name differing only by case exists, rclone will transparently fixup the name. This fixup happens only when an existing file is requested. Case sensitivity of file names created anew by rclone is controlled by the underlying remote.

-

Note that case sensitivity of the operating system running rclone (the target) may differ from case sensitivity of a file system presented by rclone (the source). The flag controls whether "fixup" is performed to satisfy the target.

-

If the flag is not provided on the command line, then its default value depends on the operating system where rclone runs: "true" on Windows and macOS, "false" otherwise. If the flag is provided without a value, then it is "true".

-

VFS Disk Options

-

This flag allows you to manually set the statistics about the filing system. It can be useful when those statistics cannot be read correctly automatically.

-
--vfs-disk-space-total-size    Manually set the total disk space size (example: 256G, default: -1)
-

Alternate report of used bytes

-

Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df on the filesystem, then pass the flag --vfs-used-is-size to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size and compute the total used space itself.

-

WARNING. Contrary to rclone size, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.

-

Auth Proxy

-

If you supply the parameter --auth-proxy /path/to/program then rclone will use that program to generate backends on the fly which then are used to authenticate incoming requests. This uses a simple JSON based protocol with input on STDIN and output on STDOUT.

-

PLEASE NOTE: --auth-proxy and --authorized-keys cannot be used together, if --auth-proxy is set the authorized keys option will be ignored.

-

There is an example program bin/test_proxy.py in the rclone source code.

-

The program's job is to take a user and pass on the input and turn those into the config for a backend on STDOUT in JSON format. This config will have any default parameters for the backend added, but it won't use configuration from environment variables or command line options - it is the job of the proxy program to make a complete config.

-

This config generated must have this extra parameter - _root - root to use for the backend

-

And it may have this parameter - _obscure - comma separated strings for parameters to obscure

-

If password authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:

-
{
-    "user": "me",
-    "pass": "mypassword"
-}
-

If public-key authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:

-
{
-    "user": "me",
-    "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
-}
-

And as an example return this on STDOUT

-
{
-    "type": "sftp",
-    "_root": "",
-    "_obscure": "pass",
-    "user": "me",
-    "pass": "mypassword",
-    "host": "sftp.example.com"
-}
-

This would mean that an SFTP backend would be created on the fly for the user and pass/public_key returned in the output to the host given. Note that since _obscure is set to pass, rclone will obscure the pass parameter before creating the backend (which is required for sftp backends).

-

The program can manipulate the supplied user in any way, for example to make proxy to many different sftp backends, you could make the user be user@example.com and then set the host to example.com in the output and the user to user. For security you'd probably want to restrict the host to a limited list.

-

Note that an internal cache is keyed on user so only use that for configuration, don't use pass or public_key. This also means that if a user's password or public-key is changed the cache will need to expire (which takes 5 mins) before it takes effect.

-

This can be used to build general purpose proxies to any kind of backend that rclone supports.

-
rclone serve http remote:path [flags]
-

Options

-
      --addr stringArray                       IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
-      --allow-origin string                    Origin which cross-domain request (CORS) can be executed from
-      --auth-proxy string                      A program to use to create the backend from the auth
-      --baseurl string                         Prefix for URLs - leave blank for root
-      --cert string                            TLS PEM key (concatenation of certificate and CA certificate)
-      --client-ca string                       Client certificate authority to verify clients with
-      --dir-cache-time Duration                Time to cache directory entries for (default 5m0s)
-      --dir-perms FileMode                     Directory permissions (default 0777)
-      --file-perms FileMode                    File permissions (default 0666)
-      --gid uint32                             Override the gid field set by the filesystem (not supported on Windows) (default 1000)
-  -h, --help                                   help for http
-      --htpasswd string                        A htpasswd file - if not provided no authentication is done
-      --key string                             TLS PEM Private key
-      --max-header-bytes int                   Maximum size of request header (default 4096)
-      --min-tls-version string                 Minimum TLS version that is acceptable (default "tls1.0")
-      --no-checksum                            Don't compare checksums on up/download
-      --no-modtime                             Don't read/write the modification time (can speed things up)
-      --no-seek                                Don't allow seeking in files
-      --pass string                            Password for authentication
-      --poll-interval Duration                 Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
-      --read-only                              Only allow read-only access
-      --realm string                           Realm for authentication
-      --salt string                            Password hashing salt (default "dlPL2MqE")
-      --server-read-timeout Duration           Timeout for server reading data (default 1h0m0s)
-      --server-write-timeout Duration          Timeout for server writing data (default 1h0m0s)
-      --template string                        User-specified template
-      --uid uint32                             Override the uid field set by the filesystem (not supported on Windows) (default 1000)
-      --umask int                              Override the permission bits set by the filesystem (not supported on Windows) (default 2)
-      --user string                            User name for authentication
-      --vfs-cache-max-age Duration             Max time since last access of objects in the cache (default 1h0m0s)
-      --vfs-cache-max-size SizeSuffix          Max total size of objects in the cache (default off)
-      --vfs-cache-min-free-space SizeSuffix    Target minimum free space on the disk containing the cache (default off)
-      --vfs-cache-mode CacheMode               Cache mode off|minimal|writes|full (default off)
-      --vfs-cache-poll-interval Duration       Interval to poll the cache for stale objects (default 1m0s)
-      --vfs-case-insensitive                   If a file name not found, find a case insensitive match
-      --vfs-disk-space-total-size SizeSuffix   Specify the total space of disk (default off)
-      --vfs-fast-fingerprint                   Use fast (less accurate) fingerprints for change detection
-      --vfs-read-ahead SizeSuffix              Extra read ahead over --buffer-size when using cache-mode full
-      --vfs-read-chunk-size SizeSuffix         Read the source objects in chunks (default 128Mi)
-      --vfs-read-chunk-size-limit SizeSuffix   If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
-      --vfs-read-wait Duration                 Time to wait for in-sequence read before seeking (default 20ms)
-      --vfs-refresh                            Refreshes the directory cache recursively on start
-      --vfs-used-is-size rclone size           Use the rclone size algorithm for Used size
-      --vfs-write-back Duration                Time to writeback files after last use when using cache (default 5s)
-      --vfs-write-wait Duration                Time to wait for in-sequence write before giving error (default 1s)
-

Filter Options

-

Flags for filtering directory listings.

-
      --delete-excluded                     Delete files on dest excluded from sync
-      --exclude stringArray                 Exclude files matching pattern
-      --exclude-from stringArray            Read file exclude patterns from file (use - to read from stdin)
-      --exclude-if-present stringArray      Exclude directories if filename is present
-      --files-from stringArray              Read list of source-file names from file (use - to read from stdin)
-      --files-from-raw stringArray          Read list of source-file names from file without any processing of lines (use - to read from stdin)
-  -f, --filter stringArray                  Add a file filtering rule
-      --filter-from stringArray             Read file filtering patterns from a file (use - to read from stdin)
-      --ignore-case                         Ignore case in filters (case insensitive)
-      --include stringArray                 Include files matching pattern
-      --include-from stringArray            Read file include patterns from file (use - to read from stdin)
-      --max-age Duration                    Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
-      --max-depth int                       If set limits the recursion depth to this (default -1)
-      --max-size SizeSuffix                 Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
-      --metadata-exclude stringArray        Exclude metadatas matching pattern
-      --metadata-exclude-from stringArray   Read metadata exclude patterns from file (use - to read from stdin)
-      --metadata-filter stringArray         Add a metadata filtering rule
-      --metadata-filter-from stringArray    Read metadata filtering patterns from a file (use - to read from stdin)
-      --metadata-include stringArray        Include metadatas matching pattern
-      --metadata-include-from stringArray   Read metadata include patterns from file (use - to read from stdin)
-      --min-age Duration                    Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
-      --min-size SizeSuffix                 Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
-

See the global flags page for global options not listed here.

-

SEE ALSO

- -

rclone serve nfs

-

Serve the remote as an NFS mount

-

Synopsis

-

Create an NFS server that serves the given remote over the network.

-

The primary purpose for this command is to enable mount command on recent macOS versions where installing FUSE is very cumbersome.

-

Since this is running on NFSv3, no authentication method is available. Any client will be able to access the data. To limit access, you can use serve NFS on loopback address and rely on secure tunnels (such as SSH). For this reason, by default, a random TCP port is chosen and loopback interface is used for the listening address; meaning that it is only available to the local machine. If you want other machines to access the NFS mount over local network, you need to specify the listening address and port using --addr flag.

-

Modifying files through NFS protocol requires VFS caching. Usually you will need to specify --vfs-cache-mode in order to be able to write to the mountpoint (full is recommended). If you don't specify VFS cache mode, the mount will be read-only.

-

To serve NFS over the network use following command:

-
rclone serve nfs remote: --addr 0.0.0.0:$PORT --vfs-cache-mode=full
-

We specify a specific port that we can use in the mount command:

-

To mount the server under Linux/macOS, use the following command:

-
mount -oport=$PORT,mountport=$PORT $HOSTNAME: path/to/mountpoint
-

Where $PORT is the same port number we used in the serve nfs command.

-

This feature is only available on Unix platforms.

-

VFS - Virtual File System

-

This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.

-

Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.

-

The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.

VFS Directory Cache

Using the --dir-cache-time flag, you can control how long a directory should be considered up to date and not refreshed from the backend. Changes made through the VFS will appear immediately or invalidate the cache.

--dir-cache-time duration   Time to cache directory entries for (default 5m0s)
@@ -4738,21 +4836,67 @@ htpasswd -B htpasswd anotherUser

Alternate report of used bytes

Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df on the filesystem, then pass the flag --vfs-used-is-size to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size and compute the total used space itself.

WARNING. Contrary to rclone size, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.

-
rclone serve nfs remote:path [flags]
+

Auth Proxy

+

If you supply the parameter --auth-proxy /path/to/program then rclone will use that program to generate backends on the fly which then are used to authenticate incoming requests. This uses a simple JSON based protocol with input on STDIN and output on STDOUT.

+

PLEASE NOTE: --auth-proxy and --authorized-keys cannot be used together, if --auth-proxy is set the authorized keys option will be ignored.

+

There is an example program bin/test_proxy.py in the rclone source code.

+

The program's job is to take a user and pass on the input and turn those into the config for a backend on STDOUT in JSON format. This config will have any default parameters for the backend added, but it won't use configuration from environment variables or command line options - it is the job of the proxy program to make a complete config.

+

This config generated must have this extra parameter - _root - root to use for the backend

+

And it may have this parameter - _obscure - comma separated strings for parameters to obscure

+

If password authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:

+
{
+    "user": "me",
+    "pass": "mypassword"
+}
+

If public-key authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:

+
{
+    "user": "me",
+    "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
+}
+

And as an example return this on STDOUT

+
{
+    "type": "sftp",
+    "_root": "",
+    "_obscure": "pass",
+    "user": "me",
+    "pass": "mypassword",
+    "host": "sftp.example.com"
+}
+

This would mean that an SFTP backend would be created on the fly for the user and pass/public_key returned in the output to the host given. Note that since _obscure is set to pass, rclone will obscure the pass parameter before creating the backend (which is required for sftp backends).

+

The program can manipulate the supplied user in any way, for example to make proxy to many different sftp backends, you could make the user be user@example.com and then set the host to example.com in the output and the user to user. For security you'd probably want to restrict the host to a limited list.

+

Note that an internal cache is keyed on user so only use that for configuration, don't use pass or public_key. This also means that if a user's password or public-key is changed the cache will need to expire (which takes 5 mins) before it takes effect.

+

This can be used to build general purpose proxies to any kind of backend that rclone supports.

+
rclone serve http remote:path [flags]

Options

-
      --addr string                            IPaddress:Port or :Port to bind server to
+
      --addr stringArray                       IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
+      --allow-origin string                    Origin which cross-domain request (CORS) can be executed from
+      --auth-proxy string                      A program to use to create the backend from the auth
+      --baseurl string                         Prefix for URLs - leave blank for root
+      --cert string                            TLS PEM key (concatenation of certificate and CA certificate)
+      --client-ca string                       Client certificate authority to verify clients with
       --dir-cache-time Duration                Time to cache directory entries for (default 5m0s)
       --dir-perms FileMode                     Directory permissions (default 0777)
       --file-perms FileMode                    File permissions (default 0666)
       --gid uint32                             Override the gid field set by the filesystem (not supported on Windows) (default 1000)
-  -h, --help                                   help for nfs
+  -h, --help                                   help for http
+      --htpasswd string                        A htpasswd file - if not provided no authentication is done
+      --key string                             TLS PEM Private key
+      --max-header-bytes int                   Maximum size of request header (default 4096)
+      --min-tls-version string                 Minimum TLS version that is acceptable (default "tls1.0")
       --no-checksum                            Don't compare checksums on up/download
       --no-modtime                             Don't read/write the modification time (can speed things up)
       --no-seek                                Don't allow seeking in files
+      --pass string                            Password for authentication
       --poll-interval Duration                 Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
       --read-only                              Only allow read-only access
+      --realm string                           Realm for authentication
+      --salt string                            Password hashing salt (default "dlPL2MqE")
+      --server-read-timeout Duration           Timeout for server reading data (default 1h0m0s)
+      --server-write-timeout Duration          Timeout for server writing data (default 1h0m0s)
+      --template string                        User-specified template
       --uid uint32                             Override the uid field set by the filesystem (not supported on Windows) (default 1000)
       --umask int                              Override the permission bits set by the filesystem (not supported on Windows) (default 2)
+      --user string                            User name for authentication
       --vfs-cache-max-age Duration             Max time since last access of objects in the cache (default 1h0m0s)
       --vfs-cache-max-size SizeSuffix          Max total size of objects in the cache (default off)
       --vfs-cache-min-free-space SizeSuffix    Target minimum free space on the disk containing the cache (default off)
@@ -4765,7 +4909,7 @@ htpasswd -B htpasswd anotherUser

       --vfs-read-chunk-size SizeSuffix         Read the source objects in chunks (default 128Mi)
       --vfs-read-chunk-size-limit SizeSuffix   If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
       --vfs-read-wait Duration                 Time to wait for in-sequence read before seeking (default 20ms)
-      --vfs-refresh                            Refreshes the directory cache recursively on start
+      --vfs-refresh                            Refreshes the directory cache recursively in the background on start
       --vfs-used-is-size rclone size           Use the rclone size algorithm for Used size
       --vfs-write-back Duration                Time to writeback files after last use when using cache (default 5s)
       --vfs-write-wait Duration                Time to wait for in-sequence write before giving error (default 1s)
@@ -4798,170 +4942,21 @@ htpasswd -B htpasswd anotherUser -

rclone serve restic

-

Serve the remote for restic's REST API.

+

rclone serve nfs

+

Serve the remote as an NFS mount

Synopsis

-

Run a basic web server to serve a remote over restic's REST backend API over HTTP. This allows restic to use rclone as a data storage mechanism for cloud providers that restic does not support directly.

-

Restic is a command-line program for doing backups.

-

The server will log errors. Use -v to see access logs.

-

--bwlimit will be respected for file transfers. Use --stats to control the stats printing.

-

Setting up rclone for use by restic

-

First set up a remote for your chosen cloud provider.

-

Once you have set up the remote, check it is working with, for example "rclone lsd remote:". You may have called the remote something other than "remote:" - just substitute whatever you called it in the following instructions.

-

Now start the rclone restic server

-
rclone serve restic -v remote:backup
-

Where you can replace "backup" in the above by whatever path in the remote you wish to use.

-

By default this will serve on "localhost:8080" you can change this with use of the --addr flag.

-

You might wish to start this server on boot.

-

Adding --cache-objects=false will cause rclone to stop caching objects returned from the List call. Caching is normally desirable as it speeds up downloading objects, saves transactions and uses very little memory.

-

Setting up restic to use rclone

-

Now you can follow the restic instructions on setting up restic.

-

Note that you will need restic 0.8.2 or later to interoperate with rclone.

-

For the example above you will want to use "http://localhost:8080/" as the URL for the REST server.

-

For example:

-
$ export RESTIC_REPOSITORY=rest:http://localhost:8080/
-$ export RESTIC_PASSWORD=yourpassword
-$ restic init
-created restic backend 8b1a4b56ae at rest:http://localhost:8080/
-
-Please note that knowledge of your password is required to access
-the repository. Losing your password means that your data is
-irrecoverably lost.
-$ restic backup /path/to/files/to/backup
-scan [/path/to/files/to/backup]
-scanned 189 directories, 312 files in 0:00
-[0:00] 100.00%  38.128 MiB / 38.128 MiB  501 / 501 items  0 errors  ETA 0:00
-duration: 0:00
-snapshot 45c8fdd8 saved
-

Multiple repositories

-

Note that you can use the endpoint to host multiple repositories. Do this by adding a directory name or path after the URL. Note that these must end with /. Eg

-
$ export RESTIC_REPOSITORY=rest:http://localhost:8080/user1repo/
-# backup user1 stuff
-$ export RESTIC_REPOSITORY=rest:http://localhost:8080/user2repo/
-# backup user2 stuff
-

Private repositories

-

The--private-repos flag can be used to limit users to repositories starting with a path of /<username>/.

-

Server options

-

Use --addr to specify which IP address and port the server should listen on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. By default it only listens on localhost. You can use port :0 to let the OS choose an available port.

-

If you set --addr to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.

-

You can use a unix socket by setting the url to unix:///path/to/socket or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions.

-

--addr may be repeated to listen on multiple IPs/ports/sockets.

-

--server-read-timeout and --server-write-timeout can be used to control the timeouts on the server. Note that this is the total time for a transfer.

-

--max-header-bytes controls the maximum number of bytes the server will accept in the HTTP header.

-

--baseurl controls the URL prefix that rclone serves from. By default rclone will serve from the root. If you used --baseurl "/rclone" then rclone would serve from a URL starting with "/rclone/". This is useful if you wish to proxy rclone serve. Rclone automatically inserts leading and trailing "/" on --baseurl, so --baseurl "rclone", --baseurl "/rclone" and --baseurl "/rclone/" are all treated identically.

-

TLS (SSL)

-

By default this will serve over http. If you want you can serve over https. You will need to supply the --cert and --key flags. If you wish to do client side certificate validation then you will need to supply --client-ca also.

-

--cert should be a either a PEM encoded certificate or a concatenation of that with the CA certificate. --key should be the PEM encoded private key and --client-ca should be the PEM encoded client certificate authority certificate.

-

--min-tls-version is minimum TLS version that is acceptable. Valid values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0").

-

Authentication

-

By default this will serve files without needing a login.

-

You can either use an htpasswd file which can take lots of users, or set a single username and password with the --user and --pass flags.

-

If no static users are configured by either of the above methods, and client certificates are required by the --client-ca flag passed to the server, the client certificate common name will be considered as the username.

-

Use --htpasswd /path/to/htpasswd to provide an htpasswd file. This is in standard apache format and supports MD5, SHA1 and BCrypt for basic authentication. Bcrypt is recommended.

-

To create an htpasswd file:

-
touch htpasswd
-htpasswd -B htpasswd user
-htpasswd -B htpasswd anotherUser
-

The password file can be updated while rclone is running.

-

Use --realm to set the authentication realm.

-

Use --salt to change the password hashing salt from the default.

-
rclone serve restic remote:path [flags]
-

Options

-
      --addr stringArray                IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
-      --allow-origin string             Origin which cross-domain request (CORS) can be executed from
-      --append-only                     Disallow deletion of repository data
-      --baseurl string                  Prefix for URLs - leave blank for root
-      --cache-objects                   Cache listed objects (default true)
-      --cert string                     TLS PEM key (concatenation of certificate and CA certificate)
-      --client-ca string                Client certificate authority to verify clients with
-  -h, --help                            help for restic
-      --htpasswd string                 A htpasswd file - if not provided no authentication is done
-      --key string                      TLS PEM Private key
-      --max-header-bytes int            Maximum size of request header (default 4096)
-      --min-tls-version string          Minimum TLS version that is acceptable (default "tls1.0")
-      --pass string                     Password for authentication
-      --private-repos                   Users can only access their private repo
-      --realm string                    Realm for authentication
-      --salt string                     Password hashing salt (default "dlPL2MqE")
-      --server-read-timeout Duration    Timeout for server reading data (default 1h0m0s)
-      --server-write-timeout Duration   Timeout for server writing data (default 1h0m0s)
-      --stdio                           Run an HTTP2 server on stdin/stdout
-      --user string                     User name for authentication
-

See the global flags page for global options not listed here.

-

SEE ALSO

- -

rclone serve s3

-

Serve remote:path over s3.

-

Synopsis

-

serve s3 implements a basic s3 server that serves a remote via s3. This can be viewed with an s3 client, or you can make an s3 type remote to read and write to it with rclone.

-

serve s3 is considered Experimental so use with care.

-

S3 server supports Signature Version 4 authentication. Just use --auth-key accessKey,secretKey and set the Authorization header correctly in the request. (See the AWS docs).

-

--auth-key can be repeated for multiple auth pairs. If --auth-key is not provided then serve s3 will allow anonymous access.

-

Please note that some clients may require HTTPS endpoints. See the SSL docs for more information.

-

This command uses the VFS directory cache. All the functionality will work with --vfs-cache-mode off. Using --vfs-cache-mode full (or writes) can be used to cache objects locally to improve performance.

-

Use --force-path-style=false if you want to use the bucket name as a part of the hostname (such as mybucket.local)

-

Use --etag-hash if you want to change the hash uses for the ETag. Note that using anything other than MD5 (the default) is likely to cause problems for S3 clients which rely on the Etag being the MD5.

-

Quickstart

-

For a simple set up, to serve remote:path over s3, run the server like this:

-
rclone serve s3 --auth-key ACCESS_KEY_ID,SECRET_ACCESS_KEY remote:path
-

This will be compatible with an rclone remote which is defined like this:

-
[serves3]
-type = s3
-provider = Rclone
-endpoint = http://127.0.0.1:8080/
-access_key_id = ACCESS_KEY_ID
-secret_access_key = SECRET_ACCESS_KEY
-use_multipart_uploads = false
-

Note that setting disable_multipart_uploads = true is to work around a bug which will be fixed in due course.

-

Bugs

-

When uploading multipart files serve s3 holds all the parts in memory (see #7453). This is a limitaton of the library rclone uses for serving S3 and will hopefully be fixed at some point.

-

Multipart server side copies do not work (see #7454). These take a very long time and eventually fail. The default threshold for multipart server side copies is 5G which is the maximum it can be, so files above this side will fail to be server side copied.

-

For a current list of serve s3 bugs see the serve s3 bug category on GitHub.

-

Limitations

-

serve s3 will treat all directories in the root as buckets and ignore all files in the root. You can use CreateBucket to create folders under the root, but you can't create empty folders under other folders not in the root.

-

When using PutObject or DeleteObject, rclone will automatically create or clean up empty folders. If you don't want to clean up empty folders automatically, use --no-cleanup.

-

When using ListObjects, rclone will use / when the delimiter is empty. This reduces backend requests with no effect on most operations, but if the delimiter is something other than / and empty, rclone will do a full recursive search of the backend, which can take some time.

-

Versioning is not currently supported.

-

Metadata will only be saved in memory other than the rclone mtime metadata which will be set as the modification time of the file.

-

Supported operations

-

serve s3 currently supports the following operations.

- -

Other operations will return error Unimplemented.

-

Server options

-

Use --addr to specify which IP address and port the server should listen on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. By default it only listens on localhost. You can use port :0 to let the OS choose an available port.

-

If you set --addr to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.

-

You can use a unix socket by setting the url to unix:///path/to/socket or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions.

-

--addr may be repeated to listen on multiple IPs/ports/sockets.

-

--server-read-timeout and --server-write-timeout can be used to control the timeouts on the server. Note that this is the total time for a transfer.

-

--max-header-bytes controls the maximum number of bytes the server will accept in the HTTP header.

-

--baseurl controls the URL prefix that rclone serves from. By default rclone will serve from the root. If you used --baseurl "/rclone" then rclone would serve from a URL starting with "/rclone/". This is useful if you wish to proxy rclone serve. Rclone automatically inserts leading and trailing "/" on --baseurl, so --baseurl "rclone", --baseurl "/rclone" and --baseurl "/rclone/" are all treated identically.

-

TLS (SSL)

-

By default this will serve over http. If you want you can serve over https. You will need to supply the --cert and --key flags. If you wish to do client side certificate validation then you will need to supply --client-ca also.

-

--cert should be a either a PEM encoded certificate or a concatenation of that with the CA certificate. --key should be the PEM encoded private key and --client-ca should be the PEM encoded client certificate authority certificate.

-

--min-tls-version is minimum TLS version that is acceptable. Valid values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). ## VFS - Virtual File System

+

Create an NFS server that serves the given remote over the network.

+

The primary purpose for this command is to enable mount command on recent macOS versions where installing FUSE is very cumbersome.

+

Since this is running on NFSv3, no authentication method is available. Any client will be able to access the data. To limit access, you can use serve NFS on loopback address and rely on secure tunnels (such as SSH). For this reason, by default, a random TCP port is chosen and loopback interface is used for the listening address; meaning that it is only available to the local machine. If you want other machines to access the NFS mount over local network, you need to specify the listening address and port using --addr flag.

+

Modifying files through NFS protocol requires VFS caching. Usually you will need to specify --vfs-cache-mode in order to be able to write to the mountpoint (full is recommended). If you don't specify VFS cache mode, the mount will be read-only.

+

To serve NFS over the network use following command:

+
rclone serve nfs remote: --addr 0.0.0.0:$PORT --vfs-cache-mode=full
+

We specify a specific port that we can use in the mount command:

+

To mount the server under Linux/macOS, use the following command:

+
mount -oport=$PORT,mountport=$PORT $HOSTNAME: path/to/mountpoint
+

Where $PORT is the same port number we used in the serve nfs command.

+

This feature is only available on Unix platforms.

+

VFS - Virtual File System

This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.

Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.

The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.

@@ -5078,32 +5073,19 @@ use_multipart_uploads = false

Alternate report of used bytes

Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df on the filesystem, then pass the flag --vfs-used-is-size to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size and compute the total used space itself.

WARNING. Contrary to rclone size, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.

-
rclone serve s3 remote:path [flags]
-

Options

-
      --addr stringArray                       IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
-      --allow-origin string                    Origin which cross-domain request (CORS) can be executed from
-      --auth-key stringArray                   Set key pair for v4 authorization: access_key_id,secret_access_key
-      --baseurl string                         Prefix for URLs - leave blank for root
-      --cert string                            TLS PEM key (concatenation of certificate and CA certificate)
-      --client-ca string                       Client certificate authority to verify clients with
+
rclone serve nfs remote:path [flags]

+
Options

+
      --addr string                            IPaddress:Port or :Port to bind server to
       --dir-cache-time Duration                Time to cache directory entries for (default 5m0s)
       --dir-perms FileMode                     Directory permissions (default 0777)
-      --etag-hash string                       Which hash to use for the ETag, or auto or blank for off (default "MD5")
       --file-perms FileMode                    File permissions (default 0666)
-      --force-path-style                       If true use path style access if false use virtual hosted style (default true) (default true)
       --gid uint32                             Override the gid field set by the filesystem (not supported on Windows) (default 1000)
-  -h, --help                                   help for s3
-      --key string                             TLS PEM Private key
-      --max-header-bytes int                   Maximum size of request header (default 4096)
-      --min-tls-version string                 Minimum TLS version that is acceptable (default "tls1.0")
+  -h, --help                                   help for nfs
       --no-checksum                            Don't compare checksums on up/download
-      --no-cleanup                             Not to cleanup empty folder after object is deleted
       --no-modtime                             Don't read/write the modification time (can speed things up)
       --no-seek                                Don't allow seeking in files
       --poll-interval Duration                 Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
       --read-only                              Only allow read-only access
-      --server-read-timeout Duration           Timeout for server reading data (default 1h0m0s)
-      --server-write-timeout Duration          Timeout for server writing data (default 1h0m0s)
       --uid uint32                             Override the uid field set by the filesystem (not supported on Windows) (default 1000)
       --umask int                              Override the permission bits set by the filesystem (not supported on Windows) (default 2)
       --vfs-cache-max-age Duration             Max time since last access of objects in the cache (default 1h0m0s)
@@ -5118,7 +5100,7 @@ use_multipart_uploads = false

       --vfs-read-chunk-size SizeSuffix         Read the source objects in chunks (default 128Mi)
       --vfs-read-chunk-size-limit SizeSuffix   If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
       --vfs-read-wait Duration                 Time to wait for in-sequence read before seeking (default 20ms)
-      --vfs-refresh                            Refreshes the directory cache recursively on start
+      --vfs-refresh                            Refreshes the directory cache recursively in the background on start
       --vfs-used-is-size rclone size           Use the rclone size algorithm for Used size
       --vfs-write-back Duration                Time to writeback files after last use when using cache (default 5s)
       --vfs-write-wait Duration                Time to wait for in-sequence write before giving error (default 1s)
@@ -5147,28 +5129,174 @@ use_multipart_uploads = false --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)

See the global flags page for global options not listed here.

+

SEE ALSO

+ +

rclone serve restic

+

Serve the remote for restic's REST API.

+

Synopsis

+

Run a basic web server to serve a remote over restic's REST backend API over HTTP. This allows restic to use rclone as a data storage mechanism for cloud providers that restic does not support directly.

+

Restic is a command-line program for doing backups.

+

The server will log errors. Use -v to see access logs.

+

--bwlimit will be respected for file transfers. Use --stats to control the stats printing.

+

Setting up rclone for use by restic

+

First set up a remote for your chosen cloud provider.

+

Once you have set up the remote, check it is working with, for example "rclone lsd remote:". You may have called the remote something other than "remote:" - just substitute whatever you called it in the following instructions.

+

Now start the rclone restic server

+
rclone serve restic -v remote:backup
+

Where you can replace "backup" in the above by whatever path in the remote you wish to use.

+

By default this will serve on "localhost:8080" you can change this with use of the --addr flag.

+

You might wish to start this server on boot.

+

Adding --cache-objects=false will cause rclone to stop caching objects returned from the List call. Caching is normally desirable as it speeds up downloading objects, saves transactions and uses very little memory.

+

Setting up restic to use rclone

+

Now you can follow the restic instructions on setting up restic.

+

Note that you will need restic 0.8.2 or later to interoperate with rclone.

+

For the example above you will want to use "http://localhost:8080/" as the URL for the REST server.

+

For example:

+
$ export RESTIC_REPOSITORY=rest:http://localhost:8080/
+$ export RESTIC_PASSWORD=yourpassword
+$ restic init
+created restic backend 8b1a4b56ae at rest:http://localhost:8080/
+
+Please note that knowledge of your password is required to access
+the repository. Losing your password means that your data is
+irrecoverably lost.
+$ restic backup /path/to/files/to/backup
+scan [/path/to/files/to/backup]
+scanned 189 directories, 312 files in 0:00
+[0:00] 100.00%  38.128 MiB / 38.128 MiB  501 / 501 items  0 errors  ETA 0:00
+duration: 0:00
+snapshot 45c8fdd8 saved
+

Multiple repositories

+

Note that you can use the endpoint to host multiple repositories. Do this by adding a directory name or path after the URL. Note that these must end with /. Eg

+
$ export RESTIC_REPOSITORY=rest:http://localhost:8080/user1repo/
+# backup user1 stuff
+$ export RESTIC_REPOSITORY=rest:http://localhost:8080/user2repo/
+# backup user2 stuff
+

Private repositories

+

The--private-repos flag can be used to limit users to repositories starting with a path of /<username>/.

+

Server options

+

Use --addr to specify which IP address and port the server should listen on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. By default it only listens on localhost. You can use port :0 to let the OS choose an available port.

+

If you set --addr to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.

+

You can use a unix socket by setting the url to unix:///path/to/socket or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions.

+

--addr may be repeated to listen on multiple IPs/ports/sockets.

+

--server-read-timeout and --server-write-timeout can be used to control the timeouts on the server. Note that this is the total time for a transfer.

+

--max-header-bytes controls the maximum number of bytes the server will accept in the HTTP header.

+

--baseurl controls the URL prefix that rclone serves from. By default rclone will serve from the root. If you used --baseurl "/rclone" then rclone would serve from a URL starting with "/rclone/". This is useful if you wish to proxy rclone serve. Rclone automatically inserts leading and trailing "/" on --baseurl, so --baseurl "rclone", --baseurl "/rclone" and --baseurl "/rclone/" are all treated identically.

+

TLS (SSL)

+

By default this will serve over http. If you want you can serve over https. You will need to supply the --cert and --key flags. If you wish to do client side certificate validation then you will need to supply --client-ca also.

+

--cert should be a either a PEM encoded certificate or a concatenation of that with the CA certificate. --key should be the PEM encoded private key and --client-ca should be the PEM encoded client certificate authority certificate.

+

--min-tls-version is minimum TLS version that is acceptable. Valid values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0").

+

Authentication

+

By default this will serve files without needing a login.

+

You can either use an htpasswd file which can take lots of users, or set a single username and password with the --user and --pass flags.

+

If no static users are configured by either of the above methods, and client certificates are required by the --client-ca flag passed to the server, the client certificate common name will be considered as the username.

+

Use --htpasswd /path/to/htpasswd to provide an htpasswd file. This is in standard apache format and supports MD5, SHA1 and BCrypt for basic authentication. Bcrypt is recommended.

+

To create an htpasswd file:

+
touch htpasswd
+htpasswd -B htpasswd user
+htpasswd -B htpasswd anotherUser
+

The password file can be updated while rclone is running.

+

Use --realm to set the authentication realm.

+

Use --salt to change the password hashing salt from the default.

+
rclone serve restic remote:path [flags]
+

Options

+
      --addr stringArray                IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
+      --allow-origin string             Origin which cross-domain request (CORS) can be executed from
+      --append-only                     Disallow deletion of repository data
+      --baseurl string                  Prefix for URLs - leave blank for root
+      --cache-objects                   Cache listed objects (default true)
+      --cert string                     TLS PEM key (concatenation of certificate and CA certificate)
+      --client-ca string                Client certificate authority to verify clients with
+  -h, --help                            help for restic
+      --htpasswd string                 A htpasswd file - if not provided no authentication is done
+      --key string                      TLS PEM Private key
+      --max-header-bytes int            Maximum size of request header (default 4096)
+      --min-tls-version string          Minimum TLS version that is acceptable (default "tls1.0")
+      --pass string                     Password for authentication
+      --private-repos                   Users can only access their private repo
+      --realm string                    Realm for authentication
+      --salt string                     Password hashing salt (default "dlPL2MqE")
+      --server-read-timeout Duration    Timeout for server reading data (default 1h0m0s)
+      --server-write-timeout Duration   Timeout for server writing data (default 1h0m0s)
+      --stdio                           Run an HTTP2 server on stdin/stdout
+      --user string                     User name for authentication
+

See the global flags page for global options not listed here.

SEE ALSO

-

rclone serve sftp

-

Serve the remote over SFTP.

+

rclone serve s3

+

Serve remote:path over s3.

Synopsis

-

Run an SFTP server to serve a remote over SFTP. This can be used with an SFTP client or you can make a remote of type sftp to use with it.

-

You can use the filter flags (e.g. --include, --exclude) to control what is served.

-

The server will respond to a small number of shell commands, mainly md5sum, sha1sum and df, which enable it to provide support for checksums and the about feature when accessed from an sftp remote.

-

Note that this server uses standard 32 KiB packet payload size, which means you must not configure the client to expect anything else, e.g. with the chunk_size option on an sftp remote.

-

The server will log errors. Use -v to see access logs.

-

--bwlimit will be respected for file transfers. Use --stats to control the stats printing.

-

You must provide some means of authentication, either with --user/--pass, an authorized keys file (specify location with --authorized-keys - the default is the same as ssh), an --auth-proxy, or set the --no-auth flag for no authentication when logging in.

-

If you don't supply a host --key then rclone will generate rsa, ecdsa and ed25519 variants, and cache them for later use in rclone's cache directory (see rclone help flags cache-dir) in the "serve-sftp" directory.

-

By default the server binds to localhost:2022 - if you want it to be reachable externally then supply --addr :2022 for example.

-

Note that the default of --vfs-cache-mode off is fine for the rclone sftp backend, but it may not be with other SFTP clients.

-

If --stdio is specified, rclone will serve SFTP over stdio, which can be used with sshd via ~/.ssh/authorized_keys, for example:

-
restrict,command="rclone serve sftp --stdio ./photos" ssh-rsa ...
-

On the client you need to set --transfers 1 when using --stdio. Otherwise multiple instances of the rclone server are started by OpenSSH which can lead to "corrupted on transfer" errors. This is the case because the client chooses indiscriminately which server to send commands to while the servers all have different views of the state of the filing system.

-

The "restrict" in authorized_keys prevents SHA1SUMs and MD5SUMs from being used. Omitting "restrict" and using --sftp-path-override to enable checksumming is possible but less secure and you could use the SFTP server provided by OpenSSH in this case.

-

VFS - Virtual File System

+

serve s3 implements a basic s3 server that serves a remote via s3. This can be viewed with an s3 client, or you can make an s3 type remote to read and write to it with rclone.

+

serve s3 is considered Experimental so use with care.

+

S3 server supports Signature Version 4 authentication. Just use --auth-key accessKey,secretKey and set the Authorization header correctly in the request. (See the AWS docs).

+

--auth-key can be repeated for multiple auth pairs. If --auth-key is not provided then serve s3 will allow anonymous access.

+

Please note that some clients may require HTTPS endpoints. See the SSL docs for more information.

+

This command uses the VFS directory cache. All the functionality will work with --vfs-cache-mode off. Using --vfs-cache-mode full (or writes) can be used to cache objects locally to improve performance.

+

Use --force-path-style=false if you want to use the bucket name as a part of the hostname (such as mybucket.local)

+

Use --etag-hash if you want to change the hash uses for the ETag. Note that using anything other than MD5 (the default) is likely to cause problems for S3 clients which rely on the Etag being the MD5.

+

Quickstart

+

For a simple set up, to serve remote:path over s3, run the server like this:

+
rclone serve s3 --auth-key ACCESS_KEY_ID,SECRET_ACCESS_KEY remote:path
+

This will be compatible with an rclone remote which is defined like this:

+
[serves3]
+type = s3
+provider = Rclone
+endpoint = http://127.0.0.1:8080/
+access_key_id = ACCESS_KEY_ID
+secret_access_key = SECRET_ACCESS_KEY
+use_multipart_uploads = false
+

Note that setting disable_multipart_uploads = true is to work around a bug which will be fixed in due course.

+

Bugs

+

When uploading multipart files serve s3 holds all the parts in memory (see #7453). This is a limitaton of the library rclone uses for serving S3 and will hopefully be fixed at some point.

+

Multipart server side copies do not work (see #7454). These take a very long time and eventually fail. The default threshold for multipart server side copies is 5G which is the maximum it can be, so files above this side will fail to be server side copied.

+

For a current list of serve s3 bugs see the serve s3 bug category on GitHub.

+

Limitations

+

serve s3 will treat all directories in the root as buckets and ignore all files in the root. You can use CreateBucket to create folders under the root, but you can't create empty folders under other folders not in the root.

+

When using PutObject or DeleteObject, rclone will automatically create or clean up empty folders. If you don't want to clean up empty folders automatically, use --no-cleanup.

+

When using ListObjects, rclone will use / when the delimiter is empty. This reduces backend requests with no effect on most operations, but if the delimiter is something other than / and empty, rclone will do a full recursive search of the backend, which can take some time.

+

Versioning is not currently supported.

+

Metadata will only be saved in memory other than the rclone mtime metadata which will be set as the modification time of the file.

+

Supported operations

+

serve s3 currently supports the following operations.

+ +

Other operations will return error Unimplemented.

+

Server options

+

Use --addr to specify which IP address and port the server should listen on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. By default it only listens on localhost. You can use port :0 to let the OS choose an available port.

+

If you set --addr to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.

+

You can use a unix socket by setting the url to unix:///path/to/socket or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions.

+

--addr may be repeated to listen on multiple IPs/ports/sockets.

+

--server-read-timeout and --server-write-timeout can be used to control the timeouts on the server. Note that this is the total time for a transfer.

+

--max-header-bytes controls the maximum number of bytes the server will accept in the HTTP header.

+

--baseurl controls the URL prefix that rclone serves from. By default rclone will serve from the root. If you used --baseurl "/rclone" then rclone would serve from a URL starting with "/rclone/". This is useful if you wish to proxy rclone serve. Rclone automatically inserts leading and trailing "/" on --baseurl, so --baseurl "rclone", --baseurl "/rclone" and --baseurl "/rclone/" are all treated identically.

+

TLS (SSL)

+

By default this will serve over http. If you want you can serve over https. You will need to supply the --cert and --key flags. If you wish to do client side certificate validation then you will need to supply --client-ca also.

+

--cert should be a either a PEM encoded certificate or a concatenation of that with the CA certificate. --key should be the PEM encoded private key and --client-ca should be the PEM encoded client certificate authority certificate.

+

--min-tls-version is minimum TLS version that is acceptable. Valid values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). ## VFS - Virtual File System

This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.

Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.

The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.

@@ -5285,58 +5413,34 @@ use_multipart_uploads = false

Alternate report of used bytes

Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df on the filesystem, then pass the flag --vfs-used-is-size to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size and compute the total used space itself.

WARNING. Contrary to rclone size, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.

-

Auth Proxy

-

If you supply the parameter --auth-proxy /path/to/program then rclone will use that program to generate backends on the fly which then are used to authenticate incoming requests. This uses a simple JSON based protocol with input on STDIN and output on STDOUT.

-

PLEASE NOTE: --auth-proxy and --authorized-keys cannot be used together, if --auth-proxy is set the authorized keys option will be ignored.

-

There is an example program bin/test_proxy.py in the rclone source code.

-

The program's job is to take a user and pass on the input and turn those into the config for a backend on STDOUT in JSON format. This config will have any default parameters for the backend added, but it won't use configuration from environment variables or command line options - it is the job of the proxy program to make a complete config.

-

This config generated must have this extra parameter - _root - root to use for the backend

-

And it may have this parameter - _obscure - comma separated strings for parameters to obscure

-

If password authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:

-
{
-    "user": "me",
-    "pass": "mypassword"
-}
-

If public-key authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:

-
{
-    "user": "me",
-    "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
-}
-

And as an example return this on STDOUT

-
{
-    "type": "sftp",
-    "_root": "",
-    "_obscure": "pass",
-    "user": "me",
-    "pass": "mypassword",
-    "host": "sftp.example.com"
-}
-

This would mean that an SFTP backend would be created on the fly for the user and pass/public_key returned in the output to the host given. Note that since _obscure is set to pass, rclone will obscure the pass parameter before creating the backend (which is required for sftp backends).

-

The program can manipulate the supplied user in any way, for example to make proxy to many different sftp backends, you could make the user be user@example.com and then set the host to example.com in the output and the user to user. For security you'd probably want to restrict the host to a limited list.

-

Note that an internal cache is keyed on user so only use that for configuration, don't use pass or public_key. This also means that if a user's password or public-key is changed the cache will need to expire (which takes 5 mins) before it takes effect.

-

This can be used to build general purpose proxies to any kind of backend that rclone supports.

-
rclone serve sftp remote:path [flags]
+
rclone serve s3 remote:path [flags]

Options

-
      --addr string                            IPaddress:Port or :Port to bind server to (default "localhost:2022")
-      --auth-proxy string                      A program to use to create the backend from the auth
-      --authorized-keys string                 Authorized keys file (default "~/.ssh/authorized_keys")
+
      --addr stringArray                       IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
+      --allow-origin string                    Origin which cross-domain request (CORS) can be executed from
+      --auth-key stringArray                   Set key pair for v4 authorization: access_key_id,secret_access_key
+      --baseurl string                         Prefix for URLs - leave blank for root
+      --cert string                            TLS PEM key (concatenation of certificate and CA certificate)
+      --client-ca string                       Client certificate authority to verify clients with
       --dir-cache-time Duration                Time to cache directory entries for (default 5m0s)
       --dir-perms FileMode                     Directory permissions (default 0777)
+      --etag-hash string                       Which hash to use for the ETag, or auto or blank for off (default "MD5")
       --file-perms FileMode                    File permissions (default 0666)
+      --force-path-style                       If true use path style access if false use virtual hosted style (default true) (default true)
       --gid uint32                             Override the gid field set by the filesystem (not supported on Windows) (default 1000)
-  -h, --help                                   help for sftp
-      --key stringArray                        SSH private host key file (Can be multi-valued, leave blank to auto generate)
-      --no-auth                                Allow connections with no authentication if set
+  -h, --help                                   help for s3
+      --key string                             TLS PEM Private key
+      --max-header-bytes int                   Maximum size of request header (default 4096)
+      --min-tls-version string                 Minimum TLS version that is acceptable (default "tls1.0")
       --no-checksum                            Don't compare checksums on up/download
+      --no-cleanup                             Not to cleanup empty folder after object is deleted
       --no-modtime                             Don't read/write the modification time (can speed things up)
       --no-seek                                Don't allow seeking in files
-      --pass string                            Password for authentication
       --poll-interval Duration                 Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
       --read-only                              Only allow read-only access
-      --stdio                                  Run an sftp server on stdin/stdout
+      --server-read-timeout Duration           Timeout for server reading data (default 1h0m0s)
+      --server-write-timeout Duration          Timeout for server writing data (default 1h0m0s)
       --uid uint32                             Override the uid field set by the filesystem (not supported on Windows) (default 1000)
       --umask int                              Override the permission bits set by the filesystem (not supported on Windows) (default 2)
-      --user string                            User name for authentication
       --vfs-cache-max-age Duration             Max time since last access of objects in the cache (default 1h0m0s)
       --vfs-cache-max-size SizeSuffix          Max total size of objects in the cache (default off)
       --vfs-cache-min-free-space SizeSuffix    Target minimum free space on the disk containing the cache (default off)
@@ -5349,7 +5453,7 @@ use_multipart_uploads = false

       --vfs-read-chunk-size SizeSuffix         Read the source objects in chunks (default 128Mi)
       --vfs-read-chunk-size-limit SizeSuffix   If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
       --vfs-read-wait Duration                 Time to wait for in-sequence read before seeking (default 20ms)
-      --vfs-refresh                            Refreshes the directory cache recursively on start
+      --vfs-refresh                            Refreshes the directory cache recursively in the background on start
       --vfs-used-is-size rclone size           Use the rclone size algorithm for Used size
       --vfs-write-back Duration                Time to writeback files after last use when using cache (default 5s)
       --vfs-write-wait Duration                Time to wait for in-sequence write before giving error (default 1s)
@@ -5382,9 +5486,240 @@ use_multipart_uploads = false +

rclone serve sftp

+

Serve the remote over SFTP.

+

Synopsis

+

Run an SFTP server to serve a remote over SFTP. This can be used with an SFTP client or you can make a remote of type sftp to use with it.

+

You can use the filter flags (e.g. --include, --exclude) to control what is served.

+

The server will respond to a small number of shell commands, mainly md5sum, sha1sum and df, which enable it to provide support for checksums and the about feature when accessed from an sftp remote.

+

Note that this server uses standard 32 KiB packet payload size, which means you must not configure the client to expect anything else, e.g. with the chunk_size option on an sftp remote.

+

The server will log errors. Use -v to see access logs.

+

--bwlimit will be respected for file transfers. Use --stats to control the stats printing.

+

You must provide some means of authentication, either with --user/--pass, an authorized keys file (specify location with --authorized-keys - the default is the same as ssh), an --auth-proxy, or set the --no-auth flag for no authentication when logging in.

+

If you don't supply a host --key then rclone will generate rsa, ecdsa and ed25519 variants, and cache them for later use in rclone's cache directory (see rclone help flags cache-dir) in the "serve-sftp" directory.

+

By default the server binds to localhost:2022 - if you want it to be reachable externally then supply --addr :2022 for example.

+

Note that the default of --vfs-cache-mode off is fine for the rclone sftp backend, but it may not be with other SFTP clients.

+

If --stdio is specified, rclone will serve SFTP over stdio, which can be used with sshd via ~/.ssh/authorized_keys, for example:

+
restrict,command="rclone serve sftp --stdio ./photos" ssh-rsa ...
+

On the client you need to set --transfers 1 when using --stdio. Otherwise multiple instances of the rclone server are started by OpenSSH which can lead to "corrupted on transfer" errors. This is the case because the client chooses indiscriminately which server to send commands to while the servers all have different views of the state of the filing system.

+

The "restrict" in authorized_keys prevents SHA1SUMs and MD5SUMs from being used. Omitting "restrict" and using --sftp-path-override to enable checksumming is possible but less secure and you could use the SFTP server provided by OpenSSH in this case.

+

VFS - Virtual File System

+

This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.

+

Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.

+

The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.

+

VFS Directory Cache

+

Using the --dir-cache-time flag, you can control how long a directory should be considered up to date and not refreshed from the backend. Changes made through the VFS will appear immediately or invalidate the cache.

+
--dir-cache-time duration   Time to cache directory entries for (default 5m0s)
+--poll-interval duration    Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
+

However, changes made directly on the cloud storage by the web interface or a different copy of rclone will only be picked up once the directory cache expires if the backend configured does not support polling for changes. If the backend supports polling, changes will be picked up within the polling interval.

+

You can send a SIGHUP signal to rclone for it to flush all directory caches, regardless of how old they are. Assuming only one rclone instance is running, you can reset the cache like this:

+
kill -SIGHUP $(pidof rclone)
+

If you configure rclone with a remote control then you can use rclone rc to flush the whole directory cache:

+
rclone rc vfs/forget
+

Or individual files or directories:

+
rclone rc vfs/forget file=path/to/file dir=path/to/dir
+

VFS File Buffering

+

The --buffer-size flag determines the amount of memory, that will be used to buffer data in advance.

+

Each open file will try to keep the specified amount of data in memory at all times. The buffered data is bound to one open file and won't be shared.

+

This flag is a upper limit for the used memory per open file. The buffer will only use memory for data that is downloaded but not not yet read. If the buffer is empty, only a small amount of memory will be used.

+

The maximum memory used by rclone for buffering can be up to --buffer-size * open files.

+

VFS File Caching

+

These flags control the VFS file caching options. File caching is necessary to make the VFS layer appear compatible with a normal file system. It can be disabled at the cost of some compatibility.

+

For example you'll need to enable VFS caching if you want to read and write simultaneously to a file. See below for more details.

+

Note that the VFS cache is separate from the cache backend and you may find that you need one or the other or both.

+
--cache-dir string                     Directory rclone will use for caching.
+--vfs-cache-mode CacheMode             Cache mode off|minimal|writes|full (default off)
+--vfs-cache-max-age duration           Max time since last access of objects in the cache (default 1h0m0s)
+--vfs-cache-max-size SizeSuffix        Max total size of objects in the cache (default off)
+--vfs-cache-min-free-space SizeSuffix  Target minimum free space on the disk containing the cache (default off)
+--vfs-cache-poll-interval duration     Interval to poll the cache for stale objects (default 1m0s)
+--vfs-write-back duration              Time to writeback files after last use when using cache (default 5s)
+

If run with -vv rclone will print the location of the file cache. The files are stored in the user cache file area which is OS dependent but can be controlled with --cache-dir or setting the appropriate environment variable.

+

The cache has 4 different modes selected by --vfs-cache-mode. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.

+

Note that files are written back to the remote only when they are closed and if they haven't been accessed for --vfs-write-back seconds. If rclone is quit or dies with files that haven't been uploaded, these will be uploaded next time rclone is run with the same flags.

+

If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the cache may exceed these quotas for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval. Secondly because open files cannot be evicted from the cache. When --vfs-cache-max-size or --vfs-cache-min-free-size is exceeded, rclone will attempt to evict the least accessed files from the cache first. rclone will start with files that haven't been accessed for the longest. This cache flushing strategy is efficient and more relevant files are likely to remain cached.

+

The --vfs-cache-max-age will evict files from the cache after the set time since last access has passed. The default value of 1 hour will start evicting files from cache that haven't been accessed for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0 and will wait for 1 more hour before evicting. Specify the time with standard notation, s, m, h, d, w .

+

You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir. You don't need to worry about this if the remotes in use don't overlap.

+

--vfs-cache-mode off

+

In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.

+

This will mean some operations are not possible

+ +

--vfs-cache-mode minimal

+

This is very similar to "off" except that files opened for read AND write will be buffered to disk. This means that files opened for write will be a lot more compatible, but uses the minimal disk space.

+

These operations are not possible

+ +

--vfs-cache-mode writes

+

In this mode files opened for read only are still read directly from the remote, write only and read/write files are buffered to disk first.

+

This mode should support all normal file system operations.

+

If an upload fails it will be retried at exponentially increasing intervals up to 1 minute.

+

--vfs-cache-mode full

+

In this mode all reads and writes are buffered to and from disk. When data is read from the remote this is buffered to disk as well.

+

In this mode the files in the cache will be sparse files and rclone will keep track of which bits of the files it has downloaded.

+

So if an application only reads the starts of each file, then rclone will only buffer the start of the file. These files will appear to be their full size in the cache, but they will be sparse files with only the data that has been downloaded present in them.

+

This mode should support all normal file system operations and is otherwise identical to --vfs-cache-mode writes.

+

When reading a file rclone will read --buffer-size plus --vfs-read-ahead bytes ahead. The --buffer-size is buffered in memory whereas the --vfs-read-ahead is buffered on disk.

+

When using this mode it is recommended that --buffer-size is not set too large and --vfs-read-ahead is set large if required.

+

IMPORTANT not all file systems support sparse files. In particular FAT/exFAT do not. Rclone will perform very badly if the cache directory is on a filesystem which doesn't support sparse files and it will log an ERROR message if one is detected.

+

Fingerprinting

+

Various parts of the VFS use fingerprinting to see if a local file copy has changed relative to a remote file. Fingerprints are made from:

+ +

where available on an object.

+

On some backends some of these attributes are slow to read (they take an extra API call per object, or extra work per object).

+

For example hash is slow with the local and sftp backends as they have to read the entire file and hash it, and modtime is slow with the s3, swift, ftp and qinqstor backends because they need to do an extra API call to fetch it.

+

If you use the --vfs-fast-fingerprint flag then rclone will not include the slow operations in the fingerprint. This makes the fingerprinting less accurate but much faster and will improve the opening time of cached files.

+

If you are running a vfs cache over local, s3 or swift backends then using this flag is recommended.

+

Note that if you change the value of this flag, the fingerprints of the files in the cache may be invalidated and the files will need to be downloaded again.

+

VFS Chunked Reading

+

When rclone reads files from a remote it reads them in chunks. This means that rather than requesting the whole file rclone reads the chunk specified. This can reduce the used download quota for some remotes by requesting only chunks from the remote that are actually read, at the cost of an increased number of requests.

+

These flags control the chunking:

+
--vfs-read-chunk-size SizeSuffix        Read the source objects in chunks (default 128M)
+--vfs-read-chunk-size-limit SizeSuffix  Max chunk doubling size (default off)
+

Rclone will start reading a chunk of size --vfs-read-chunk-size, and then double the size for each read. When --vfs-read-chunk-size-limit is specified, and greater than --vfs-read-chunk-size, the chunk size for each open file will get doubled only until the specified value is reached. If the value is "off", which is the default, the limit is disabled and the chunk size will grow indefinitely.

+

With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.

+

Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading.

+

VFS Performance

+

These flags may be used to enable/disable features of the VFS for performance or other reasons. See also the chunked reading feature.

+

In particular S3 and Swift benefit hugely from the --no-modtime flag (or use --use-server-modtime for a slightly different effect) as each read of the modification time takes a transaction.

+
--no-checksum     Don't compare checksums on up/download.
+--no-modtime      Don't read/write the modification time (can speed things up).
+--no-seek         Don't allow seeking in files.
+--read-only       Only allow read-only access.
+

Sometimes rclone is delivered reads or writes out of order. Rather than seeking rclone will wait a short time for the in sequence read or write to come in. These flags only come into effect when not using an on disk cache file.

+
--vfs-read-wait duration   Time to wait for in-sequence read before seeking (default 20ms)
+--vfs-write-wait duration  Time to wait for in-sequence write before giving error (default 1s)
+

When using VFS write caching (--vfs-cache-mode with value writes or full), the global flag --transfers can be set to adjust the number of parallel uploads of modified files from the cache (the related global flag --checkers has no effect on the VFS).

+
--transfers int  Number of file transfers to run in parallel (default 4)
+

VFS Case Sensitivity

+

Linux file systems are case-sensitive: two files can differ only by case, and the exact case must be used when opening a file.

+

File systems in modern Windows are case-insensitive but case-preserving: although existing files can be opened using any case, the exact case used to create the file is preserved and available for programs to query. It is not allowed for two files in the same directory to differ only by case.

+

Usually file systems on macOS are case-insensitive. It is possible to make macOS file systems case-sensitive but that is not the default.

+

The --vfs-case-insensitive VFS flag controls how rclone handles these two cases. If its value is "false", rclone passes file names to the remote as-is. If the flag is "true" (or appears without a value on the command line), rclone may perform a "fixup" as explained below.

+

The user may specify a file name to open/delete/rename/etc with a case different than what is stored on the remote. If an argument refers to an existing file with exactly the same name, then the case of the existing file on the disk will be used. However, if a file name with exactly the same name is not found but a name differing only by case exists, rclone will transparently fixup the name. This fixup happens only when an existing file is requested. Case sensitivity of file names created anew by rclone is controlled by the underlying remote.

+

Note that case sensitivity of the operating system running rclone (the target) may differ from case sensitivity of a file system presented by rclone (the source). The flag controls whether "fixup" is performed to satisfy the target.

+

If the flag is not provided on the command line, then its default value depends on the operating system where rclone runs: "true" on Windows and macOS, "false" otherwise. If the flag is provided without a value, then it is "true".

+

VFS Disk Options

+

This flag allows you to manually set the statistics about the filing system. It can be useful when those statistics cannot be read correctly automatically.

+
--vfs-disk-space-total-size    Manually set the total disk space size (example: 256G, default: -1)
+

Alternate report of used bytes

+

Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df on the filesystem, then pass the flag --vfs-used-is-size to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size and compute the total used space itself.

+

WARNING. Contrary to rclone size, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.

+

Auth Proxy

+

If you supply the parameter --auth-proxy /path/to/program then rclone will use that program to generate backends on the fly which then are used to authenticate incoming requests. This uses a simple JSON based protocol with input on STDIN and output on STDOUT.

+

PLEASE NOTE: --auth-proxy and --authorized-keys cannot be used together, if --auth-proxy is set the authorized keys option will be ignored.

+

There is an example program bin/test_proxy.py in the rclone source code.

+

The program's job is to take a user and pass on the input and turn those into the config for a backend on STDOUT in JSON format. This config will have any default parameters for the backend added, but it won't use configuration from environment variables or command line options - it is the job of the proxy program to make a complete config.

+

This config generated must have this extra parameter - _root - root to use for the backend

+

And it may have this parameter - _obscure - comma separated strings for parameters to obscure

+

If password authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:

+
{
+    "user": "me",
+    "pass": "mypassword"
+}
+

If public-key authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:

+
{
+    "user": "me",
+    "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
+}
+

And as an example return this on STDOUT

+
{
+    "type": "sftp",
+    "_root": "",
+    "_obscure": "pass",
+    "user": "me",
+    "pass": "mypassword",
+    "host": "sftp.example.com"
+}
+

This would mean that an SFTP backend would be created on the fly for the user and pass/public_key returned in the output to the host given. Note that since _obscure is set to pass, rclone will obscure the pass parameter before creating the backend (which is required for sftp backends).

+

The program can manipulate the supplied user in any way, for example to make proxy to many different sftp backends, you could make the user be user@example.com and then set the host to example.com in the output and the user to user. For security you'd probably want to restrict the host to a limited list.

+

Note that an internal cache is keyed on user so only use that for configuration, don't use pass or public_key. This also means that if a user's password or public-key is changed the cache will need to expire (which takes 5 mins) before it takes effect.

+

This can be used to build general purpose proxies to any kind of backend that rclone supports.

+
rclone serve sftp remote:path [flags]
+

Options

+
      --addr string                            IPaddress:Port or :Port to bind server to (default "localhost:2022")
+      --auth-proxy string                      A program to use to create the backend from the auth
+      --authorized-keys string                 Authorized keys file (default "~/.ssh/authorized_keys")
+      --dir-cache-time Duration                Time to cache directory entries for (default 5m0s)
+      --dir-perms FileMode                     Directory permissions (default 0777)
+      --file-perms FileMode                    File permissions (default 0666)
+      --gid uint32                             Override the gid field set by the filesystem (not supported on Windows) (default 1000)
+  -h, --help                                   help for sftp
+      --key stringArray                        SSH private host key file (Can be multi-valued, leave blank to auto generate)
+      --no-auth                                Allow connections with no authentication if set
+      --no-checksum                            Don't compare checksums on up/download
+      --no-modtime                             Don't read/write the modification time (can speed things up)
+      --no-seek                                Don't allow seeking in files
+      --pass string                            Password for authentication
+      --poll-interval Duration                 Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
+      --read-only                              Only allow read-only access
+      --stdio                                  Run an sftp server on stdin/stdout
+      --uid uint32                             Override the uid field set by the filesystem (not supported on Windows) (default 1000)
+      --umask int                              Override the permission bits set by the filesystem (not supported on Windows) (default 2)
+      --user string                            User name for authentication
+      --vfs-cache-max-age Duration             Max time since last access of objects in the cache (default 1h0m0s)
+      --vfs-cache-max-size SizeSuffix          Max total size of objects in the cache (default off)
+      --vfs-cache-min-free-space SizeSuffix    Target minimum free space on the disk containing the cache (default off)
+      --vfs-cache-mode CacheMode               Cache mode off|minimal|writes|full (default off)
+      --vfs-cache-poll-interval Duration       Interval to poll the cache for stale objects (default 1m0s)
+      --vfs-case-insensitive                   If a file name not found, find a case insensitive match
+      --vfs-disk-space-total-size SizeSuffix   Specify the total space of disk (default off)
+      --vfs-fast-fingerprint                   Use fast (less accurate) fingerprints for change detection
+      --vfs-read-ahead SizeSuffix              Extra read ahead over --buffer-size when using cache-mode full
+      --vfs-read-chunk-size SizeSuffix         Read the source objects in chunks (default 128Mi)
+      --vfs-read-chunk-size-limit SizeSuffix   If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
+      --vfs-read-wait Duration                 Time to wait for in-sequence read before seeking (default 20ms)
+      --vfs-refresh                            Refreshes the directory cache recursively in the background on start
+      --vfs-used-is-size rclone size           Use the rclone size algorithm for Used size
+      --vfs-write-back Duration                Time to writeback files after last use when using cache (default 5s)
+      --vfs-write-wait Duration                Time to wait for in-sequence write before giving error (default 1s)
+

Filter Options

+

Flags for filtering directory listings.

+
      --delete-excluded                     Delete files on dest excluded from sync
+      --exclude stringArray                 Exclude files matching pattern
+      --exclude-from stringArray            Read file exclude patterns from file (use - to read from stdin)
+      --exclude-if-present stringArray      Exclude directories if filename is present
+      --files-from stringArray              Read list of source-file names from file (use - to read from stdin)
+      --files-from-raw stringArray          Read list of source-file names from file without any processing of lines (use - to read from stdin)
+  -f, --filter stringArray                  Add a file filtering rule
+      --filter-from stringArray             Read file filtering patterns from a file (use - to read from stdin)
+      --ignore-case                         Ignore case in filters (case insensitive)
+      --include stringArray                 Include files matching pattern
+      --include-from stringArray            Read file include patterns from file (use - to read from stdin)
+      --max-age Duration                    Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+      --max-depth int                       If set limits the recursion depth to this (default -1)
+      --max-size SizeSuffix                 Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
+      --metadata-exclude stringArray        Exclude metadatas matching pattern
+      --metadata-exclude-from stringArray   Read metadata exclude patterns from file (use - to read from stdin)
+      --metadata-filter stringArray         Add a metadata filtering rule
+      --metadata-filter-from stringArray    Read metadata filtering patterns from a file (use - to read from stdin)
+      --metadata-include stringArray        Include metadatas matching pattern
+      --metadata-include-from stringArray   Read metadata include patterns from file (use - to read from stdin)
+      --min-age Duration                    Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+      --min-size SizeSuffix                 Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
+

See the global flags page for global options not listed here.

+

SEE ALSO

+

rclone serve webdav

Serve remote:path over WebDAV.

-

Synopsis

+

Synopsis

Run a basic WebDAV server to serve a remote over HTTP via the WebDAV protocol. This can be viewed with a WebDAV client, through a web browser, or you can make a remote of type WebDAV to read and write it.

WebDAV options

--etag-hash

@@ -5533,7 +5868,7 @@ htpasswd -B htpasswd anotherUser

This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.

Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.

The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.

-

VFS Directory Cache

+

VFS Directory Cache

Using the --dir-cache-time flag, you can control how long a directory should be considered up to date and not refreshed from the backend. Changes made through the VFS will appear immediately or invalidate the cache.

--dir-cache-time duration   Time to cache directory entries for (default 5m0s)
 --poll-interval duration    Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
@@ -5544,12 +5879,12 @@ htpasswd -B htpasswd anotherUser 
rclone rc vfs/forget

Or individual files or directories:

rclone rc vfs/forget file=path/to/file dir=path/to/dir
-

VFS File Buffering

+

VFS File Buffering

The --buffer-size flag determines the amount of memory, that will be used to buffer data in advance.

Each open file will try to keep the specified amount of data in memory at all times. The buffered data is bound to one open file and won't be shared.

This flag is a upper limit for the used memory per open file. The buffer will only use memory for data that is downloaded but not not yet read. If the buffer is empty, only a small amount of memory will be used.

The maximum memory used by rclone for buffering can be up to --buffer-size * open files.

-

VFS File Caching

+

VFS File Caching

These flags control the VFS file caching options. File caching is necessary to make the VFS layer appear compatible with a normal file system. It can be disabled at the cost of some compatibility.

For example you'll need to enable VFS caching if you want to read and write simultaneously to a file. See below for more details.

Note that the VFS cache is separate from the cache backend and you may find that you need one or the other or both.

@@ -5566,7 +5901,7 @@ htpasswd -B htpasswd anotherUser

If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the cache may exceed these quotas for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval. Secondly because open files cannot be evicted from the cache. When --vfs-cache-max-size or --vfs-cache-min-free-size is exceeded, rclone will attempt to evict the least accessed files from the cache first. rclone will start with files that haven't been accessed for the longest. This cache flushing strategy is efficient and more relevant files are likely to remain cached.

The --vfs-cache-max-age will evict files from the cache after the set time since last access has passed. The default value of 1 hour will start evicting files from cache that haven't been accessed for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0 and will wait for 1 more hour before evicting. Specify the time with standard notation, s, m, h, d, w .

You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir. You don't need to worry about this if the remotes in use don't overlap.

-

--vfs-cache-mode off

+

--vfs-cache-mode off

In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.

This will mean some operations are not possible

-

--vfs-cache-mode minimal

+

--vfs-cache-mode minimal

This is very similar to "off" except that files opened for read AND write will be buffered to disk. This means that files opened for write will be a lot more compatible, but uses the minimal disk space.

These operations are not possible

-

--vfs-cache-mode writes

+

--vfs-cache-mode writes

In this mode files opened for read only are still read directly from the remote, write only and read/write files are buffered to disk first.

This mode should support all normal file system operations.

If an upload fails it will be retried at exponentially increasing intervals up to 1 minute.

-

--vfs-cache-mode full

+

--vfs-cache-mode full

In this mode all reads and writes are buffered to and from disk. When data is read from the remote this is buffered to disk as well.

In this mode the files in the cache will be sparse files and rclone will keep track of which bits of the files it has downloaded.

So if an application only reads the starts of each file, then rclone will only buffer the start of the file. These files will appear to be their full size in the cache, but they will be sparse files with only the data that has been downloaded present in them.

@@ -5599,7 +5934,7 @@ htpasswd -B htpasswd anotherUser

When reading a file rclone will read --buffer-size plus --vfs-read-ahead bytes ahead. The --buffer-size is buffered in memory whereas the --vfs-read-ahead is buffered on disk.

When using this mode it is recommended that --buffer-size is not set too large and --vfs-read-ahead is set large if required.

IMPORTANT not all file systems support sparse files. In particular FAT/exFAT do not. Rclone will perform very badly if the cache directory is on a filesystem which doesn't support sparse files and it will log an ERROR message if one is detected.

-

Fingerprinting

+

Fingerprinting

Various parts of the VFS use fingerprinting to see if a local file copy has changed relative to a remote file. Fingerprints are made from:

-
{
-    "SrcFs": "gdrive:",
-    "SrcFsType": "drive",
-    "DstFs": "newdrive:user",
-    "DstFsType": "onedrive",
-    "Remote": "test.txt",
-    "Size": 6,
-    "MimeType": "text/plain; charset=utf-8",
-    "ModTime": "2022-10-11T17:53:10.286745272+01:00",
-    "IsDir": false,
-    "ID": "xyz",
-    "Metadata": {
-        "btime": "2022-10-11T16:53:11Z",
-        "content-type": "text/plain; charset=utf-8",
-        "mtime": "2022-10-11T17:53:10.286745272+01:00",
-        "owner": "user1@domain1.com",
-        "permissions": "...",
-        "description": "my nice file",
-        "starred": "false"
-    }
-}
+
{
+    "SrcFs": "gdrive:",
+    "SrcFsType": "drive",
+    "DstFs": "newdrive:user",
+    "DstFsType": "onedrive",
+    "Remote": "test.txt",
+    "Size": 6,
+    "MimeType": "text/plain; charset=utf-8",
+    "ModTime": "2022-10-11T17:53:10.286745272+01:00",
+    "IsDir": false,
+    "ID": "xyz",
+    "Metadata": {
+        "btime": "2022-10-11T16:53:11Z",
+        "content-type": "text/plain; charset=utf-8",
+        "mtime": "2022-10-11T17:53:10.286745272+01:00",
+        "owner": "user1@domain1.com",
+        "permissions": "...",
+        "description": "my nice file",
+        "starred": "false"
+    }
+}

The program should then modify the input as desired and send it to STDOUT. The returned Metadata field will be used in its entirety for the destination object. Any other fields will be ignored. Note in this example we translate user names and permissions and add something to the description:

-
{
-    "Metadata": {
-        "btime": "2022-10-11T16:53:11Z",
-        "content-type": "text/plain; charset=utf-8",
-        "mtime": "2022-10-11T17:53:10.286745272+01:00",
-        "owner": "user1@domain2.com",
-        "permissions": "...",
-        "description": "my nice file [migrated from domain1]",
-        "starred": "false"
-    }
-}
+
{
+    "Metadata": {
+        "btime": "2022-10-11T16:53:11Z",
+        "content-type": "text/plain; charset=utf-8",
+        "mtime": "2022-10-11T17:53:10.286745272+01:00",
+        "owner": "user1@domain2.com",
+        "permissions": "...",
+        "description": "my nice file [migrated from domain1]",
+        "starred": "false"
+    }
+}

Metadata can be removed here too.

An example python program might look something like this to implement the above transformations.

-
import sys, json
-
-i = json.load(sys.stdin)
-metadata = i["Metadata"]
-# Add tag to description
-if "description" in metadata:
-    metadata["description"] += " [migrated from domain1]"
-else:
-    metadata["description"] = "[migrated from domain1]"
-# Modify owner
-if "owner" in metadata:
-    metadata["owner"] = metadata["owner"].replace("domain1.com", "domain2.com")
-o = { "Metadata": metadata }
-json.dump(o, sys.stdout, indent="\t")
-

You can find this example (slightly expanded) in the rclone source code at bin/test_metadata_mapper.py.

+
import sys, json
+
+i = json.load(sys.stdin)
+metadata = i["Metadata"]
+# Add tag to description
+if "description" in metadata:
+    metadata["description"] += " [migrated from domain1]"
+else:
+    metadata["description"] = "[migrated from domain1]"
+# Modify owner
+if "owner" in metadata:
+    metadata["owner"] = metadata["owner"].replace("domain1.com", "domain2.com")
+o = { "Metadata": metadata }
+json.dump(o, sys.stdout, indent="\t")
+

You can find this example (slightly expanded) in the rclone source code at bin/test_metadata_mapper.py.

If you want to see the input to the metadata mapper and the output returned from it in the log you can use -vv --dump mapper.

See the metadata section for more info.

--metadata-set key=value

@@ -6666,7 +7001,7 @@ y/n/s/!/q> n

In this case the value of this option is used (default 64Mi).

--multi-thread-cutoff=SIZE

When transferring files above SIZE to capable backends, rclone will use multiple threads to transfer the file (default 256M).

-

Capable backends are marked in the overview as MultithreadUpload. (They need to implement either the OpenWriterAt or OpenChunkedWriter internal interfaces). These include include, local, s3, azureblob, b2, oracleobjectstorage and smb at the time of writing.

+

Capable backends are marked in the overview as MultithreadUpload. (They need to implement either the OpenWriterAt or OpenChunkWriter internal interfaces). These include include, local, s3, azureblob, b2, oracleobjectstorage and smb at the time of writing.

On the local disk, rclone preallocates the file (using fallocate(FALLOC_FL_KEEP_SIZE) on unix or NTSetInformationFile on Windows both of which takes no time) then each thread writes directly into the file at the correct place. This means that rclone won't create fragmented or sparse files and there won't be any assembly time at the end of the transfer.

The number of threads used to transfer is controlled by --multi-thread-streams.

Use -vv if you wish to see info about the threads.

@@ -6726,7 +7061,7 @@ y/n/s/!/q> n
  • --order-by name - send the files with alphabetically by path first

    • If the --order-by flag is not supplied or it is supplied with an empty string then the default ordering will be used which is as scanned. With --checkers 1 this is mostly alphabetical, however with the default --checkers 8 it is somewhat random.

    -

    Limitations

    +

    Limitations

    The --order-by flag does not do a separate pass over the data. This means that it may transfer some files out of the order specified if

    Environment Variables

    Rclone can be configured entirely using environment variables. These can be used to set defaults for options or config file entries.

    -

    Options

    +

    Options

    Every option in rclone can have its default set by environment variable.

    To find the name of the environment variable, first, take the long option name, strip the leading --, change - to _, make upper case and prepend RCLONE_.

    For example, to always set --stats 5s, set the environment variable RCLONE_STATS=5s. If you set stats on the command line this will override the environment variable setting.

    @@ -10761,7 +11096,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total --tpslimit float Limit HTTP transactions per second to this --tpslimit-burst int Max burst of transactions for --tpslimit (default 1) --use-cookies Enable session cookiejar - --user-agent string Set the user-agent to a specified string (default "rclone/v1.65.0") + --user-agent string Set the user-agent to a specified string (default "rclone/v1.65.1")

    Performance

    Flags helpful for increasing performance.

          --buffer-size SizeSuffix   In memory buffer size when reading files for each --transfer (default 16Mi)
@@ -12034,7 +12369,7 @@ rclone copy Path2 Path1 --create-empty-src-dirs

    Note that while concurrent bisync runs are allowed, be very cautious that there is no overlap in the trees being synched between concurrent runs, lest there be replicated files, deleted files and general mayhem.

    Return codes

    rclone bisync returns the following codes to calling program: - 0 on a successful run, - 1 for a non-critical failing run (a rerun may be successful), - 2 for a critically aborted run (requires a --resync to recover).

    -

    Limitations

    +

    Limitations

    Supported backends

    Bisync is considered BETA and has been tested with the following backends: - Local filesystem - Google Drive - Dropbox - OneDrive - S3 - SFTP - Yandex Disk

    It has not been fully tested with other services yet. If it works, or sorta works, please let us know and we'll update the list. Run the test suite to check for proper operation as described below.

    @@ -12724,7 +13059,7 @@ y/e/d> y
  • Type: Encoding
  • Default: Slash,LtGt,DoubleQuote,SingleQuote,BackQuote,Dollar,BackSlash,Del,Ctl,LeftSpace,RightSpace,InvalidUtf8,Dot
    • -

    Limitations

    +

    Limitations

    rclone about is not supported by the 1Fichier backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.

    See List of backends that do not support rclone about and rclone about

    Alias

    @@ -12994,7 +13329,7 @@ y/e/d> y
  • Type: Encoding
  • Default: Slash,InvalidUtf8,Dot
    • -

    Limitations

    +

    Limitations

    Note that Amazon Drive is case insensitive so you can't have a file called "Hello.doc" and one called "hello.doc".

    Amazon Drive has rate limiting so you may notice errors in the sync (429 errors). rclone will automatically retry the sync up to 3 times by default (see --retries flag) which should hopefully work around this problem.

    Amazon Drive has an internal limit of file sizes that can be uploaded to the service. This limit is not officially published, but all files larger than this will fail.

    @@ -13434,6 +13769,7 @@ $ rclone -q --s3-versions ls s3:cleanup-test
  • GetObject
  • PutObject
  • PutObjectACL
    • +
  • CreateBucket (unless using s3-no-check-bucket)

    • When using the lsd subcommand, the ListAllMyBuckets permission is required.

    Example policy:

    @@ -13468,6 +13804,7 @@ $ rclone -q --s3-versions ls s3:cleanup-test
    1. This is a policy that can be used when creating bucket. It assumes that USER_NAME has been created.
    2. The Resource entry must include both resource ARNs, as one implies the bucket and the other implies the bucket's objects.
      3. +
    3. When using s3-no-check-bucket and the bucket already exsits, the "arn:aws:s3:::BUCKET_NAME" doesn't have to be included.

    For reference, here's an Ansible script that will generate one or more buckets that will work with rclone sync.

    Key Management System (KMS)

    @@ -16928,7 +17265,7 @@ cos s3

    For Netease NOS configure as per the configurator rclone config setting the provider Netease. This will automatically set force_path_style = false which is necessary for it to run properly.

    Petabox

    Here is an example of making a Petabox configuration. First run:

    -
    rclone config
    +
    rclone config

    This will guide you through an interactive setup process.

    No remotes found, make a new one?
 n) New remote
@@ -17139,7 +17476,7 @@ y/n> n

    Use the the native protocol to take advantage of client-side encryption as well as to achieve the best possible download performance. Uploads will be erasure-coded locally, thus a 1gb upload will result in 2.68gb of data being uploaded to storage nodes across the network.

    Use this backend and the S3 compatible Hosted Gateway to increase upload performance and reduce the load on your systems and network. Uploads will be encrypted and erasure-coded server-side, thus a 1GB upload will result in only in 1GB of data being uploaded to storage nodes across the network.

    For more detailed comparison please check the documentation of the storj backend.

    -

    Limitations

    +

    Limitations

    rclone about is not supported by the S3 backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.

    See List of backends that do not support rclone about and rclone about

    Synology C2 Object Storage

    @@ -33981,7 +34318,7 @@ server_command = sudo /usr/libexec/openssh/sftp-server
  • Type: bool
  • Default: false
    • -

    Limitations

    +

    Limitations

    On some SFTP servers (e.g. Synology) the paths are different for SSH and SFTP so the hashes can't be calculated properly. For them using disable_hashcheck is a good idea.

    The only ssh agent supported under Windows is Putty's pageant.

    The Go SSH library disables the use of the aes128-cbc cipher by default, due to security concerns. This can be re-enabled on a per-connection basis by setting the use_insecure_cipher setting in the configuration file to true. Further details on the insecurity of this cipher can be found in this paper.

    @@ -34469,7 +34806,7 @@ y/e/d> y 
    rclone sync --interactive --progress remote-us:bucket/path/to/dir/ remote-europe:bucket/path/to/dir/

    Or even between another cloud storage and Storj.

    rclone sync --interactive --progress s3:bucket/path/to/dir/ storj:bucket/path/to/dir/
    -

    Limitations

    +

    Limitations

    rclone about is not supported by the rclone Storj backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.

    See List of backends that do not support rclone about and rclone about

    Known issues

    @@ -34663,7 +35000,7 @@ y/e/d> y
  • Type: Encoding
  • Default: Slash,Ctl,InvalidUtf8,Dot
    • -

    Limitations

    +

    Limitations

    rclone about is not supported by the SugarSync backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.

    See List of backends that do not support rclone about and rclone about

    Tardigrade

    @@ -34785,7 +35122,7 @@ y/e/d>
  • Type: Encoding
  • Default: Slash,LtGt,DoubleQuote,BackQuote,Del,Ctl,LeftSpace,InvalidUtf8,Dot
    • -

    Limitations

    +

    Limitations

    Uptobox will delete inactive files that have not been accessed in 60 days.

    rclone about is not supported by this backend an overview of used space can however been seen in the uptobox web interface.

    Union

    @@ -34936,7 +35273,7 @@ e/n/d/r/c/s/q> q

    To check if your upstream supports the field, run rclone about remote: [flags] and see if the required field exists.

    -

    Filters

    +

    Filters

    Policies basically search upstream remotes and create a list of files / paths for functions to work on. The policy is responsible for filtering and sorting. The policy type defines the sorting but filtering is mostly uniform as described below.

    -

    Limitations

    +

    Limitations

    When uploading very large files (bigger than about 5 GiB) you will need to increase the --timeout parameter. This is because Yandex pauses (perhaps to calculate the MD5SUM for the entire file) before returning confirmation that the file has been uploaded. The default handling of timeouts in rclone is to assume a 5 minute pause is an error and close the connection - you'll see net/http: timeout awaiting response headers errors in the logs if this is happening. Setting the timeout to twice the max size of file in GiB should be enough, so if you want to upload a 30 GiB file set a timeout of 2 * 30 = 60m, that is --timeout 60m.

    Having a Yandex Mail account is mandatory to use the Yandex.Disk subscription. Token generation will work without a mail account, but Rclone won't be able to complete any actions.

    [403 - DiskUnsupportedUserAccountTypeError] User account type is not supported.
    @@ -36350,6 +36687,89 @@ $ tree /tmp/b
  • "error": return an error based on option value

    • Changelog

    +

    v1.65.1 - 2024-01-08

    +

    See commits

    +

    v1.65.0 - 2023-11-26

    See commits

    Bugs and Limitations

    -

    Limitations

    +

    Limitations

    Directory timestamps aren't preserved

    Rclone doesn't currently preserve the timestamps of directories. This is because rclone only really considers objects when syncing.

    Rclone struggles with millions of files in a directory/bucket

    @@ -43798,7 +44218,7 @@ THE SOFTWARE.
  • Scott McGillivray
  • Bjørn Erik Pedersen
  • Lukas Loesche
    • -
  • emyarod
    • +
  • emyarod
  • T.C. Ferguson
  • Brandur
  • Dario Giovannetti
    • diff --git a/MANUAL.md b/MANUAL.md index 13bee7b75..a9f888b25 100644 --- a/MANUAL.md +++ b/MANUAL.md @@ -1,6 +1,6 @@ % rclone(1) User Manual % Nick Craig-Wood -% Nov 26, 2023 +% Jan 08, 2024 # Rclone syncs your files to cloud storage @@ -5855,7 +5855,7 @@ rclone mount remote:path /path/to/mountpoint [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -6164,6 +6164,900 @@ See the [global flags page](https://rclone.org/flags/) for global options not li * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. +# rclone nfsmount + +Mount the remote as file system on a mountpoint. + +## Synopsis + +rclone nfsmount allows Linux, FreeBSD, macOS and Windows to +mount any of Rclone's cloud storage systems as a file system with +FUSE. + +First set up your remote using `rclone config`. Check it works with `rclone ls` etc. + +On Linux and macOS, you can run mount in either foreground or background (aka +daemon) mode. Mount runs in foreground mode by default. Use the `--daemon` flag +to force background mode. On Windows you can run mount in foreground only, +the flag is ignored. + +In background mode rclone acts as a generic Unix mount program: the main +program starts, spawns background rclone process to setup and maintain the +mount, waits until success or timeout and exits with appropriate code +(killing the child process if it fails). + +On Linux/macOS/FreeBSD start the mount like this, where `/path/to/local/mount` +is an **empty** **existing** directory: + + rclone nfsmount remote:path/to/files /path/to/local/mount + +On Windows you can start a mount in different ways. See [below](#mounting-modes-on-windows) +for details. If foreground mount is used interactively from a console window, +rclone will serve the mount and occupy the console so another window should be +used to work with the mount until rclone is interrupted e.g. by pressing Ctrl-C. + +The following examples will mount to an automatically assigned drive, +to specific drive letter `X:`, to path `C:\path\parent\mount` +(where parent directory or drive must exist, and mount must **not** exist, +and is not supported when [mounting as a network drive](#mounting-modes-on-windows)), and +the last example will mount as network share `\\cloud\remote` and map it to an +automatically assigned drive: + + rclone nfsmount remote:path/to/files * + rclone nfsmount remote:path/to/files X: + rclone nfsmount remote:path/to/files C:\path\parent\mount + rclone nfsmount remote:path/to/files \\cloud\remote + +When the program ends while in foreground mode, either via Ctrl+C or receiving +a SIGINT or SIGTERM signal, the mount should be automatically stopped. + +When running in background mode the user will have to stop the mount manually: + + # Linux + fusermount -u /path/to/local/mount + # OS X + umount /path/to/local/mount + +The umount operation can fail, for example when the mountpoint is busy. +When that happens, it is the user's responsibility to stop the mount manually. + +The size of the mounted file system will be set according to information retrieved +from the remote, the same as returned by the [rclone about](https://rclone.org/commands/rclone_about/) +command. Remotes with unlimited storage may report the used size only, +then an additional 1 PiB of free space is assumed. If the remote does not +[support](https://rclone.org/overview/#optional-features) the about feature +at all, then 1 PiB is set as both the total and the free size. + +## Installing on Windows + +To run rclone nfsmount on Windows, you will need to +download and install [WinFsp](http://www.secfs.net/winfsp/). + +[WinFsp](https://github.com/winfsp/winfsp) is an open-source +Windows File System Proxy which makes it easy to write user space file +systems for Windows. It provides a FUSE emulation layer which rclone +uses combination with [cgofuse](https://github.com/winfsp/cgofuse). +Both of these packages are by Bill Zissimopoulos who was very helpful +during the implementation of rclone nfsmount for Windows. + +### Mounting modes on windows + +Unlike other operating systems, Microsoft Windows provides a different filesystem +type for network and fixed drives. It optimises access on the assumption fixed +disk drives are fast and reliable, while network drives have relatively high latency +and less reliability. Some settings can also be differentiated between the two types, +for example that Windows Explorer should just display icons and not create preview +thumbnails for image and video files on network drives. + +In most cases, rclone will mount the remote as a normal, fixed disk drive by default. +However, you can also choose to mount it as a remote network drive, often described +as a network share. If you mount an rclone remote using the default, fixed drive mode +and experience unexpected program errors, freezes or other issues, consider mounting +as a network drive instead. + +When mounting as a fixed disk drive you can either mount to an unused drive letter, +or to a path representing a **nonexistent** subdirectory of an **existing** parent +directory or drive. Using the special value `*` will tell rclone to +automatically assign the next available drive letter, starting with Z: and moving backward. +Examples: + + rclone nfsmount remote:path/to/files * + rclone nfsmount remote:path/to/files X: + rclone nfsmount remote:path/to/files C:\path\parent\mount + rclone nfsmount remote:path/to/files X: + +Option `--volname` can be used to set a custom volume name for the mounted +file system. The default is to use the remote name and path. + +To mount as network drive, you can add option `--network-mode` +to your nfsmount command. Mounting to a directory path is not supported in +this mode, it is a limitation Windows imposes on junctions, so the remote must always +be mounted to a drive letter. + + rclone nfsmount remote:path/to/files X: --network-mode + +A volume name specified with `--volname` will be used to create the network share path. +A complete UNC path, such as `\\cloud\remote`, optionally with path +`\\cloud\remote\madeup\path`, will be used as is. Any other +string will be used as the share part, after a default prefix `\\server\`. +If no volume name is specified then `\\server\share` will be used. +You must make sure the volume name is unique when you are mounting more than one drive, +or else the mount command will fail. The share name will treated as the volume label for +the mapped drive, shown in Windows Explorer etc, while the complete +`\\server\share` will be reported as the remote UNC path by +`net use` etc, just like a normal network drive mapping. + +If you specify a full network share UNC path with `--volname`, this will implicitly +set the `--network-mode` option, so the following two examples have same result: + + rclone nfsmount remote:path/to/files X: --network-mode + rclone nfsmount remote:path/to/files X: --volname \\server\share + +You may also specify the network share UNC path as the mountpoint itself. Then rclone +will automatically assign a drive letter, same as with `*` and use that as +mountpoint, and instead use the UNC path specified as the volume name, as if it were +specified with the `--volname` option. This will also implicitly set +the `--network-mode` option. This means the following two examples have same result: + + rclone nfsmount remote:path/to/files \\cloud\remote + rclone nfsmount remote:path/to/files * --volname \\cloud\remote + +There is yet another way to enable network mode, and to set the share path, +and that is to pass the "native" libfuse/WinFsp option directly: +`--fuse-flag --VolumePrefix=\server\share`. Note that the path +must be with just a single backslash prefix in this case. + + +*Note:* In previous versions of rclone this was the only supported method. + +[Read more about drive mapping](https://en.wikipedia.org/wiki/Drive_mapping) + +See also [Limitations](#limitations) section below. + +### Windows filesystem permissions + +The FUSE emulation layer on Windows must convert between the POSIX-based +permission model used in FUSE, and the permission model used in Windows, +based on access-control lists (ACL). + +The mounted filesystem will normally get three entries in its access-control list (ACL), +representing permissions for the POSIX permission scopes: Owner, group and others. +By default, the owner and group will be taken from the current user, and the built-in +group "Everyone" will be used to represent others. The user/group can be customized +with FUSE options "UserName" and "GroupName", +e.g. `-o UserName=user123 -o GroupName="Authenticated Users"`. +The permissions on each entry will be set according to [options](#options) +`--dir-perms` and `--file-perms`, which takes a value in traditional Unix +[numeric notation](https://en.wikipedia.org/wiki/File-system_permissions#Numeric_notation). + +The default permissions corresponds to `--file-perms 0666 --dir-perms 0777`, +i.e. read and write permissions to everyone. This means you will not be able +to start any programs from the mount. To be able to do that you must add +execute permissions, e.g. `--file-perms 0777 --dir-perms 0777` to add it +to everyone. If the program needs to write files, chances are you will +have to enable [VFS File Caching](#vfs-file-caching) as well (see also +[limitations](#limitations)). Note that the default write permission have +some restrictions for accounts other than the owner, specifically it lacks +the "write extended attributes", as explained next. + +The mapping of permissions is not always trivial, and the result you see in +Windows Explorer may not be exactly like you expected. For example, when setting +a value that includes write access for the group or others scope, this will be +mapped to individual permissions "write attributes", "write data" and +"append data", but not "write extended attributes". Windows will then show this +as basic permission "Special" instead of "Write", because "Write" also covers +the "write extended attributes" permission. When setting digit 0 for group or +others, to indicate no permissions, they will still get individual permissions +"read attributes", "read extended attributes" and "read permissions". This is +done for compatibility reasons, e.g. to allow users without additional +permissions to be able to read basic metadata about files like in Unix. + +WinFsp 2021 (version 1.9) introduced a new FUSE option "FileSecurity", +that allows the complete specification of file security descriptors using +[SDDL](https://docs.microsoft.com/en-us/windows/win32/secauthz/security-descriptor-string-format). +With this you get detailed control of the resulting permissions, compared +to use of the POSIX permissions described above, and no additional permissions +will be added automatically for compatibility with Unix. Some example use +cases will following. + +If you set POSIX permissions for only allowing access to the owner, +using `--file-perms 0600 --dir-perms 0700`, the user group and the built-in +"Everyone" group will still be given some special permissions, as described +above. Some programs may then (incorrectly) interpret this as the file being +accessible by everyone, for example an SSH client may warn about "unprotected +private key file". You can work around this by specifying +`-o FileSecurity="D:P(A;;FA;;;OW)"`, which sets file all access (FA) to the +owner (OW), and nothing else. + +When setting write permissions then, except for the owner, this does not +include the "write extended attributes" permission, as mentioned above. +This may prevent applications from writing to files, giving permission denied +error instead. To set working write permissions for the built-in "Everyone" +group, similar to what it gets by default but with the addition of the +"write extended attributes", you can specify +`-o FileSecurity="D:P(A;;FRFW;;;WD)"`, which sets file read (FR) and file +write (FW) to everyone (WD). If file execute (FX) is also needed, then change +to `-o FileSecurity="D:P(A;;FRFWFX;;;WD)"`, or set file all access (FA) to +get full access permissions, including delete, with +`-o FileSecurity="D:P(A;;FA;;;WD)"`. + +### Windows caveats + +Drives created as Administrator are not visible to other accounts, +not even an account that was elevated to Administrator with the +User Account Control (UAC) feature. A result of this is that if you mount +to a drive letter from a Command Prompt run as Administrator, and then try +to access the same drive from Windows Explorer (which does not run as +Administrator), you will not be able to see the mounted drive. + +If you don't need to access the drive from applications running with +administrative privileges, the easiest way around this is to always +create the mount from a non-elevated command prompt. + +To make mapped drives available to the user account that created them +regardless if elevated or not, there is a special Windows setting called +[linked connections](https://docs.microsoft.com/en-us/troubleshoot/windows-client/networking/mapped-drives-not-available-from-elevated-command#detail-to-configure-the-enablelinkedconnections-registry-entry) +that can be enabled. + +It is also possible to make a drive mount available to everyone on the system, +by running the process creating it as the built-in SYSTEM account. +There are several ways to do this: One is to use the command-line +utility [PsExec](https://docs.microsoft.com/en-us/sysinternals/downloads/psexec), +from Microsoft's Sysinternals suite, which has option `-s` to start +processes as the SYSTEM account. Another alternative is to run the mount +command from a Windows Scheduled Task, or a Windows Service, configured +to run as the SYSTEM account. A third alternative is to use the +[WinFsp.Launcher infrastructure](https://github.com/winfsp/winfsp/wiki/WinFsp-Service-Architecture)). +Read more in the [install documentation](https://rclone.org/install/). +Note that when running rclone as another user, it will not use +the configuration file from your profile unless you tell it to +with the [`--config`](https://rclone.org/docs/#config-config-file) option. +Note also that it is now the SYSTEM account that will have the owner +permissions, and other accounts will have permissions according to the +group or others scopes. As mentioned above, these will then not get the +"write extended attributes" permission, and this may prevent writing to +files. You can work around this with the FileSecurity option, see +example above. + +Note that mapping to a directory path, instead of a drive letter, +does not suffer from the same limitations. + +## Mounting on macOS + +Mounting on macOS can be done either via [built-in NFS server](https://rclone.org/commands/rclone_serve_nfs/), [macFUSE](https://osxfuse.github.io/) +(also known as osxfuse) or [FUSE-T](https://www.fuse-t.org/). macFUSE is a traditional +FUSE driver utilizing a macOS kernel extension (kext). FUSE-T is an alternative FUSE system +which "mounts" via an NFSv4 local server. + +# NFS mount + +This method spins up an NFS server using [serve nfs](https://rclone.org/commands/rclone_serve_nfs/) command and mounts +it to the specified mountpoint. If you run this in background mode using |--daemon|, you will need to +send SIGTERM signal to the rclone process using |kill| command to stop the mount. + +### macFUSE Notes + +If installing macFUSE using [dmg packages](https://github.com/osxfuse/osxfuse/releases) from +the website, rclone will locate the macFUSE libraries without any further intervention. +If however, macFUSE is installed using the [macports](https://www.macports.org/) package manager, +the following addition steps are required. + + sudo mkdir /usr/local/lib + cd /usr/local/lib + sudo ln -s /opt/local/lib/libfuse.2.dylib + +### FUSE-T Limitations, Caveats, and Notes + +There are some limitations, caveats, and notes about how it works. These are current as +of FUSE-T version 1.0.14. + +#### ModTime update on read + +As per the [FUSE-T wiki](https://github.com/macos-fuse-t/fuse-t/wiki#caveats): + +> File access and modification times cannot be set separately as it seems to be an +> issue with the NFS client which always modifies both. Can be reproduced with +> 'touch -m' and 'touch -a' commands + +This means that viewing files with various tools, notably macOS Finder, will cause rlcone +to update the modification time of the file. This may make rclone upload a full new copy +of the file. + +#### Unicode Normalization + +Rclone includes flags for unicode normalization with macFUSE that should be updated +for FUSE-T. See [this forum post](https://forum.rclone.org/t/some-unicode-forms-break-mount-on-macos-with-fuse-t/36403) +and [FUSE-T issue #16](https://github.com/macos-fuse-t/fuse-t/issues/16). The following +flag should be added to the `rclone mount` command. + + -o modules=iconv,from_code=UTF-8,to_code=UTF-8 + +#### Read Only mounts + +When mounting with `--read-only`, attempts to write to files will fail *silently* as +opposed to with a clear warning as in macFUSE. + +## Limitations + +Without the use of `--vfs-cache-mode` this can only write files +sequentially, it can only seek when reading. This means that many +applications won't work with their files on an rclone mount without +`--vfs-cache-mode writes` or `--vfs-cache-mode full`. +See the [VFS File Caching](#vfs-file-caching) section for more info. +When using NFS mount on macOS, if you don't specify |--vfs-cache-mode| +the mount point will be read-only. + +The bucket-based remotes (e.g. Swift, S3, Google Compute Storage, B2) +do not support the concept of empty directories, so empty +directories will have a tendency to disappear once they fall out of +the directory cache. + +When `rclone mount` is invoked on Unix with `--daemon` flag, the main rclone +program will wait for the background mount to become ready or until the timeout +specified by the `--daemon-wait` flag. On Linux it can check mount status using +ProcFS so the flag in fact sets **maximum** time to wait, while the real wait +can be less. On macOS / BSD the time to wait is constant and the check is +performed only at the end. We advise you to set wait time on macOS reasonably. + +Only supported on Linux, FreeBSD, OS X and Windows at the moment. + +## rclone nfsmount vs rclone sync/copy + +File systems expect things to be 100% reliable, whereas cloud storage +systems are a long way from 100% reliable. The rclone sync/copy +commands cope with this with lots of retries. However rclone nfsmount +can't use retries in the same way without making local copies of the +uploads. Look at the [VFS File Caching](#vfs-file-caching) +for solutions to make nfsmount more reliable. + +## Attribute caching + +You can use the flag `--attr-timeout` to set the time the kernel caches +the attributes (size, modification time, etc.) for directory entries. + +The default is `1s` which caches files just long enough to avoid +too many callbacks to rclone from the kernel. + +In theory 0s should be the correct value for filesystems which can +change outside the control of the kernel. However this causes quite a +few problems such as +[rclone using too much memory](https://github.com/rclone/rclone/issues/2157), +[rclone not serving files to samba](https://forum.rclone.org/t/rclone-1-39-vs-1-40-mount-issue/5112) +and [excessive time listing directories](https://github.com/rclone/rclone/issues/2095#issuecomment-371141147). + +The kernel can cache the info about a file for the time given by +`--attr-timeout`. You may see corruption if the remote file changes +length during this window. It will show up as either a truncated file +or a file with garbage on the end. With `--attr-timeout 1s` this is +very unlikely but not impossible. The higher you set `--attr-timeout` +the more likely it is. The default setting of "1s" is the lowest +setting which mitigates the problems above. + +If you set it higher (`10s` or `1m` say) then the kernel will call +back to rclone less often making it more efficient, however there is +more chance of the corruption issue above. + +If files don't change on the remote outside of the control of rclone +then there is no chance of corruption. + +This is the same as setting the attr_timeout option in mount.fuse. + +## Filters + +Note that all the rclone filters can be used to select a subset of the +files to be visible in the mount. + +## systemd + +When running rclone nfsmount as a systemd service, it is possible +to use Type=notify. In this case the service will enter the started state +after the mountpoint has been successfully set up. +Units having the rclone nfsmount service specified as a requirement +will see all files and folders immediately in this mode. + +Note that systemd runs mount units without any environment variables including +`PATH` or `HOME`. This means that tilde (`~`) expansion will not work +and you should provide `--config` and `--cache-dir` explicitly as absolute +paths via rclone arguments. +Since mounting requires the `fusermount` program, rclone will use the fallback +PATH of `/bin:/usr/bin` in this scenario. Please ensure that `fusermount` +is present on this PATH. + +## Rclone as Unix mount helper + +The core Unix program `/bin/mount` normally takes the `-t FSTYPE` argument +then runs the `/sbin/mount.FSTYPE` helper program passing it mount options +as `-o key=val,...` or `--opt=...`. Automount (classic or systemd) behaves +in a similar way. + +rclone by default expects GNU-style flags `--key val`. To run it as a mount +helper you should symlink rclone binary to `/sbin/mount.rclone` and optionally +`/usr/bin/rclonefs`, e.g. `ln -s /usr/bin/rclone /sbin/mount.rclone`. +rclone will detect it and translate command-line arguments appropriately. + +Now you can run classic mounts like this: +``` +mount sftp1:subdir /mnt/data -t rclone -o vfs_cache_mode=writes,sftp_key_file=/path/to/pem +``` + +or create systemd mount units: +``` +# /etc/systemd/system/mnt-data.mount +[Unit] +Description=Mount for /mnt/data +[Mount] +Type=rclone +What=sftp1:subdir +Where=/mnt/data +Options=rw,_netdev,allow_other,args2env,vfs-cache-mode=writes,config=/etc/rclone.conf,cache-dir=/var/rclone +``` + +optionally accompanied by systemd automount unit +``` +# /etc/systemd/system/mnt-data.automount +[Unit] +Description=AutoMount for /mnt/data +[Automount] +Where=/mnt/data +TimeoutIdleSec=600 +[Install] +WantedBy=multi-user.target +``` + +or add in `/etc/fstab` a line like +``` +sftp1:subdir /mnt/data rclone rw,noauto,nofail,_netdev,x-systemd.automount,args2env,vfs_cache_mode=writes,config=/etc/rclone.conf,cache_dir=/var/cache/rclone 0 0 +``` + +or use classic Automountd. +Remember to provide explicit `config=...,cache-dir=...` as a workaround for +mount units being run without `HOME`. + +Rclone in the mount helper mode will split `-o` argument(s) by comma, replace `_` +by `-` and prepend `--` to get the command-line flags. Options containing commas +or spaces can be wrapped in single or double quotes. Any inner quotes inside outer +quotes of the same type should be doubled. + +Mount option syntax includes a few extra options treated specially: + +- `env.NAME=VALUE` will set an environment variable for the mount process. + This helps with Automountd and Systemd.mount which don't allow setting + custom environment for mount helpers. + Typically you will use `env.HTTPS_PROXY=proxy.host:3128` or `env.HOME=/root` +- `command=cmount` can be used to run `cmount` or any other rclone command + rather than the default `mount`. +- `args2env` will pass mount options to the mount helper running in background + via environment variables instead of command line arguments. This allows to + hide secrets from such commands as `ps` or `pgrep`. +- `vv...` will be transformed into appropriate `--verbose=N` +- standard mount options like `x-systemd.automount`, `_netdev`, `nosuid` and alike + are intended only for Automountd and ignored by rclone. +## VFS - Virtual File System + +This command uses the VFS layer. This adapts the cloud storage objects +that rclone uses into something which looks much more like a disk +filing system. + +Cloud storage objects have lots of properties which aren't like disk +files - you can't extend them or write to the middle of them, so the +VFS layer has to deal with that. Because there is no one right way of +doing this there are various options explained below. + +The VFS layer also implements a directory cache - this caches info +about files and directories (but not the data) in memory. + +## VFS Directory Cache + +Using the `--dir-cache-time` flag, you can control how long a +directory should be considered up to date and not refreshed from the +backend. Changes made through the VFS will appear immediately or +invalidate the cache. + + --dir-cache-time duration Time to cache directory entries for (default 5m0s) + --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s) + +However, changes made directly on the cloud storage by the web +interface or a different copy of rclone will only be picked up once +the directory cache expires if the backend configured does not support +polling for changes. If the backend supports polling, changes will be +picked up within the polling interval. + +You can send a `SIGHUP` signal to rclone for it to flush all +directory caches, regardless of how old they are. Assuming only one +rclone instance is running, you can reset the cache like this: + + kill -SIGHUP $(pidof rclone) + +If you configure rclone with a [remote control](/rc) then you can use +rclone rc to flush the whole directory cache: + + rclone rc vfs/forget + +Or individual files or directories: + + rclone rc vfs/forget file=path/to/file dir=path/to/dir + +## VFS File Buffering + +The `--buffer-size` flag determines the amount of memory, +that will be used to buffer data in advance. + +Each open file will try to keep the specified amount of data in memory +at all times. The buffered data is bound to one open file and won't be +shared. + +This flag is a upper limit for the used memory per open file. The +buffer will only use memory for data that is downloaded but not not +yet read. If the buffer is empty, only a small amount of memory will +be used. + +The maximum memory used by rclone for buffering can be up to +`--buffer-size * open files`. + +## VFS File Caching + +These flags control the VFS file caching options. File caching is +necessary to make the VFS layer appear compatible with a normal file +system. It can be disabled at the cost of some compatibility. + +For example you'll need to enable VFS caching if you want to read and +write simultaneously to a file. See below for more details. + +Note that the VFS cache is separate from the cache backend and you may +find that you need one or the other or both. + + --cache-dir string Directory rclone will use for caching. + --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) + --vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s) + --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) + --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) + --vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s) + --vfs-write-back duration Time to writeback files after last use when using cache (default 5s) + +If run with `-vv` rclone will print the location of the file cache. The +files are stored in the user cache file area which is OS dependent but +can be controlled with `--cache-dir` or setting the appropriate +environment variable. + +The cache has 4 different modes selected by `--vfs-cache-mode`. +The higher the cache mode the more compatible rclone becomes at the +cost of using disk space. + +Note that files are written back to the remote only when they are +closed and if they haven't been accessed for `--vfs-write-back` +seconds. If rclone is quit or dies with files that haven't been +uploaded, these will be uploaded next time rclone is run with the same +flags. + +If using `--vfs-cache-max-size` or `--vfs-cache-min-free-size` note +that the cache may exceed these quotas for two reasons. Firstly +because it is only checked every `--vfs-cache-poll-interval`. Secondly +because open files cannot be evicted from the cache. When +`--vfs-cache-max-size` or `--vfs-cache-min-free-size` is exceeded, +rclone will attempt to evict the least accessed files from the cache +first. rclone will start with files that haven't been accessed for the +longest. This cache flushing strategy is efficient and more relevant +files are likely to remain cached. + +The `--vfs-cache-max-age` will evict files from the cache +after the set time since last access has passed. The default value of +1 hour will start evicting files from cache that haven't been accessed +for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0 +and will wait for 1 more hour before evicting. Specify the time with +standard notation, s, m, h, d, w . + +You **should not** run two copies of rclone using the same VFS cache +with the same or overlapping remotes if using `--vfs-cache-mode > off`. +This can potentially cause data corruption if you do. You can work +around this by giving each rclone its own cache hierarchy with +`--cache-dir`. You don't need to worry about this if the remotes in +use don't overlap. + +### --vfs-cache-mode off + +In this mode (the default) the cache will read directly from the remote and write +directly to the remote without caching anything on disk. + +This will mean some operations are not possible + + * Files can't be opened for both read AND write + * Files opened for write can't be seeked + * Existing files opened for write must have O_TRUNC set + * Files open for read with O_TRUNC will be opened write only + * Files open for write only will behave as if O_TRUNC was supplied + * Open modes O_APPEND, O_TRUNC are ignored + * If an upload fails it can't be retried + +### --vfs-cache-mode minimal + +This is very similar to "off" except that files opened for read AND +write will be buffered to disk. This means that files opened for +write will be a lot more compatible, but uses the minimal disk space. + +These operations are not possible + + * Files opened for write only can't be seeked + * Existing files opened for write must have O_TRUNC set + * Files opened for write only will ignore O_APPEND, O_TRUNC + * If an upload fails it can't be retried + +### --vfs-cache-mode writes + +In this mode files opened for read only are still read directly from +the remote, write only and read/write files are buffered to disk +first. + +This mode should support all normal file system operations. + +If an upload fails it will be retried at exponentially increasing +intervals up to 1 minute. + +### --vfs-cache-mode full + +In this mode all reads and writes are buffered to and from disk. When +data is read from the remote this is buffered to disk as well. + +In this mode the files in the cache will be sparse files and rclone +will keep track of which bits of the files it has downloaded. + +So if an application only reads the starts of each file, then rclone +will only buffer the start of the file. These files will appear to be +their full size in the cache, but they will be sparse files with only +the data that has been downloaded present in them. + +This mode should support all normal file system operations and is +otherwise identical to `--vfs-cache-mode` writes. + +When reading a file rclone will read `--buffer-size` plus +`--vfs-read-ahead` bytes ahead. The `--buffer-size` is buffered in memory +whereas the `--vfs-read-ahead` is buffered on disk. + +When using this mode it is recommended that `--buffer-size` is not set +too large and `--vfs-read-ahead` is set large if required. + +**IMPORTANT** not all file systems support sparse files. In particular +FAT/exFAT do not. Rclone will perform very badly if the cache +directory is on a filesystem which doesn't support sparse files and it +will log an ERROR message if one is detected. + +### Fingerprinting + +Various parts of the VFS use fingerprinting to see if a local file +copy has changed relative to a remote file. Fingerprints are made +from: + +- size +- modification time +- hash + +where available on an object. + +On some backends some of these attributes are slow to read (they take +an extra API call per object, or extra work per object). + +For example `hash` is slow with the `local` and `sftp` backends as +they have to read the entire file and hash it, and `modtime` is slow +with the `s3`, `swift`, `ftp` and `qinqstor` backends because they +need to do an extra API call to fetch it. + +If you use the `--vfs-fast-fingerprint` flag then rclone will not +include the slow operations in the fingerprint. This makes the +fingerprinting less accurate but much faster and will improve the +opening time of cached files. + +If you are running a vfs cache over `local`, `s3` or `swift` backends +then using this flag is recommended. + +Note that if you change the value of this flag, the fingerprints of +the files in the cache may be invalidated and the files will need to +be downloaded again. + +## VFS Chunked Reading + +When rclone reads files from a remote it reads them in chunks. This +means that rather than requesting the whole file rclone reads the +chunk specified. This can reduce the used download quota for some +remotes by requesting only chunks from the remote that are actually +read, at the cost of an increased number of requests. + +These flags control the chunking: + + --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) + --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + +Rclone will start reading a chunk of size `--vfs-read-chunk-size`, +and then double the size for each read. When `--vfs-read-chunk-size-limit` is +specified, and greater than `--vfs-read-chunk-size`, the chunk size for each +open file will get doubled only until the specified value is reached. If the +value is "off", which is the default, the limit is disabled and the chunk size +will grow indefinitely. + +With `--vfs-read-chunk-size 100M` and `--vfs-read-chunk-size-limit 0` +the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. +When `--vfs-read-chunk-size-limit 500M` is specified, the result would be +0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on. + +Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading. + +## VFS Performance + +These flags may be used to enable/disable features of the VFS for +performance or other reasons. See also the [chunked reading](#vfs-chunked-reading) +feature. + +In particular S3 and Swift benefit hugely from the `--no-modtime` flag +(or use `--use-server-modtime` for a slightly different effect) as each +read of the modification time takes a transaction. + + --no-checksum Don't compare checksums on up/download. + --no-modtime Don't read/write the modification time (can speed things up). + --no-seek Don't allow seeking in files. + --read-only Only allow read-only access. + +Sometimes rclone is delivered reads or writes out of order. Rather +than seeking rclone will wait a short time for the in sequence read or +write to come in. These flags only come into effect when not using an +on disk cache file. + + --vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms) + --vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s) + +When using VFS write caching (`--vfs-cache-mode` with value writes or full), +the global flag `--transfers` can be set to adjust the number of parallel uploads of +modified files from the cache (the related global flag `--checkers` has no effect on the VFS). + + --transfers int Number of file transfers to run in parallel (default 4) + +## VFS Case Sensitivity + +Linux file systems are case-sensitive: two files can differ only +by case, and the exact case must be used when opening a file. + +File systems in modern Windows are case-insensitive but case-preserving: +although existing files can be opened using any case, the exact case used +to create the file is preserved and available for programs to query. +It is not allowed for two files in the same directory to differ only by case. + +Usually file systems on macOS are case-insensitive. It is possible to make macOS +file systems case-sensitive but that is not the default. + +The `--vfs-case-insensitive` VFS flag controls how rclone handles these +two cases. If its value is "false", rclone passes file names to the remote +as-is. If the flag is "true" (or appears without a value on the +command line), rclone may perform a "fixup" as explained below. + +The user may specify a file name to open/delete/rename/etc with a case +different than what is stored on the remote. If an argument refers +to an existing file with exactly the same name, then the case of the existing +file on the disk will be used. However, if a file name with exactly the same +name is not found but a name differing only by case exists, rclone will +transparently fixup the name. This fixup happens only when an existing file +is requested. Case sensitivity of file names created anew by rclone is +controlled by the underlying remote. + +Note that case sensitivity of the operating system running rclone (the target) +may differ from case sensitivity of a file system presented by rclone (the source). +The flag controls whether "fixup" is performed to satisfy the target. + +If the flag is not provided on the command line, then its default value depends +on the operating system where rclone runs: "true" on Windows and macOS, "false" +otherwise. If the flag is provided without a value, then it is "true". + +## VFS Disk Options + +This flag allows you to manually set the statistics about the filing system. +It can be useful when those statistics cannot be read correctly automatically. + + --vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1) + +## Alternate report of used bytes + +Some backends, most notably S3, do not report the amount of bytes used. +If you need this information to be available when running `df` on the +filesystem, then pass the flag `--vfs-used-is-size` to rclone. +With this flag set, instead of relying on the backend to report this +information, rclone will scan the whole remote similar to `rclone size` +and compute the total used space itself. + +_WARNING._ Contrary to `rclone size`, this flag ignores filters so that the +result is accurate. However, this is very inefficient and may cost lots of API +calls resulting in extra charges. Use it as a last resort and only with caching. + + +``` +rclone nfsmount remote:path /path/to/mountpoint [flags] +``` + +## Options + +``` + --allow-non-empty Allow mounting over a non-empty directory (not supported on Windows) + --allow-other Allow access to other users (not supported on Windows) + --allow-root Allow access to root user (not supported on Windows) + --async-read Use asynchronous reads (not supported on Windows) (default true) + --attr-timeout Duration Time for which file/directory attributes are cached (default 1s) + --daemon Run mount in background and exit parent process (as background output is suppressed, use --log-file with --log-format=pid,... to monitor) (not supported on Windows) + --daemon-timeout Duration Time limit for rclone to respond to kernel (not supported on Windows) (default 0s) + --daemon-wait Duration Time to wait for ready mount from daemon (maximum time on Linux, constant sleep time on OSX/BSD) (not supported on Windows) (default 1m0s) + --debug-fuse Debug the FUSE internals - needs -v + --default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows) + --devname string Set the device name - default is remote:path + --dir-cache-time Duration Time to cache directory entries for (default 5m0s) + --dir-perms FileMode Directory permissions (default 0777) + --file-perms FileMode File permissions (default 0666) + --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required) + --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) + -h, --help help for nfsmount + --max-read-ahead SizeSuffix The number of bytes that can be prefetched for sequential reads (not supported on Windows) (default 128Ki) + --mount-case-insensitive Tristate Tell the OS the mount is case insensitive (true) or sensitive (false) regardless of the backend (auto) (default unset) + --network-mode Mount as remote network drive, instead of fixed disk drive (supported on Windows only) + --no-checksum Don't compare checksums on up/download + --no-modtime Don't read/write the modification time (can speed things up) + --no-seek Don't allow seeking in files + --noappledouble Ignore Apple Double (._) and .DS_Store files (supported on OSX only) (default true) + --noapplexattr Ignore all "com.apple.*" extended attributes (supported on OSX only) + -o, --option stringArray Option for libfuse/WinFsp (repeat if required) + --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) + --read-only Only allow read-only access + --sudo Use sudo to run the mount command as root. + --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) + --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) + --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) + --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) + --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) + --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s) + --vfs-case-insensitive If a file name not found, find a case insensitive match + --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off) + --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection + --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full + --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) + --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) + --vfs-refresh Refreshes the directory cache recursively in the background on start + --vfs-used-is-size rclone size Use the rclone size algorithm for Used size + --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) + --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) + --volname string Set the volume name (supported on Windows and OSX only) + --write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows) +``` + + +## Filter Options + +Flags for filtering directory listings. + +``` + --delete-excluded Delete files on dest excluded from sync + --exclude stringArray Exclude files matching pattern + --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) + --exclude-if-present stringArray Exclude directories if filename is present + --files-from stringArray Read list of source-file names from file (use - to read from stdin) + --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) + -f, --filter stringArray Add a file filtering rule + --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) + --ignore-case Ignore case in filters (case insensitive) + --include stringArray Include files matching pattern + --include-from stringArray Read file include patterns from file (use - to read from stdin) + --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) + --max-depth int If set limits the recursion depth to this (default -1) + --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) + --metadata-exclude stringArray Exclude metadatas matching pattern + --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) + --metadata-filter stringArray Add a metadata filtering rule + --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) + --metadata-include stringArray Include metadatas matching pattern + --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) + --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) + --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) +``` + +See the [global flags page](https://rclone.org/flags/) for global options not listed here. + +# SEE ALSO + +* [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. + # rclone obscure Obscure password for use in the rclone config file. @@ -7109,7 +8003,7 @@ rclone serve dlna remote:path [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -7581,7 +8475,7 @@ rclone serve docker [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -7994,7 +8888,7 @@ together, if `--auth-proxy` is set the authorized keys option will be ignored. There is an example program -[bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/test_proxy.py) +[bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/bin/test_proxy.py) in the rclone source code. The program's job is to take a `user` and `pass` on the input and turn @@ -8103,7 +8997,7 @@ rclone serve ftp remote:path [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -8615,7 +9509,7 @@ together, if `--auth-proxy` is set the authorized keys option will be ignored. There is an example program -[bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/test_proxy.py) +[bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/bin/test_proxy.py) in the rclone source code. The program's job is to take a `user` and `pass` on the input and turn @@ -8733,7 +9627,7 @@ rclone serve http remote:path [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -9174,7 +10068,7 @@ rclone serve nfs remote:path [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -9960,7 +10854,7 @@ rclone serve s3 remote:path [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -10403,7 +11297,7 @@ together, if `--auth-proxy` is set the authorized keys option will be ignored. There is an example program -[bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/test_proxy.py) +[bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/bin/test_proxy.py) in the rclone source code. The program's job is to take a `user` and `pass` on the input and turn @@ -10512,7 +11406,7 @@ rclone serve sftp remote:path [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -11053,7 +11947,7 @@ together, if `--auth-proxy` is set the authorized keys option will be ignored. There is an example program -[bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/test_proxy.py) +[bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/bin/test_proxy.py) in the rclone source code. The program's job is to take a `user` and `pass` on the input and turn @@ -11173,7 +12067,7 @@ rclone serve webdav remote:path [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -13145,7 +14039,7 @@ json.dump(o, sys.stdout, indent="\t") ``` You can find this example (slightly expanded) in the rclone source code at -[bin/test_metadata_mapper.py](https://github.com/rclone/rclone/blob/master/test_metadata_mapper.py). +[bin/test_metadata_mapper.py](https://github.com/rclone/rclone/blob/master/bin/test_metadata_mapper.py). If you want to see the input to the metadata mapper and the output returned from it in the log you can use `-vv --dump mapper`. @@ -13205,7 +14099,7 @@ use multiple threads to transfer the file (default 256M). Capable backends are marked in the [overview](https://rclone.org/overview/#optional-features) as `MultithreadUpload`. (They -need to implement either the `OpenWriterAt` or `OpenChunkedWriter` +need to implement either the `OpenWriterAt` or `OpenChunkWriter` internal interfaces). These include include, `local`, `s3`, `azureblob`, `b2`, `oracleobjectstorage` and `smb` at the time of writing. @@ -18256,7 +19150,7 @@ General networking and HTTP stuff. --tpslimit float Limit HTTP transactions per second to this --tpslimit-burst int Max burst of transactions for --tpslimit (default 1) --use-cookies Enable session cookiejar - --user-agent string Set the user-agent to a specified string (default "rclone/v1.65.0") + --user-agent string Set the user-agent to a specified string (default "rclone/v1.65.1") ``` @@ -22323,6 +23217,7 @@ permissions are required to be available on the bucket being written to: * `GetObject` * `PutObject` * `PutObjectACL` +* `CreateBucket` (unless using [s3-no-check-bucket](#s3-no-check-bucket)) When using the `lsd` subcommand, the `ListAllMyBuckets` permission is required. @@ -22364,6 +23259,7 @@ Notes on above: that `USER_NAME` has been created. 2. The Resource entry must include both resource ARNs, as one implies the bucket and the other implies the bucket's objects. +3. When using [s3-no-check-bucket](#s3-no-check-bucket) and the bucket already exsits, the `"arn:aws:s3:::BUCKET_NAME"` doesn't have to be included. For reference, [here's an Ansible script](https://gist.github.com/ebridges/ebfc9042dd7c756cd101cfa807b7ae2b) that will generate one or more buckets that will work with `rclone sync`. @@ -49516,6 +50412,56 @@ Options: # Changelog +## v1.65.1 - 2024-01-08 + +[See commits](https://github.com/rclone/rclone/compare/v1.65.0...v1.65.1) + +* Bug Fixes + * build + * Bump golang.org/x/crypto to fix ssh terrapin CVE-2023-48795 (dependabot) + * Update to go1.21.5 to fix Windows path problems (Nick Craig-Wood) + * Fix docker build on arm/v6 (Nick Craig-Wood) + * install.sh: fix harmless error message on install (Nick Craig-Wood) + * accounting: fix stats to show server side transfers (Nick Craig-Wood) + * doc fixes (albertony, ben-ba, Eli Orzitzer, emyarod, keongalvin, rarspace01) + * nfsmount: Compile for all unix oses, add `--sudo` and fix error/option handling (Nick Craig-Wood) + * operations: Fix files moved by rclone move not being counted as transfers (Nick Craig-Wood) + * oauthutil: Avoid panic when `*token` and `*ts.token` are the same (rkonfj) + * serve s3: Fix listing oddities (Nick Craig-Wood) +* VFS + * Note that `--vfs-refresh` runs in the background (Nick Craig-Wood) +* Azurefiles + * Fix storage base url (Oksana) +* Crypt + * Fix rclone move a file over itself deleting the file (Nick Craig-Wood) +* Chunker + * Fix rclone move a file over itself deleting the file (Nick Craig-Wood) +* Compress + * Fix rclone move a file over itself deleting the file (Nick Craig-Wood) +* Dropbox + * Fix used space on dropbox team accounts (Nick Craig-Wood) +* FTP + * Fix multi-thread copy (WeidiDeng) +* Googlephotos + * Fix nil pointer exception when batch failed (Nick Craig-Wood) +* Hasher + * Fix rclone move a file over itself deleting the file (Nick Craig-Wood) + * Fix invalid memory address error when MaxAge == 0 (nielash) +* Onedrive + * Fix error listing: unknown object type `` (Nick Craig-Wood) + * Fix "unauthenticated: Unauthenticated" errors when uploading (Nick Craig-Wood) +* Oracleobjectstorage + * Fix object storage endpoint for custom endpoints (Manoj Ghosh) + * Multipart copy create bucket if it doesn't exist. (Manoj Ghosh) +* Protondrive + * Fix CVE-2023-45286 / GHSA-xwh9-gc39-5298 (Nick Craig-Wood) +* S3 + * Fix crash if no UploadId in multipart upload (Nick Craig-Wood) +* Smb + * Fix shares not listed by updating go-smb2 (halms) +* Union + * Fix rclone move a file over itself deleting the file (Nick Craig-Wood) + ## v1.65.0 - 2023-11-26 [See commits](https://github.com/rclone/rclone/compare/v1.64.0...v1.65.0) @@ -54799,7 +55745,7 @@ put them back in again.` >}} * Scott McGillivray * Bjørn Erik Pedersen * Lukas Loesche - * emyarod + * emyarod * T.C. Ferguson * Brandur * Dario Giovannetti diff --git a/MANUAL.txt b/MANUAL.txt index 11a38590b..b26c87bb7 100644 --- a/MANUAL.txt +++ b/MANUAL.txt @@ -1,6 +1,6 @@ rclone(1) User Manual Nick Craig-Wood -Nov 26, 2023 +Jan 08, 2024 Rclone syncs your files to cloud storage @@ -5332,7 +5332,7 @@ Options --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -5608,6 +5608,889 @@ SEE ALSO - rclone - Show help for rclone commands, flags and backends. +rclone nfsmount + +Mount the remote as file system on a mountpoint. + +Synopsis + +rclone nfsmount allows Linux, FreeBSD, macOS and Windows to mount any of +Rclone's cloud storage systems as a file system with FUSE. + +First set up your remote using rclone config. Check it works with +rclone ls etc. + +On Linux and macOS, you can run mount in either foreground or background +(aka daemon) mode. Mount runs in foreground mode by default. Use the +--daemon flag to force background mode. On Windows you can run mount in +foreground only, the flag is ignored. + +In background mode rclone acts as a generic Unix mount program: the main +program starts, spawns background rclone process to setup and maintain +the mount, waits until success or timeout and exits with appropriate +code (killing the child process if it fails). + +On Linux/macOS/FreeBSD start the mount like this, where +/path/to/local/mount is an empty existing directory: + + rclone nfsmount remote:path/to/files /path/to/local/mount + +On Windows you can start a mount in different ways. See below for +details. If foreground mount is used interactively from a console +window, rclone will serve the mount and occupy the console so another +window should be used to work with the mount until rclone is interrupted +e.g. by pressing Ctrl-C. + +The following examples will mount to an automatically assigned drive, to +specific drive letter X:, to path C:\path\parent\mount (where parent +directory or drive must exist, and mount must not exist, and is not +supported when mounting as a network drive), and the last example will +mount as network share \\cloud\remote and map it to an automatically +assigned drive: + + rclone nfsmount remote:path/to/files * + rclone nfsmount remote:path/to/files X: + rclone nfsmount remote:path/to/files C:\path\parent\mount + rclone nfsmount remote:path/to/files \\cloud\remote + +When the program ends while in foreground mode, either via Ctrl+C or +receiving a SIGINT or SIGTERM signal, the mount should be automatically +stopped. + +When running in background mode the user will have to stop the mount +manually: + + # Linux + fusermount -u /path/to/local/mount + # OS X + umount /path/to/local/mount + +The umount operation can fail, for example when the mountpoint is busy. +When that happens, it is the user's responsibility to stop the mount +manually. + +The size of the mounted file system will be set according to information +retrieved from the remote, the same as returned by the rclone about +command. Remotes with unlimited storage may report the used size only, +then an additional 1 PiB of free space is assumed. If the remote does +not support the about feature at all, then 1 PiB is set as both the +total and the free size. + +Installing on Windows + +To run rclone nfsmount on Windows, you will need to download and install +WinFsp. + +WinFsp is an open-source Windows File System Proxy which makes it easy +to write user space file systems for Windows. It provides a FUSE +emulation layer which rclone uses combination with cgofuse. Both of +these packages are by Bill Zissimopoulos who was very helpful during the +implementation of rclone nfsmount for Windows. + +Mounting modes on windows + +Unlike other operating systems, Microsoft Windows provides a different +filesystem type for network and fixed drives. It optimises access on the +assumption fixed disk drives are fast and reliable, while network drives +have relatively high latency and less reliability. Some settings can +also be differentiated between the two types, for example that Windows +Explorer should just display icons and not create preview thumbnails for +image and video files on network drives. + +In most cases, rclone will mount the remote as a normal, fixed disk +drive by default. However, you can also choose to mount it as a remote +network drive, often described as a network share. If you mount an +rclone remote using the default, fixed drive mode and experience +unexpected program errors, freezes or other issues, consider mounting as +a network drive instead. + +When mounting as a fixed disk drive you can either mount to an unused +drive letter, or to a path representing a nonexistent subdirectory of an +existing parent directory or drive. Using the special value * will tell +rclone to automatically assign the next available drive letter, starting +with Z: and moving backward. Examples: + + rclone nfsmount remote:path/to/files * + rclone nfsmount remote:path/to/files X: + rclone nfsmount remote:path/to/files C:\path\parent\mount + rclone nfsmount remote:path/to/files X: + +Option --volname can be used to set a custom volume name for the mounted +file system. The default is to use the remote name and path. + +To mount as network drive, you can add option --network-mode to your +nfsmount command. Mounting to a directory path is not supported in this +mode, it is a limitation Windows imposes on junctions, so the remote +must always be mounted to a drive letter. + + rclone nfsmount remote:path/to/files X: --network-mode + +A volume name specified with --volname will be used to create the +network share path. A complete UNC path, such as \\cloud\remote, +optionally with path \\cloud\remote\madeup\path, will be used as is. Any +other string will be used as the share part, after a default prefix +\\server\. If no volume name is specified then \\server\share will be +used. You must make sure the volume name is unique when you are mounting +more than one drive, or else the mount command will fail. The share name +will treated as the volume label for the mapped drive, shown in Windows +Explorer etc, while the complete \\server\share will be reported as the +remote UNC path by net use etc, just like a normal network drive +mapping. + +If you specify a full network share UNC path with --volname, this will +implicitly set the --network-mode option, so the following two examples +have same result: + + rclone nfsmount remote:path/to/files X: --network-mode + rclone nfsmount remote:path/to/files X: --volname \\server\share + +You may also specify the network share UNC path as the mountpoint +itself. Then rclone will automatically assign a drive letter, same as +with * and use that as mountpoint, and instead use the UNC path +specified as the volume name, as if it were specified with the --volname +option. This will also implicitly set the --network-mode option. This +means the following two examples have same result: + + rclone nfsmount remote:path/to/files \\cloud\remote + rclone nfsmount remote:path/to/files * --volname \\cloud\remote + +There is yet another way to enable network mode, and to set the share +path, and that is to pass the "native" libfuse/WinFsp option directly: +--fuse-flag --VolumePrefix=\server\share. Note that the path must be +with just a single backslash prefix in this case. + +Note: In previous versions of rclone this was the only supported method. + +Read more about drive mapping + +See also Limitations section below. + +Windows filesystem permissions + +The FUSE emulation layer on Windows must convert between the POSIX-based +permission model used in FUSE, and the permission model used in Windows, +based on access-control lists (ACL). + +The mounted filesystem will normally get three entries in its +access-control list (ACL), representing permissions for the POSIX +permission scopes: Owner, group and others. By default, the owner and +group will be taken from the current user, and the built-in group +"Everyone" will be used to represent others. The user/group can be +customized with FUSE options "UserName" and "GroupName", e.g. +-o UserName=user123 -o GroupName="Authenticated Users". The permissions +on each entry will be set according to options --dir-perms and +--file-perms, which takes a value in traditional Unix numeric notation. + +The default permissions corresponds to +--file-perms 0666 --dir-perms 0777, i.e. read and write permissions to +everyone. This means you will not be able to start any programs from the +mount. To be able to do that you must add execute permissions, e.g. +--file-perms 0777 --dir-perms 0777 to add it to everyone. If the program +needs to write files, chances are you will have to enable VFS File +Caching as well (see also limitations). Note that the default write +permission have some restrictions for accounts other than the owner, +specifically it lacks the "write extended attributes", as explained +next. + +The mapping of permissions is not always trivial, and the result you see +in Windows Explorer may not be exactly like you expected. For example, +when setting a value that includes write access for the group or others +scope, this will be mapped to individual permissions "write attributes", +"write data" and "append data", but not "write extended attributes". +Windows will then show this as basic permission "Special" instead of +"Write", because "Write" also covers the "write extended attributes" +permission. When setting digit 0 for group or others, to indicate no +permissions, they will still get individual permissions "read +attributes", "read extended attributes" and "read permissions". This is +done for compatibility reasons, e.g. to allow users without additional +permissions to be able to read basic metadata about files like in Unix. + +WinFsp 2021 (version 1.9) introduced a new FUSE option "FileSecurity", +that allows the complete specification of file security descriptors +using SDDL. With this you get detailed control of the resulting +permissions, compared to use of the POSIX permissions described above, +and no additional permissions will be added automatically for +compatibility with Unix. Some example use cases will following. + +If you set POSIX permissions for only allowing access to the owner, +using --file-perms 0600 --dir-perms 0700, the user group and the +built-in "Everyone" group will still be given some special permissions, +as described above. Some programs may then (incorrectly) interpret this +as the file being accessible by everyone, for example an SSH client may +warn about "unprotected private key file". You can work around this by +specifying -o FileSecurity="D:P(A;;FA;;;OW)", which sets file all access +(FA) to the owner (OW), and nothing else. + +When setting write permissions then, except for the owner, this does not +include the "write extended attributes" permission, as mentioned above. +This may prevent applications from writing to files, giving permission +denied error instead. To set working write permissions for the built-in +"Everyone" group, similar to what it gets by default but with the +addition of the "write extended attributes", you can specify +-o FileSecurity="D:P(A;;FRFW;;;WD)", which sets file read (FR) and file +write (FW) to everyone (WD). If file execute (FX) is also needed, then +change to -o FileSecurity="D:P(A;;FRFWFX;;;WD)", or set file all access +(FA) to get full access permissions, including delete, with +-o FileSecurity="D:P(A;;FA;;;WD)". + +Windows caveats + +Drives created as Administrator are not visible to other accounts, not +even an account that was elevated to Administrator with the User Account +Control (UAC) feature. A result of this is that if you mount to a drive +letter from a Command Prompt run as Administrator, and then try to +access the same drive from Windows Explorer (which does not run as +Administrator), you will not be able to see the mounted drive. + +If you don't need to access the drive from applications running with +administrative privileges, the easiest way around this is to always +create the mount from a non-elevated command prompt. + +To make mapped drives available to the user account that created them +regardless if elevated or not, there is a special Windows setting called +linked connections that can be enabled. + +It is also possible to make a drive mount available to everyone on the +system, by running the process creating it as the built-in SYSTEM +account. There are several ways to do this: One is to use the +command-line utility PsExec, from Microsoft's Sysinternals suite, which +has option -s to start processes as the SYSTEM account. Another +alternative is to run the mount command from a Windows Scheduled Task, +or a Windows Service, configured to run as the SYSTEM account. A third +alternative is to use the WinFsp.Launcher infrastructure). Read more in +the install documentation. Note that when running rclone as another +user, it will not use the configuration file from your profile unless +you tell it to with the --config option. Note also that it is now the +SYSTEM account that will have the owner permissions, and other accounts +will have permissions according to the group or others scopes. As +mentioned above, these will then not get the "write extended attributes" +permission, and this may prevent writing to files. You can work around +this with the FileSecurity option, see example above. + +Note that mapping to a directory path, instead of a drive letter, does +not suffer from the same limitations. + +Mounting on macOS + +Mounting on macOS can be done either via built-in NFS server, macFUSE +(also known as osxfuse) or FUSE-T. macFUSE is a traditional FUSE driver +utilizing a macOS kernel extension (kext). FUSE-T is an alternative FUSE +system which "mounts" via an NFSv4 local server. + +NFS mount + +This method spins up an NFS server using serve nfs command and mounts it +to the specified mountpoint. If you run this in background mode using +|--daemon|, you will need to send SIGTERM signal to the rclone process +using |kill| command to stop the mount. + +macFUSE Notes + +If installing macFUSE using dmg packages from the website, rclone will +locate the macFUSE libraries without any further intervention. If +however, macFUSE is installed using the macports package manager, the +following addition steps are required. + + sudo mkdir /usr/local/lib + cd /usr/local/lib + sudo ln -s /opt/local/lib/libfuse.2.dylib + +FUSE-T Limitations, Caveats, and Notes + +There are some limitations, caveats, and notes about how it works. These +are current as of FUSE-T version 1.0.14. + +ModTime update on read + +As per the FUSE-T wiki: + + File access and modification times cannot be set separately as it + seems to be an issue with the NFS client which always modifies both. + Can be reproduced with 'touch -m' and 'touch -a' commands + +This means that viewing files with various tools, notably macOS Finder, +will cause rlcone to update the modification time of the file. This may +make rclone upload a full new copy of the file. + +Unicode Normalization + +Rclone includes flags for unicode normalization with macFUSE that should +be updated for FUSE-T. See this forum post and FUSE-T issue #16. The +following flag should be added to the rclone mount command. + + -o modules=iconv,from_code=UTF-8,to_code=UTF-8 + +Read Only mounts + +When mounting with --read-only, attempts to write to files will fail +silently as opposed to with a clear warning as in macFUSE. + +Limitations + +Without the use of --vfs-cache-mode this can only write files +sequentially, it can only seek when reading. This means that many +applications won't work with their files on an rclone mount without +--vfs-cache-mode writes or --vfs-cache-mode full. See the VFS File +Caching section for more info. When using NFS mount on macOS, if you +don't specify |--vfs-cache-mode| the mount point will be read-only. + +The bucket-based remotes (e.g. Swift, S3, Google Compute Storage, B2) do +not support the concept of empty directories, so empty directories will +have a tendency to disappear once they fall out of the directory cache. + +When rclone mount is invoked on Unix with --daemon flag, the main rclone +program will wait for the background mount to become ready or until the +timeout specified by the --daemon-wait flag. On Linux it can check mount +status using ProcFS so the flag in fact sets maximum time to wait, while +the real wait can be less. On macOS / BSD the time to wait is constant +and the check is performed only at the end. We advise you to set wait +time on macOS reasonably. + +Only supported on Linux, FreeBSD, OS X and Windows at the moment. + +rclone nfsmount vs rclone sync/copy + +File systems expect things to be 100% reliable, whereas cloud storage +systems are a long way from 100% reliable. The rclone sync/copy commands +cope with this with lots of retries. However rclone nfsmount can't use +retries in the same way without making local copies of the uploads. Look +at the VFS File Caching for solutions to make nfsmount more reliable. + +Attribute caching + +You can use the flag --attr-timeout to set the time the kernel caches +the attributes (size, modification time, etc.) for directory entries. + +The default is 1s which caches files just long enough to avoid too many +callbacks to rclone from the kernel. + +In theory 0s should be the correct value for filesystems which can +change outside the control of the kernel. However this causes quite a +few problems such as rclone using too much memory, rclone not serving +files to samba and excessive time listing directories. + +The kernel can cache the info about a file for the time given by +--attr-timeout. You may see corruption if the remote file changes length +during this window. It will show up as either a truncated file or a file +with garbage on the end. With --attr-timeout 1s this is very unlikely +but not impossible. The higher you set --attr-timeout the more likely it +is. The default setting of "1s" is the lowest setting which mitigates +the problems above. + +If you set it higher (10s or 1m say) then the kernel will call back to +rclone less often making it more efficient, however there is more chance +of the corruption issue above. + +If files don't change on the remote outside of the control of rclone +then there is no chance of corruption. + +This is the same as setting the attr_timeout option in mount.fuse. + +Filters + +Note that all the rclone filters can be used to select a subset of the +files to be visible in the mount. + +systemd + +When running rclone nfsmount as a systemd service, it is possible to use +Type=notify. In this case the service will enter the started state after +the mountpoint has been successfully set up. Units having the rclone +nfsmount service specified as a requirement will see all files and +folders immediately in this mode. + +Note that systemd runs mount units without any environment variables +including PATH or HOME. This means that tilde (~) expansion will not +work and you should provide --config and --cache-dir explicitly as +absolute paths via rclone arguments. Since mounting requires the +fusermount program, rclone will use the fallback PATH of /bin:/usr/bin +in this scenario. Please ensure that fusermount is present on this PATH. + +Rclone as Unix mount helper + +The core Unix program /bin/mount normally takes the -t FSTYPE argument +then runs the /sbin/mount.FSTYPE helper program passing it mount options +as -o key=val,... or --opt=.... Automount (classic or systemd) behaves +in a similar way. + +rclone by default expects GNU-style flags --key val. To run it as a +mount helper you should symlink rclone binary to /sbin/mount.rclone and +optionally /usr/bin/rclonefs, e.g. +ln -s /usr/bin/rclone /sbin/mount.rclone. rclone will detect it and +translate command-line arguments appropriately. + +Now you can run classic mounts like this: + + mount sftp1:subdir /mnt/data -t rclone -o vfs_cache_mode=writes,sftp_key_file=/path/to/pem + +or create systemd mount units: + + # /etc/systemd/system/mnt-data.mount + [Unit] + Description=Mount for /mnt/data + [Mount] + Type=rclone + What=sftp1:subdir + Where=/mnt/data + Options=rw,_netdev,allow_other,args2env,vfs-cache-mode=writes,config=/etc/rclone.conf,cache-dir=/var/rclone + +optionally accompanied by systemd automount unit + + # /etc/systemd/system/mnt-data.automount + [Unit] + Description=AutoMount for /mnt/data + [Automount] + Where=/mnt/data + TimeoutIdleSec=600 + [Install] + WantedBy=multi-user.target + +or add in /etc/fstab a line like + + sftp1:subdir /mnt/data rclone rw,noauto,nofail,_netdev,x-systemd.automount,args2env,vfs_cache_mode=writes,config=/etc/rclone.conf,cache_dir=/var/cache/rclone 0 0 + +or use classic Automountd. Remember to provide explicit +config=...,cache-dir=... as a workaround for mount units being run +without HOME. + +Rclone in the mount helper mode will split -o argument(s) by comma, +replace _ by - and prepend -- to get the command-line flags. Options +containing commas or spaces can be wrapped in single or double quotes. +Any inner quotes inside outer quotes of the same type should be doubled. + +Mount option syntax includes a few extra options treated specially: + +- env.NAME=VALUE will set an environment variable for the mount + process. This helps with Automountd and Systemd.mount which don't + allow setting custom environment for mount helpers. Typically you + will use env.HTTPS_PROXY=proxy.host:3128 or env.HOME=/root +- command=cmount can be used to run cmount or any other rclone command + rather than the default mount. +- args2env will pass mount options to the mount helper running in + background via environment variables instead of command line + arguments. This allows to hide secrets from such commands as ps or + pgrep. +- vv... will be transformed into appropriate --verbose=N +- standard mount options like x-systemd.automount, _netdev, nosuid and + alike are intended only for Automountd and ignored by rclone. ## + VFS - Virtual File System + +This command uses the VFS layer. This adapts the cloud storage objects +that rclone uses into something which looks much more like a disk filing +system. + +Cloud storage objects have lots of properties which aren't like disk +files - you can't extend them or write to the middle of them, so the VFS +layer has to deal with that. Because there is no one right way of doing +this there are various options explained below. + +The VFS layer also implements a directory cache - this caches info about +files and directories (but not the data) in memory. + +VFS Directory Cache + +Using the --dir-cache-time flag, you can control how long a directory +should be considered up to date and not refreshed from the backend. +Changes made through the VFS will appear immediately or invalidate the +cache. + + --dir-cache-time duration Time to cache directory entries for (default 5m0s) + --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s) + +However, changes made directly on the cloud storage by the web interface +or a different copy of rclone will only be picked up once the directory +cache expires if the backend configured does not support polling for +changes. If the backend supports polling, changes will be picked up +within the polling interval. + +You can send a SIGHUP signal to rclone for it to flush all directory +caches, regardless of how old they are. Assuming only one rclone +instance is running, you can reset the cache like this: + + kill -SIGHUP $(pidof rclone) + +If you configure rclone with a remote control then you can use rclone rc +to flush the whole directory cache: + + rclone rc vfs/forget + +Or individual files or directories: + + rclone rc vfs/forget file=path/to/file dir=path/to/dir + +VFS File Buffering + +The --buffer-size flag determines the amount of memory, that will be +used to buffer data in advance. + +Each open file will try to keep the specified amount of data in memory +at all times. The buffered data is bound to one open file and won't be +shared. + +This flag is a upper limit for the used memory per open file. The buffer +will only use memory for data that is downloaded but not not yet read. +If the buffer is empty, only a small amount of memory will be used. + +The maximum memory used by rclone for buffering can be up to +--buffer-size * open files. + +VFS File Caching + +These flags control the VFS file caching options. File caching is +necessary to make the VFS layer appear compatible with a normal file +system. It can be disabled at the cost of some compatibility. + +For example you'll need to enable VFS caching if you want to read and +write simultaneously to a file. See below for more details. + +Note that the VFS cache is separate from the cache backend and you may +find that you need one or the other or both. + + --cache-dir string Directory rclone will use for caching. + --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) + --vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s) + --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) + --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) + --vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s) + --vfs-write-back duration Time to writeback files after last use when using cache (default 5s) + +If run with -vv rclone will print the location of the file cache. The +files are stored in the user cache file area which is OS dependent but +can be controlled with --cache-dir or setting the appropriate +environment variable. + +The cache has 4 different modes selected by --vfs-cache-mode. The higher +the cache mode the more compatible rclone becomes at the cost of using +disk space. + +Note that files are written back to the remote only when they are closed +and if they haven't been accessed for --vfs-write-back seconds. If +rclone is quit or dies with files that haven't been uploaded, these will +be uploaded next time rclone is run with the same flags. + +If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the +cache may exceed these quotas for two reasons. Firstly because it is +only checked every --vfs-cache-poll-interval. Secondly because open +files cannot be evicted from the cache. When --vfs-cache-max-size or +--vfs-cache-min-free-size is exceeded, rclone will attempt to evict the +least accessed files from the cache first. rclone will start with files +that haven't been accessed for the longest. This cache flushing strategy +is efficient and more relevant files are likely to remain cached. + +The --vfs-cache-max-age will evict files from the cache after the set +time since last access has passed. The default value of 1 hour will +start evicting files from cache that haven't been accessed for 1 hour. +When a cached file is accessed the 1 hour timer is reset to 0 and will +wait for 1 more hour before evicting. Specify the time with standard +notation, s, m, h, d, w . + +You should not run two copies of rclone using the same VFS cache with +the same or overlapping remotes if using --vfs-cache-mode > off. This +can potentially cause data corruption if you do. You can work around +this by giving each rclone its own cache hierarchy with --cache-dir. You +don't need to worry about this if the remotes in use don't overlap. + +--vfs-cache-mode off + +In this mode (the default) the cache will read directly from the remote +and write directly to the remote without caching anything on disk. + +This will mean some operations are not possible + +- Files can't be opened for both read AND write +- Files opened for write can't be seeked +- Existing files opened for write must have O_TRUNC set +- Files open for read with O_TRUNC will be opened write only +- Files open for write only will behave as if O_TRUNC was supplied +- Open modes O_APPEND, O_TRUNC are ignored +- If an upload fails it can't be retried + +--vfs-cache-mode minimal + +This is very similar to "off" except that files opened for read AND +write will be buffered to disk. This means that files opened for write +will be a lot more compatible, but uses the minimal disk space. + +These operations are not possible + +- Files opened for write only can't be seeked +- Existing files opened for write must have O_TRUNC set +- Files opened for write only will ignore O_APPEND, O_TRUNC +- If an upload fails it can't be retried + +--vfs-cache-mode writes + +In this mode files opened for read only are still read directly from the +remote, write only and read/write files are buffered to disk first. + +This mode should support all normal file system operations. + +If an upload fails it will be retried at exponentially increasing +intervals up to 1 minute. + +--vfs-cache-mode full + +In this mode all reads and writes are buffered to and from disk. When +data is read from the remote this is buffered to disk as well. + +In this mode the files in the cache will be sparse files and rclone will +keep track of which bits of the files it has downloaded. + +So if an application only reads the starts of each file, then rclone +will only buffer the start of the file. These files will appear to be +their full size in the cache, but they will be sparse files with only +the data that has been downloaded present in them. + +This mode should support all normal file system operations and is +otherwise identical to --vfs-cache-mode writes. + +When reading a file rclone will read --buffer-size plus --vfs-read-ahead +bytes ahead. The --buffer-size is buffered in memory whereas the +--vfs-read-ahead is buffered on disk. + +When using this mode it is recommended that --buffer-size is not set too +large and --vfs-read-ahead is set large if required. + +IMPORTANT not all file systems support sparse files. In particular +FAT/exFAT do not. Rclone will perform very badly if the cache directory +is on a filesystem which doesn't support sparse files and it will log an +ERROR message if one is detected. + +Fingerprinting + +Various parts of the VFS use fingerprinting to see if a local file copy +has changed relative to a remote file. Fingerprints are made from: + +- size +- modification time +- hash + +where available on an object. + +On some backends some of these attributes are slow to read (they take an +extra API call per object, or extra work per object). + +For example hash is slow with the local and sftp backends as they have +to read the entire file and hash it, and modtime is slow with the s3, +swift, ftp and qinqstor backends because they need to do an extra API +call to fetch it. + +If you use the --vfs-fast-fingerprint flag then rclone will not include +the slow operations in the fingerprint. This makes the fingerprinting +less accurate but much faster and will improve the opening time of +cached files. + +If you are running a vfs cache over local, s3 or swift backends then +using this flag is recommended. + +Note that if you change the value of this flag, the fingerprints of the +files in the cache may be invalidated and the files will need to be +downloaded again. + +VFS Chunked Reading + +When rclone reads files from a remote it reads them in chunks. This +means that rather than requesting the whole file rclone reads the chunk +specified. This can reduce the used download quota for some remotes by +requesting only chunks from the remote that are actually read, at the +cost of an increased number of requests. + +These flags control the chunking: + + --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) + --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + +Rclone will start reading a chunk of size --vfs-read-chunk-size, and +then double the size for each read. When --vfs-read-chunk-size-limit is +specified, and greater than --vfs-read-chunk-size, the chunk size for +each open file will get doubled only until the specified value is +reached. If the value is "off", which is the default, the limit is +disabled and the chunk size will grow indefinitely. + +With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the +following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, +300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, +the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, +1200M-1700M and so on. + +Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading. + +VFS Performance + +These flags may be used to enable/disable features of the VFS for +performance or other reasons. See also the chunked reading feature. + +In particular S3 and Swift benefit hugely from the --no-modtime flag (or +use --use-server-modtime for a slightly different effect) as each read +of the modification time takes a transaction. + + --no-checksum Don't compare checksums on up/download. + --no-modtime Don't read/write the modification time (can speed things up). + --no-seek Don't allow seeking in files. + --read-only Only allow read-only access. + +Sometimes rclone is delivered reads or writes out of order. Rather than +seeking rclone will wait a short time for the in sequence read or write +to come in. These flags only come into effect when not using an on disk +cache file. + + --vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms) + --vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s) + +When using VFS write caching (--vfs-cache-mode with value writes or +full), the global flag --transfers can be set to adjust the number of +parallel uploads of modified files from the cache (the related global +flag --checkers has no effect on the VFS). + + --transfers int Number of file transfers to run in parallel (default 4) + +VFS Case Sensitivity + +Linux file systems are case-sensitive: two files can differ only by +case, and the exact case must be used when opening a file. + +File systems in modern Windows are case-insensitive but case-preserving: +although existing files can be opened using any case, the exact case +used to create the file is preserved and available for programs to +query. It is not allowed for two files in the same directory to differ +only by case. + +Usually file systems on macOS are case-insensitive. It is possible to +make macOS file systems case-sensitive but that is not the default. + +The --vfs-case-insensitive VFS flag controls how rclone handles these +two cases. If its value is "false", rclone passes file names to the +remote as-is. If the flag is "true" (or appears without a value on the +command line), rclone may perform a "fixup" as explained below. + +The user may specify a file name to open/delete/rename/etc with a case +different than what is stored on the remote. If an argument refers to an +existing file with exactly the same name, then the case of the existing +file on the disk will be used. However, if a file name with exactly the +same name is not found but a name differing only by case exists, rclone +will transparently fixup the name. This fixup happens only when an +existing file is requested. Case sensitivity of file names created anew +by rclone is controlled by the underlying remote. + +Note that case sensitivity of the operating system running rclone (the +target) may differ from case sensitivity of a file system presented by +rclone (the source). The flag controls whether "fixup" is performed to +satisfy the target. + +If the flag is not provided on the command line, then its default value +depends on the operating system where rclone runs: "true" on Windows and +macOS, "false" otherwise. If the flag is provided without a value, then +it is "true". + +VFS Disk Options + +This flag allows you to manually set the statistics about the filing +system. It can be useful when those statistics cannot be read correctly +automatically. + + --vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1) + +Alternate report of used bytes + +Some backends, most notably S3, do not report the amount of bytes used. +If you need this information to be available when running df on the +filesystem, then pass the flag --vfs-used-is-size to rclone. With this +flag set, instead of relying on the backend to report this information, +rclone will scan the whole remote similar to rclone size and compute the +total used space itself. + +WARNING. Contrary to rclone size, this flag ignores filters so that the +result is accurate. However, this is very inefficient and may cost lots +of API calls resulting in extra charges. Use it as a last resort and +only with caching. + + rclone nfsmount remote:path /path/to/mountpoint [flags] + +Options + + --allow-non-empty Allow mounting over a non-empty directory (not supported on Windows) + --allow-other Allow access to other users (not supported on Windows) + --allow-root Allow access to root user (not supported on Windows) + --async-read Use asynchronous reads (not supported on Windows) (default true) + --attr-timeout Duration Time for which file/directory attributes are cached (default 1s) + --daemon Run mount in background and exit parent process (as background output is suppressed, use --log-file with --log-format=pid,... to monitor) (not supported on Windows) + --daemon-timeout Duration Time limit for rclone to respond to kernel (not supported on Windows) (default 0s) + --daemon-wait Duration Time to wait for ready mount from daemon (maximum time on Linux, constant sleep time on OSX/BSD) (not supported on Windows) (default 1m0s) + --debug-fuse Debug the FUSE internals - needs -v + --default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows) + --devname string Set the device name - default is remote:path + --dir-cache-time Duration Time to cache directory entries for (default 5m0s) + --dir-perms FileMode Directory permissions (default 0777) + --file-perms FileMode File permissions (default 0666) + --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required) + --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) + -h, --help help for nfsmount + --max-read-ahead SizeSuffix The number of bytes that can be prefetched for sequential reads (not supported on Windows) (default 128Ki) + --mount-case-insensitive Tristate Tell the OS the mount is case insensitive (true) or sensitive (false) regardless of the backend (auto) (default unset) + --network-mode Mount as remote network drive, instead of fixed disk drive (supported on Windows only) + --no-checksum Don't compare checksums on up/download + --no-modtime Don't read/write the modification time (can speed things up) + --no-seek Don't allow seeking in files + --noappledouble Ignore Apple Double (._) and .DS_Store files (supported on OSX only) (default true) + --noapplexattr Ignore all "com.apple.*" extended attributes (supported on OSX only) + -o, --option stringArray Option for libfuse/WinFsp (repeat if required) + --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) + --read-only Only allow read-only access + --sudo Use sudo to run the mount command as root. + --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) + --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) + --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) + --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) + --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) + --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s) + --vfs-case-insensitive If a file name not found, find a case insensitive match + --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off) + --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection + --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full + --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) + --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) + --vfs-refresh Refreshes the directory cache recursively in the background on start + --vfs-used-is-size rclone size Use the rclone size algorithm for Used size + --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) + --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) + --volname string Set the volume name (supported on Windows and OSX only) + --write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows) + +Filter Options + +Flags for filtering directory listings. + + --delete-excluded Delete files on dest excluded from sync + --exclude stringArray Exclude files matching pattern + --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) + --exclude-if-present stringArray Exclude directories if filename is present + --files-from stringArray Read list of source-file names from file (use - to read from stdin) + --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) + -f, --filter stringArray Add a file filtering rule + --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) + --ignore-case Ignore case in filters (case insensitive) + --include stringArray Include files matching pattern + --include-from stringArray Read file include patterns from file (use - to read from stdin) + --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) + --max-depth int If set limits the recursion depth to this (default -1) + --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) + --metadata-exclude stringArray Exclude metadatas matching pattern + --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) + --metadata-filter stringArray Add a metadata filtering rule + --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) + --metadata-include stringArray Include metadatas matching pattern + --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) + --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) + --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) + +See the global flags page for global options not listed here. + +SEE ALSO + +- rclone - Show help for rclone commands, flags and backends. + rclone obscure Obscure password for use in the rclone config file. @@ -6531,7 +7414,7 @@ Options --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -6994,7 +7877,7 @@ Options --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -7497,7 +8380,7 @@ Options --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -8146,7 +9029,7 @@ Options --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -8583,7 +9466,7 @@ Options --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -9343,7 +10226,7 @@ Options --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -9877,7 +10760,7 @@ Options --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -10558,7 +11441,7 @@ Options --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -12521,7 +13404,7 @@ When transferring files above SIZE to capable backends, rclone will use multiple threads to transfer the file (default 256M). Capable backends are marked in the overview as MultithreadUpload. (They -need to implement either the OpenWriterAt or OpenChunkedWriter internal +need to implement either the OpenWriterAt or OpenChunkWriter internal interfaces). These include include, local, s3, azureblob, b2, oracleobjectstorage and smb at the time of writing. @@ -17609,7 +18492,7 @@ General networking and HTTP stuff. --tpslimit float Limit HTTP transactions per second to this --tpslimit-burst int Max burst of transactions for --tpslimit (default 1) --use-cookies Enable session cookiejar - --user-agent string Set the user-agent to a specified string (default "rclone/v1.65.0") + --user-agent string Set the user-agent to a specified string (default "rclone/v1.65.1") Performance @@ -21602,6 +22485,7 @@ permissions are required to be available on the bucket being written to: - GetObject - PutObject - PutObjectACL +- CreateBucket (unless using s3-no-check-bucket) When using the lsd subcommand, the ListAllMyBuckets permission is required. @@ -21642,6 +22526,8 @@ Notes on above: that USER_NAME has been created. 2. The Resource entry must include both resource ARNs, as one implies the bucket and the other implies the bucket's objects. +3. When using s3-no-check-bucket and the bucket already exsits, the + "arn:aws:s3:::BUCKET_NAME" doesn't have to be included. For reference, here's an Ansible script that will generate one or more buckets that will work with rclone sync. @@ -47039,6 +47925,70 @@ Options: Changelog +v1.65.1 - 2024-01-08 + +See commits + +- Bug Fixes + - build + - Bump golang.org/x/crypto to fix ssh terrapin CVE-2023-48795 + (dependabot) + - Update to go1.21.5 to fix Windows path problems (Nick + Craig-Wood) + - Fix docker build on arm/v6 (Nick Craig-Wood) + - install.sh: fix harmless error message on install (Nick + Craig-Wood) + - accounting: fix stats to show server side transfers (Nick + Craig-Wood) + - doc fixes (albertony, ben-ba, Eli Orzitzer, emyarod, keongalvin, + rarspace01) + - nfsmount: Compile for all unix oses, add --sudo and fix + error/option handling (Nick Craig-Wood) + - operations: Fix files moved by rclone move not being counted as + transfers (Nick Craig-Wood) + - oauthutil: Avoid panic when *token and *ts.token are the same + (rkonfj) + - serve s3: Fix listing oddities (Nick Craig-Wood) +- VFS + - Note that --vfs-refresh runs in the background (Nick Craig-Wood) +- Azurefiles + - Fix storage base url (Oksana) +- Crypt + - Fix rclone move a file over itself deleting the file (Nick + Craig-Wood) +- Chunker + - Fix rclone move a file over itself deleting the file (Nick + Craig-Wood) +- Compress + - Fix rclone move a file over itself deleting the file (Nick + Craig-Wood) +- Dropbox + - Fix used space on dropbox team accounts (Nick Craig-Wood) +- FTP + - Fix multi-thread copy (WeidiDeng) +- Googlephotos + - Fix nil pointer exception when batch failed (Nick Craig-Wood) +- Hasher + - Fix rclone move a file over itself deleting the file (Nick + Craig-Wood) + - Fix invalid memory address error when MaxAge == 0 (nielash) +- Onedrive + - Fix error listing: unknown object type (Nick Craig-Wood) + - Fix "unauthenticated: Unauthenticated" errors when uploading + (Nick Craig-Wood) +- Oracleobjectstorage + - Fix object storage endpoint for custom endpoints (Manoj Ghosh) + - Multipart copy create bucket if it doesn't exist. (Manoj Ghosh) +- Protondrive + - Fix CVE-2023-45286 / GHSA-xwh9-gc39-5298 (Nick Craig-Wood) +- S3 + - Fix crash if no UploadId in multipart upload (Nick Craig-Wood) +- Smb + - Fix shares not listed by updating go-smb2 (halms) +- Union + - Fix rclone move a file over itself deleting the file (Nick + Craig-Wood) + v1.65.0 - 2023-11-26 See commits @@ -53850,7 +54800,7 @@ email addresses removed from here need to be added to bin/.ignore-emails to make - Scott McGillivray scott.mcgillivray@gmail.com - Bjørn Erik Pedersen bjorn.erik.pedersen@gmail.com - Lukas Loesche lukas@mesosphere.io -- emyarod allllaboutyou@gmail.com +- emyarod emyarod@users.noreply.github.com - T.C. Ferguson tcf909@gmail.com - Brandur brandur@mutelight.org - Dario Giovannetti dev@dariogiovannetti.net diff --git a/docs/content/changelog.md b/docs/content/changelog.md index 17387e5b0..d9364b460 100644 --- a/docs/content/changelog.md +++ b/docs/content/changelog.md @@ -5,6 +5,56 @@ description: "Rclone Changelog" # Changelog +## v1.65.1 - 2024-01-08 + +[See commits](https://github.com/rclone/rclone/compare/v1.65.0...v1.65.1) + +* Bug Fixes + * build + * Bump golang.org/x/crypto to fix ssh terrapin CVE-2023-48795 (dependabot) + * Update to go1.21.5 to fix Windows path problems (Nick Craig-Wood) + * Fix docker build on arm/v6 (Nick Craig-Wood) + * install.sh: fix harmless error message on install (Nick Craig-Wood) + * accounting: fix stats to show server side transfers (Nick Craig-Wood) + * doc fixes (albertony, ben-ba, Eli Orzitzer, emyarod, keongalvin, rarspace01) + * nfsmount: Compile for all unix oses, add `--sudo` and fix error/option handling (Nick Craig-Wood) + * operations: Fix files moved by rclone move not being counted as transfers (Nick Craig-Wood) + * oauthutil: Avoid panic when `*token` and `*ts.token` are the same (rkonfj) + * serve s3: Fix listing oddities (Nick Craig-Wood) +* VFS + * Note that `--vfs-refresh` runs in the background (Nick Craig-Wood) +* Azurefiles + * Fix storage base url (Oksana) +* Crypt + * Fix rclone move a file over itself deleting the file (Nick Craig-Wood) +* Chunker + * Fix rclone move a file over itself deleting the file (Nick Craig-Wood) +* Compress + * Fix rclone move a file over itself deleting the file (Nick Craig-Wood) +* Dropbox + * Fix used space on dropbox team accounts (Nick Craig-Wood) +* FTP + * Fix multi-thread copy (WeidiDeng) +* Googlephotos + * Fix nil pointer exception when batch failed (Nick Craig-Wood) +* Hasher + * Fix rclone move a file over itself deleting the file (Nick Craig-Wood) + * Fix invalid memory address error when MaxAge == 0 (nielash) +* Onedrive + * Fix error listing: unknown object type `` (Nick Craig-Wood) + * Fix "unauthenticated: Unauthenticated" errors when uploading (Nick Craig-Wood) +* Oracleobjectstorage + * Fix object storage endpoint for custom endpoints (Manoj Ghosh) + * Multipart copy create bucket if it doesn't exist. (Manoj Ghosh) +* Protondrive + * Fix CVE-2023-45286 / GHSA-xwh9-gc39-5298 (Nick Craig-Wood) +* S3 + * Fix crash if no UploadId in multipart upload (Nick Craig-Wood) +* Smb + * Fix shares not listed by updating go-smb2 (halms) +* Union + * Fix rclone move a file over itself deleting the file (Nick Craig-Wood) + ## v1.65.0 - 2023-11-26 [See commits](https://github.com/rclone/rclone/compare/v1.64.0...v1.65.0) diff --git a/docs/content/commands/rclone.md b/docs/content/commands/rclone.md index e1489389d..dd1fa1ffd 100644 --- a/docs/content/commands/rclone.md +++ b/docs/content/commands/rclone.md @@ -831,7 +831,7 @@ rclone [flags] --use-json-log Use json log format --use-mmap Use mmap allocator (see docs) --use-server-modtime Use server modified time instead of object metadata - --user-agent string Set the user-agent to a specified string (default "rclone/v1.65.0") + --user-agent string Set the user-agent to a specified string (default "rclone/v1.65.1") -v, --verbose count Print lots more stuff (repeat for more) -V, --version Print the version number --webdav-bearer-token string Bearer token instead of user/pass (e.g. a Macaroon) @@ -895,6 +895,7 @@ rclone [flags] * [rclone move](/commands/rclone_move/) - Move files from source to dest. * [rclone moveto](/commands/rclone_moveto/) - Move file or directory from source to dest. * [rclone ncdu](/commands/rclone_ncdu/) - Explore a remote with a text based user interface. +* [rclone nfsmount](/commands/rclone_nfsmount/) - Mount the remote as file system on a mountpoint. * [rclone obscure](/commands/rclone_obscure/) - Obscure password for use in the rclone config file. * [rclone purge](/commands/rclone_purge/) - Remove the path and all of its contents. * [rclone rc](/commands/rclone_rc/) - Run a command against a running rclone. diff --git a/docs/content/commands/rclone_mount.md b/docs/content/commands/rclone_mount.md index c22fb2e7e..b85bf7482 100644 --- a/docs/content/commands/rclone_mount.md +++ b/docs/content/commands/rclone_mount.md @@ -856,7 +856,7 @@ rclone mount remote:path /path/to/mountpoint [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) diff --git a/docs/content/commands/rclone_nfsmount.md b/docs/content/commands/rclone_nfsmount.md new file mode 100644 index 000000000..04a92d762 --- /dev/null +++ b/docs/content/commands/rclone_nfsmount.md @@ -0,0 +1,904 @@ +--- +title: "rclone nfsmount" +description: "Mount the remote as file system on a mountpoint." +slug: rclone_nfsmount +url: /commands/rclone_nfsmount/ +groups: Filter +status: Experimental +versionIntroduced: v1.65 +# autogenerated - DO NOT EDIT, instead edit the source code in cmd/nfsmount/ and as part of making a release run "make commanddocs" +--- +# rclone nfsmount + +Mount the remote as file system on a mountpoint. + +## Synopsis + +rclone nfsmount allows Linux, FreeBSD, macOS and Windows to +mount any of Rclone's cloud storage systems as a file system with +FUSE. + +First set up your remote using `rclone config`. Check it works with `rclone ls` etc. + +On Linux and macOS, you can run mount in either foreground or background (aka +daemon) mode. Mount runs in foreground mode by default. Use the `--daemon` flag +to force background mode. On Windows you can run mount in foreground only, +the flag is ignored. + +In background mode rclone acts as a generic Unix mount program: the main +program starts, spawns background rclone process to setup and maintain the +mount, waits until success or timeout and exits with appropriate code +(killing the child process if it fails). + +On Linux/macOS/FreeBSD start the mount like this, where `/path/to/local/mount` +is an **empty** **existing** directory: + + rclone nfsmount remote:path/to/files /path/to/local/mount + +On Windows you can start a mount in different ways. See [below](#mounting-modes-on-windows) +for details. If foreground mount is used interactively from a console window, +rclone will serve the mount and occupy the console so another window should be +used to work with the mount until rclone is interrupted e.g. by pressing Ctrl-C. + +The following examples will mount to an automatically assigned drive, +to specific drive letter `X:`, to path `C:\path\parent\mount` +(where parent directory or drive must exist, and mount must **not** exist, +and is not supported when [mounting as a network drive](#mounting-modes-on-windows)), and +the last example will mount as network share `\\cloud\remote` and map it to an +automatically assigned drive: + + rclone nfsmount remote:path/to/files * + rclone nfsmount remote:path/to/files X: + rclone nfsmount remote:path/to/files C:\path\parent\mount + rclone nfsmount remote:path/to/files \\cloud\remote + +When the program ends while in foreground mode, either via Ctrl+C or receiving +a SIGINT or SIGTERM signal, the mount should be automatically stopped. + +When running in background mode the user will have to stop the mount manually: + + # Linux + fusermount -u /path/to/local/mount + # OS X + umount /path/to/local/mount + +The umount operation can fail, for example when the mountpoint is busy. +When that happens, it is the user's responsibility to stop the mount manually. + +The size of the mounted file system will be set according to information retrieved +from the remote, the same as returned by the [rclone about](https://rclone.org/commands/rclone_about/) +command. Remotes with unlimited storage may report the used size only, +then an additional 1 PiB of free space is assumed. If the remote does not +[support](https://rclone.org/overview/#optional-features) the about feature +at all, then 1 PiB is set as both the total and the free size. + +## Installing on Windows + +To run rclone nfsmount on Windows, you will need to +download and install [WinFsp](http://www.secfs.net/winfsp/). + +[WinFsp](https://github.com/winfsp/winfsp) is an open-source +Windows File System Proxy which makes it easy to write user space file +systems for Windows. It provides a FUSE emulation layer which rclone +uses combination with [cgofuse](https://github.com/winfsp/cgofuse). +Both of these packages are by Bill Zissimopoulos who was very helpful +during the implementation of rclone nfsmount for Windows. + +### Mounting modes on windows + +Unlike other operating systems, Microsoft Windows provides a different filesystem +type for network and fixed drives. It optimises access on the assumption fixed +disk drives are fast and reliable, while network drives have relatively high latency +and less reliability. Some settings can also be differentiated between the two types, +for example that Windows Explorer should just display icons and not create preview +thumbnails for image and video files on network drives. + +In most cases, rclone will mount the remote as a normal, fixed disk drive by default. +However, you can also choose to mount it as a remote network drive, often described +as a network share. If you mount an rclone remote using the default, fixed drive mode +and experience unexpected program errors, freezes or other issues, consider mounting +as a network drive instead. + +When mounting as a fixed disk drive you can either mount to an unused drive letter, +or to a path representing a **nonexistent** subdirectory of an **existing** parent +directory or drive. Using the special value `*` will tell rclone to +automatically assign the next available drive letter, starting with Z: and moving backward. +Examples: + + rclone nfsmount remote:path/to/files * + rclone nfsmount remote:path/to/files X: + rclone nfsmount remote:path/to/files C:\path\parent\mount + rclone nfsmount remote:path/to/files X: + +Option `--volname` can be used to set a custom volume name for the mounted +file system. The default is to use the remote name and path. + +To mount as network drive, you can add option `--network-mode` +to your nfsmount command. Mounting to a directory path is not supported in +this mode, it is a limitation Windows imposes on junctions, so the remote must always +be mounted to a drive letter. + + rclone nfsmount remote:path/to/files X: --network-mode + +A volume name specified with `--volname` will be used to create the network share path. +A complete UNC path, such as `\\cloud\remote`, optionally with path +`\\cloud\remote\madeup\path`, will be used as is. Any other +string will be used as the share part, after a default prefix `\\server\`. +If no volume name is specified then `\\server\share` will be used. +You must make sure the volume name is unique when you are mounting more than one drive, +or else the mount command will fail. The share name will treated as the volume label for +the mapped drive, shown in Windows Explorer etc, while the complete +`\\server\share` will be reported as the remote UNC path by +`net use` etc, just like a normal network drive mapping. + +If you specify a full network share UNC path with `--volname`, this will implicitly +set the `--network-mode` option, so the following two examples have same result: + + rclone nfsmount remote:path/to/files X: --network-mode + rclone nfsmount remote:path/to/files X: --volname \\server\share + +You may also specify the network share UNC path as the mountpoint itself. Then rclone +will automatically assign a drive letter, same as with `*` and use that as +mountpoint, and instead use the UNC path specified as the volume name, as if it were +specified with the `--volname` option. This will also implicitly set +the `--network-mode` option. This means the following two examples have same result: + + rclone nfsmount remote:path/to/files \\cloud\remote + rclone nfsmount remote:path/to/files * --volname \\cloud\remote + +There is yet another way to enable network mode, and to set the share path, +and that is to pass the "native" libfuse/WinFsp option directly: +`--fuse-flag --VolumePrefix=\server\share`. Note that the path +must be with just a single backslash prefix in this case. + + +*Note:* In previous versions of rclone this was the only supported method. + +[Read more about drive mapping](https://en.wikipedia.org/wiki/Drive_mapping) + +See also [Limitations](#limitations) section below. + +### Windows filesystem permissions + +The FUSE emulation layer on Windows must convert between the POSIX-based +permission model used in FUSE, and the permission model used in Windows, +based on access-control lists (ACL). + +The mounted filesystem will normally get three entries in its access-control list (ACL), +representing permissions for the POSIX permission scopes: Owner, group and others. +By default, the owner and group will be taken from the current user, and the built-in +group "Everyone" will be used to represent others. The user/group can be customized +with FUSE options "UserName" and "GroupName", +e.g. `-o UserName=user123 -o GroupName="Authenticated Users"`. +The permissions on each entry will be set according to [options](#options) +`--dir-perms` and `--file-perms`, which takes a value in traditional Unix +[numeric notation](https://en.wikipedia.org/wiki/File-system_permissions#Numeric_notation). + +The default permissions corresponds to `--file-perms 0666 --dir-perms 0777`, +i.e. read and write permissions to everyone. This means you will not be able +to start any programs from the mount. To be able to do that you must add +execute permissions, e.g. `--file-perms 0777 --dir-perms 0777` to add it +to everyone. If the program needs to write files, chances are you will +have to enable [VFS File Caching](#vfs-file-caching) as well (see also +[limitations](#limitations)). Note that the default write permission have +some restrictions for accounts other than the owner, specifically it lacks +the "write extended attributes", as explained next. + +The mapping of permissions is not always trivial, and the result you see in +Windows Explorer may not be exactly like you expected. For example, when setting +a value that includes write access for the group or others scope, this will be +mapped to individual permissions "write attributes", "write data" and +"append data", but not "write extended attributes". Windows will then show this +as basic permission "Special" instead of "Write", because "Write" also covers +the "write extended attributes" permission. When setting digit 0 for group or +others, to indicate no permissions, they will still get individual permissions +"read attributes", "read extended attributes" and "read permissions". This is +done for compatibility reasons, e.g. to allow users without additional +permissions to be able to read basic metadata about files like in Unix. + +WinFsp 2021 (version 1.9) introduced a new FUSE option "FileSecurity", +that allows the complete specification of file security descriptors using +[SDDL](https://docs.microsoft.com/en-us/windows/win32/secauthz/security-descriptor-string-format). +With this you get detailed control of the resulting permissions, compared +to use of the POSIX permissions described above, and no additional permissions +will be added automatically for compatibility with Unix. Some example use +cases will following. + +If you set POSIX permissions for only allowing access to the owner, +using `--file-perms 0600 --dir-perms 0700`, the user group and the built-in +"Everyone" group will still be given some special permissions, as described +above. Some programs may then (incorrectly) interpret this as the file being +accessible by everyone, for example an SSH client may warn about "unprotected +private key file". You can work around this by specifying +`-o FileSecurity="D:P(A;;FA;;;OW)"`, which sets file all access (FA) to the +owner (OW), and nothing else. + +When setting write permissions then, except for the owner, this does not +include the "write extended attributes" permission, as mentioned above. +This may prevent applications from writing to files, giving permission denied +error instead. To set working write permissions for the built-in "Everyone" +group, similar to what it gets by default but with the addition of the +"write extended attributes", you can specify +`-o FileSecurity="D:P(A;;FRFW;;;WD)"`, which sets file read (FR) and file +write (FW) to everyone (WD). If file execute (FX) is also needed, then change +to `-o FileSecurity="D:P(A;;FRFWFX;;;WD)"`, or set file all access (FA) to +get full access permissions, including delete, with +`-o FileSecurity="D:P(A;;FA;;;WD)"`. + +### Windows caveats + +Drives created as Administrator are not visible to other accounts, +not even an account that was elevated to Administrator with the +User Account Control (UAC) feature. A result of this is that if you mount +to a drive letter from a Command Prompt run as Administrator, and then try +to access the same drive from Windows Explorer (which does not run as +Administrator), you will not be able to see the mounted drive. + +If you don't need to access the drive from applications running with +administrative privileges, the easiest way around this is to always +create the mount from a non-elevated command prompt. + +To make mapped drives available to the user account that created them +regardless if elevated or not, there is a special Windows setting called +[linked connections](https://docs.microsoft.com/en-us/troubleshoot/windows-client/networking/mapped-drives-not-available-from-elevated-command#detail-to-configure-the-enablelinkedconnections-registry-entry) +that can be enabled. + +It is also possible to make a drive mount available to everyone on the system, +by running the process creating it as the built-in SYSTEM account. +There are several ways to do this: One is to use the command-line +utility [PsExec](https://docs.microsoft.com/en-us/sysinternals/downloads/psexec), +from Microsoft's Sysinternals suite, which has option `-s` to start +processes as the SYSTEM account. Another alternative is to run the mount +command from a Windows Scheduled Task, or a Windows Service, configured +to run as the SYSTEM account. A third alternative is to use the +[WinFsp.Launcher infrastructure](https://github.com/winfsp/winfsp/wiki/WinFsp-Service-Architecture)). +Read more in the [install documentation](https://rclone.org/install/). +Note that when running rclone as another user, it will not use +the configuration file from your profile unless you tell it to +with the [`--config`](https://rclone.org/docs/#config-config-file) option. +Note also that it is now the SYSTEM account that will have the owner +permissions, and other accounts will have permissions according to the +group or others scopes. As mentioned above, these will then not get the +"write extended attributes" permission, and this may prevent writing to +files. You can work around this with the FileSecurity option, see +example above. + +Note that mapping to a directory path, instead of a drive letter, +does not suffer from the same limitations. + +## Mounting on macOS + +Mounting on macOS can be done either via [built-in NFS server](/commands/rclone_serve_nfs/), [macFUSE](https://osxfuse.github.io/) +(also known as osxfuse) or [FUSE-T](https://www.fuse-t.org/). macFUSE is a traditional +FUSE driver utilizing a macOS kernel extension (kext). FUSE-T is an alternative FUSE system +which "mounts" via an NFSv4 local server. + +# NFS mount + +This method spins up an NFS server using [serve nfs](/commands/rclone_serve_nfs/) command and mounts +it to the specified mountpoint. If you run this in background mode using |--daemon|, you will need to +send SIGTERM signal to the rclone process using |kill| command to stop the mount. + +### macFUSE Notes + +If installing macFUSE using [dmg packages](https://github.com/osxfuse/osxfuse/releases) from +the website, rclone will locate the macFUSE libraries without any further intervention. +If however, macFUSE is installed using the [macports](https://www.macports.org/) package manager, +the following addition steps are required. + + sudo mkdir /usr/local/lib + cd /usr/local/lib + sudo ln -s /opt/local/lib/libfuse.2.dylib + +### FUSE-T Limitations, Caveats, and Notes + +There are some limitations, caveats, and notes about how it works. These are current as +of FUSE-T version 1.0.14. + +#### ModTime update on read + +As per the [FUSE-T wiki](https://github.com/macos-fuse-t/fuse-t/wiki#caveats): + +> File access and modification times cannot be set separately as it seems to be an +> issue with the NFS client which always modifies both. Can be reproduced with +> 'touch -m' and 'touch -a' commands + +This means that viewing files with various tools, notably macOS Finder, will cause rlcone +to update the modification time of the file. This may make rclone upload a full new copy +of the file. + +#### Unicode Normalization + +Rclone includes flags for unicode normalization with macFUSE that should be updated +for FUSE-T. See [this forum post](https://forum.rclone.org/t/some-unicode-forms-break-mount-on-macos-with-fuse-t/36403) +and [FUSE-T issue #16](https://github.com/macos-fuse-t/fuse-t/issues/16). The following +flag should be added to the `rclone mount` command. + + -o modules=iconv,from_code=UTF-8,to_code=UTF-8 + +#### Read Only mounts + +When mounting with `--read-only`, attempts to write to files will fail *silently* as +opposed to with a clear warning as in macFUSE. + +## Limitations + +Without the use of `--vfs-cache-mode` this can only write files +sequentially, it can only seek when reading. This means that many +applications won't work with their files on an rclone mount without +`--vfs-cache-mode writes` or `--vfs-cache-mode full`. +See the [VFS File Caching](#vfs-file-caching) section for more info. +When using NFS mount on macOS, if you don't specify |--vfs-cache-mode| +the mount point will be read-only. + +The bucket-based remotes (e.g. Swift, S3, Google Compute Storage, B2) +do not support the concept of empty directories, so empty +directories will have a tendency to disappear once they fall out of +the directory cache. + +When `rclone mount` is invoked on Unix with `--daemon` flag, the main rclone +program will wait for the background mount to become ready or until the timeout +specified by the `--daemon-wait` flag. On Linux it can check mount status using +ProcFS so the flag in fact sets **maximum** time to wait, while the real wait +can be less. On macOS / BSD the time to wait is constant and the check is +performed only at the end. We advise you to set wait time on macOS reasonably. + +Only supported on Linux, FreeBSD, OS X and Windows at the moment. + +## rclone nfsmount vs rclone sync/copy + +File systems expect things to be 100% reliable, whereas cloud storage +systems are a long way from 100% reliable. The rclone sync/copy +commands cope with this with lots of retries. However rclone nfsmount +can't use retries in the same way without making local copies of the +uploads. Look at the [VFS File Caching](#vfs-file-caching) +for solutions to make nfsmount more reliable. + +## Attribute caching + +You can use the flag `--attr-timeout` to set the time the kernel caches +the attributes (size, modification time, etc.) for directory entries. + +The default is `1s` which caches files just long enough to avoid +too many callbacks to rclone from the kernel. + +In theory 0s should be the correct value for filesystems which can +change outside the control of the kernel. However this causes quite a +few problems such as +[rclone using too much memory](https://github.com/rclone/rclone/issues/2157), +[rclone not serving files to samba](https://forum.rclone.org/t/rclone-1-39-vs-1-40-mount-issue/5112) +and [excessive time listing directories](https://github.com/rclone/rclone/issues/2095#issuecomment-371141147). + +The kernel can cache the info about a file for the time given by +`--attr-timeout`. You may see corruption if the remote file changes +length during this window. It will show up as either a truncated file +or a file with garbage on the end. With `--attr-timeout 1s` this is +very unlikely but not impossible. The higher you set `--attr-timeout` +the more likely it is. The default setting of "1s" is the lowest +setting which mitigates the problems above. + +If you set it higher (`10s` or `1m` say) then the kernel will call +back to rclone less often making it more efficient, however there is +more chance of the corruption issue above. + +If files don't change on the remote outside of the control of rclone +then there is no chance of corruption. + +This is the same as setting the attr_timeout option in mount.fuse. + +## Filters + +Note that all the rclone filters can be used to select a subset of the +files to be visible in the mount. + +## systemd + +When running rclone nfsmount as a systemd service, it is possible +to use Type=notify. In this case the service will enter the started state +after the mountpoint has been successfully set up. +Units having the rclone nfsmount service specified as a requirement +will see all files and folders immediately in this mode. + +Note that systemd runs mount units without any environment variables including +`PATH` or `HOME`. This means that tilde (`~`) expansion will not work +and you should provide `--config` and `--cache-dir` explicitly as absolute +paths via rclone arguments. +Since mounting requires the `fusermount` program, rclone will use the fallback +PATH of `/bin:/usr/bin` in this scenario. Please ensure that `fusermount` +is present on this PATH. + +## Rclone as Unix mount helper + +The core Unix program `/bin/mount` normally takes the `-t FSTYPE` argument +then runs the `/sbin/mount.FSTYPE` helper program passing it mount options +as `-o key=val,...` or `--opt=...`. Automount (classic or systemd) behaves +in a similar way. + +rclone by default expects GNU-style flags `--key val`. To run it as a mount +helper you should symlink rclone binary to `/sbin/mount.rclone` and optionally +`/usr/bin/rclonefs`, e.g. `ln -s /usr/bin/rclone /sbin/mount.rclone`. +rclone will detect it and translate command-line arguments appropriately. + +Now you can run classic mounts like this: +``` +mount sftp1:subdir /mnt/data -t rclone -o vfs_cache_mode=writes,sftp_key_file=/path/to/pem +``` + +or create systemd mount units: +``` +# /etc/systemd/system/mnt-data.mount +[Unit] +Description=Mount for /mnt/data +[Mount] +Type=rclone +What=sftp1:subdir +Where=/mnt/data +Options=rw,_netdev,allow_other,args2env,vfs-cache-mode=writes,config=/etc/rclone.conf,cache-dir=/var/rclone +``` + +optionally accompanied by systemd automount unit +``` +# /etc/systemd/system/mnt-data.automount +[Unit] +Description=AutoMount for /mnt/data +[Automount] +Where=/mnt/data +TimeoutIdleSec=600 +[Install] +WantedBy=multi-user.target +``` + +or add in `/etc/fstab` a line like +``` +sftp1:subdir /mnt/data rclone rw,noauto,nofail,_netdev,x-systemd.automount,args2env,vfs_cache_mode=writes,config=/etc/rclone.conf,cache_dir=/var/cache/rclone 0 0 +``` + +or use classic Automountd. +Remember to provide explicit `config=...,cache-dir=...` as a workaround for +mount units being run without `HOME`. + +Rclone in the mount helper mode will split `-o` argument(s) by comma, replace `_` +by `-` and prepend `--` to get the command-line flags. Options containing commas +or spaces can be wrapped in single or double quotes. Any inner quotes inside outer +quotes of the same type should be doubled. + +Mount option syntax includes a few extra options treated specially: + +- `env.NAME=VALUE` will set an environment variable for the mount process. + This helps with Automountd and Systemd.mount which don't allow setting + custom environment for mount helpers. + Typically you will use `env.HTTPS_PROXY=proxy.host:3128` or `env.HOME=/root` +- `command=cmount` can be used to run `cmount` or any other rclone command + rather than the default `mount`. +- `args2env` will pass mount options to the mount helper running in background + via environment variables instead of command line arguments. This allows to + hide secrets from such commands as `ps` or `pgrep`. +- `vv...` will be transformed into appropriate `--verbose=N` +- standard mount options like `x-systemd.automount`, `_netdev`, `nosuid` and alike + are intended only for Automountd and ignored by rclone. +## VFS - Virtual File System + +This command uses the VFS layer. This adapts the cloud storage objects +that rclone uses into something which looks much more like a disk +filing system. + +Cloud storage objects have lots of properties which aren't like disk +files - you can't extend them or write to the middle of them, so the +VFS layer has to deal with that. Because there is no one right way of +doing this there are various options explained below. + +The VFS layer also implements a directory cache - this caches info +about files and directories (but not the data) in memory. + +## VFS Directory Cache + +Using the `--dir-cache-time` flag, you can control how long a +directory should be considered up to date and not refreshed from the +backend. Changes made through the VFS will appear immediately or +invalidate the cache. + + --dir-cache-time duration Time to cache directory entries for (default 5m0s) + --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s) + +However, changes made directly on the cloud storage by the web +interface or a different copy of rclone will only be picked up once +the directory cache expires if the backend configured does not support +polling for changes. If the backend supports polling, changes will be +picked up within the polling interval. + +You can send a `SIGHUP` signal to rclone for it to flush all +directory caches, regardless of how old they are. Assuming only one +rclone instance is running, you can reset the cache like this: + + kill -SIGHUP $(pidof rclone) + +If you configure rclone with a [remote control](/rc) then you can use +rclone rc to flush the whole directory cache: + + rclone rc vfs/forget + +Or individual files or directories: + + rclone rc vfs/forget file=path/to/file dir=path/to/dir + +## VFS File Buffering + +The `--buffer-size` flag determines the amount of memory, +that will be used to buffer data in advance. + +Each open file will try to keep the specified amount of data in memory +at all times. The buffered data is bound to one open file and won't be +shared. + +This flag is a upper limit for the used memory per open file. The +buffer will only use memory for data that is downloaded but not not +yet read. If the buffer is empty, only a small amount of memory will +be used. + +The maximum memory used by rclone for buffering can be up to +`--buffer-size * open files`. + +## VFS File Caching + +These flags control the VFS file caching options. File caching is +necessary to make the VFS layer appear compatible with a normal file +system. It can be disabled at the cost of some compatibility. + +For example you'll need to enable VFS caching if you want to read and +write simultaneously to a file. See below for more details. + +Note that the VFS cache is separate from the cache backend and you may +find that you need one or the other or both. + + --cache-dir string Directory rclone will use for caching. + --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) + --vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s) + --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) + --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) + --vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s) + --vfs-write-back duration Time to writeback files after last use when using cache (default 5s) + +If run with `-vv` rclone will print the location of the file cache. The +files are stored in the user cache file area which is OS dependent but +can be controlled with `--cache-dir` or setting the appropriate +environment variable. + +The cache has 4 different modes selected by `--vfs-cache-mode`. +The higher the cache mode the more compatible rclone becomes at the +cost of using disk space. + +Note that files are written back to the remote only when they are +closed and if they haven't been accessed for `--vfs-write-back` +seconds. If rclone is quit or dies with files that haven't been +uploaded, these will be uploaded next time rclone is run with the same +flags. + +If using `--vfs-cache-max-size` or `--vfs-cache-min-free-size` note +that the cache may exceed these quotas for two reasons. Firstly +because it is only checked every `--vfs-cache-poll-interval`. Secondly +because open files cannot be evicted from the cache. When +`--vfs-cache-max-size` or `--vfs-cache-min-free-size` is exceeded, +rclone will attempt to evict the least accessed files from the cache +first. rclone will start with files that haven't been accessed for the +longest. This cache flushing strategy is efficient and more relevant +files are likely to remain cached. + +The `--vfs-cache-max-age` will evict files from the cache +after the set time since last access has passed. The default value of +1 hour will start evicting files from cache that haven't been accessed +for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0 +and will wait for 1 more hour before evicting. Specify the time with +standard notation, s, m, h, d, w . + +You **should not** run two copies of rclone using the same VFS cache +with the same or overlapping remotes if using `--vfs-cache-mode > off`. +This can potentially cause data corruption if you do. You can work +around this by giving each rclone its own cache hierarchy with +`--cache-dir`. You don't need to worry about this if the remotes in +use don't overlap. + +### --vfs-cache-mode off + +In this mode (the default) the cache will read directly from the remote and write +directly to the remote without caching anything on disk. + +This will mean some operations are not possible + + * Files can't be opened for both read AND write + * Files opened for write can't be seeked + * Existing files opened for write must have O_TRUNC set + * Files open for read with O_TRUNC will be opened write only + * Files open for write only will behave as if O_TRUNC was supplied + * Open modes O_APPEND, O_TRUNC are ignored + * If an upload fails it can't be retried + +### --vfs-cache-mode minimal + +This is very similar to "off" except that files opened for read AND +write will be buffered to disk. This means that files opened for +write will be a lot more compatible, but uses the minimal disk space. + +These operations are not possible + + * Files opened for write only can't be seeked + * Existing files opened for write must have O_TRUNC set + * Files opened for write only will ignore O_APPEND, O_TRUNC + * If an upload fails it can't be retried + +### --vfs-cache-mode writes + +In this mode files opened for read only are still read directly from +the remote, write only and read/write files are buffered to disk +first. + +This mode should support all normal file system operations. + +If an upload fails it will be retried at exponentially increasing +intervals up to 1 minute. + +### --vfs-cache-mode full + +In this mode all reads and writes are buffered to and from disk. When +data is read from the remote this is buffered to disk as well. + +In this mode the files in the cache will be sparse files and rclone +will keep track of which bits of the files it has downloaded. + +So if an application only reads the starts of each file, then rclone +will only buffer the start of the file. These files will appear to be +their full size in the cache, but they will be sparse files with only +the data that has been downloaded present in them. + +This mode should support all normal file system operations and is +otherwise identical to `--vfs-cache-mode` writes. + +When reading a file rclone will read `--buffer-size` plus +`--vfs-read-ahead` bytes ahead. The `--buffer-size` is buffered in memory +whereas the `--vfs-read-ahead` is buffered on disk. + +When using this mode it is recommended that `--buffer-size` is not set +too large and `--vfs-read-ahead` is set large if required. + +**IMPORTANT** not all file systems support sparse files. In particular +FAT/exFAT do not. Rclone will perform very badly if the cache +directory is on a filesystem which doesn't support sparse files and it +will log an ERROR message if one is detected. + +### Fingerprinting + +Various parts of the VFS use fingerprinting to see if a local file +copy has changed relative to a remote file. Fingerprints are made +from: + +- size +- modification time +- hash + +where available on an object. + +On some backends some of these attributes are slow to read (they take +an extra API call per object, or extra work per object). + +For example `hash` is slow with the `local` and `sftp` backends as +they have to read the entire file and hash it, and `modtime` is slow +with the `s3`, `swift`, `ftp` and `qinqstor` backends because they +need to do an extra API call to fetch it. + +If you use the `--vfs-fast-fingerprint` flag then rclone will not +include the slow operations in the fingerprint. This makes the +fingerprinting less accurate but much faster and will improve the +opening time of cached files. + +If you are running a vfs cache over `local`, `s3` or `swift` backends +then using this flag is recommended. + +Note that if you change the value of this flag, the fingerprints of +the files in the cache may be invalidated and the files will need to +be downloaded again. + +## VFS Chunked Reading + +When rclone reads files from a remote it reads them in chunks. This +means that rather than requesting the whole file rclone reads the +chunk specified. This can reduce the used download quota for some +remotes by requesting only chunks from the remote that are actually +read, at the cost of an increased number of requests. + +These flags control the chunking: + + --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) + --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) + +Rclone will start reading a chunk of size `--vfs-read-chunk-size`, +and then double the size for each read. When `--vfs-read-chunk-size-limit` is +specified, and greater than `--vfs-read-chunk-size`, the chunk size for each +open file will get doubled only until the specified value is reached. If the +value is "off", which is the default, the limit is disabled and the chunk size +will grow indefinitely. + +With `--vfs-read-chunk-size 100M` and `--vfs-read-chunk-size-limit 0` +the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. +When `--vfs-read-chunk-size-limit 500M` is specified, the result would be +0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on. + +Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading. + +## VFS Performance + +These flags may be used to enable/disable features of the VFS for +performance or other reasons. See also the [chunked reading](#vfs-chunked-reading) +feature. + +In particular S3 and Swift benefit hugely from the `--no-modtime` flag +(or use `--use-server-modtime` for a slightly different effect) as each +read of the modification time takes a transaction. + + --no-checksum Don't compare checksums on up/download. + --no-modtime Don't read/write the modification time (can speed things up). + --no-seek Don't allow seeking in files. + --read-only Only allow read-only access. + +Sometimes rclone is delivered reads or writes out of order. Rather +than seeking rclone will wait a short time for the in sequence read or +write to come in. These flags only come into effect when not using an +on disk cache file. + + --vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms) + --vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s) + +When using VFS write caching (`--vfs-cache-mode` with value writes or full), +the global flag `--transfers` can be set to adjust the number of parallel uploads of +modified files from the cache (the related global flag `--checkers` has no effect on the VFS). + + --transfers int Number of file transfers to run in parallel (default 4) + +## VFS Case Sensitivity + +Linux file systems are case-sensitive: two files can differ only +by case, and the exact case must be used when opening a file. + +File systems in modern Windows are case-insensitive but case-preserving: +although existing files can be opened using any case, the exact case used +to create the file is preserved and available for programs to query. +It is not allowed for two files in the same directory to differ only by case. + +Usually file systems on macOS are case-insensitive. It is possible to make macOS +file systems case-sensitive but that is not the default. + +The `--vfs-case-insensitive` VFS flag controls how rclone handles these +two cases. If its value is "false", rclone passes file names to the remote +as-is. If the flag is "true" (or appears without a value on the +command line), rclone may perform a "fixup" as explained below. + +The user may specify a file name to open/delete/rename/etc with a case +different than what is stored on the remote. If an argument refers +to an existing file with exactly the same name, then the case of the existing +file on the disk will be used. However, if a file name with exactly the same +name is not found but a name differing only by case exists, rclone will +transparently fixup the name. This fixup happens only when an existing file +is requested. Case sensitivity of file names created anew by rclone is +controlled by the underlying remote. + +Note that case sensitivity of the operating system running rclone (the target) +may differ from case sensitivity of a file system presented by rclone (the source). +The flag controls whether "fixup" is performed to satisfy the target. + +If the flag is not provided on the command line, then its default value depends +on the operating system where rclone runs: "true" on Windows and macOS, "false" +otherwise. If the flag is provided without a value, then it is "true". + +## VFS Disk Options + +This flag allows you to manually set the statistics about the filing system. +It can be useful when those statistics cannot be read correctly automatically. + + --vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1) + +## Alternate report of used bytes + +Some backends, most notably S3, do not report the amount of bytes used. +If you need this information to be available when running `df` on the +filesystem, then pass the flag `--vfs-used-is-size` to rclone. +With this flag set, instead of relying on the backend to report this +information, rclone will scan the whole remote similar to `rclone size` +and compute the total used space itself. + +_WARNING._ Contrary to `rclone size`, this flag ignores filters so that the +result is accurate. However, this is very inefficient and may cost lots of API +calls resulting in extra charges. Use it as a last resort and only with caching. + + +``` +rclone nfsmount remote:path /path/to/mountpoint [flags] +``` + +## Options + +``` + --allow-non-empty Allow mounting over a non-empty directory (not supported on Windows) + --allow-other Allow access to other users (not supported on Windows) + --allow-root Allow access to root user (not supported on Windows) + --async-read Use asynchronous reads (not supported on Windows) (default true) + --attr-timeout Duration Time for which file/directory attributes are cached (default 1s) + --daemon Run mount in background and exit parent process (as background output is suppressed, use --log-file with --log-format=pid,... to monitor) (not supported on Windows) + --daemon-timeout Duration Time limit for rclone to respond to kernel (not supported on Windows) (default 0s) + --daemon-wait Duration Time to wait for ready mount from daemon (maximum time on Linux, constant sleep time on OSX/BSD) (not supported on Windows) (default 1m0s) + --debug-fuse Debug the FUSE internals - needs -v + --default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows) + --devname string Set the device name - default is remote:path + --dir-cache-time Duration Time to cache directory entries for (default 5m0s) + --dir-perms FileMode Directory permissions (default 0777) + --file-perms FileMode File permissions (default 0666) + --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required) + --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) + -h, --help help for nfsmount + --max-read-ahead SizeSuffix The number of bytes that can be prefetched for sequential reads (not supported on Windows) (default 128Ki) + --mount-case-insensitive Tristate Tell the OS the mount is case insensitive (true) or sensitive (false) regardless of the backend (auto) (default unset) + --network-mode Mount as remote network drive, instead of fixed disk drive (supported on Windows only) + --no-checksum Don't compare checksums on up/download + --no-modtime Don't read/write the modification time (can speed things up) + --no-seek Don't allow seeking in files + --noappledouble Ignore Apple Double (._) and .DS_Store files (supported on OSX only) (default true) + --noapplexattr Ignore all "com.apple.*" extended attributes (supported on OSX only) + -o, --option stringArray Option for libfuse/WinFsp (repeat if required) + --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) + --read-only Only allow read-only access + --sudo Use sudo to run the mount command as root. + --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) + --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) + --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) + --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) + --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) + --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s) + --vfs-case-insensitive If a file name not found, find a case insensitive match + --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off) + --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection + --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full + --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) + --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) + --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) + --vfs-refresh Refreshes the directory cache recursively in the background on start + --vfs-used-is-size rclone size Use the rclone size algorithm for Used size + --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) + --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) + --volname string Set the volume name (supported on Windows and OSX only) + --write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows) +``` + + +## Filter Options + +Flags for filtering directory listings. + +``` + --delete-excluded Delete files on dest excluded from sync + --exclude stringArray Exclude files matching pattern + --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) + --exclude-if-present stringArray Exclude directories if filename is present + --files-from stringArray Read list of source-file names from file (use - to read from stdin) + --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) + -f, --filter stringArray Add a file filtering rule + --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) + --ignore-case Ignore case in filters (case insensitive) + --include stringArray Include files matching pattern + --include-from stringArray Read file include patterns from file (use - to read from stdin) + --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) + --max-depth int If set limits the recursion depth to this (default -1) + --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) + --metadata-exclude stringArray Exclude metadatas matching pattern + --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) + --metadata-filter stringArray Add a metadata filtering rule + --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) + --metadata-include stringArray Include metadatas matching pattern + --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) + --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) + --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) +``` + +See the [global flags page](/flags/) for global options not listed here. + +# SEE ALSO + +* [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends. + diff --git a/docs/content/commands/rclone_serve_dlna.md b/docs/content/commands/rclone_serve_dlna.md index abe864473..f4106ad1f 100644 --- a/docs/content/commands/rclone_serve_dlna.md +++ b/docs/content/commands/rclone_serve_dlna.md @@ -404,7 +404,7 @@ rclone serve dlna remote:path [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) diff --git a/docs/content/commands/rclone_serve_docker.md b/docs/content/commands/rclone_serve_docker.md index 0366cf6c9..0d09b7a33 100644 --- a/docs/content/commands/rclone_serve_docker.md +++ b/docs/content/commands/rclone_serve_docker.md @@ -437,7 +437,7 @@ rclone serve docker [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) diff --git a/docs/content/commands/rclone_serve_ftp.md b/docs/content/commands/rclone_serve_ftp.md index 6ba8142e9..52382d0df 100644 --- a/docs/content/commands/rclone_serve_ftp.md +++ b/docs/content/commands/rclone_serve_ftp.md @@ -376,7 +376,7 @@ together, if `--auth-proxy` is set the authorized keys option will be ignored. There is an example program -[bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/test_proxy.py) +[bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/bin/test_proxy.py) in the rclone source code. The program's job is to take a `user` and `pass` on the input and turn @@ -485,7 +485,7 @@ rclone serve ftp remote:path [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) diff --git a/docs/content/commands/rclone_serve_http.md b/docs/content/commands/rclone_serve_http.md index f599e5284..e1ebede7a 100644 --- a/docs/content/commands/rclone_serve_http.md +++ b/docs/content/commands/rclone_serve_http.md @@ -477,7 +477,7 @@ together, if `--auth-proxy` is set the authorized keys option will be ignored. There is an example program -[bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/test_proxy.py) +[bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/bin/test_proxy.py) in the rclone source code. The program's job is to take a `user` and `pass` on the input and turn @@ -595,7 +595,7 @@ rclone serve http remote:path [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) diff --git a/docs/content/commands/rclone_serve_nfs.md b/docs/content/commands/rclone_serve_nfs.md index bbd28ccf8..c412bfb6d 100644 --- a/docs/content/commands/rclone_serve_nfs.md +++ b/docs/content/commands/rclone_serve_nfs.md @@ -4,6 +4,7 @@ description: "Serve the remote as an NFS mount" slug: rclone_serve_nfs url: /commands/rclone_serve_nfs/ groups: Filter +status: Experimental versionIntroduced: v1.65 # autogenerated - DO NOT EDIT, instead edit the source code in cmd/serve/nfs/ and as part of making a release run "make commanddocs" --- @@ -406,7 +407,7 @@ rclone serve nfs remote:path [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) diff --git a/docs/content/commands/rclone_serve_s3.md b/docs/content/commands/rclone_serve_s3.md index 986d31119..2cfeb03b6 100644 --- a/docs/content/commands/rclone_serve_s3.md +++ b/docs/content/commands/rclone_serve_s3.md @@ -553,7 +553,7 @@ rclone serve s3 remote:path [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) diff --git a/docs/content/commands/rclone_serve_sftp.md b/docs/content/commands/rclone_serve_sftp.md index 9181e7be4..9b563989b 100644 --- a/docs/content/commands/rclone_serve_sftp.md +++ b/docs/content/commands/rclone_serve_sftp.md @@ -408,7 +408,7 @@ together, if `--auth-proxy` is set the authorized keys option will be ignored. There is an example program -[bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/test_proxy.py) +[bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/bin/test_proxy.py) in the rclone source code. The program's job is to take a `user` and `pass` on the input and turn @@ -517,7 +517,7 @@ rclone serve sftp remote:path [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) diff --git a/docs/content/commands/rclone_serve_webdav.md b/docs/content/commands/rclone_serve_webdav.md index cbd0ff0c0..2b5ebcc3b 100644 --- a/docs/content/commands/rclone_serve_webdav.md +++ b/docs/content/commands/rclone_serve_webdav.md @@ -626,7 +626,7 @@ rclone serve webdav remote:path [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) diff --git a/docs/content/flags.md b/docs/content/flags.md index 4ce4c079f..3448e29f5 100644 --- a/docs/content/flags.md +++ b/docs/content/flags.md @@ -112,7 +112,7 @@ General networking and HTTP stuff. --tpslimit float Limit HTTP transactions per second to this --tpslimit-burst int Max burst of transactions for --tpslimit (default 1) --use-cookies Enable session cookiejar - --user-agent string Set the user-agent to a specified string (default "rclone/v1.65.0") + --user-agent string Set the user-agent to a specified string (default "rclone/v1.65.1") ``` diff --git a/rclone.1 b/rclone.1 index 347ad1846..af68093e5 100644 --- a/rclone.1 +++ b/rclone.1 @@ -1,7 +1,7 @@ .\"t .\" Automatically generated by Pandoc 2.9.2.1 .\" -.TH "rclone" "1" "Nov 26, 2023" "User Manual" "" +.TH "rclone" "1" "Jan 08, 2024" "User Manual" "" .hy .SH Rclone syncs your files to cloud storage .PP @@ -7055,7 +7055,7 @@ rclone mount remote:path /path/to/mountpoint [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -7389,6 +7389,1112 @@ not listed here. .IP \[bu] 2 rclone (https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. +.SH rclone nfsmount +.PP +Mount the remote as file system on a mountpoint. +.SS Synopsis +.PP +rclone nfsmount allows Linux, FreeBSD, macOS and Windows to mount any of +Rclone\[aq]s cloud storage systems as a file system with FUSE. +.PP +First set up your remote using \f[C]rclone config\f[R]. +Check it works with \f[C]rclone ls\f[R] etc. +.PP +On Linux and macOS, you can run mount in either foreground or background +(aka daemon) mode. +Mount runs in foreground mode by default. +Use the \f[C]--daemon\f[R] flag to force background mode. +On Windows you can run mount in foreground only, the flag is ignored. +.PP +In background mode rclone acts as a generic Unix mount program: the main +program starts, spawns background rclone process to setup and maintain +the mount, waits until success or timeout and exits with appropriate +code (killing the child process if it fails). +.PP +On Linux/macOS/FreeBSD start the mount like this, where +\f[C]/path/to/local/mount\f[R] is an \f[B]empty\f[R] \f[B]existing\f[R] +directory: +.IP +.nf +\f[C] +rclone nfsmount remote:path/to/files /path/to/local/mount +\f[R] +.fi +.PP +On Windows you can start a mount in different ways. +See below for details. +If foreground mount is used interactively from a console window, rclone +will serve the mount and occupy the console so another window should be +used to work with the mount until rclone is interrupted e.g. +by pressing Ctrl-C. +.PP +The following examples will mount to an automatically assigned drive, to +specific drive letter \f[C]X:\f[R], to path +\f[C]C:\[rs]path\[rs]parent\[rs]mount\f[R] (where parent directory or +drive must exist, and mount must \f[B]not\f[R] exist, and is not +supported when mounting as a network drive), and the last example will +mount as network share \f[C]\[rs]\[rs]cloud\[rs]remote\f[R] and map it +to an automatically assigned drive: +.IP +.nf +\f[C] +rclone nfsmount remote:path/to/files * +rclone nfsmount remote:path/to/files X: +rclone nfsmount remote:path/to/files C:\[rs]path\[rs]parent\[rs]mount +rclone nfsmount remote:path/to/files \[rs]\[rs]cloud\[rs]remote +\f[R] +.fi +.PP +When the program ends while in foreground mode, either via Ctrl+C or +receiving a SIGINT or SIGTERM signal, the mount should be automatically +stopped. +.PP +When running in background mode the user will have to stop the mount +manually: +.IP +.nf +\f[C] +# Linux +fusermount -u /path/to/local/mount +# OS X +umount /path/to/local/mount +\f[R] +.fi +.PP +The umount operation can fail, for example when the mountpoint is busy. +When that happens, it is the user\[aq]s responsibility to stop the mount +manually. +.PP +The size of the mounted file system will be set according to information +retrieved from the remote, the same as returned by the rclone +about (https://rclone.org/commands/rclone_about/) command. +Remotes with unlimited storage may report the used size only, then an +additional 1 PiB of free space is assumed. +If the remote does not +support (https://rclone.org/overview/#optional-features) the about +feature at all, then 1 PiB is set as both the total and the free size. +.SS Installing on Windows +.PP +To run rclone nfsmount on Windows, you will need to download and install +WinFsp (http://www.secfs.net/winfsp/). +.PP +WinFsp (https://github.com/winfsp/winfsp) is an open-source Windows File +System Proxy which makes it easy to write user space file systems for +Windows. +It provides a FUSE emulation layer which rclone uses combination with +cgofuse (https://github.com/winfsp/cgofuse). +Both of these packages are by Bill Zissimopoulos who was very helpful +during the implementation of rclone nfsmount for Windows. +.SS Mounting modes on windows +.PP +Unlike other operating systems, Microsoft Windows provides a different +filesystem type for network and fixed drives. +It optimises access on the assumption fixed disk drives are fast and +reliable, while network drives have relatively high latency and less +reliability. +Some settings can also be differentiated between the two types, for +example that Windows Explorer should just display icons and not create +preview thumbnails for image and video files on network drives. +.PP +In most cases, rclone will mount the remote as a normal, fixed disk +drive by default. +However, you can also choose to mount it as a remote network drive, +often described as a network share. +If you mount an rclone remote using the default, fixed drive mode and +experience unexpected program errors, freezes or other issues, consider +mounting as a network drive instead. +.PP +When mounting as a fixed disk drive you can either mount to an unused +drive letter, or to a path representing a \f[B]nonexistent\f[R] +subdirectory of an \f[B]existing\f[R] parent directory or drive. +Using the special value \f[C]*\f[R] will tell rclone to automatically +assign the next available drive letter, starting with Z: and moving +backward. +Examples: +.IP +.nf +\f[C] +rclone nfsmount remote:path/to/files * +rclone nfsmount remote:path/to/files X: +rclone nfsmount remote:path/to/files C:\[rs]path\[rs]parent\[rs]mount +rclone nfsmount remote:path/to/files X: +\f[R] +.fi +.PP +Option \f[C]--volname\f[R] can be used to set a custom volume name for +the mounted file system. +The default is to use the remote name and path. +.PP +To mount as network drive, you can add option \f[C]--network-mode\f[R] +to your nfsmount command. +Mounting to a directory path is not supported in this mode, it is a +limitation Windows imposes on junctions, so the remote must always be +mounted to a drive letter. +.IP +.nf +\f[C] +rclone nfsmount remote:path/to/files X: --network-mode +\f[R] +.fi +.PP +A volume name specified with \f[C]--volname\f[R] will be used to create +the network share path. +A complete UNC path, such as \f[C]\[rs]\[rs]cloud\[rs]remote\f[R], +optionally with path +\f[C]\[rs]\[rs]cloud\[rs]remote\[rs]madeup\[rs]path\f[R], will be used +as is. +Any other string will be used as the share part, after a default prefix +\f[C]\[rs]\[rs]server\[rs]\f[R]. +If no volume name is specified then \f[C]\[rs]\[rs]server\[rs]share\f[R] +will be used. +You must make sure the volume name is unique when you are mounting more +than one drive, or else the mount command will fail. +The share name will treated as the volume label for the mapped drive, +shown in Windows Explorer etc, while the complete +\f[C]\[rs]\[rs]server\[rs]share\f[R] will be reported as the remote UNC +path by \f[C]net use\f[R] etc, just like a normal network drive mapping. +.PP +If you specify a full network share UNC path with \f[C]--volname\f[R], +this will implicitly set the \f[C]--network-mode\f[R] option, so the +following two examples have same result: +.IP +.nf +\f[C] +rclone nfsmount remote:path/to/files X: --network-mode +rclone nfsmount remote:path/to/files X: --volname \[rs]\[rs]server\[rs]share +\f[R] +.fi +.PP +You may also specify the network share UNC path as the mountpoint +itself. +Then rclone will automatically assign a drive letter, same as with +\f[C]*\f[R] and use that as mountpoint, and instead use the UNC path +specified as the volume name, as if it were specified with the +\f[C]--volname\f[R] option. +This will also implicitly set the \f[C]--network-mode\f[R] option. +This means the following two examples have same result: +.IP +.nf +\f[C] +rclone nfsmount remote:path/to/files \[rs]\[rs]cloud\[rs]remote +rclone nfsmount remote:path/to/files * --volname \[rs]\[rs]cloud\[rs]remote +\f[R] +.fi +.PP +There is yet another way to enable network mode, and to set the share +path, and that is to pass the \[dq]native\[dq] libfuse/WinFsp option +directly: \f[C]--fuse-flag --VolumePrefix=\[rs]server\[rs]share\f[R]. +Note that the path must be with just a single backslash prefix in this +case. +.PP +\f[I]Note:\f[R] In previous versions of rclone this was the only +supported method. +.PP +Read more about drive +mapping (https://en.wikipedia.org/wiki/Drive_mapping) +.PP +See also Limitations section below. +.SS Windows filesystem permissions +.PP +The FUSE emulation layer on Windows must convert between the POSIX-based +permission model used in FUSE, and the permission model used in Windows, +based on access-control lists (ACL). +.PP +The mounted filesystem will normally get three entries in its +access-control list (ACL), representing permissions for the POSIX +permission scopes: Owner, group and others. +By default, the owner and group will be taken from the current user, and +the built-in group \[dq]Everyone\[dq] will be used to represent others. +The user/group can be customized with FUSE options \[dq]UserName\[dq] +and \[dq]GroupName\[dq], e.g. +\f[C]-o UserName=user123 -o GroupName=\[dq]Authenticated Users\[dq]\f[R]. +The permissions on each entry will be set according to options +\f[C]--dir-perms\f[R] and \f[C]--file-perms\f[R], which takes a value in +traditional Unix numeric +notation (https://en.wikipedia.org/wiki/File-system_permissions#Numeric_notation). +.PP +The default permissions corresponds to +\f[C]--file-perms 0666 --dir-perms 0777\f[R], i.e. +read and write permissions to everyone. +This means you will not be able to start any programs from the mount. +To be able to do that you must add execute permissions, e.g. +\f[C]--file-perms 0777 --dir-perms 0777\f[R] to add it to everyone. +If the program needs to write files, chances are you will have to enable +VFS File Caching as well (see also limitations). +Note that the default write permission have some restrictions for +accounts other than the owner, specifically it lacks the \[dq]write +extended attributes\[dq], as explained next. +.PP +The mapping of permissions is not always trivial, and the result you see +in Windows Explorer may not be exactly like you expected. +For example, when setting a value that includes write access for the +group or others scope, this will be mapped to individual permissions +\[dq]write attributes\[dq], \[dq]write data\[dq] and \[dq]append +data\[dq], but not \[dq]write extended attributes\[dq]. +Windows will then show this as basic permission \[dq]Special\[dq] +instead of \[dq]Write\[dq], because \[dq]Write\[dq] also covers the +\[dq]write extended attributes\[dq] permission. +When setting digit 0 for group or others, to indicate no permissions, +they will still get individual permissions \[dq]read attributes\[dq], +\[dq]read extended attributes\[dq] and \[dq]read permissions\[dq]. +This is done for compatibility reasons, e.g. +to allow users without additional permissions to be able to read basic +metadata about files like in Unix. +.PP +WinFsp 2021 (version 1.9) introduced a new FUSE option +\[dq]FileSecurity\[dq], that allows the complete specification of file +security descriptors using +SDDL (https://docs.microsoft.com/en-us/windows/win32/secauthz/security-descriptor-string-format). +With this you get detailed control of the resulting permissions, +compared to use of the POSIX permissions described above, and no +additional permissions will be added automatically for compatibility +with Unix. +Some example use cases will following. +.PP +If you set POSIX permissions for only allowing access to the owner, +using \f[C]--file-perms 0600 --dir-perms 0700\f[R], the user group and +the built-in \[dq]Everyone\[dq] group will still be given some special +permissions, as described above. +Some programs may then (incorrectly) interpret this as the file being +accessible by everyone, for example an SSH client may warn about +\[dq]unprotected private key file\[dq]. +You can work around this by specifying +\f[C]-o FileSecurity=\[dq]D:P(A;;FA;;;OW)\[dq]\f[R], which sets file all +access (FA) to the owner (OW), and nothing else. +.PP +When setting write permissions then, except for the owner, this does not +include the \[dq]write extended attributes\[dq] permission, as mentioned +above. +This may prevent applications from writing to files, giving permission +denied error instead. +To set working write permissions for the built-in \[dq]Everyone\[dq] +group, similar to what it gets by default but with the addition of the +\[dq]write extended attributes\[dq], you can specify +\f[C]-o FileSecurity=\[dq]D:P(A;;FRFW;;;WD)\[dq]\f[R], which sets file +read (FR) and file write (FW) to everyone (WD). +If file execute (FX) is also needed, then change to +\f[C]-o FileSecurity=\[dq]D:P(A;;FRFWFX;;;WD)\[dq]\f[R], or set file all +access (FA) to get full access permissions, including delete, with +\f[C]-o FileSecurity=\[dq]D:P(A;;FA;;;WD)\[dq]\f[R]. +.SS Windows caveats +.PP +Drives created as Administrator are not visible to other accounts, not +even an account that was elevated to Administrator with the User Account +Control (UAC) feature. +A result of this is that if you mount to a drive letter from a Command +Prompt run as Administrator, and then try to access the same drive from +Windows Explorer (which does not run as Administrator), you will not be +able to see the mounted drive. +.PP +If you don\[aq]t need to access the drive from applications running with +administrative privileges, the easiest way around this is to always +create the mount from a non-elevated command prompt. +.PP +To make mapped drives available to the user account that created them +regardless if elevated or not, there is a special Windows setting called +linked +connections (https://docs.microsoft.com/en-us/troubleshoot/windows-client/networking/mapped-drives-not-available-from-elevated-command#detail-to-configure-the-enablelinkedconnections-registry-entry) +that can be enabled. +.PP +It is also possible to make a drive mount available to everyone on the +system, by running the process creating it as the built-in SYSTEM +account. +There are several ways to do this: One is to use the command-line +utility +PsExec (https://docs.microsoft.com/en-us/sysinternals/downloads/psexec), +from Microsoft\[aq]s Sysinternals suite, which has option \f[C]-s\f[R] +to start processes as the SYSTEM account. +Another alternative is to run the mount command from a Windows Scheduled +Task, or a Windows Service, configured to run as the SYSTEM account. +A third alternative is to use the WinFsp.Launcher +infrastructure (https://github.com/winfsp/winfsp/wiki/WinFsp-Service-Architecture)). +Read more in the install documentation (https://rclone.org/install/). +Note that when running rclone as another user, it will not use the +configuration file from your profile unless you tell it to with the +\f[C]--config\f[R] (https://rclone.org/docs/#config-config-file) option. +Note also that it is now the SYSTEM account that will have the owner +permissions, and other accounts will have permissions according to the +group or others scopes. +As mentioned above, these will then not get the \[dq]write extended +attributes\[dq] permission, and this may prevent writing to files. +You can work around this with the FileSecurity option, see example +above. +.PP +Note that mapping to a directory path, instead of a drive letter, does +not suffer from the same limitations. +.SS Mounting on macOS +.PP +Mounting on macOS can be done either via built-in NFS +server (https://rclone.org/commands/rclone_serve_nfs/), +macFUSE (https://osxfuse.github.io/) (also known as osxfuse) or +FUSE-T (https://www.fuse-t.org/). +macFUSE is a traditional FUSE driver utilizing a macOS kernel extension +(kext). +FUSE-T is an alternative FUSE system which \[dq]mounts\[dq] via an NFSv4 +local server. +.SH NFS mount +.PP +This method spins up an NFS server using serve +nfs (https://rclone.org/commands/rclone_serve_nfs/) command and mounts +it to the specified mountpoint. +If you run this in background mode using |--daemon|, you will need to +send SIGTERM signal to the rclone process using |kill| command to stop +the mount. +.SS macFUSE Notes +.PP +If installing macFUSE using dmg +packages (https://github.com/osxfuse/osxfuse/releases) from the website, +rclone will locate the macFUSE libraries without any further +intervention. +If however, macFUSE is installed using the +macports (https://www.macports.org/) package manager, the following +addition steps are required. +.IP +.nf +\f[C] +sudo mkdir /usr/local/lib +cd /usr/local/lib +sudo ln -s /opt/local/lib/libfuse.2.dylib +\f[R] +.fi +.SS FUSE-T Limitations, Caveats, and Notes +.PP +There are some limitations, caveats, and notes about how it works. +These are current as of FUSE-T version 1.0.14. +.SS ModTime update on read +.PP +As per the FUSE-T +wiki (https://github.com/macos-fuse-t/fuse-t/wiki#caveats): +.RS +.PP +File access and modification times cannot be set separately as it seems +to be an issue with the NFS client which always modifies both. +Can be reproduced with \[aq]touch -m\[aq] and \[aq]touch -a\[aq] +commands +.RE +.PP +This means that viewing files with various tools, notably macOS Finder, +will cause rlcone to update the modification time of the file. +This may make rclone upload a full new copy of the file. +.SS Unicode Normalization +.PP +Rclone includes flags for unicode normalization with macFUSE that should +be updated for FUSE-T. +See this forum +post (https://forum.rclone.org/t/some-unicode-forms-break-mount-on-macos-with-fuse-t/36403) +and FUSE-T issue #16 (https://github.com/macos-fuse-t/fuse-t/issues/16). +The following flag should be added to the \f[C]rclone mount\f[R] +command. +.IP +.nf +\f[C] +-o modules=iconv,from_code=UTF-8,to_code=UTF-8 +\f[R] +.fi +.SS Read Only mounts +.PP +When mounting with \f[C]--read-only\f[R], attempts to write to files +will fail \f[I]silently\f[R] as opposed to with a clear warning as in +macFUSE. +.SS Limitations +.PP +Without the use of \f[C]--vfs-cache-mode\f[R] this can only write files +sequentially, it can only seek when reading. +This means that many applications won\[aq]t work with their files on an +rclone mount without \f[C]--vfs-cache-mode writes\f[R] or +\f[C]--vfs-cache-mode full\f[R]. +See the VFS File Caching section for more info. +When using NFS mount on macOS, if you don\[aq]t specify +|--vfs-cache-mode| the mount point will be read-only. +.PP +The bucket-based remotes (e.g. +Swift, S3, Google Compute Storage, B2) do not support the concept of +empty directories, so empty directories will have a tendency to +disappear once they fall out of the directory cache. +.PP +When \f[C]rclone mount\f[R] is invoked on Unix with \f[C]--daemon\f[R] +flag, the main rclone program will wait for the background mount to +become ready or until the timeout specified by the +\f[C]--daemon-wait\f[R] flag. +On Linux it can check mount status using ProcFS so the flag in fact sets +\f[B]maximum\f[R] time to wait, while the real wait can be less. +On macOS / BSD the time to wait is constant and the check is performed +only at the end. +We advise you to set wait time on macOS reasonably. +.PP +Only supported on Linux, FreeBSD, OS X and Windows at the moment. +.SS rclone nfsmount vs rclone sync/copy +.PP +File systems expect things to be 100% reliable, whereas cloud storage +systems are a long way from 100% reliable. +The rclone sync/copy commands cope with this with lots of retries. +However rclone nfsmount can\[aq]t use retries in the same way without +making local copies of the uploads. +Look at the VFS File Caching for solutions to make nfsmount more +reliable. +.SS Attribute caching +.PP +You can use the flag \f[C]--attr-timeout\f[R] to set the time the kernel +caches the attributes (size, modification time, etc.) for directory +entries. +.PP +The default is \f[C]1s\f[R] which caches files just long enough to avoid +too many callbacks to rclone from the kernel. +.PP +In theory 0s should be the correct value for filesystems which can +change outside the control of the kernel. +However this causes quite a few problems such as rclone using too much +memory (https://github.com/rclone/rclone/issues/2157), rclone not +serving files to +samba (https://forum.rclone.org/t/rclone-1-39-vs-1-40-mount-issue/5112) +and excessive time listing +directories (https://github.com/rclone/rclone/issues/2095#issuecomment-371141147). +.PP +The kernel can cache the info about a file for the time given by +\f[C]--attr-timeout\f[R]. +You may see corruption if the remote file changes length during this +window. +It will show up as either a truncated file or a file with garbage on the +end. +With \f[C]--attr-timeout 1s\f[R] this is very unlikely but not +impossible. +The higher you set \f[C]--attr-timeout\f[R] the more likely it is. +The default setting of \[dq]1s\[dq] is the lowest setting which +mitigates the problems above. +.PP +If you set it higher (\f[C]10s\f[R] or \f[C]1m\f[R] say) then the kernel +will call back to rclone less often making it more efficient, however +there is more chance of the corruption issue above. +.PP +If files don\[aq]t change on the remote outside of the control of rclone +then there is no chance of corruption. +.PP +This is the same as setting the attr_timeout option in mount.fuse. +.SS Filters +.PP +Note that all the rclone filters can be used to select a subset of the +files to be visible in the mount. +.SS systemd +.PP +When running rclone nfsmount as a systemd service, it is possible to use +Type=notify. +In this case the service will enter the started state after the +mountpoint has been successfully set up. +Units having the rclone nfsmount service specified as a requirement will +see all files and folders immediately in this mode. +.PP +Note that systemd runs mount units without any environment variables +including \f[C]PATH\f[R] or \f[C]HOME\f[R]. +This means that tilde (\f[C]\[ti]\f[R]) expansion will not work and you +should provide \f[C]--config\f[R] and \f[C]--cache-dir\f[R] explicitly +as absolute paths via rclone arguments. +Since mounting requires the \f[C]fusermount\f[R] program, rclone will +use the fallback PATH of \f[C]/bin:/usr/bin\f[R] in this scenario. +Please ensure that \f[C]fusermount\f[R] is present on this PATH. +.SS Rclone as Unix mount helper +.PP +The core Unix program \f[C]/bin/mount\f[R] normally takes the +\f[C]-t FSTYPE\f[R] argument then runs the \f[C]/sbin/mount.FSTYPE\f[R] +helper program passing it mount options as \f[C]-o key=val,...\f[R] or +\f[C]--opt=...\f[R]. +Automount (classic or systemd) behaves in a similar way. +.PP +rclone by default expects GNU-style flags \f[C]--key val\f[R]. +To run it as a mount helper you should symlink rclone binary to +\f[C]/sbin/mount.rclone\f[R] and optionally \f[C]/usr/bin/rclonefs\f[R], +e.g. +\f[C]ln -s /usr/bin/rclone /sbin/mount.rclone\f[R]. +rclone will detect it and translate command-line arguments +appropriately. +.PP +Now you can run classic mounts like this: +.IP +.nf +\f[C] +mount sftp1:subdir /mnt/data -t rclone -o vfs_cache_mode=writes,sftp_key_file=/path/to/pem +\f[R] +.fi +.PP +or create systemd mount units: +.IP +.nf +\f[C] +# /etc/systemd/system/mnt-data.mount +[Unit] +Description=Mount for /mnt/data +[Mount] +Type=rclone +What=sftp1:subdir +Where=/mnt/data +Options=rw,_netdev,allow_other,args2env,vfs-cache-mode=writes,config=/etc/rclone.conf,cache-dir=/var/rclone +\f[R] +.fi +.PP +optionally accompanied by systemd automount unit +.IP +.nf +\f[C] +# /etc/systemd/system/mnt-data.automount +[Unit] +Description=AutoMount for /mnt/data +[Automount] +Where=/mnt/data +TimeoutIdleSec=600 +[Install] +WantedBy=multi-user.target +\f[R] +.fi +.PP +or add in \f[C]/etc/fstab\f[R] a line like +.IP +.nf +\f[C] +sftp1:subdir /mnt/data rclone rw,noauto,nofail,_netdev,x-systemd.automount,args2env,vfs_cache_mode=writes,config=/etc/rclone.conf,cache_dir=/var/cache/rclone 0 0 +\f[R] +.fi +.PP +or use classic Automountd. +Remember to provide explicit \f[C]config=...,cache-dir=...\f[R] as a +workaround for mount units being run without \f[C]HOME\f[R]. +.PP +Rclone in the mount helper mode will split \f[C]-o\f[R] argument(s) by +comma, replace \f[C]_\f[R] by \f[C]-\f[R] and prepend \f[C]--\f[R] to +get the command-line flags. +Options containing commas or spaces can be wrapped in single or double +quotes. +Any inner quotes inside outer quotes of the same type should be doubled. +.PP +Mount option syntax includes a few extra options treated specially: +.IP \[bu] 2 +\f[C]env.NAME=VALUE\f[R] will set an environment variable for the mount +process. +This helps with Automountd and Systemd.mount which don\[aq]t allow +setting custom environment for mount helpers. +Typically you will use \f[C]env.HTTPS_PROXY=proxy.host:3128\f[R] or +\f[C]env.HOME=/root\f[R] +.IP \[bu] 2 +\f[C]command=cmount\f[R] can be used to run \f[C]cmount\f[R] or any +other rclone command rather than the default \f[C]mount\f[R]. +.IP \[bu] 2 +\f[C]args2env\f[R] will pass mount options to the mount helper running +in background via environment variables instead of command line +arguments. +This allows to hide secrets from such commands as \f[C]ps\f[R] or +\f[C]pgrep\f[R]. +.IP \[bu] 2 +\f[C]vv...\f[R] will be transformed into appropriate +\f[C]--verbose=N\f[R] +.IP \[bu] 2 +standard mount options like \f[C]x-systemd.automount\f[R], +\f[C]_netdev\f[R], \f[C]nosuid\f[R] and alike are intended only for +Automountd and ignored by rclone. +## VFS - Virtual File System +.PP +This command uses the VFS layer. +This adapts the cloud storage objects that rclone uses into something +which looks much more like a disk filing system. +.PP +Cloud storage objects have lots of properties which aren\[aq]t like disk +files - you can\[aq]t extend them or write to the middle of them, so the +VFS layer has to deal with that. +Because there is no one right way of doing this there are various +options explained below. +.PP +The VFS layer also implements a directory cache - this caches info about +files and directories (but not the data) in memory. +.SS VFS Directory Cache +.PP +Using the \f[C]--dir-cache-time\f[R] flag, you can control how long a +directory should be considered up to date and not refreshed from the +backend. +Changes made through the VFS will appear immediately or invalidate the +cache. +.IP +.nf +\f[C] +--dir-cache-time duration Time to cache directory entries for (default 5m0s) +--poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s) +\f[R] +.fi +.PP +However, changes made directly on the cloud storage by the web interface +or a different copy of rclone will only be picked up once the directory +cache expires if the backend configured does not support polling for +changes. +If the backend supports polling, changes will be picked up within the +polling interval. +.PP +You can send a \f[C]SIGHUP\f[R] signal to rclone for it to flush all +directory caches, regardless of how old they are. +Assuming only one rclone instance is running, you can reset the cache +like this: +.IP +.nf +\f[C] +kill -SIGHUP $(pidof rclone) +\f[R] +.fi +.PP +If you configure rclone with a remote control then you can use rclone rc +to flush the whole directory cache: +.IP +.nf +\f[C] +rclone rc vfs/forget +\f[R] +.fi +.PP +Or individual files or directories: +.IP +.nf +\f[C] +rclone rc vfs/forget file=path/to/file dir=path/to/dir +\f[R] +.fi +.SS VFS File Buffering +.PP +The \f[C]--buffer-size\f[R] flag determines the amount of memory, that +will be used to buffer data in advance. +.PP +Each open file will try to keep the specified amount of data in memory +at all times. +The buffered data is bound to one open file and won\[aq]t be shared. +.PP +This flag is a upper limit for the used memory per open file. +The buffer will only use memory for data that is downloaded but not not +yet read. +If the buffer is empty, only a small amount of memory will be used. +.PP +The maximum memory used by rclone for buffering can be up to +\f[C]--buffer-size * open files\f[R]. +.SS VFS File Caching +.PP +These flags control the VFS file caching options. +File caching is necessary to make the VFS layer appear compatible with a +normal file system. +It can be disabled at the cost of some compatibility. +.PP +For example you\[aq]ll need to enable VFS caching if you want to read +and write simultaneously to a file. +See below for more details. +.PP +Note that the VFS cache is separate from the cache backend and you may +find that you need one or the other or both. +.IP +.nf +\f[C] +--cache-dir string Directory rclone will use for caching. +--vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) +--vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s) +--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) +--vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) +--vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s) +--vfs-write-back duration Time to writeback files after last use when using cache (default 5s) +\f[R] +.fi +.PP +If run with \f[C]-vv\f[R] rclone will print the location of the file +cache. +The files are stored in the user cache file area which is OS dependent +but can be controlled with \f[C]--cache-dir\f[R] or setting the +appropriate environment variable. +.PP +The cache has 4 different modes selected by \f[C]--vfs-cache-mode\f[R]. +The higher the cache mode the more compatible rclone becomes at the cost +of using disk space. +.PP +Note that files are written back to the remote only when they are closed +and if they haven\[aq]t been accessed for \f[C]--vfs-write-back\f[R] +seconds. +If rclone is quit or dies with files that haven\[aq]t been uploaded, +these will be uploaded next time rclone is run with the same flags. +.PP +If using \f[C]--vfs-cache-max-size\f[R] or +\f[C]--vfs-cache-min-free-size\f[R] note that the cache may exceed these +quotas for two reasons. +Firstly because it is only checked every +\f[C]--vfs-cache-poll-interval\f[R]. +Secondly because open files cannot be evicted from the cache. +When \f[C]--vfs-cache-max-size\f[R] or +\f[C]--vfs-cache-min-free-size\f[R] is exceeded, rclone will attempt to +evict the least accessed files from the cache first. +rclone will start with files that haven\[aq]t been accessed for the +longest. +This cache flushing strategy is efficient and more relevant files are +likely to remain cached. +.PP +The \f[C]--vfs-cache-max-age\f[R] will evict files from the cache after +the set time since last access has passed. +The default value of 1 hour will start evicting files from cache that +haven\[aq]t been accessed for 1 hour. +When a cached file is accessed the 1 hour timer is reset to 0 and will +wait for 1 more hour before evicting. +Specify the time with standard notation, s, m, h, d, w . +.PP +You \f[B]should not\f[R] run two copies of rclone using the same VFS +cache with the same or overlapping remotes if using +\f[C]--vfs-cache-mode > off\f[R]. +This can potentially cause data corruption if you do. +You can work around this by giving each rclone its own cache hierarchy +with \f[C]--cache-dir\f[R]. +You don\[aq]t need to worry about this if the remotes in use don\[aq]t +overlap. +.SS --vfs-cache-mode off +.PP +In this mode (the default) the cache will read directly from the remote +and write directly to the remote without caching anything on disk. +.PP +This will mean some operations are not possible +.IP \[bu] 2 +Files can\[aq]t be opened for both read AND write +.IP \[bu] 2 +Files opened for write can\[aq]t be seeked +.IP \[bu] 2 +Existing files opened for write must have O_TRUNC set +.IP \[bu] 2 +Files open for read with O_TRUNC will be opened write only +.IP \[bu] 2 +Files open for write only will behave as if O_TRUNC was supplied +.IP \[bu] 2 +Open modes O_APPEND, O_TRUNC are ignored +.IP \[bu] 2 +If an upload fails it can\[aq]t be retried +.SS --vfs-cache-mode minimal +.PP +This is very similar to \[dq]off\[dq] except that files opened for read +AND write will be buffered to disk. +This means that files opened for write will be a lot more compatible, +but uses the minimal disk space. +.PP +These operations are not possible +.IP \[bu] 2 +Files opened for write only can\[aq]t be seeked +.IP \[bu] 2 +Existing files opened for write must have O_TRUNC set +.IP \[bu] 2 +Files opened for write only will ignore O_APPEND, O_TRUNC +.IP \[bu] 2 +If an upload fails it can\[aq]t be retried +.SS --vfs-cache-mode writes +.PP +In this mode files opened for read only are still read directly from the +remote, write only and read/write files are buffered to disk first. +.PP +This mode should support all normal file system operations. +.PP +If an upload fails it will be retried at exponentially increasing +intervals up to 1 minute. +.SS --vfs-cache-mode full +.PP +In this mode all reads and writes are buffered to and from disk. +When data is read from the remote this is buffered to disk as well. +.PP +In this mode the files in the cache will be sparse files and rclone will +keep track of which bits of the files it has downloaded. +.PP +So if an application only reads the starts of each file, then rclone +will only buffer the start of the file. +These files will appear to be their full size in the cache, but they +will be sparse files with only the data that has been downloaded present +in them. +.PP +This mode should support all normal file system operations and is +otherwise identical to \f[C]--vfs-cache-mode\f[R] writes. +.PP +When reading a file rclone will read \f[C]--buffer-size\f[R] plus +\f[C]--vfs-read-ahead\f[R] bytes ahead. +The \f[C]--buffer-size\f[R] is buffered in memory whereas the +\f[C]--vfs-read-ahead\f[R] is buffered on disk. +.PP +When using this mode it is recommended that \f[C]--buffer-size\f[R] is +not set too large and \f[C]--vfs-read-ahead\f[R] is set large if +required. +.PP +\f[B]IMPORTANT\f[R] not all file systems support sparse files. +In particular FAT/exFAT do not. +Rclone will perform very badly if the cache directory is on a filesystem +which doesn\[aq]t support sparse files and it will log an ERROR message +if one is detected. +.SS Fingerprinting +.PP +Various parts of the VFS use fingerprinting to see if a local file copy +has changed relative to a remote file. +Fingerprints are made from: +.IP \[bu] 2 +size +.IP \[bu] 2 +modification time +.IP \[bu] 2 +hash +.PP +where available on an object. +.PP +On some backends some of these attributes are slow to read (they take an +extra API call per object, or extra work per object). +.PP +For example \f[C]hash\f[R] is slow with the \f[C]local\f[R] and +\f[C]sftp\f[R] backends as they have to read the entire file and hash +it, and \f[C]modtime\f[R] is slow with the \f[C]s3\f[R], +\f[C]swift\f[R], \f[C]ftp\f[R] and \f[C]qinqstor\f[R] backends because +they need to do an extra API call to fetch it. +.PP +If you use the \f[C]--vfs-fast-fingerprint\f[R] flag then rclone will +not include the slow operations in the fingerprint. +This makes the fingerprinting less accurate but much faster and will +improve the opening time of cached files. +.PP +If you are running a vfs cache over \f[C]local\f[R], \f[C]s3\f[R] or +\f[C]swift\f[R] backends then using this flag is recommended. +.PP +Note that if you change the value of this flag, the fingerprints of the +files in the cache may be invalidated and the files will need to be +downloaded again. +.SS VFS Chunked Reading +.PP +When rclone reads files from a remote it reads them in chunks. +This means that rather than requesting the whole file rclone reads the +chunk specified. +This can reduce the used download quota for some remotes by requesting +only chunks from the remote that are actually read, at the cost of an +increased number of requests. +.PP +These flags control the chunking: +.IP +.nf +\f[C] +--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) +--vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) +\f[R] +.fi +.PP +Rclone will start reading a chunk of size +\f[C]--vfs-read-chunk-size\f[R], and then double the size for each read. +When \f[C]--vfs-read-chunk-size-limit\f[R] is specified, and greater +than \f[C]--vfs-read-chunk-size\f[R], the chunk size for each open file +will get doubled only until the specified value is reached. +If the value is \[dq]off\[dq], which is the default, the limit is +disabled and the chunk size will grow indefinitely. +.PP +With \f[C]--vfs-read-chunk-size 100M\f[R] and +\f[C]--vfs-read-chunk-size-limit 0\f[R] the following parts will be +downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. +When \f[C]--vfs-read-chunk-size-limit 500M\f[R] is specified, the result +would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so +on. +.PP +Setting \f[C]--vfs-read-chunk-size\f[R] to \f[C]0\f[R] or \[dq]off\[dq] +disables chunked reading. +.SS VFS Performance +.PP +These flags may be used to enable/disable features of the VFS for +performance or other reasons. +See also the chunked reading feature. +.PP +In particular S3 and Swift benefit hugely from the +\f[C]--no-modtime\f[R] flag (or use \f[C]--use-server-modtime\f[R] for a +slightly different effect) as each read of the modification time takes a +transaction. +.IP +.nf +\f[C] +--no-checksum Don\[aq]t compare checksums on up/download. +--no-modtime Don\[aq]t read/write the modification time (can speed things up). +--no-seek Don\[aq]t allow seeking in files. +--read-only Only allow read-only access. +\f[R] +.fi +.PP +Sometimes rclone is delivered reads or writes out of order. +Rather than seeking rclone will wait a short time for the in sequence +read or write to come in. +These flags only come into effect when not using an on disk cache file. +.IP +.nf +\f[C] +--vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms) +--vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s) +\f[R] +.fi +.PP +When using VFS write caching (\f[C]--vfs-cache-mode\f[R] with value +writes or full), the global flag \f[C]--transfers\f[R] can be set to +adjust the number of parallel uploads of modified files from the cache +(the related global flag \f[C]--checkers\f[R] has no effect on the VFS). +.IP +.nf +\f[C] +--transfers int Number of file transfers to run in parallel (default 4) +\f[R] +.fi +.SS VFS Case Sensitivity +.PP +Linux file systems are case-sensitive: two files can differ only by +case, and the exact case must be used when opening a file. +.PP +File systems in modern Windows are case-insensitive but case-preserving: +although existing files can be opened using any case, the exact case +used to create the file is preserved and available for programs to +query. +It is not allowed for two files in the same directory to differ only by +case. +.PP +Usually file systems on macOS are case-insensitive. +It is possible to make macOS file systems case-sensitive but that is not +the default. +.PP +The \f[C]--vfs-case-insensitive\f[R] VFS flag controls how rclone +handles these two cases. +If its value is \[dq]false\[dq], rclone passes file names to the remote +as-is. +If the flag is \[dq]true\[dq] (or appears without a value on the command +line), rclone may perform a \[dq]fixup\[dq] as explained below. +.PP +The user may specify a file name to open/delete/rename/etc with a case +different than what is stored on the remote. +If an argument refers to an existing file with exactly the same name, +then the case of the existing file on the disk will be used. +However, if a file name with exactly the same name is not found but a +name differing only by case exists, rclone will transparently fixup the +name. +This fixup happens only when an existing file is requested. +Case sensitivity of file names created anew by rclone is controlled by +the underlying remote. +.PP +Note that case sensitivity of the operating system running rclone (the +target) may differ from case sensitivity of a file system presented by +rclone (the source). +The flag controls whether \[dq]fixup\[dq] is performed to satisfy the +target. +.PP +If the flag is not provided on the command line, then its default value +depends on the operating system where rclone runs: \[dq]true\[dq] on +Windows and macOS, \[dq]false\[dq] otherwise. +If the flag is provided without a value, then it is \[dq]true\[dq]. +.SS VFS Disk Options +.PP +This flag allows you to manually set the statistics about the filing +system. +It can be useful when those statistics cannot be read correctly +automatically. +.IP +.nf +\f[C] +--vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1) +\f[R] +.fi +.SS Alternate report of used bytes +.PP +Some backends, most notably S3, do not report the amount of bytes used. +If you need this information to be available when running \f[C]df\f[R] +on the filesystem, then pass the flag \f[C]--vfs-used-is-size\f[R] to +rclone. +With this flag set, instead of relying on the backend to report this +information, rclone will scan the whole remote similar to +\f[C]rclone size\f[R] and compute the total used space itself. +.PP +\f[I]WARNING.\f[R] Contrary to \f[C]rclone size\f[R], this flag ignores +filters so that the result is accurate. +However, this is very inefficient and may cost lots of API calls +resulting in extra charges. +Use it as a last resort and only with caching. +.IP +.nf +\f[C] +rclone nfsmount remote:path /path/to/mountpoint [flags] +\f[R] +.fi +.SS Options +.IP +.nf +\f[C] + --allow-non-empty Allow mounting over a non-empty directory (not supported on Windows) + --allow-other Allow access to other users (not supported on Windows) + --allow-root Allow access to root user (not supported on Windows) + --async-read Use asynchronous reads (not supported on Windows) (default true) + --attr-timeout Duration Time for which file/directory attributes are cached (default 1s) + --daemon Run mount in background and exit parent process (as background output is suppressed, use --log-file with --log-format=pid,... to monitor) (not supported on Windows) + --daemon-timeout Duration Time limit for rclone to respond to kernel (not supported on Windows) (default 0s) + --daemon-wait Duration Time to wait for ready mount from daemon (maximum time on Linux, constant sleep time on OSX/BSD) (not supported on Windows) (default 1m0s) + --debug-fuse Debug the FUSE internals - needs -v + --default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows) + --devname string Set the device name - default is remote:path + --dir-cache-time Duration Time to cache directory entries for (default 5m0s) + --dir-perms FileMode Directory permissions (default 0777) + --file-perms FileMode File permissions (default 0666) + --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required) + --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) + -h, --help help for nfsmount + --max-read-ahead SizeSuffix The number of bytes that can be prefetched for sequential reads (not supported on Windows) (default 128Ki) + --mount-case-insensitive Tristate Tell the OS the mount is case insensitive (true) or sensitive (false) regardless of the backend (auto) (default unset) + --network-mode Mount as remote network drive, instead of fixed disk drive (supported on Windows only) + --no-checksum Don\[aq]t compare checksums on up/download + --no-modtime Don\[aq]t read/write the modification time (can speed things up) + --no-seek Don\[aq]t allow seeking in files + --noappledouble Ignore Apple Double (._) and .DS_Store files (supported on OSX only) (default true) + --noapplexattr Ignore all \[dq]com.apple.*\[dq] extended attributes (supported on OSX only) + -o, --option stringArray Option for libfuse/WinFsp (repeat if required) + --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) + --read-only Only allow read-only access + --sudo Use sudo to run the mount command as root. + --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) + --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) + --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) + --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) + --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) + --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) + --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s) + --vfs-case-insensitive If a file name not found, find a case insensitive match + --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off) + --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection + --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full + --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) + --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off) + --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) + --vfs-refresh Refreshes the directory cache recursively in the background on start + --vfs-used-is-size rclone size Use the rclone size algorithm for Used size + --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) + --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) + --volname string Set the volume name (supported on Windows and OSX only) + --write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows) +\f[R] +.fi +.SS Filter Options +.PP +Flags for filtering directory listings. +.IP +.nf +\f[C] + --delete-excluded Delete files on dest excluded from sync + --exclude stringArray Exclude files matching pattern + --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) + --exclude-if-present stringArray Exclude directories if filename is present + --files-from stringArray Read list of source-file names from file (use - to read from stdin) + --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) + -f, --filter stringArray Add a file filtering rule + --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) + --ignore-case Ignore case in filters (case insensitive) + --include stringArray Include files matching pattern + --include-from stringArray Read file include patterns from file (use - to read from stdin) + --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) + --max-depth int If set limits the recursion depth to this (default -1) + --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) + --metadata-exclude stringArray Exclude metadatas matching pattern + --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) + --metadata-filter stringArray Add a metadata filtering rule + --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) + --metadata-include stringArray Include metadatas matching pattern + --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) + --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) + --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) +\f[R] +.fi +.PP +See the global flags page (https://rclone.org/flags/) for global options +not listed here. +.SH SEE ALSO +.IP \[bu] 2 +rclone (https://rclone.org/commands/rclone/) - Show help for rclone +commands, flags and backends. .SH rclone obscure .PP Obscure password for use in the rclone config file. @@ -8612,7 +9718,7 @@ rclone serve dlna remote:path [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -9175,7 +10281,7 @@ rclone serve docker [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -9667,7 +10773,7 @@ STDOUT. ignored. .PP There is an example program -bin/test_proxy.py (https://github.com/rclone/rclone/blob/master/test_proxy.py) +bin/test_proxy.py (https://github.com/rclone/rclone/blob/master/bin/test_proxy.py) in the rclone source code. .PP The program\[aq]s job is to take a \f[C]user\f[R] and \f[C]pass\f[R] on @@ -9788,7 +10894,7 @@ rclone serve ftp remote:path [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -10492,7 +11598,7 @@ STDOUT. ignored. .PP There is an example program -bin/test_proxy.py (https://github.com/rclone/rclone/blob/master/test_proxy.py) +bin/test_proxy.py (https://github.com/rclone/rclone/blob/master/bin/test_proxy.py) in the rclone source code. .PP The program\[aq]s job is to take a \f[C]user\f[R] and \f[C]pass\f[R] on @@ -10622,7 +11728,7 @@ rclone serve http remote:path [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -11158,7 +12264,7 @@ rclone serve nfs remote:path [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -12084,7 +13190,7 @@ rclone serve s3 remote:path [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -12616,7 +13722,7 @@ STDOUT. ignored. .PP There is an example program -bin/test_proxy.py (https://github.com/rclone/rclone/blob/master/test_proxy.py) +bin/test_proxy.py (https://github.com/rclone/rclone/blob/master/bin/test_proxy.py) in the rclone source code. .PP The program\[aq]s job is to take a \f[C]user\f[R] and \f[C]pass\f[R] on @@ -12737,7 +13843,7 @@ rclone serve sftp remote:path [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -13474,7 +14580,7 @@ STDOUT. ignored. .PP There is an example program -bin/test_proxy.py (https://github.com/rclone/rclone/blob/master/test_proxy.py) +bin/test_proxy.py (https://github.com/rclone/rclone/blob/master/bin/test_proxy.py) in the rclone source code. .PP The program\[aq]s job is to take a \f[C]user\f[R] and \f[C]pass\f[R] on @@ -13606,7 +14712,7 @@ rclone serve webdav remote:path [flags] --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached (\[aq]off\[aq] is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) - --vfs-refresh Refreshes the directory cache recursively on start + --vfs-refresh Refreshes the directory cache recursively in the background on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -16061,7 +17167,7 @@ json.dump(o, sys.stdout, indent=\[dq]\[rs]t\[dq]) .PP You can find this example (slightly expanded) in the rclone source code at -bin/test_metadata_mapper.py (https://github.com/rclone/rclone/blob/master/test_metadata_mapper.py). +bin/test_metadata_mapper.py (https://github.com/rclone/rclone/blob/master/bin/test_metadata_mapper.py). .PP If you want to see the input to the metadata mapper and the output returned from it in the log you can use \f[C]-vv --dump mapper\f[R]. @@ -16122,7 +17228,7 @@ Capable backends are marked in the overview (https://rclone.org/overview/#optional-features) as \f[C]MultithreadUpload\f[R]. (They need to implement either the \f[C]OpenWriterAt\f[R] or -\f[C]OpenChunkedWriter\f[R] internal interfaces). +\f[C]OpenChunkWriter\f[R] internal interfaces). These include include, \f[C]local\f[R], \f[C]s3\f[R], \f[C]azureblob\f[R], \f[C]b2\f[R], \f[C]oracleobjectstorage\f[R] and \f[C]smb\f[R] at the time of writing. @@ -24524,7 +25630,7 @@ General networking and HTTP stuff. --tpslimit float Limit HTTP transactions per second to this --tpslimit-burst int Max burst of transactions for --tpslimit (default 1) --use-cookies Enable session cookiejar - --user-agent string Set the user-agent to a specified string (default \[dq]rclone/v1.65.0\[dq]) + --user-agent string Set the user-agent to a specified string (default \[dq]rclone/v1.65.1\[dq]) \f[R] .fi .SS Performance @@ -29605,6 +30711,8 @@ being written to: \f[C]PutObject\f[R] .IP \[bu] 2 \f[C]PutObjectACL\f[R] +.IP \[bu] 2 +\f[C]CreateBucket\f[R] (unless using s3-no-check-bucket) .PP When using the \f[C]lsd\f[R] subcommand, the \f[C]ListAllMyBuckets\f[R] permission is required. @@ -29650,6 +30758,10 @@ It assumes that \f[C]USER_NAME\f[R] has been created. .IP "2." 3 The Resource entry must include both resource ARNs, as one implies the bucket and the other implies the bucket\[aq]s objects. +.IP "3." 3 +When using s3-no-check-bucket and the bucket already exsits, the +\f[C]\[dq]arn:aws:s3:::BUCKET_NAME\[dq]\f[R] doesn\[aq]t have to be +included. .PP For reference, here\[aq]s an Ansible script (https://gist.github.com/ebridges/ebfc9042dd7c756cd101cfa807b7ae2b) @@ -58931,6 +60043,140 @@ Options: .IP \[bu] 2 \[dq]error\[dq]: return an error based on option value .SH Changelog +.SS v1.65.1 - 2024-01-08 +.PP +See commits (https://github.com/rclone/rclone/compare/v1.65.0...v1.65.1) +.IP \[bu] 2 +Bug Fixes +.RS 2 +.IP \[bu] 2 +build +.RS 2 +.IP \[bu] 2 +Bump golang.org/x/crypto to fix ssh terrapin CVE-2023-48795 (dependabot) +.IP \[bu] 2 +Update to go1.21.5 to fix Windows path problems (Nick Craig-Wood) +.IP \[bu] 2 +Fix docker build on arm/v6 (Nick Craig-Wood) +.RE +.IP \[bu] 2 +install.sh: fix harmless error message on install (Nick Craig-Wood) +.IP \[bu] 2 +accounting: fix stats to show server side transfers (Nick Craig-Wood) +.IP \[bu] 2 +doc fixes (albertony, ben-ba, Eli Orzitzer, emyarod, keongalvin, +rarspace01) +.IP \[bu] 2 +nfsmount: Compile for all unix oses, add \f[C]--sudo\f[R] and fix +error/option handling (Nick Craig-Wood) +.IP \[bu] 2 +operations: Fix files moved by rclone move not being counted as +transfers (Nick Craig-Wood) +.IP \[bu] 2 +oauthutil: Avoid panic when \f[C]*token\f[R] and \f[C]*ts.token\f[R] are +the same (rkonfj) +.IP \[bu] 2 +serve s3: Fix listing oddities (Nick Craig-Wood) +.RE +.IP \[bu] 2 +VFS +.RS 2 +.IP \[bu] 2 +Note that \f[C]--vfs-refresh\f[R] runs in the background (Nick +Craig-Wood) +.RE +.IP \[bu] 2 +Azurefiles +.RS 2 +.IP \[bu] 2 +Fix storage base url (Oksana) +.RE +.IP \[bu] 2 +Crypt +.RS 2 +.IP \[bu] 2 +Fix rclone move a file over itself deleting the file (Nick Craig-Wood) +.RE +.IP \[bu] 2 +Chunker +.RS 2 +.IP \[bu] 2 +Fix rclone move a file over itself deleting the file (Nick Craig-Wood) +.RE +.IP \[bu] 2 +Compress +.RS 2 +.IP \[bu] 2 +Fix rclone move a file over itself deleting the file (Nick Craig-Wood) +.RE +.IP \[bu] 2 +Dropbox +.RS 2 +.IP \[bu] 2 +Fix used space on dropbox team accounts (Nick Craig-Wood) +.RE +.IP \[bu] 2 +FTP +.RS 2 +.IP \[bu] 2 +Fix multi-thread copy (WeidiDeng) +.RE +.IP \[bu] 2 +Googlephotos +.RS 2 +.IP \[bu] 2 +Fix nil pointer exception when batch failed (Nick Craig-Wood) +.RE +.IP \[bu] 2 +Hasher +.RS 2 +.IP \[bu] 2 +Fix rclone move a file over itself deleting the file (Nick Craig-Wood) +.IP \[bu] 2 +Fix invalid memory address error when MaxAge == 0 (nielash) +.RE +.IP \[bu] 2 +Onedrive +.RS 2 +.IP \[bu] 2 +Fix error listing: unknown object type \f[C]\f[R] (Nick Craig-Wood) +.IP \[bu] 2 +Fix \[dq]unauthenticated: Unauthenticated\[dq] errors when uploading +(Nick Craig-Wood) +.RE +.IP \[bu] 2 +Oracleobjectstorage +.RS 2 +.IP \[bu] 2 +Fix object storage endpoint for custom endpoints (Manoj Ghosh) +.IP \[bu] 2 +Multipart copy create bucket if it doesn\[aq]t exist. +(Manoj Ghosh) +.RE +.IP \[bu] 2 +Protondrive +.RS 2 +.IP \[bu] 2 +Fix CVE-2023-45286 / GHSA-xwh9-gc39-5298 (Nick Craig-Wood) +.RE +.IP \[bu] 2 +S3 +.RS 2 +.IP \[bu] 2 +Fix crash if no UploadId in multipart upload (Nick Craig-Wood) +.RE +.IP \[bu] 2 +Smb +.RS 2 +.IP \[bu] 2 +Fix shares not listed by updating go-smb2 (halms) +.RE +.IP \[bu] 2 +Union +.RS 2 +.IP \[bu] 2 +Fix rclone move a file over itself deleting the file (Nick Craig-Wood) +.RE .SS v1.65.0 - 2023-11-26 .PP See commits (https://github.com/rclone/rclone/compare/v1.64.0...v1.65.0) @@ -72323,7 +73569,7 @@ Bj\[/o]rn Erik Pedersen .IP \[bu] 2 Lukas Loesche .IP \[bu] 2 -emyarod +emyarod .IP \[bu] 2 T.C. Ferguson