Commit Graph

736 Commits

Author SHA1 Message Date
Nick Craig-Wood c553ad5158 serve sftp: fix authentication on one connection blocking others - fixes #4882
Before this change, if one connection was authenticating this would
block any others from authenticating.

This was due to ssh.NewServerConn not being called in a go routine
after the Accept call.

This is fixed by running the ssh authentication in a go routine.

Thanks to @FiloSottile for advice on how to fix this.

See: https://github.com/golang/go/issues/43521
2021-01-06 15:34:07 +00:00
Ilyess Bachiri 5d1f947f32
docs: fix a typo in the dedupe docs 2021-01-05 15:46:57 +00:00
Nick Craig-Wood 2af7b61fc3 cmd: fix wording for no retries message
See: https://forum.rclone.org/t/immutable-should-set-retries-1-when-source-and-dest-are-different/21326/
2021-01-02 11:20:02 +00:00
Nick Craig-Wood 5601652d65 ncdu: add ! (errror) and . (unreadable) file flags to go with e (empty) 2020-12-28 17:25:46 +00:00
Adam Plánský b218bc5bed ncdu: add empty folder flag into ncdu browser
Implemented empty folder flag for ncdu browser interface. If there is
empty folder in the list the flag e is prepended before size. If there
is no empty folder this flag is ommited. It has the same behaviour as
original ncdu browser. (https://dev.yorhel.nl/ncdu/man)
2020-12-28 17:25:46 +00:00
Claudio Bantaloukas f7fe1d766b
cmd/ncdu: highlight read errors instead of aborting - fixes #4014
When a directory cannot be walk-ed because of a permissions error
- or any error for that matter -, ncdu mode keeps track of the error
and highlights directories that could not be read.

Previously, the error would cause ncdu to abort.

Now, directories with unreadable sub-directories are displayed in yellow and
a message warns that the total may be underestimated.

Unreadable directories themselves are displayed in red along with the error message
2020-12-28 14:08:12 +00:00
albertony 55aec19389 mount: docs: add section about windows filesystem permissions 2020-12-28 13:59:34 +00:00
albertony f50e15c77c mount: docs: add note about mounted file system size 2020-12-28 13:59:34 +00:00
albertony e3191d096f mount: just set default options without checking if customized by user, because it will be overridden anyway 2020-12-28 13:59:34 +00:00
albertony 07c40780b3 mount: also detect if uid or gid are set in same option string: -o uid=123,gid=456 2020-12-28 13:59:34 +00:00
albertony 67b82b4a28 mount: docs: update documentation according to new syntax on windows 2020-12-28 13:59:34 +00:00
albertony 5f47e1e034 mount: cleanup os specific option handling and documentation 2020-12-28 13:59:34 +00:00
albertony e92cb9e8f8 mount: more user friendly mounting as network drive on windows
Add --network-mode option to activate mounting as network drive without having to set volume prefix.
Add support for automatic drive letter assignment (not specific to network drive mounting).
Allow full network share unc path in --volname, which will also implicitely activate network drive mounting.
Allow full network share unc path as mountpoint, which will also implicitely activate network drive mounting, and the specified path will be used as volume prefix and the remote will be mounted on an automatically assigned drive letter instead.
2020-12-28 13:59:34 +00:00
Nick Craig-Wood 629c0d0f65 serve http: fix serving files of unknown length
Before this change serving files of unknown length were always
returned as 0 length files.

This change serves them correctly, but does not support Range:
requests on them.

See: https://forum.rclone.org/t/serve-http-behavior-when-the-size-is-unknown/21319
2020-12-27 22:01:41 +00:00
lostheli c8cfa43ccc Add a download flag to hashsum and related commands to force rclone to download and hash files locally
This commit modifies the operations.hashSum function by adding an alternate code path. This code path is triggered by passing downloadFlag = True. When activated, rclone will download files from the remote and hash them locally. downloadFlag = False preserves the existing behavior of using the remote to retrieve the hash.

This commit modifies HashLister to support the new hashSum method as well as consolidating the roles of HashLister, HashListerBase64, Md5sum, and Sha1sum.  The printing of hashes from the function defined in HashLister has been revised to work with --progress.  There are light changes to operations.syncFprintf and cmd.startProgress for this.

The unit test operations_test.TestHashSums is modified to support this change and test the download functionality.

The command functions hashsum, md5sum, sha1sum, and dbhashsum are modified to support this change.  A download flag has been added and an output-file flag has been added.  The output-file flag writes hashes to a file instead of stdout to avoid the need to redirect stdout.
2020-12-27 15:40:44 +00:00
Nick Craig-Wood 73bb9322f5 rc: prefer actual listener address if using ":port" or "addr:0" only
Before this change rclone would turn `localhost:8888` into
`127.0.0.1:8888` which apparently does not work with some browsers.

See: https://github.com/rclone/rclone-webui-react/issues/117
2020-12-26 16:28:54 +00:00
Nick Craig-Wood e45716cac2 mount: add "." and ".." to directories to match cmount and expectations
See: https://forum.rclone.org/t/empty-directorys-size-for-a-mounted-crypt-remote/21077
2020-12-17 12:14:22 +00:00
Nick Craig-Wood c98dd8755c log: fix enabling systemd logging when using --log-file
This also moves all the systemd logging decisions to fs/log
2020-12-17 11:55:27 +00:00
Nick Craig-Wood 86014cebd7 dedupe: add --dedupe-mode list to just list dupes, changing nothing 2020-12-02 16:52:12 +00:00
Nick Craig-Wood 507f861c67 dedupe: add --by-hash to dedupe on hash not file name - fixes #1674 2020-12-02 16:52:12 +00:00
Nick Craig-Wood 8bf4697dc2 serve http/webdav: redirect requests to the base url without the /
When using `--baseurl` before this patch, if a request was made to the
base URL without a trailing / then rclone would return a 404 error.

Unfortunately GVFS / Nautilus makes the request without the /
regardless of what the user put in.

This patch redirects the request to the base URL with a /. So if the
user was using `--baseurl rclone` then a request to
http://localhost/rclone would be redirected with a 308 response to
http://localhost/rclone/

Fixes #4814
2020-12-01 18:00:07 +00:00
Nick Craig-Wood f7d9b15707 cmount: don't call host.Umount if a signal has been received
Before this change cgofuse and libatexit would race to see who could
unmount the file system with unpredicatable results. On Linux it could
report an error or not, depending.

This change checks to see if umount is beng called from a signal and
if so leaves the unmounting to cgofuse/libfuse.

See #4804
2020-11-29 17:44:00 +00:00
Nick Craig-Wood 1cfce703b2 mountlib: make sure we don't call umount more than once #4804
Before this change when using CTRL-C with rclone cmount the
mount would be unmounted twice.
2020-11-29 17:44:00 +00:00
Nick Craig-Wood c22d04aa30 filter: deglobalise to put filter config into the context #4685 2020-11-27 17:28:42 +00:00
Nick Craig-Wood 354b4f19ec rc: add context to flag Reload function #4685 2020-11-27 17:28:42 +00:00
edwardxml 19a8b66cee
docs: update rclone about docs
Create a full loop of documentation for rclone about, backends overview
and individual backend pages.

Discussion:
https://github.com/rclone/rclone/pull/4774 relates

Previously pull was requested, in part, under ref
https://github.com/rclone/rclone/pull/4801

Notes:
Introduce a tentative draft see-link format the end of sections to try
rather than lots of in-para links.

Update about.go incl link to list of backends not supporting about.

In list of backends not supporting about, include link to about command
reference.

I appreciate there may be decisions to make going forward about whether
command links should be code formatted, and using proper pretty url
links, but I have fudged that for now.

Update backend pages that do not support about with wording used
previously for ftp - it is in passive voice but I can live with it. (my
own wording and fault). The note is applied to a limitations section. If
one does not already exist it is created (even if there are other
limitations with their own sections)
2020-11-27 14:08:52 +00:00
Anagh Kumar Baranwal 07dee18d6b cmount: Add optional `brew` tag to throw an error when using mount in the
binaries installed via Homebrew - Fixes #4775

Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-11-26 22:56:59 +00:00
Nick Craig-Wood 9d574c0d63 fshttp: read config from ctx not passed in ConfigInfo #4685 2020-11-26 16:40:12 +00:00
Nick Craig-Wood 2e21c58e6a fs: deglobalise the config #4685
This is done by making fs.Config private and attaching it to the
context instead.

The Config should be obtained with fs.GetConfig and fs.AddConfig
should be used to get a new mutable config that can be changed.
2020-11-26 16:40:12 +00:00
albertony 79474b2e4c docs: update documentation of commands delete,purge,rmdir,rmdirs 2020-11-23 17:10:08 +00:00
Deepak Sah 5b9115d87a
serve ftp: add options to enable TLS - fixes #3640 2020-11-23 16:07:51 +00:00
Nick Craig-Wood f0905499e3 random: seed math/rand in one place with crypto strong seed #4783
This shouldn't be read as encouraging the use of math/rand instead of
crypto/rand in security sensitive contexts, rather as a safer default
if that does happen by accident.
2020-11-18 17:48:44 +00:00
Nick Craig-Wood ceeac84cfe serve restic: implement object cache
This caches all the objects returned from the List call. This makes
opening them much quicker so speeds up prune and restores. It also
uses fewer transactions. It can be disabled with
`--cache-objects=false`.

This was discovered when using the B2 backend when the budget was
being blown on list object calls which can avoided with a bit of
caching.

For typical 1 million file backup for a latop or server this will only
use a small amount more memory.
2020-11-12 17:58:46 +00:00
albertony 2708a7569e mount: docs: make note about mounting as network drive less confusing 2020-11-11 20:37:57 +00:00
Nick Craig-Wood e204f89685 servetest: add -sub-run flag for running a subset of the backend tests
Use like this (eg in cmd/serve/sftp)

    go test -v -run TestSftp/Normal -sub-run "TestIntegration/FsMkdir/FsPutFiles/FsDirMove"
2020-11-10 16:47:25 +00:00
Nick Craig-Wood f7efce594b config: add context.Context #3257 #4685
This add config to the Config callback in the backends and the related
config functions.
2020-11-09 18:05:54 +00:00
Nick Craig-Wood 1fb6ad700f accounting: add context.Context #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood d846210978 fs: Add context to NewFs #3257 #4685
This adds a context.Context parameter to NewFs and related calls.

This is necessary as part of reading config from the context -
backends need to be able to read the global config.
2020-11-09 18:05:54 +00:00
Adam Plánský f89ff3872d ncdu: add toggle option for average size in directory - key 'a'
Add toggle option to show average size in directory. This toggle
function is for ncdu and is binded to key 'a'.
2020-10-30 15:33:54 +00:00
Nick Craig-Wood bfcd4113c3 mount: implement mknod to make NFS file creation work - fixes #2115
It turns out that NFS calls mknod in FUSE even though we have create
defined. This was causing EIO errors when creating files.

This patch fixes it by implementing mknod. The way it is implemented
means that to write to an NFS file system you'll need --vfs-cache-mode
writes.
2020-10-29 15:12:36 +00:00
Nick Craig-Wood 0e7fc7613f mount: make mount be cmount under macOS #4393
This also adds an alias to the mount command so it responds as `rclone
cmount` as well as `rclone mount`.
2020-10-29 13:34:39 +00:00
Nick Craig-Wood 8ac2f52b6e mount: disable bazil/fuse based mount on macOS #4393
The library is no longer supported on macOS.
2020-10-29 13:34:39 +00:00
Josh Soref 8bc9b2b883 docs: grammar: examples are examples
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-28 18:16:23 +00:00
Josh Soref a15f50254a docs: grammar: if, then
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-28 18:16:23 +00:00
Josh Soref 3e1cb8302a docs: spelling: etc.
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-28 18:16:23 +00:00
Josh Soref e4a87f772f docs: spelling: e.g.
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-28 18:16:23 +00:00
Josh Soref bbe7eb35f1 docs: spelling: server-side
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-28 18:16:23 +00:00
Adam Plánský e00bf3d723
ncdu: add sort by average size in directory
Add keyboard shortcut 'A' which sort by average size in directory.

If files/folders have same avgSize sort by actual size

Fixes: #4699
2020-10-27 13:28:38 +00:00
Nick Craig-Wood 00a5086ff2 Remove accidentally committed binary and fix formatting
These were both committed in

b7253fc1c1 mount: docs: add note that allow-root and allow-other is not relevant on windows
2020-10-26 19:07:42 +00:00
albertony b7253fc1c1 mount: docs: add note that allow-root and allow-other is not relevant on windows 2020-10-26 16:21:43 +00:00