community
/
rclone
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
6,810 Commits 2 Branches 109 Tags 198 MiB
Commit Graph

123 Commits

Author SHA1 Message Date
Nick Craig-Wood dd6e229327 move: if --check-first and --order-by are set then delete with perfect ordering 
If using rclone move and --check-first and --order-by then rclone uses
the transfer routine to delete files to ensure perfect ordering.

This will cause the transfer stats to have a larger than expected
number of items in it so we don't enable this by default.

Fixes #6033
 2023-03-10 08:23:32 +00:00
Nick Craig-Wood a5390dbbeb sync,operations: fix correct concurrency: use --checkers unless transferring files 
There were some places (e.g. deleting files) where we were using
--transfers instead of --checkers to control the concurrency when
files weren't being transferred.

These have been updated to use --checkers.
 2023-03-01 11:10:38 +00:00
Nick Craig-Wood 019a486d5b accounting: Make checkers show what they are doing 
Before this change, all types of checkers showed "checking" after the
file name despite the fact that not all of them were checking.

After this change, they can show

- checking
- deleting
- hashing
- importing
- listing
- merging
- moving
- renaming

See: https://forum.rclone.org/t/what-is-rclone-checking-during-a-purge/35931/
 2023-03-01 11:10:38 +00:00
Eng Zer Jun 8e507075d1 test: replace defer cleanup with `t.Cleanup` 
Reference: https://pkg.go.dev/testing#T.Cleanup
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2022-12-09 14:38:05 +00:00
Nick Craig-Wood 6fd9e3d717 build: reformat comments to pass go1.19 vet 
See: https://go.dev/doc/go1.19#go-doc
 2022-08-05 16:35:41 +01:00
Nick Craig-Wood a875320e37 sync,operations: optimise --copy-dest and --compare-dest 
Before this change --compare-dest and --copy-dest would check to see
if the compare/copy object existed first, before seeing if the
destination object was present.

This is inefficient, because in most --copy-dest syncs the destination
will be present and the compare/copy object need never be tested.
--compare-dest syncs may also be speeded up if they are done to the
same directory repeatedly.

This fixes the problem by re-arranging the logic so if the transfer is
not needed then the compare/copy object is never tested.

See: https://forum.rclone.org/t/union-with-copy-dest-enabled-is-slower-than-expected/32172
 2022-08-03 17:44:47 +01:00
albertony fdd2f8e6d2 Error strings should not be capitalized 
Reported by staticcheck 2022.1.2 (v0.3.2)

See: staticcheck.io
 2022-06-23 23:26:02 +02:00
albertony de5ccaab8e docs: cross link doc pages for related commands 2022-06-21 23:43:00 +02:00
Nick Craig-Wood 4ac875a811 sync: fix --max-duration and --cutoff-mode soft 
Before this change using --max-duration and --cutoff-mode soft would
work like --cutoff-mode hard.

This bug was introduced in this commit which made transfers be
cancelable - before that transfers couldn't be canceled.

122a47fba6 accounting: Allow transfers to be canceled with context #3257

This change adds the timeout to the input context for reading files
rather than the transfer context so the files transfers themselves
aren't canceled if --cutoff-mode soft is in action.

This also adds a test for cutoff mode soft and max duration which was
missing.

See: https://forum.rclone.org/t/max-duration-and-retries-not-working-correctly/27738
 2022-06-16 22:13:50 +01:00
Nick Craig-Wood cfe0911e0d sync: fix tests for overlapping with filter 
In commit

3ccf222acb sync: overlap check is now filter-sensitive

The tests were attempting to write invalid objects on some backends
due to a leading / on the object name.

This fix also adds a few more test cases and makes sure the tests can
be run individually.
 2022-06-09 14:26:43 +01:00
albertony f4f0e444bf filter: allow multiple --exclude-if-present flags - fixes #6219 2022-06-08 17:11:52 +02:00
Nick 3ccf222acb
sync: overlap check is now filter-sensitive 
Previously, the overlap check was based on simple prefix checks of the source and destination paths. Now it actually checks whether the destination is excluded via any filter rule or a "--exclude-if-present"-file.
 2022-06-01 18:24:54 +01:00
Nick Craig-Wood 21355b4208 sync: Fix --max-duration so it doesn't retry when the duration is exceeded 
Before this change, if the --max-duration limit was reached then
rclone would retry the sync as a fatal error wasn't raised.

This checks the deadline and raises a fatal error if necessary at the
end of the sync.

Fixes #6002
 2022-03-03 17:08:16 +00:00
database64128 a7a8372976
🧪 fstest: fix time tests on Windows and add convenience methods to check local and remote fs with precision 
Previously only the fs being checked on gets passed to
GetModifyWindow(). However, in most tests, the test files are
generated in the local fs and transferred to the remote fs. So the
local fs time precision has to be taken into account.

This meant that on Windows the time tests failed because the
local fs has a time precision of 100ns. Checking remote items uploaded
from local fs on Windows also requires a modify window of 100ns.
 2021-11-09 11:43:36 +00:00
Nick Craig-Wood e43b5ce5e5 Remove github.com/pkg/errors and replace with std library version 
This is possible now that we no longer support go1.12 and brings
rclone into line with standard practices in the Go world.

This also removes errors.New and errors.Errorf from lib/errors and
prefers the stdlib errors package over lib/errors.
 2021-11-07 11:53:30 +00:00
Ivan Andreev b323bf34e2 sync/test: skip test ConcurrentTruncate on uptobox (take 2) 
The test is not applicable to uptobox which can't upload empty files.
The test was not skipped as intended because the direct error was compared.
This fix will compare error Cause because Sync wraps the error.
 2021-11-02 19:24:23 +03:00
Ivan Andreev 70297c3aed sync/test: TestConcurrentTruncate needs empty files - skip on uptobox 2021-10-28 17:04:56 +03:00
albertony e2f47ecdeb docs: punctuation cleanup 
See #5538
 2021-10-20 22:56:19 +02:00
Nathan Collins c32d5dd1f3 fs: move with --ignore-existing will not delete skipped files - #5463 2021-08-01 17:46:45 +01:00
Nick Craig-Wood ba5c559fec fs/sync: fix tests by only --compare-dest timestamp if have hash 
This fixes the integration test errors introduced in #5410
 2021-07-07 16:59:51 +01:00
Ivan Andreev 4680c0776d
backend/local: skip entries removed concurrently with List() (#5297) 
This change fixes the bug described below:
if a file is removed while the local backend List() runs,
the call will flag an accounting error.
The bug manifests itself if local backend is the Sync target
due to intrinsic concurrency.
The odds to hit this bug depend on --checkers and --transfers.
Chunker over local backend is affected even more because
updating a composite object with a smaller size content
translates into removing chunks on the underlying file system
and involves a number of List() calls.
 2021-07-07 16:50:19 +03:00
Nick Gaya 40024990b7 fs/operations: Don't update timestamps of files in --compare-dest 2021-07-05 16:29:44 +01:00
albertony 98579608ec docs: cleanup spelling of size and rate units 2021-04-27 02:25:52 +03:00
Nick Craig-Wood 54771e4402 sync: fix incorrect error reported by graceful cutoff - fixes #5203 
Before this change, a sync which was finished with a graceful transfer
cutoff could return "context canceled" instead of the correct error.

This fixes the problem by ignoring "context canceled" errors if we
have done a graceful stop.
 2021-04-06 13:08:42 +01:00
Nick Gaya 9db7c51eaa
sync: don't warn about --no-traverse when --files-from is set 2021-04-05 20:36:39 +01:00
Nick Craig-Wood 25d5ed763c fs: make sync and operations tests use context instead of global variables 2021-03-15 19:22:07 +00:00
K265 930bca2478 feat: add multiple paths support to `--compare-dest` and `--copy-dest` flag 2021-02-16 18:17:04 +00:00
Nick Craig-Wood f1147fe1dd rc: sync,copy,move: document createEmptySrcDirs parameter - fixes #4489 2021-02-08 12:25:40 +00:00
Nick Craig-Wood 0a932dc1f2 Add --bwlimit for upload and download #1873 2021-01-30 17:39:30 +00:00
Nick Craig-Wood 3b6df71838 accounting: refactor bwlimit code to allow for multiple slots 2021-01-30 17:39:30 +00:00
Nick Craig-Wood 06f1c0c61c move: fix data loss when moving the same object 
This checks to see if IDs are the same of the source and destination
object before deleting one of them and potentially causing data loss.

See: https://forum.rclone.org/t/files-deleted-and-not-moved/21830
 2021-01-28 16:14:16 +00:00
Nick Craig-Wood 963c0f28b9 sync: Only print "There was nothing to transfer" if no errors 
See: https://forum.rclone.org/t/immutable-should-set-retries-1-when-source-and-dest-are-different/21326
 2020-12-31 18:16:51 +00:00
Nick Craig-Wood b3815dc0c2 sync: fix --immutable errors retrying many times 
See: https://forum.rclone.org/t/immutable-should-set-retries-1-when-source-and-dest-are-different/21326
 2020-12-31 18:16:51 +00:00
Nick Craig-Wood c22d04aa30 filter: deglobalise to put filter config into the context #4685 2020-11-27 17:28:42 +00:00
Nick Craig-Wood 2e21c58e6a fs: deglobalise the config #4685 
This is done by making fs.Config private and attaching it to the
context instead.

The Config should be obtained with fs.GetConfig and fs.AddConfig
should be used to get a new mutable config that can be changed.
 2020-11-26 16:40:12 +00:00
Nick Craig-Wood 1fb6ad700f accounting: add context.Context #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood e3fe31f7cb fs: add context.Context to fs.GetModifyWindow #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood d846210978 fs: Add context to NewFs #3257 #4685 
This adds a context.Context parameter to NewFs and related calls.

This is necessary as part of reading config from the context -
backends need to be able to read the global config.
 2020-11-09 18:05:54 +00:00
Josh Soref e4a87f772f docs: spelling: e.g. 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref bbe7eb35f1 docs: spelling: server-side 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref d0888edc0a Spelling fixes 
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-14 15:21:31 +01:00
Nick Craig-Wood db4bbf9521 operations: fix use of --suffix without --backup-dir 
As part of the original work adding this feature it was overlooked
that this didn't actually work for full rclone copy/sync.

This commit fixes the problem and adds a test to make sure it stays
working.

See: https://forum.rclone.org/t/suffix-not-working-on-folder-upload-via-ssh-sftp/19526
 2020-10-04 16:40:20 +01:00
Nick Craig-Wood 75de30cfa8 sync: fix --cutoff-mode soft & cautious so it doesn't end the transfer early 
Before ths fix --cutoff-mode soft and cautious would emit a Fatal
error which stopped the sync immediately.

This fix introduces a new error which is checked in the sync error
processing which stops the sync gracefully.

Fixes #4576
 2020-09-09 12:53:21 +01:00
Nick Craig-Wood 122a47fba6 accounting: Allow transfers to be canceled with context #3257 
This makes all transfers cancelable even if the backend doesn't
support context as all transfers are done using the Accounting
framework.
 2020-07-28 16:41:17 +01:00
Nick Craig-Wood 8f42532b6d sync: add --track-renames-strategy leaf 
See: https://forum.rclone.org/t/how-to-minimize-bandwith-w-r-t-renames-during-sync/16928/22
 2020-07-28 11:34:27 +01:00
Nick Craig-Wood 811b30d116 sync: fix deadlock with --track-renames-strategy modtime - fixes #4427 
Before this change we could exit the popRenameMap function with the
lock held.

This fixes the problem by defer-ring the unlock.

See: https://forum.rclone.org/t/track-renames-strategy-modtime-doesnt-work/16992
 2020-07-17 17:09:58 +01:00
Nick Craig-Wood fb06427c69 sync: fix --track-renames-strategy modtime 
Before this change `--track-renames-strategy` was broken. The hashing
method it used could declare times that were very close together to be
different.

The time hash was discarded and instead we check the modification time
window on every hash match.

Provided that the user doesn't use `--track-renames-strategy` on a
huge number of identically sized files this will perform just fine.

See: https://forum.rclone.org/t/track-renames-strategy-modtime-doesnt-work/16992/5
 2020-06-12 15:38:35 +01:00
Nick Craig-Wood b23cf58a41 operations: Add skip all, do all, quit operations to --interactive - fixes #3886 
This also adds SkipDestructive into all the remaing places --dry-run
was used and adds documentation.
 2020-06-10 12:33:53 +01:00
Martin Michlmayr 041b201abd doc: fix typos throughout docs and code 2020-05-25 11:23:58 +01:00
Nick Craig-Wood 518d39815c sync,copy,move: add --check-first to do all checking before starting transfers 
See: https://forum.rclone.org/t/rclone-sync-doing-transfer-and-checking-in-paralel/16352/
 2020-05-22 17:50:07 +01:00