community
/
rclone
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
5,377 Commits 2 Branches 109 Tags 198 MiB
Commit Graph

99 Commits

Author SHA1 Message Date
Nick Craig-Wood 563c4f76c6 sync: fix incorrect error reported by graceful cutoff - fixes #5203 
Before this change, a sync which was finished with a graceful transfer
cutoff could return "context canceled" instead of the correct error.

This fixes the problem by ignoring "context canceled" errors if we
have done a graceful stop.
 2021-04-10 11:49:44 +01:00
Nick Craig-Wood 25d5ed763c fs: make sync and operations tests use context instead of global variables 2021-03-15 19:22:07 +00:00
K265 930bca2478 feat: add multiple paths support to `--compare-dest` and `--copy-dest` flag 2021-02-16 18:17:04 +00:00
Nick Craig-Wood f1147fe1dd rc: sync,copy,move: document createEmptySrcDirs parameter - fixes #4489 2021-02-08 12:25:40 +00:00
Nick Craig-Wood 0a932dc1f2 Add --bwlimit for upload and download #1873 2021-01-30 17:39:30 +00:00
Nick Craig-Wood 3b6df71838 accounting: refactor bwlimit code to allow for multiple slots 2021-01-30 17:39:30 +00:00
Nick Craig-Wood 06f1c0c61c move: fix data loss when moving the same object 
This checks to see if IDs are the same of the source and destination
object before deleting one of them and potentially causing data loss.

See: https://forum.rclone.org/t/files-deleted-and-not-moved/21830
 2021-01-28 16:14:16 +00:00
Nick Craig-Wood 963c0f28b9 sync: Only print "There was nothing to transfer" if no errors 
See: https://forum.rclone.org/t/immutable-should-set-retries-1-when-source-and-dest-are-different/21326
 2020-12-31 18:16:51 +00:00
Nick Craig-Wood b3815dc0c2 sync: fix --immutable errors retrying many times 
See: https://forum.rclone.org/t/immutable-should-set-retries-1-when-source-and-dest-are-different/21326
 2020-12-31 18:16:51 +00:00
Nick Craig-Wood c22d04aa30 filter: deglobalise to put filter config into the context #4685 2020-11-27 17:28:42 +00:00
Nick Craig-Wood 2e21c58e6a fs: deglobalise the config #4685 
This is done by making fs.Config private and attaching it to the
context instead.

The Config should be obtained with fs.GetConfig and fs.AddConfig
should be used to get a new mutable config that can be changed.
 2020-11-26 16:40:12 +00:00
Nick Craig-Wood 1fb6ad700f accounting: add context.Context #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood e3fe31f7cb fs: add context.Context to fs.GetModifyWindow #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood d846210978 fs: Add context to NewFs #3257 #4685 
This adds a context.Context parameter to NewFs and related calls.

This is necessary as part of reading config from the context -
backends need to be able to read the global config.
 2020-11-09 18:05:54 +00:00
Josh Soref e4a87f772f docs: spelling: e.g. 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref bbe7eb35f1 docs: spelling: server-side 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref d0888edc0a Spelling fixes 
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-14 15:21:31 +01:00
Nick Craig-Wood db4bbf9521 operations: fix use of --suffix without --backup-dir 
As part of the original work adding this feature it was overlooked
that this didn't actually work for full rclone copy/sync.

This commit fixes the problem and adds a test to make sure it stays
working.

See: https://forum.rclone.org/t/suffix-not-working-on-folder-upload-via-ssh-sftp/19526
 2020-10-04 16:40:20 +01:00
Nick Craig-Wood 75de30cfa8 sync: fix --cutoff-mode soft & cautious so it doesn't end the transfer early 
Before ths fix --cutoff-mode soft and cautious would emit a Fatal
error which stopped the sync immediately.

This fix introduces a new error which is checked in the sync error
processing which stops the sync gracefully.

Fixes #4576
 2020-09-09 12:53:21 +01:00
Nick Craig-Wood 122a47fba6 accounting: Allow transfers to be canceled with context #3257 
This makes all transfers cancelable even if the backend doesn't
support context as all transfers are done using the Accounting
framework.
 2020-07-28 16:41:17 +01:00
Nick Craig-Wood 8f42532b6d sync: add --track-renames-strategy leaf 
See: https://forum.rclone.org/t/how-to-minimize-bandwith-w-r-t-renames-during-sync/16928/22
 2020-07-28 11:34:27 +01:00
Nick Craig-Wood 811b30d116 sync: fix deadlock with --track-renames-strategy modtime - fixes #4427 
Before this change we could exit the popRenameMap function with the
lock held.

This fixes the problem by defer-ring the unlock.

See: https://forum.rclone.org/t/track-renames-strategy-modtime-doesnt-work/16992
 2020-07-17 17:09:58 +01:00
Nick Craig-Wood fb06427c69 sync: fix --track-renames-strategy modtime 
Before this change `--track-renames-strategy` was broken. The hashing
method it used could declare times that were very close together to be
different.

The time hash was discarded and instead we check the modification time
window on every hash match.

Provided that the user doesn't use `--track-renames-strategy` on a
huge number of identically sized files this will perform just fine.

See: https://forum.rclone.org/t/track-renames-strategy-modtime-doesnt-work/16992/5
 2020-06-12 15:38:35 +01:00
Nick Craig-Wood b23cf58a41 operations: Add skip all, do all, quit operations to --interactive - fixes #3886 
This also adds SkipDestructive into all the remaing places --dry-run
was used and adds documentation.
 2020-06-10 12:33:53 +01:00
Martin Michlmayr 041b201abd doc: fix typos throughout docs and code 2020-05-25 11:23:58 +01:00
Nick Craig-Wood 518d39815c sync,copy,move: add --check-first to do all checking before starting transfers 
See: https://forum.rclone.org/t/rclone-sync-doing-transfer-and-checking-in-paralel/16352/
 2020-05-22 17:50:07 +01:00
Nick Craig-Wood 147f97d1f7 sync: allow --max-backlog to be -ve meaning as large as possible 2020-05-22 17:50:07 +01:00
Martin Michlmayr 4aee962233 doc: fix typos throughout docs and code 2020-05-20 15:54:51 +01:00
Ben Zenker 899c8e0697 march: added flag to allow Unicode filenames to remain unique 
If your filenames contain two near-identical Unicode characters,
rclone will normalize these, making them identical. This flag
gives you the ability to keep them unique. This might
create unintended side effects, such as duplicating files that
contain certain Unicode characters, when downloading them from
certain cloud providers to a macOS filesystem.

Fixes #4228
 2020-05-15 12:28:01 +01:00
Nick Craig-Wood b2c9ef23fa sync: make --track-renames tests only check rename count if expecting renames 2020-03-31 10:58:49 +01:00
Nick Craig-Wood 5f9be3dd05 sync: make --track-renames tests less fragile by using rename stat 
Before this change these tests attempted to measure transfers and
checks in lieu of having a rename statistic with a very complicated
heuristic.

The change switches over to using the rename statistic which should be
100% reliable.
 2020-03-30 18:30:33 +01:00
Nick Craig-Wood ad9c7ff7ed sync: Fix incorrect "nothing to transfer" message using --delete-before 
Before this change the first pass of --delete-before would output
"There was nothing to transfer" and then proceed to transfer things.

This makes sure the message isn't printed in the delete phase.

See: https://forum.rclone.org/t/incorrect-debug-output/15267
 2020-03-30 16:45:02 +01:00
Nick Craig-Wood 32df634cb6 sync: fix --track-renames-strategy modtime test on remotes which don't support modtime 2020-03-22 11:52:40 +00:00
Nick Craig-Wood 93f5125f51 sync: fix --track-renames-strategy tests 
This commit corrects the logic for --track-renames-strategy which
broke the integration tests.

It also improves the parsing of the argument and adds a test for that.
 2020-03-21 17:39:51 +00:00
Bernd Schoolmann 158870bcdb
fs: Add --track-renames-strategy for configurable matching criteria for --track-renames 
This commit adds the `--track-renames-strategy` flag which allows the
user to choose the strategy for tracking renames when using the
`--track-renames` flag.

This can be "hash" or "modtime" or both currently.

This, when used with `--track-renames-strategy modtime` enables
support for tracking renames in encrypted remotes.

Fixes #3696
Fixes #2721
 2020-03-20 13:04:56 +00:00
Nick Craig-Wood c227a90b52 sync: implement --order-by xxx,mixed 2020-03-16 15:50:04 +00:00
Nick Craig-Wood 1e3d899db8 sync: replace container/heap with github.com/aalpar/deheap 2020-03-16 15:50:04 +00:00
Anuar Serdaliyev f14871caf7
accounting: Correct exitcode on Transfer Limit Exceeded flag. Fixes #3203 
Before this change the exit code for transfer limit exceeded was
incorrect. This was because the `resolveExitCode` function unwraps the
error thus reading the underlying error which is not the same as the
error it was comparing to (`ErrorMaxTransferLimitReached`).

This change fixes it by splitting the error definition in two so that
when the Fatal error is unwrapped we match against
`ErrorMaxTransferLimitReached` however when we return the error we
return `ErrorMaxTransferLimitReachedFatal`.
 2020-03-10 12:00:10 +00:00
Ishuah Kariuki 0c6f14c694
copy/sync: only create empty directories when they don't exist on the remote 
Sync/copy now only creates empty directories when they don't exist on the remote (--create-empty-src-dirs flag) - fixes #2800
 2020-03-03 16:24:22 +00:00
Nick Craig-Wood 2b268f9724 build: fixup formatting after go1.14 go fmt changes 2020-02-28 16:58:33 +00:00
Tim Gallant 1d73f071f6 fs: improve log output when no changes are made - fixes #3454 
- changes a few log messages to debug level
- adds a log message for when 0 bytes are transferred
 2020-02-11 12:16:15 +00:00
Nick Craig-Wood 9c9cdf1712 fs: don't run tests for --max-duration on remote backends 
This is a timing dependent test and to make it long enough so that it
would work with the remotes would make it too long for local tests.

The code paths are identical for local vs non-local so just run on
local.

This fixes the integration tests.
 2020-01-26 09:23:03 +00:00
boosh 0d7573dd81 fs: Add --max-duration flag to control the maximum duration of a transfer session 
This gives you more control over how long rclone will run for, making
it easier to script backups, e.g. via cron. Once the `--max-duration`
time limit is reached, no new transfers will be initiated, but those
already in-flight will be allowed to complete.

Fixes #985
 2020-01-24 13:28:56 +00:00
Nick Craig-Wood 375d25f158 sync: implement --order-by flag to order transfers - fixes #1205 2020-01-16 15:24:36 +00:00
Nick Craig-Wood 207474abab sync: add --no-check-dest flag - fixes #3616 2019-12-29 16:47:57 +00:00
Ankur Gupta 75a6c49f87 Fix error counter - fixes #3650 
For few commands, RClone counts a error multiple times. This was fixed by
creating a new error type which keeps a flag to remember if the error has
already been counted or not. The CountError function now wraps the original
error eith the above new error type and returns it.
 2019-11-18 14:13:02 +00:00
Nick Craig-Wood 543100070a sync: free objects after they come out of the transfer pipe 
This reduces memory when the transfer pipe shrinks

See: https://forum.rclone.org/t/rclone-memory-consumption-increasing-linearly/12244
 2019-10-16 10:27:07 +01:00
Nick Craig-Wood f3b0f8a9f0 sync: --update/-u not transfer files that haven't changed - fixes #3232 
Before this change --update would transfer any file which was newer
than the destination regardless of whether it had changed or not.
This is needlessly wasteful of bandwidth.

After this change --update will only transfer files if they are newer
**and** they are different (checked with checksum and size).
 2019-10-12 11:54:56 +01:00
Ivan Andreev ccecfa9cb1 chunker: finish meta-format before release 
changes:
- chunker: remove GetTier and SetTier
- remove wdmrcompat metaformat
- remove fastopen strategy
- make hash_type option non-advanced
- adverise hash support when possible
- add metadata field "ver", run strict checks
- describe internal behavior in comments
- improve documentation

note:
wdmrcompat used to write file name in the metadata, so maximum metadata
size was 1K; removing it allows to cap size by 200 bytes now.
 2019-09-25 11:03:33 +01:00
Ivan Andreev 1e4691f951 tests/sync: adjust transfer counts for chunker 2019-09-24 12:45:12 +01:00