community
/
rclone
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
5,377 Commits 2 Branches 109 Tags 198 MiB
Commit Graph

202 Commits

Author SHA1 Message Date
Nick Craig-Wood fc57648b75 lib/rest: fix multipart uploads stopping on context cancel 
Before this change when the context was cancelled (due to
--max-duration for example) this could deadlock when uploading
multipart uploads.

This change fixes the problem by introducing another go routine to
monitor the context and close the pipe with an error when the context
errors.
 2021-03-29 19:09:47 +01:00
albertony ce182adf46
copyurl: add option to print resulting auto-filename (#5095) 
Fixes #5094
 2021-03-18 10:04:59 +01:00
Nick Craig-Wood 25d5ed763c fs: make sync and operations tests use context instead of global variables 2021-03-15 19:22:07 +00:00
Nick Craig-Wood 4013bc4a4c Fix excessive retries missing --max-duration timeout - fixes #4504 
This change checks the context whenever rclone might retry, and
doesn't retry if the current context has an error.

This fixes the pathological behaviour of `--max-duration` refusing to
exit because all the context deadline exceeded errors were being
retried.

This unfortunately meant changing the shouldRetry logic in every
backend and doing a lot of context propagation.

See: https://forum.rclone.org/t/add-flag-to-exit-immediately-when-max-duration-reached/22723
 2021-03-13 09:25:44 +00:00
Saksham Khanna 4d8ef7bca7
cmd/dedupe: make largest directory primary to minimize data moved (#3648) 
This change makes dedupe recursively count elements in same-named directories
and make the largest one primary. This allows to minimize the amount of data
moved (or at least the amount of API calls) when dedupe merges them.
It also adds a new fs.Object interface `ParentIDer` with function `ParentID` and
implements it for the drive and opendrive backends. This function returns
parent directory ID for objects on filesystems that allow same-named dirs.
We use it to correctly count sizes of same-named directories.

Fixes #2568

Co-authored-by: Ivan Andreev <ivandeex@gmail.com>
 2021-03-11 20:40:29 +03:00
Nick Craig-Wood 4b5fe3adad delete,rmdirs: make --rmdirs obey the filters 
See: https://forum.rclone.org/t/a-problem-with-rclone-delete-from-list/22143
 2021-02-19 10:32:28 +00:00
K265 930bca2478 feat: add multiple paths support to `--compare-dest` and `--copy-dest` flag 2021-02-16 18:17:04 +00:00
Ankur Gupta 47b69d6300 operations: Made copy and sync operations obey a RetryAfterError 2021-02-11 17:47:34 +00:00
Nick Craig-Wood 06f1c0c61c move: fix data loss when moving the same object 
This checks to see if IDs are the same of the source and destination
object before deleting one of them and potentially causing data loss.

See: https://forum.rclone.org/t/files-deleted-and-not-moved/21830
 2021-01-28 16:14:16 +00:00
Nick Craig-Wood dd67a3d5f5 operations: add size if known to skipped items and JSON log - fixes #4624 2021-01-05 19:44:25 +00:00
Nick Craig-Wood 35da38e93f operations: fix --immutable error message 
This was accidentally changed in

53a1a0e3ef accounting: add reference to completed transfers
 2020-12-31 18:16:51 +00:00
Ingo Weiss 1d40bc1901 fs: Accumulate stats when using --dry-run 
Fixes #4624
 2020-12-29 21:11:12 +00:00
lostheli c8cfa43ccc Add a download flag to hashsum and related commands to force rclone to download and hash files locally 
This commit modifies the operations.hashSum function by adding an alternate code path. This code path is triggered by passing downloadFlag = True. When activated, rclone will download files from the remote and hash them locally. downloadFlag = False preserves the existing behavior of using the remote to retrieve the hash.

This commit modifies HashLister to support the new hashSum method as well as consolidating the roles of HashLister, HashListerBase64, Md5sum, and Sha1sum.  The printing of hashes from the function defined in HashLister has been revised to work with --progress.  There are light changes to operations.syncFprintf and cmd.startProgress for this.

The unit test operations_test.TestHashSums is modified to support this change and test the download functionality.

The command functions hashsum, md5sum, sha1sum, and dbhashsum are modified to support this change.  A download flag has been added and an output-file flag has been added.  The output-file flag writes hashes to a file instead of stdout to avoid the need to redirect stdout.
 2020-12-27 15:40:44 +00:00
Nick Craig-Wood 86014cebd7 dedupe: add --dedupe-mode list to just list dupes, changing nothing 2020-12-02 16:52:12 +00:00
Nick Craig-Wood 507f861c67 dedupe: add --by-hash to dedupe on hash not file name - fixes #1674 2020-12-02 16:52:12 +00:00
Nick Craig-Wood c22d04aa30 filter: deglobalise to put filter config into the context #4685 2020-11-27 17:28:42 +00:00
Nick Craig-Wood 9d574c0d63 fshttp: read config from ctx not passed in ConfigInfo #4685 2020-11-26 16:40:12 +00:00
Nick Craig-Wood 2e21c58e6a fs: deglobalise the config #4685 
This is done by making fs.Config private and attaching it to the
context instead.

The Config should be obtained with fs.GetConfig and fs.AddConfig
should be used to get a new mutable config that can be changed.
 2020-11-26 16:40:12 +00:00
Nick Craig-Wood 1fb6ad700f accounting: add context.Context #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood e3fe31f7cb fs: add context.Context to fs.GetModifyWindow #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood d69b96a94c test: Add context to mockfs.NewFs #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood d846210978 fs: Add context to NewFs #3257 #4685 
This adds a context.Context parameter to NewFs and related calls.

This is necessary as part of reading config from the context -
backends need to be able to read the global config.
 2020-11-09 18:05:54 +00:00
Anagh Kumar Baranwal 30c8b1b84f
fs: Fix nil pointer on copy & move operations directly to remote 
Fix the copy and move operations that broke in 127f0fc when copying directly
to a remote without a specific destination.

Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
 2020-11-09 12:36:10 +00:00
Nick Craig-Wood af55a74bd2 stats: add counter for deleted directories - fixes #4676 2020-11-03 11:47:00 +00:00
Nick Craig-Wood d00c126cef operations: fix --cutof-mode hard not cutting off immediately 
This failure was noted on the integration tests server.

The fix was to be more careful about which error message was emitted
with which --cutoff-mode
 2020-11-02 17:13:19 +00:00
Adam Plánský 127f0fc64c operations: move and copy log name of the destination object in verbose 
If the object is moved or copied rclone in verbose mode prints name of the
destination object into the info log.
 2020-10-30 15:31:54 +00:00
Josh Soref e4a87f772f docs: spelling: e.g. 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref bbe7eb35f1 docs: spelling: server-side 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Nick Craig-Wood 4e2b5389d7 check: make the error count match up in the log message 
Before this change we counted the final summary error as an error,
producing confusing log messages like:

    Failed to check with 54 errors: last error was: 53 differences found

This change marks the summary error as already being counted, so the
error message becomes:

    Failed to check with 53 errors: last error was: 53 differences found

This change also returns a listing failure in preference to a summary error.

See: https://forum.rclone.org/t/slow-checksum-validation/19763/22
 2020-10-15 12:57:48 +01:00
Josh Soref d0888edc0a Spelling fixes 
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-14 15:21:31 +01:00
Nick Craig-Wood 20b00db390 operations: fix spurious "--checksum is in use but the source and destination have no hashes in common" 
Before this change rclone would emit the message

    --checksum is in use but the source and destination have no hashes in common; falling back to --size-only

When the source or destination hash was missing as well as when the
source and destination had no hashes in common.

This first case is very confusing for users when the source and
destination do have a hash in common.

This change fixes that and makes sure the error message is not emitted
on missing hashes even when there is a hash in common.

See: https://forum.rclone.org/t/source-and-destination-have-no-hashes-in-common-for-unencrypted-drive-to-local-sync/19531
 2020-10-05 16:07:05 +01:00
Nick Craig-Wood db4bbf9521 operations: fix use of --suffix without --backup-dir 
As part of the original work adding this feature it was overlooked
that this didn't actually work for full rclone copy/sync.

This commit fixes the problem and adds a test to make sure it stays
working.

See: https://forum.rclone.org/t/suffix-not-working-on-folder-upload-via-ssh-sftp/19526
 2020-10-04 16:40:20 +01:00
Nick Craig-Wood 2b7994e739 operations: add logs when need to upload files to set mod times #1505 2020-09-29 17:04:29 +01:00
Nick Craig-Wood 75de30cfa8 sync: fix --cutoff-mode soft & cautious so it doesn't end the transfer early 
Before ths fix --cutoff-mode soft and cautious would emit a Fatal
error which stopped the sync immediately.

This fix introduces a new error which is checked in the sync error
processing which stops the sync gracefully.

Fixes #4576
 2020-09-09 12:53:21 +01:00
Chaitanya Bankanhal 61c7ea4085
rc: fix rc/uploadfile only working for root of the fs 2020-08-10 17:09:46 +01:00
Nick Craig-Wood a2afa9aadd fs: Add directory to optional Purge interface - fixes #1891 
- add a directory to the optional Purge interface
- fix up all the backends
- add an additional integration test to test for the feature
- use the new feature in operations.Purge

Many of the backends had been prepared in advance for this so the
change was trivial for them.
 2020-07-31 17:43:17 +01:00
Nick Craig-Wood 421585dd72 accounting: add context to Account and propagate changes #3257 
This is preparation for getting the Accounting to check the context,
buf first we need to get it in place. Since this is one of those
changes that makes lots of noise, this is in a seperate commit.
 2020-07-28 16:41:17 +01:00
Nick Craig-Wood 62f0bbb598 dedupe: Make it obey the --size-only flag for duplicate detection #4321 2020-07-28 11:40:37 +01:00
Nick Craig-Wood ff84351655 operations: factor Check and related functions into its own files 2020-07-21 22:08:13 +01:00
Nick Craig-Wood 8b6f2bbb4b check,cryptcheck: add reporting of filenames for same/missing/changed #3264 
See: https://forum.rclone.org/t/rclone-check-v-doesnt-show-once-per-minute-update-counts/17402
 2020-07-21 22:08:13 +01:00
Chaitanya Bankanhal a2dd23efd3 rc: Add tests for operations/uploadfile 
rc: Go import file rc_test.go
 2020-06-25 11:38:24 +01:00
Chaitanya Bankanhal fa43d02874 rc: Add operations/uploadfile to upload a file through rc using encoding multipart/form-data 2020-06-25 11:38:24 +01:00
Chaitanya d0de39ebcd rc: add NeedsRequest to call. 2020-06-25 11:38:24 +01:00
Nick Craig-Wood ce394426b0 check: fix successful retries with --download counting errors 
See: https://forum.rclone.org/t/tons-of-data-corruption-after-rclone-copy-to-mega-can-rclone-correct-it/17017
 2020-06-20 14:52:04 +01:00
Nick Craig-Wood 40611fc4fc check: retry downloads if they fail when using the --download flag 
See: https://forum.rclone.org/t/tons-of-data-corruption-after-rclone-copy-to-mega-can-rclone-correct-it/17017/7
 2020-06-18 16:16:19 +01:00
Nick Craig-Wood 7c4ba9fcb2 check: Fix misleading message which printed errors instead of differences 
See: https://forum.rclone.org/t/tons-of-data-corruption-after-rclone-copy-to-mega-can-rclone-correct-it/17017/7
 2020-06-18 16:16:19 +01:00
Nick Craig-Wood 3efdf5e095 operations: implement --refresh-times flag to set modtimes on hashless backends 2020-06-17 10:48:13 +01:00
Nick Craig-Wood a3f6fe5287 dedupe: fix logging to be easier to understand #4321 2020-06-16 11:41:56 +01:00
Nick Craig-Wood d55053098f check: make check do --checkers files concurrently - fixes #4318 2020-06-10 17:20:54 +01:00
Nick Craig-Wood 63cf0b1cdd check: make check command obey --dry-run/-i/--interactive - fixes #4325 2020-06-10 17:20:54 +01:00