community
/
rclone
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,855 Commits 2 Branches 109 Tags 198 MiB
Commit Graph

291 Commits

Author SHA1 Message Date
Josh Soref e4a87f772f docs: spelling: e.g. 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref bbe7eb35f1 docs: spelling: server-side 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Nick Craig-Wood bed83b0b64 test: add ListRetries config parameter to integration tests 
Occasionally the b2 tests fail because the integration tests don't
retry hard enough with their new setting of -list-retries 3. Override
this setting to 5 for the b2 tests only.
 2020-10-25 18:10:50 +00:00
Nick Craig-Wood 85d35ef03c test: remove TestS3Ceph: and TestSwiftCeph: from integration tests 
Unfortunately we don't have access to this server any more
 2020-10-25 18:10:49 +00:00
Josh Soref d0888edc0a Spelling fixes 
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-14 15:21:31 +01:00
Ivan Andreev c797494d88
Merge pull request #4608 from ivandeex/pr-chunker-crypt 
chunker: fix upload over crypt (fixes #4570)
 2020-09-18 17:58:44 +03:00
Ivan Andreev e2a57182be mailru: re-enable fixed chunker tests 
This reverts commit 9d3d397f50.
 2020-09-18 17:56:34 +03:00
Nick Craig-Wood 5bf53fe3ac test_all: only run the backend tests for 1fichier 
Running all the tests for 1fichier takes too long due to the directory
reading rate limiter.

The backend tests do complete in a reasonable time (21 mins).
 2020-09-01 16:13:47 +01:00
Nick Craig-Wood e2816629d0 fstest: fix upwrapping tests for bucket based remotes 
TestIntegration/FsRmdirNotFound was failing on crypt wrapping a bucket based remote.

This was spotted by the integration tests.
 2020-09-01 15:43:41 +01:00
Nick Craig-Wood 9d3d397f50 test_all: disable chunker + mailru tests while mailru is broken #4376 2020-08-20 12:50:20 +01:00
Nick Craig-Wood 38e8415e77 test_all: remove Digital Ocean s3 integration tests due to excessive rate limiting 
This is what I wrote to Digital Ocean support on July 10, 2020 - alas
it didn't result in the rate limits dropping, so reluctantly I'm going
to remove DO from the integration tests since they never pass and have
no hope of ever passing while this rate limit is in effect.

----

Somewhere towards the end of June 2020 or the start of July 2020 my
integration tests between rclone ( https://rclone.org ) and Digital
Ocean started failing.

I tried moving the tests to different regions (currently they are
using AMS1 because I'm in Europe) with no improvement.

Rclone seems to be hitting this rate limit as documented here:
https://www.digitalocean.com/docs/spaces/#limits

- 2 COPYs per 5 minutes on any individual object in a Space

Rclone creates small objects about 100 bytes in size and renames them
a few times - this involves using the COPY call as S3 does not have a
rename API. The tests do this more than twice per object so hit the 5
minute timeout I think. Rclone does exponential backoff and fails
after 10 retries not having reached 5 minutes delay after 10 retries.

Having a 5 minute lockout on an S3 compatible API is surprising!

Rclone integration tests with about 30 other providers, none of which
have a rate limit like this.

I understand the need for a COPY rate limit as server side copying
large files can be resource intensive. However a 5 minute lockout for
copying 100 byte files seems excessive!

Might I humbly suggest that you reduce or eliminate this rate limit
for small files?

----

This was the reply

Unfortunately it is not possible to raise this limit or remove it
currently on our platform. I do see how this would interfere with type
of applications that need to copy many small files and will be happy
to take the feedback to our engineering team to see how we can improve
the spaces system in the future
 2020-08-20 12:50:04 +01:00
Nick Craig-Wood fb9edbe34e test_all: export more internal variables to index.json for analysis 2020-08-20 12:23:33 +01:00
Nick Craig-Wood 63e4d2952b fstests: Suggest that Purge on a nonexistent dir should return fs.ErrorDirNotFound 2020-08-19 18:03:42 +01:00
Nick Craig-Wood a2afa9aadd fs: Add directory to optional Purge interface - fixes #1891 
- add a directory to the optional Purge interface
- fix up all the backends
- add an additional integration test to test for the feature
- use the new feature in operations.Purge

Many of the backends had been prepared in advance for this so the
change was trivial for them.
 2020-07-31 17:43:17 +01:00
Nick Craig-Wood 25662b9e05 fstest: add ability for mock objects and filesystems to have hashes 2020-06-30 12:01:36 +01:00
Nick Craig-Wood d8eea0e397 build: run gofmt -s to simplify the code: suggested by Go Report Card 2020-06-18 18:45:39 +01:00
Nick Craig-Wood 4b4ee72796 fstest: fix PublicLink tests to send non zero expiry and work with s3 2020-06-18 17:50:12 +01:00
Nick Craig-Wood ba7fbfa8a7 testy: test utility functions 2020-06-15 21:34:37 +01:00
Roman Kredentser 55ad1354b6 link: Add --expire and --unlink flags 
This adds expire and unlink fields to the PublicLink interface.

This fixes up the affected backends and removes unlink parameters
where they are present.
 2020-06-05 14:51:05 +01:00
Nick Craig-Wood baaec5b126 testserver: don't allow starting of an empty remote 2020-05-25 15:26:06 +01:00
Martin Michlmayr 4aee962233 doc: fix typos throughout docs and code 2020-05-20 15:54:51 +01:00
Caleb Case 0ce662faad Tardigrade Backend 2020-05-12 15:56:50 +00:00
Nick Craig-Wood bd4b91bd57 test_all: revert running fichier tests one at a time 
This didn't achieve the objective of getting the tests to run clean
and makes them take 16 hours!

This reverts commit 97f6f8fe19.
 2020-05-11 08:51:33 +01:00
Nick Craig-Wood 97f6f8fe19 test_all: run fichier tests one at a time 
This is in attempt to get the tests to run cleanly without hitting the
rate limiter.
 2020-05-07 20:39:54 +01:00
Nick Craig-Wood 94920d39ae test_all: increase the test timeout to 60m from 30m 
Some tests are failing at 30m - 60m doesn't seem unreasonable
 2020-05-07 20:38:40 +01:00
Nick Craig-Wood 9403bd2990 test_all: allow -list-retries to be overridden on the command line 2020-05-07 20:36:45 +01:00
Nick Craig-Wood e098924e61 fstest: change -list-retries default to 3 from 6 
The integration tests have got much better about retrying the tests,
and we aren't testing ACD any more so we don't need it this high.
 2020-05-07 20:28:41 +01:00
Fred c754e89906 seafile: New backend for seafile server 2020-05-06 17:33:22 +00:00
Nick Craig-Wood 1aa1a2c174 backend: add new backend command for backend specific commands 
These commands are for implementing backend specific
functionality. They have documentation which is placed automatically
into the backend doc.

There is a simple test for the feature in the backend tests.
 2020-04-29 10:10:57 +01:00
Nick Craig-Wood bbb6f94377 fstest: create AssertTimeEqualWithPrecision from CheckTimeEqualWithPrecision 2020-04-22 11:33:00 +01:00
Nick Craig-Wood cd3c699f28 lib/readers: factor ErrorReader from multiple sources 2020-04-19 15:18:49 +01:00
Nick Craig-Wood 9a5178be7a test_all: optionally run Cleanup before cleaning a remote 
Add this to the QingStor remote to stop it running out of buckets
 2020-03-31 15:56:58 +01:00
Nick Craig-Wood 3855c003ce build: update to use go1.14 for the build 2020-02-26 21:26:47 +00:00
Nick Craig-Wood bd9fd629bc test: add TestS3MinioEdge to test leading edge minio too #3934 2020-02-13 11:01:06 +00:00
Nick Craig-Wood 3b19f48929 testserver: add provider to TestS3Minio #3934 
This is necessary to pass the TestIntegration/FsMkdir/FsEncoding
listing tests.
 2020-02-13 11:01:06 +00:00
Nick Craig-Wood 24ef00a258 build: implement a framework for starting test servers during tests 
Test servers are implemented by docker containers and run real servers
for rclone to test against.
 2020-01-18 16:47:37 +00:00
Nick Craig-Wood c789436580 The memory backend 
This is a bucket based remote
 2020-01-18 10:41:08 +00:00
Nick Craig-Wood bedeaf23af sugarsync: new backend - fixes #622 2020-01-17 17:39:34 +00:00
Nick Craig-Wood b81601baff test_all: ignore TestIntegration/FsMkdir/FsPutFiles/FsPutStream/0 on wasabi 
This has been reported to Wasabi and they've confirmed as a known
issue that multipart uploads can't be 0 sized even though that is
incompatible with AWS S3.
 2020-01-13 09:47:11 +00:00
Nick Craig-Wood ad2bb86d8c fstests: add test for 0 sized streaming upload 2020-01-05 12:32:55 +00:00
Ivan Andreev 77b42aa33a chunker: fix integration tests and hashsum issues 2019-10-13 10:43:46 +01:00
Ivan Andreev 9049bb62ca chunker: prevent chunk corruption, survive meta-like input 2019-10-13 10:43:46 +01:00
Nick Craig-Wood 6abaa9e22c fstests: allow skipping of the broken UTF-8 test for the cache backend 2019-10-10 10:36:18 +01:00
Nick Craig-Wood e2914c0097 test_all: ignore some encoding tests with nextcloud integration test 2019-10-02 11:34:08 +01:00
Nick Craig-Wood 23bf6bb4d8 test_all: mark expected failures for minio, wasabi and FTP 2019-10-01 15:40:32 +01:00
Nick Craig-Wood 5cef5f8b49 lib/encoder: add LeftPeriod encoding 2019-09-30 22:00:25 +01:00
Nick Craig-Wood 7e738c9d71 fstest: remove WinPath as it is no longer needed 2019-09-30 22:00:24 +01:00
Nick Craig-Wood ced2616da5 fstests: allow Purge to fail with ErrorDirNotFound 2019-09-30 22:00:24 +01:00
Nick Craig-Wood 57c1284df7 fstests: make integration tests to check all backends can store any file name 
This tests the encoder is working properly
 2019-09-30 22:00:24 +01:00
Fabian Möller 97a218903c fstest: remove WinPath from fstest.Item 2019-09-30 14:05:48 +01:00