test_all: remove Digital Ocean s3 integration tests due to excessive rate limiting

This is what I wrote to Digital Ocean support on July 10, 2020 - alas
it didn't result in the rate limits dropping, so reluctantly I'm going
to remove DO from the integration tests since they never pass and have
no hope of ever passing while this rate limit is in effect.

----

Somewhere towards the end of June 2020 or the start of July 2020 my
integration tests between rclone ( https://rclone.org ) and Digital
Ocean started failing.

I tried moving the tests to different regions (currently they are
using AMS1 because I'm in Europe) with no improvement.

Rclone seems to be hitting this rate limit as documented here:
https://www.digitalocean.com/docs/spaces/#limits

- 2 COPYs per 5 minutes on any individual object in a Space

Rclone creates small objects about 100 bytes in size and renames them
a few times - this involves using the COPY call as S3 does not have a
rename API. The tests do this more than twice per object so hit the 5
minute timeout I think. Rclone does exponential backoff and fails
after 10 retries not having reached 5 minutes delay after 10 retries.

Having a 5 minute lockout on an S3 compatible API is surprising!

Rclone integration tests with about 30 other providers, none of which
have a rate limit like this.

I understand the need for a COPY rate limit as server side copying
large files can be resource intensive. However a 5 minute lockout for
copying 100 byte files seems excessive!

Might I humbly suggest that you reduce or eliminate this rate limit
for small files?

----

This was the reply

Unfortunately it is not possible to raise this limit or remove it
currently on our platform. I do see how this would interfere with type
of applications that need to copy many small files and will be happy
to take the feedback to our engineering team to see how we can improve
the spaces system in the future
This commit is contained in:
Nick Craig-Wood 2020-08-20 12:43:43 +01:00
parent fb9edbe34e
commit 38e8415e77
1 changed files with 9 additions and 6 deletions

View File

@ -133,12 +133,15 @@ backends:
- TestIntegration/FsMkdir/FsEncoding/leading_HT
- TestIntegration/FsMkdir/FsEncoding/leading_VT
- TestIntegration/FsMkdir/FsPutFiles/FsPutStream/0
- backend: "s3"
remote: "TestS3DigitalOcean:"
fastlist: true
ignore:
- TestIntegration/FsMkdir/FsPutFiles/FsCopy
- TestIntegration/FsMkdir/FsPutFiles/SetTier
# Disabled due to excessive rate limiting at DO which cause the tests never to pass
# This hits the rate limit as documented here: https://www.digitalocean.com/docs/spaces/#limits
# 2 COPYs per 5 minutes on any individual object in a Space
# - backend: "s3"
# remote: "TestS3DigitalOcean:"
# fastlist: true
# ignore:
# - TestIntegration/FsMkdir/FsPutFiles/FsCopy
# - TestIntegration/FsMkdir/FsPutFiles/SetTier
- backend: "s3"
remote: "TestS3Ceph:"
fastlist: true