community
/
rclone
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,855 Commits 2 Branches 109 Tags 198 MiB
Commit Graph

79 Commits

Author SHA1 Message Date
Josh Soref e4a87f772f docs: spelling: e.g. 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref bbe7eb35f1 docs: spelling: server-side 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref d0888edc0a Spelling fixes 
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-14 15:21:31 +01:00
Nick Craig-Wood 3a14b1d5a9 build: make rclone build with wasm 
Needed to drop
- azureblob backend
- cache backend
- qingstor backend
- cachestats command
- ncdu command
 2020-08-10 17:32:21 +01:00
Nick Craig-Wood a2afa9aadd fs: Add directory to optional Purge interface - fixes #1891 
- add a directory to the optional Purge interface
- fix up all the backends
- add an additional integration test to test for the feature
- use the new feature in operations.Purge

Many of the backends had been prepared in advance for this so the
change was trivial for them.
 2020-07-31 17:43:17 +01:00
Nick Craig-Wood 2a3b377d34 azureblob: don't compile on < go1.13 after dependency update 2020-06-29 14:45:39 +01:00
Martin Michlmayr 041b201abd doc: fix typos throughout docs and code 2020-05-25 11:23:58 +01:00
Martin Michlmayr 4aee962233 doc: fix typos throughout docs and code 2020-05-20 15:54:51 +01:00
Nick Craig-Wood 4a1b644bfb azureblob: implement streaming of unknown sized files 
See: https://forum.rclone.org/t/rclone-rcat-azure-blob-container-sas-token-403-error/16286/3
 2020-05-14 11:56:15 +01:00
Nick Craig-Wood d342f9f942 azureblob: fix permission error on SAS URL limited to container 
Before this change, for some operations, eg rcat or copyto (of a file)
rclone would attempt to create the container when using a SAS URL
limited to a container.

After this change we assume the container does not need creating when
using a container SAS URL.

See: https://forum.rclone.org/t/rclone-rcat-azure-blob-container-sas-token-403-error/16286
 2020-05-13 09:11:51 +01:00
Nick Craig-Wood 37a53570d4 azureblob: implement memory pooling to control memory use 
This commit implements memory pooling to control excessive memory use
as was implemented in the s3 backend.
 2020-04-28 17:47:10 +01:00
Nick Craig-Wood ee7219aa20 azureblob: add --azureblob-disable-checksum flag 2020-04-28 17:47:10 +01:00
Nick Craig-Wood b1d8da484b azureblob: retry InvalidBlobOrBlock error as it may indicate block concurrency problems 
According to Microsoft support this error can be caused by

> A timing/concurrency issue where the PUT operations are happening
> about the same time for a single blob. The Put Block List operation
> writes a blob by specifying the list of block IDs that make up the
> blob. In order to be written as part of a blob, a block must have
> been successfully written to the server in a prior Put Block
> operation.
>
> Documentation reference:
>
> https://docs.microsoft.com/en-us/rest/api/storageservices/put-block
>
> This error can happen when doing concurrent upload commits after you
> have started the upload but before you commit. In that case, the
> upload fails. The application can retry this error or attempt some
> other recovery action based on the required scenario.

See: https://forum.rclone.org/t/error-while-syncing-with-azure-blob-storage-x-ms-error-code-invalidbloborblock/15561
 2020-04-28 17:47:10 +01:00
Lars Lehtonen 481c8a40ea backend/azureblob: fmt nit 2020-02-20 15:50:53 +01:00
Lars Lehtonen 25ef3a281b backend/azureblob: remove unused Object.parseTimeString() 2020-02-20 15:50:53 +01:00
Nick Craig-Wood bafe7d5a73 backends: move encoding definitions from fs/encodings 2020-01-16 14:40:36 +00:00
Nick Craig-Wood 3c620d521d backend: adjust backends to have encoding parameter 
Fixes #3761
Fixes #3836
Fixes #3841
 2020-01-16 14:40:36 +00:00
Nick Craig-Wood 2d9b8cb981 azureblob: disable logging to the Windows event log 
See: https://forum.rclone.org/t/event-log-warning/12430
 2019-10-21 11:50:31 +01:00
Nick Craig-Wood 35d6ff89bf azureblob: use lib/encoder 2019-09-30 22:00:24 +01:00
Nick Craig-Wood 23dc313fa5 azureblob: add missing type assertions for GetTier/SetTier 2019-09-14 09:18:55 +01:00
Nick Craig-Wood ffa1dac10b build: apply gofmt from go1.13 to change case of number literals 2019-09-05 13:59:06 +01:00
Nick Craig-Wood 6f16588123 s3,b2,googlecloudstorage,swift,qingstor,azureblob: fixes after code review #3421 
- change the interface of listBuckets() removing dir parameter and adding context
- add makeBucket() and use in place of Mkdir("")
    - this fixes some corner cases in Copy/Update
- mark all the listed buckets OK in ListR

Thanks to @yparitcher for the review.
 2019-08-22 23:06:59 +01:00
Nick Craig-Wood 8a0775ce3c azureblob: make all operations work from the root #3421 2019-08-17 10:30:41 +01:00
Nick Craig-Wood e502be475a azureblob/b2/dropbox/gcs/koofr/qingstor/s3: fix 0 length files 
In 0386d22cc9 we introduced a test for 0 length files read the
way mount does.

This test failed on these backends which we fix up here.
 2019-08-06 15:18:08 +01:00
Nick Craig-Wood 57d5de6fba build: fix up package paths after repo move 
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
 2019-07-28 18:47:38 +01:00
Sandeep 6907242cae
azureblob: Updated config help details to remove connection string references (#3306) 2019-06-27 18:53:33 -07:00
Sandeep fc44eb4093
Azure Storage Emulator support (#3285) 
* azureblob - Add support for Azure Storage Emulator to test things locally.

Testing - Verified changes by testing manually.

* docs: update azureblob docs to reflect support of storage emulator
 2019-06-26 20:46:22 -07:00
Aleksandar Jankovic f78cd1e043 Add context propagation to rclone 
- Change rclone/fs interfaces to accept context.Context
- Update interface implementations to use context.Context
- Change top level usage to propagate context to lover level functions

Context propagation is needed for stopping transfers and passing other
request-scoped values.
 2019-06-19 11:59:46 +01:00
Nick Craig-Wood 16d8014cbb build: drop support for go1.8 2019-04-15 21:49:58 +01:00
Dr.Rx 0ea3a57ecb azureblob: Enable MD5 checksums when uploading files bigger than the "Cutoff" 
This enables MD5 checksum calculation and publication when uploading file above the "Cutoff" limit.
It was explictely ignored in case of multi-block (a.k.a. multipart) uploads to Azure Blob Storage.
 2019-03-01 11:12:23 +01:00
Fabian Möller 61616ba864 pacer: make pacer more flexible 
Make the pacer package more flexible by extracting the pace calculation
functions into a separate interface. This also allows to move features
that require the fs package like logging and custom errors into the fs
package.

Also add a RetryAfterError sentinel error that can be used to signal a
desired retry time to the Calculator.
 2019-02-16 14:38:07 +00:00
Nick Craig-Wood ffe067d6e7 azureblob: fix SAS URL support - fixes #2969 
This was broken accidentally in 5d1d93e163 as part of #2654
 2019-02-13 17:36:14 +00:00
Fabian Möller a0d4c04687
backend: fix misspellings 2019-02-07 19:51:03 +01:00
Nick Craig-Wood d966cef14c build: fix problems found with unconvert 2019-02-02 11:45:07 +00:00
Nick Craig-Wood 97752ca8fb build: fix problems found with ineffasign linter 2019-02-02 11:45:07 +00:00
Nick Craig-Wood 6b3a9bf26a build: fix problems found by the deadcode linter 2019-02-02 11:45:07 +00:00
Nick Craig-Wood 2fc095cd3e azureblob: Stop Mkdir attempting to create existing containers 
Before this change azureblob would attempt to create already existing
containers.  This causes problems with limited permissions keys.

This change checks the container exists before trying to create it in
the same way the s3 backend does.  This uses no more requests in the
usual case of the container existing.

See: https://forum.rclone.org/t/copying-individual-files-to-azure-blob-storage/8397
 2019-01-23 09:58:46 +00:00
Nick Craig-Wood 733b072d4f azureblob: ignore directory markers in inital Fs creation too - fixes #2806 
This is a follow-up to feea0532 including the initial Fs creation
where the backend detects whether the path is pointing to a file or a
directory.
 2019-01-08 11:21:20 +00:00
Nick Craig-Wood feea0532cd azureblob: ignore directory markers - fixes #2806 
This ignores 0 length blobs if
- they end with /
- they have the metadata hdi_isfolder = true
 2018-12-06 21:47:03 +00:00
Nick Craig-Wood 941ad6bc62 azureblob: use the s3 pacer for 0 delay - fixes #2799 2018-12-02 20:55:16 +00:00
Nick Craig-Wood 5d1d93e163 azureblob: use the rclone HTTP client - fixes #2654 
This enables --dump headers and --timeout to work properly.
 2018-12-02 20:55:16 +00:00
themylogin eb41253764 azureblob: allow building azureblob backend on *BSD 
FreeBSD support was added in Azure/azure-storage-blob-go@0562badec5
OpenBSD and NetBSD support was added in Azure/azure-storage-blob-go@1d6dd77d74
 2018-11-27 12:20:48 +00:00
Nick Craig-Wood 8b8220c4f7 azureblob: wait for up to 60s to create a just deleted container 
When a container is deleted, a container with the same name cannot be
created for at least 30 seconds; the container may not be available
for more than 30 seconds if the service is still processing the
request.

We sleep so that we wait at most 60 seconds.  This is mostly useful in
the integration tests where containers get deleted and remade
immediately.
 2018-11-24 10:57:37 +00:00
Nick Craig-Wood f7ce2e8d95 azureblob: fix erroneous Rmdir error "directory not empty" 
Before this change Rmdir would check the root rather than the
directory specified for being empty and return "directory not empty"
when it shouldn't have done.
 2018-11-14 17:13:39 +00:00
brused27 d87aa33ec5 azureblob: Avoid context deadline exceeded error by setting a large TryTimeout value - Fixes #2647 2018-11-13 16:59:53 +00:00
Nick Craig-Wood c514cb752d vendor: update to latest versions of everything 2018-10-25 17:06:33 +01:00
Nick Craig-Wood 7f0b204292 azureblob: work around SDK bug which causes errors for chunk-sized files (again) 
Until https://github.com/Azure/azure-storage-blob-go/pull/75 is merged
the SDK can't upload a single blob of exactly the chunk size, so
upload files of this size with a multpart upload as a work around.

The previous fix for this 6a773289e7 turned out to cause problems
uploading files with maximum chunk size so needed to be redone.

Fixes #2653
 2018-10-15 09:05:34 +01:00
Nick Craig-Wood e1503add41 azureblob, b2, drive: implement set upload cutoff for chunked upload tests 2018-10-13 22:49:12 +01:00
Nick Craig-Wood 6a773289e7 azureblob: work around SDK bug which causes errors for chunk-sized files 
See https://github.com/Azure/azure-storage-blob-go/pull/75 for details
 2018-10-13 22:49:12 +01:00
Fabian Möller 98e2746e31 backend: add fstests.ChunkedUploadConfig 
- azureblob
- b2
- drive
- dropbox
- onedrive
- s3
- swift
 2018-10-11 14:47:58 +01:00