Commit Graph

144 Commits

Author SHA1 Message Date
Yury Stankevich 71edc75ca6 HDFS (Hadoop Distributed File System) implementation - #42
This includes an HDFS docker image to use with the integration tests.

Co-authored-by: Ivan Andreev <ivandeex@gmail.com>
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
2021-01-07 09:48:51 +00:00
buengese 66c3f2f31f new backend: zoho workdrive - fixes #4533 2020-12-30 17:56:08 +00:00
Nick Craig-Wood 3b24a4cada yandex: set Features.WriteMimeType=false as Yandex ignores mime types
Yandex appears to ignore mime types set as part of the PUT request or
as part of a PATCH request.

The docs make no mention of being able to set a mime type, so set
WriteMimeType=false indicating the backend can't set mime types on
uploaded files.
2020-11-29 17:22:43 +00:00
buengese 52688a63c6 jottacloud: don't erroniously report support for writing mime types - fixes #4817 2020-11-29 18:11:43 +01:00
edwardxml 19a8b66cee
docs: update rclone about docs
Create a full loop of documentation for rclone about, backends overview
and individual backend pages.

Discussion:
https://github.com/rclone/rclone/pull/4774 relates

Previously pull was requested, in part, under ref
https://github.com/rclone/rclone/pull/4801

Notes:
Introduce a tentative draft see-link format the end of sections to try
rather than lots of in-para links.

Update about.go incl link to list of backends not supporting about.

In list of backends not supporting about, include link to about command
reference.

I appreciate there may be decisions to make going forward about whether
command links should be code formatted, and using proper pretty url
links, but I have fudged that for now.

Update backend pages that do not support about with wording used
previously for ftp - it is in passive voice but I can live with it. (my
own wording and fault). The note is applied to a limitations section. If
one does not already exist it is created (even if there are other
limitations with their own sections)
2020-11-27 14:08:52 +00:00
Nick Craig-Wood 979bb07c86 filefabric: Implement the Enterprise File Fabric backend
Missing features
- M-Stream support
- Oauth-like flow (soon being changed to oauth)
2020-11-25 21:11:29 +00:00
Nick Craig-Wood f43a9ac17e pcloud: only use SHA1 hashes in EU region
Apparently only SHA1 hashes are supported in the EU region for
pcloud. This has been confirmed by pCloud support. The EU regions also
support SHA256 hashes which we don't support yet.

https://forum.rclone.org/t/pcloud-to-local-no-hashes-in-common/19440
2020-11-25 20:46:38 +00:00
edwardxml fb001b6c01
docs: improve "rclone about" docs and relate to union mfs limitation in ftp remote
See: https://forum.rclone.org/t/rclone-union-mfs-most-free-space-not-working-for-ftp/20346
2020-11-23 16:34:27 +00:00
Josh Soref e4a87f772f docs: spelling: e.g.
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-28 18:16:23 +00:00
Josh Soref bbe7eb35f1 docs: spelling: server-side
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-28 18:16:23 +00:00
Josh Soref d0888edc0a Spelling fixes
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-14 15:21:31 +01:00
Evan Harris b03fcbcc12 docs: Updated overview with Opendrive support of dupes 2020-09-02 11:45:11 +01:00
Nick Craig-Wood 8eb16ce89c onedrive: implement rclone cleanup #4106 2020-08-07 15:58:30 +01:00
Nick Craig-Wood d5f4c74697 s3: implement cleanup and backend command to list & remove multipart uploads
This implements `rclone cleanup` to remove multipart uploads over 24
hours old. It also implements the backend command
`list-multipart-uploads` to see which ones are available and `cleanup`
to delete them with a configurable expiry interval.

See #4302
2020-07-28 11:37:46 +01:00
buengese 6768f999ed docs/overview: pcloud now supports link sharing 2020-06-21 17:22:56 +02:00
buengese 6606602f1e docs/box: add some info regarding the CleanUp implementation 2020-06-18 23:39:59 +02:00
Nick Craig-Wood e62c032184 docs: remove manually set dates and use git dates instead 2020-06-01 13:07:46 +01:00
Nick Craig-Wood 74b8cbfb84 docs: set unsafe HTML parsing to false and fix raw HTML insertion
This means that markdown files can't contain <thing> any more.
2020-05-27 17:31:09 +01:00
Martin Michlmayr a36ef8582f doc: use consistent capitalization 2020-05-20 15:54:51 +01:00
Nick Craig-Wood 4a1b644bfb azureblob: implement streaming of unknown sized files
See: https://forum.rclone.org/t/rclone-rcat-azure-blob-container-sas-token-403-error/16286/3
2020-05-14 11:56:15 +01:00
Caleb Case 0ce662faad Tardigrade Backend 2020-05-12 15:56:50 +00:00
albertony 962fbc8257 jottacloud: update docs regarding cleanup, removed remains from old auth, and added warning about special mountpoints. 2020-05-11 11:41:17 +02:00
Fred c754e89906 seafile: New backend for seafile server 2020-05-06 17:33:22 +00:00
Nick Craig-Wood 472d4799d1 qingstor: make `rclone cleanup` remove pending multipart uploads older than 24h 2020-03-18 12:49:21 +00:00
Nick Craig-Wood c789436580 The memory backend
This is a bucket based remote
2020-01-18 10:41:08 +00:00
Nick Craig-Wood bedeaf23af sugarsync: new backend - fixes #622 2020-01-17 17:39:34 +00:00
Nick Craig-Wood 0e57335396 docs: document new backend encoder parameter 2020-01-16 14:40:36 +00:00
Fabian Möller c8d3e57418 encodings: add . and .. to all backends, except Drive 2019-09-30 14:05:49 +01:00
Fabian Möller a56c9ab61d docs: add section for restricted filenames 2019-09-30 14:05:48 +01:00
Nick Craig-Wood 4627ac5709 New backend for Citrix Sharefile - Fixes #1543
Many thanks to Bob Droog for organizing a test account and extensive
testing.
2019-09-30 12:28:33 +01:00
Ivan Andreev ba1daea072 mailru: backend for mail.ru 2019-09-09 21:56:16 +01:00
Cenk Alti a1ca65bd80 putio: add new backend 2019-08-23 12:11:36 +01:00
yparitcher ba0e1ea6ae Docs: add emptydir support to table 2019-08-17 09:45:20 +01:00
Nick Craig-Wood a1263e70cf premiumizeme: new backend for premiumize.me - Fixes #3063 2019-08-10 19:17:51 +01:00
Nick Craig-Wood 57d5de6fba build: fix up package paths after repo move
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
2019-07-28 18:47:38 +01:00
yparitcher 3f5767b94e b2: Implement link sharing #2178 2019-07-03 14:10:25 +01:00
Nick Craig-Wood a1cfe61ffd googlephotos: Backend for accessing Google Photos #369 2019-07-02 15:26:55 +01:00
Laura dde4dd0198 fichier: 1fichier support - fixes #2908
This was started by Fionera, finished off by Laura with fixes and more
docs from Nick.

Co-authored-by: Fionera <fionera@fionera.de>
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
2019-06-30 18:35:01 +01:00
Nick Craig-Wood a5bed67016 b2: implement server side copy - fixes #3210 2019-06-13 17:31:33 +01:00
Gary Kim d0ff07bdb0 mega: add cleanup support
Fixes #3138
2019-05-01 16:32:34 +01:00
Gary Kim 49d7b0d278 sftp: add About support - fixes #3107
This adds support for using About with SFTP remotes. This works by calling the df command remotely.
2019-04-25 10:51:15 +01:00
jaKa 118a8b949e koofr: implemented a backend for Koofr cloud storage service.
Implemented a Koofr REST API backend.
Added said backend to tests.
Added documentation for said backend.
2019-03-06 13:41:43 +00:00
Nick Craig-Wood c294068780 webdav: support About - fixes #2937
This means that `rclone about` will work with Webdav backends and `df`
will be correct when using them in `rclone mount`.
2019-01-28 13:45:09 +00:00
Nick Craig-Wood 8a774a3dd4 webdav: support MD5 and SHA1 hashes with Owncloud and Nextcloud - fixes #2379 2019-01-28 13:45:09 +00:00
Sebastian Bünger 703b0535a4 yandex: update docs 2018-11-22 20:14:50 +00:00
jackyzy823 411a6cc472 onedrive: add link sharing support #2178 2018-10-09 20:11:48 +08:00
albertony cfe65f1e72 jottacloud: minor update in docs 2018-09-18 10:25:30 +01:00
albertony d51501938a jottacloud: add link sharing support 2018-09-08 09:38:57 +01:00
albertony 3fccce625c jottacloud: add --fast-list support - fixes #2532 2018-09-07 12:49:39 +01:00
Sebastian Bünger c6f1c3c7f6 box: Implement link sharing. #2178 2018-09-04 22:01:22 +01:00
Sebastian Bünger 007c7757d4 Add docs for Jottacloud 2018-08-10 11:14:34 +01:00
Cnly 0f845e3a59 onedrive: implement DirMove - fixes #197 2018-08-07 09:33:19 +01:00
Nick Craig-Wood f7c0b2407d drive: add docs for --fast-list and add to integration tests 2018-08-06 21:38:50 +01:00
Nick Craig-Wood cdde8fa75a opendrive: finish off #1026
* Fix errcheck and golint warnings
  * Remove unused constants and fix comments
  * Parse error responses properly
  * Fix Open with RangeOption
  * Fix Move, Copy and DirMove
  * Implement DirCacheFlush
  * Check interfaces are correct
  * Remove debugs and update overview
  * Correct feature flags
  * Pare replacement characters down to the minimum set
  * Add to the integration tests
2018-05-12 10:10:46 +01:00
Jakub Karlicek 53292527bb opendrive: fill out the functionality #1026
* Add Mkdir, Rmdir, Purge, Delete, SetModTime, Copy, Move, DirMove
 * Update file size after upload
 * Add Open seek
 * Set private permission for new folder and uploaded file
 * Add docs
 * Update List function
 * Fix UserSessionInfo struct
 * Fix socket leaks
 * Don’t close resp.Body in Open method
 * Get hash when listing files
2018-05-12 10:07:25 +01:00
Nick Craig-Wood 1a3fb21a77 onedrive: add QuickXorHash support for OneDrive for business - fixes #2262 2018-04-20 21:03:03 +01:00
Nick Craig-Wood 1f255a8567 Add a mega.nz remote #163
Not supported yet:
  * Hash
  * ModTime
  * Server Side Copy

Otherwise fully functional and passing all the tests.
2018-04-18 21:09:54 +01:00
Nick Craig-Wood 1ac6dacf0f about: complete other providers and re-work internals
* Implement about for:
    * local, crypt, cache, drive, swift, hubic, onedrive, pcloud, dropbox
  * Implement `--json` and `---full` flag for `rclone about`
  * change About interface to return a Usage structure
  * Remove operations.About as it is too thin an interface
  * Implement Integration test

Relates to #1138 and #1564
2018-04-17 21:53:27 +01:00
a-roussos 94e277d759 about: add new command 'about' to get quota info from a remote
Implemented for drive only.

Relates to #1138 and #1564.
2018-04-17 21:50:14 +01:00
Stefan a8267d1628
link: allow creating public link to files and folders - closes #1562 2018-03-29 09:10:19 +02:00
Corban Raun a8dfc5ce3b Fix spelling in some documentation 2017-11-05 21:38:59 +00:00
Nick Craig-Wood 7f3dc9b5c4 Implement WebDAV remote #580
This has special knowledge of Owncloud and Nextcloud to enable more
functionality such as mod times.
2017-10-09 16:19:37 +01:00
Nick Craig-Wood 9e9297838f Implement pcloud remote - #418 2017-10-01 11:37:35 +01:00
Nick Craig-Wood 3fc69f4140 docs: fix daggers 2017-09-30 09:19:53 +01:00
Stefan Breunig 646a419453 docs: update overview table to reflect streaming upload ability 2017-09-24 21:59:31 +02:00
ishuah91 49816e67bd yandex: implement cleanup (empty trash) - addresses #575 2017-09-08 11:37:39 +01:00
ishuah91 dd975ab00d drive: implement cleanup (empty trash) - addresses #575 2017-09-06 16:12:00 +01:00
wuyu ee9f987234 qingstor: Support hash md5 for upload object
* Using single object to uploaded when files less than or equal to 67108864 bytes

 * Using multi-part object to uploaded when files large than 67108864 bytes, and
   calculate MD5SUMS in the upload process

 * For Mkdir and Rmdir, Add block to wait qingstor service sync status to
   handling extreme cases that try to create a just deleted bucket or delete
   a just created bucket etc
2017-08-31 16:41:08 +01:00
Stefan Breunig b78ecb1568 docs: add optional feature "streaming uploads" to overview table 2017-08-19 14:35:17 +02:00
Christian Brüggemann 80ce569874 sftp: Add support for md5 and sha1 hashes where available 2017-08-06 11:49:52 +01:00
Nick Craig-Wood 92d2e1f8d7 azureblob: rework and complete #801
* Fixup bitrot (rclone and Azure library)
  * Implement Copy
  * Add modtime to metadata under mtime key as RFC3339Nano
  * Make multipart upload work
  * Make it pass the integration tests
  * Fix uploading of zero length blobs
  * Rename to azureblob as it seems likely we will do azurefile
  * Add docs
2017-08-04 22:56:16 +01:00
wuyu ec5b72f8d5 Add new QingStor remote
Add new package qingstor to support QingStor API.

Add new unit test for its and tested through; But I commented
on some tests case because of some of the features of QingStor.

Add new docs for it.
2017-08-04 17:25:47 +01:00
Nick Craig-Wood 7b81f12dad box: add docs
* reorder remotes so they are in alphabetical order by full name everywhere
  * update CONTRIBUTING doc
2017-07-23 11:32:34 +01:00
Nick Craig-Wood b22c4c4307 http: fix, tidy and rework ready for release
* Fix remaining problems
  * Refactor to make testing easier and add a test suite
  * Make path parsing more robust.
  * Add single file operations
  * Add MimeType reading for objects
  * Add documentation
  * Note go1.7+ is required to build
2017-06-20 21:27:14 +01:00
Nick Craig-Wood 50928a5027 Implement --fast-list flag.
This is supported remotes which can do a recursive listing.  It will
use more memory.

This is related to #1277 but doesn't fix that issue yet.
2017-06-14 16:49:40 +01:00
Nick Craig-Wood 6381959850 dropbox: support Dropbox content hashing scheme - fixes #1302
* add support to hashing module
  * add dbhashsum to list the hashes
  * add support to dropbox module

This means objects up and downloaded to/from Dropbox will have their
hashes checked.

Note after this change local objects are calculating MD5, SHA1 and
DBHASH which is excessive and needs to be fixed.
2017-05-29 12:04:44 +01:00
Nick Craig-Wood 8e214e838e dropbox: Update dropbox to use the v2 API #349
This is feature complete with the old version but now supports modification times.
2017-05-29 12:04:33 +01:00
Nick Craig-Wood cdacf026e4 ftp: implement server side move and directory move 2017-05-18 20:49:36 +01:00
Antonio Messina c72fca2711 Add ftp backend - fixes #540 2017-05-18 20:49:36 +01:00
Ihor Dvoretskyi a54692d165 OneDrive vs One Drive
It's better to call this service as it's officially named.
2017-03-29 18:11:33 +01:00
Nick Craig-Wood 99f7fe736a onedrive: implement Move #197 2017-03-15 14:01:08 +00:00
Nick Craig-Wood 726cb43be9 Complete SFTP remote #521
* Add unit tests
  * Fix up remote so it passes tests
  * Add docs
2017-02-04 10:29:46 +00:00
Nick Craig-Wood 4e809c951d acd: Note that Move and DirMove are now supported - fixes #122 2016-12-14 17:45:20 +00:00
Nick Craig-Wood 945f49ab5e Make ContentType be preserved for cloud -> cloud copies - fixes #733 2016-10-03 20:02:04 +01:00
Nick Craig-Wood cf43ca2a7b Document which remotes support which optional features 2016-09-12 17:50:19 +01:00
Nick Craig-Wood 56adb52a21 Rename Amazon Cloud Drive to Amazon Drive - fixes #532 2016-07-11 12:42:44 +01:00
Nick Craig-Wood cb22583212 b2: Enable mod time syncing - fixes #348 2016-03-22 15:56:44 +00:00
klauspost 7ae40cb352 Update information on revised hash functionality. 2016-01-16 10:17:11 +00:00
Nick Craig-Wood 98778b1870 Docs for Yandex 2015-12-30 14:30:16 +00:00
Nick Craig-Wood 33b3eea6ec Implement Backblaze B2 - fixes #224 2015-12-30 10:05:07 +00:00
Nick Craig-Wood 0b338eaa28 Fix up sensitive vs insensitive in the docs and some formatting - fixes #214 2015-11-14 11:20:04 +00:00
Nick Craig-Wood fcea3777c0 Implement Hubic storage system - fixes #200 2015-11-14 08:08:52 +00:00
Nick Craig-Wood 2fcb8f5db7 Add support for Microsoft One Drive - fixes #10
* Still to do
    * Copy
    * Move
    * MoveDir
2015-11-07 10:19:10 +00:00
Nick Craig-Wood 8c3df224ef Implement Amazon Cloud Drive - fixes #45
* Optional interfaces Copier, Mover, DirMover not done
2015-09-08 21:02:48 +01:00