Commit Graph

2215 Commits

Author SHA1 Message Date
Nick Craig-Wood 21a10e58c9 rc: implement core/memstats to print internal memory usage info 2018-04-23 20:49:36 +01:00
Nick Craig-Wood d36b80f587 vendor: update bazil.org/fuse - corrects df -i - fixes #2089 2018-04-21 22:57:08 +01:00
Nick Craig-Wood 24980d7123 config: fix typo in error message #2268 2018-04-21 22:49:30 +01:00
Nick Craig-Wood 870c58f7f8 sftp: fail soft with a debug on hash failure #1474
If md5sum/sha1sum fails we debug what it outputed on stderr and return
an empty hash indicating we didn't have a hash, rather than
hash.ErrUnsupported indicating that we don't support this hash type.

This fixes lots of ERROR messages for sftp and synology NAS which,
while it supports md5sum the SFTP paths and the SSH paths are
different so md5sum doesn't work.

We also stop disabling md5sum/sha1sum on errors since typically Hashes
is only checked at the start of a sync run and isn't expected to
change dynamically.
2018-04-21 09:02:53 +01:00
Nick Craig-Wood b3c6f5f4b8 sftp: Update docs with Synology quirks 2018-04-21 09:02:53 +01:00
Nick Craig-Wood 311a962011 s3: Look in S3 named profile files for credentials - fixes #2243 2018-04-21 09:00:20 +01:00
Nick Craig-Wood da7a77ef2e ftp: Fix no error on listing non-existent directory 2018-04-20 23:22:46 +01:00
Nick Craig-Wood 9fbc40c5b9 fstests: List missing dir must return ErrorDirNotFound for non bucket based remotes
List or ListR of an non existent directory must return
ErrorDirNotFound for non bucket based remotes.  For bucket based
remotes it may return ErrorDirNotFound or it may return no error and
no entries.
2018-04-20 23:22:46 +01:00
Nick Craig-Wood 56ce784301 Add hensur to contributors 2018-04-20 21:44:12 +01:00
hensur 8fe3037301 webdav: support SharePoint cookie authentication
This enables the use of the SharePoint webdav endpoint provided by
OneDrive for Business or Office365 Education Accounts. It enables
unverified accounts to be accessed with rclone via webdav as it isn't
possible through the normal onedrive backend.

This integrates the https://github.com/hensur/onedrive-cookie-test
package to fetch the required cookies to authorize against the
SharePoint webdav endpoint.
2018-04-20 21:43:54 +01:00
hensur ba7ae2ee8c rest: Add RemoveHeader and SetCookie method
These methods extend the rest package to support the cookie header and
header deletion.

The deletion is necessary to delete an existing authorization header if
cookie auth should be used.
2018-04-20 21:43:54 +01:00
Nick Craig-Wood dc59836021 webdav: strip leading and trailing / off root - fixes #2257 2018-04-20 21:43:54 +01:00
Nick Craig-Wood 1a3fb21a77 onedrive: add QuickXorHash support for OneDrive for business - fixes #2262 2018-04-20 21:03:03 +01:00
Nick Craig-Wood bcdb7719c6 fs/hash: install QuickXorHash as a supported rclone hash type #2262 2018-04-20 21:02:57 +01:00
Nick Craig-Wood c51d97c752 hashsum: make generic tool for any hash to produce md5sum like output 2018-04-20 21:02:37 +01:00
Nick Craig-Wood 57a5b72d60 onedrive: implement quickXorHash algorithm #2262 2018-04-20 21:02:37 +01:00
Nick Craig-Wood 34ba17deec Add Chris Redekop second email to contributors 2018-04-20 20:53:15 +01:00
Nick Craig-Wood e3a1bc9cd3 Add Michael G. Noll to contributors 2018-04-20 20:51:31 +01:00
Chris Redekop a35e62e15c s3: Add an option to disable checksum uploading - fixes #2213 2018-04-20 20:51:12 +01:00
Michael G. Noll d1ca8b8959 sftp: update docs to match code, fix typos and clarify disable_hashcheck prompt 2018-04-20 20:49:49 +01:00
Nick Craig-Wood a0c65deca8 box: Parse file/directory size as a floating point number
Very large directories can have their sizes returned as floating point
numbers, eg `1.0034576985781e+14` from the box API.

Before this change this would fail to parse as an int64.

This change parses the size as a float64 instead which will be
perfectly accurate for sizes up to 2**56 which is about 9 PB.

It is unknown whether box themselves use a float64 as an intermediate
representation in the API or not - it seems likely.

Fixes #2261
2018-04-19 21:04:52 +01:00
Nick Craig-Wood 1f255a8567 Add a mega.nz remote #163
Not supported yet:
  * Hash
  * ModTime
  * Server Side Copy

Otherwise fully functional and passing all the tests.
2018-04-18 21:09:54 +01:00
Nick Craig-Wood f50b85278a vendor: github.com/t3rm1n4l for backend/mega 2018-04-18 21:09:54 +01:00
Nick Craig-Wood 9948b39dba about: don't attempt retries 2018-04-18 21:09:54 +01:00
Nick Craig-Wood 2b855751fc vfs,mount,cmount: use About to return the correct disk total/used/free
Disks total, used, free now shows correctly for mount and cmount (eg
`df` for Unix or in the Windows explorer).
2018-04-18 18:27:34 +01:00
Nick Craig-Wood ef3bcec76c fs: Extend SizeSuffix to include TB and PB for rclone about 2018-04-17 21:53:42 +01:00
Nick Craig-Wood 1ac6dacf0f about: complete other providers and re-work internals
* Implement about for:
    * local, crypt, cache, drive, swift, hubic, onedrive, pcloud, dropbox
  * Implement `--json` and `---full` flag for `rclone about`
  * change About interface to return a Usage structure
  * Remove operations.About as it is too thin an interface
  * Implement Integration test

Relates to #1138 and #1564
2018-04-17 21:53:27 +01:00
a-roussos 94e277d759 about: add new command 'about' to get quota info from a remote
Implemented for drive only.

Relates to #1138 and #1564.
2018-04-17 21:50:14 +01:00
Nick Craig-Wood b83814082b backend/http: if HEAD didn't return Content-Length use -1 as size
This means that the files will be treated as an unknown length and
will download properly.

Fixes #2247
2018-04-16 19:40:02 +01:00
Nick Craig-Wood 2b7957cc74 vfs: Only make the VFS cache if --vfs-cache-mode > Off
This stops the cache cleaner running unnecessarily and saves
resources.

This also helps with issue #2227 which was caused by a second mount
deleting objects in the first mounts cache.
2018-04-16 17:06:41 +01:00
Nick Craig-Wood 3d5106e52b drive: fix DirMove leaving a hardlinked directory behind #2245
This bug was introduced by the v3 API conversion in 07f20dd1fd.

The problem was that dircache.FindPath doesn't work for the root directory.

This adds an internal error for dircache.FindPath being called with
the root directory.  This makes a failing test, which the fix to the
drive backend fixes.

This also improves the DirCache integration test.
2018-04-15 10:12:21 +01:00
Nick Craig-Wood 29ce1c2747 fstest: fix CheckListingWithPrecision with non Windows safe chars
* Factor WinPath from fstest to fstests
  * Use it to normalize the directory names while checking them
2018-04-15 10:12:20 +01:00
Nick Craig-Wood dc247d21ff s3: add in config for all the supported S3 providers #2140
These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other.

This configures endpoints where known and makes sure config doesn't
appear where it isn't valid where possible.
2018-04-13 16:33:26 +01:00
Nick Craig-Wood 8c3740c2c5 config: Improve the Provider matching to have a negated match #2140
This makes it easier to make classes of provider in the config.
2018-04-13 16:06:37 +01:00
Giri Badanahatti acd5d4377e config,s3: hierarchical configuration support #2140
This introduces a method of making provider specific configuration
within a remote.  This is useful particularly in s3.

This commit does the basic configuration in S3 for IBM COS.
2018-04-13 16:05:35 +01:00
Matthew Holt 9e4cd55477 size: Add --json flag 2018-04-13 13:38:06 +01:00
Nick Craig-Wood 2015f98f0c Add Craig Rachel to contributors 2018-04-13 13:36:46 +01:00
Craig Rachel 0e6faa2313 s3: add One Zone Infrequent Access storage class - fixes #2240 2018-04-13 13:36:25 +01:00
Nick Craig-Wood 905e40b3e6 Add Peter Baumgartner to contributors 2018-04-13 13:33:22 +01:00
Peter Baumgartner 1db68571fd s3,swift: Add --use-server-modtime
`--use-server-modtime` stops s3 and swift retrieving the modtime from metadata which enables a fast sync mode with the `--update` flag.
2018-04-13 13:32:17 +01:00
Nick Craig-Wood 6b67489133 Add Animosity022 to contributors 2018-04-13 13:26:41 +01:00
Animosity022 27dfcf303c cache: improve docs
This adds that the cache-chunk-path needs to be cleared manually if chunk-size is changed.
2018-04-13 13:26:26 +01:00
Nick Craig-Wood e6d9720d7b Add Mateusz Piotrowski to contributors 2018-04-13 13:25:16 +01:00
Mateusz Piotrowski 196da4d903 dropbox: fix a typo in the docs 2018-04-13 13:24:58 +01:00
Nick Craig-Wood 18317a2747 vendor: update github.com/pkg/sftp because dep insisted 2018-04-13 13:23:55 +01:00
Nick Craig-Wood ef412c1985 drive: fix misplaced log in dedupe MergeDirs 2018-04-13 13:23:55 +01:00
Nick Craig-Wood d97fe3b824 fs/operations: make dedupe work with mega
* factor into its own files
  * remove assumptions about having a given hash type
  * make tests work if the remote has no hash
2018-04-13 13:23:55 +01:00
Nick Craig-Wood 792c9e185e Add Antoine GIRARD to contributors 2018-04-13 13:23:55 +01:00
Antoine GIRARD 1f681e585b fstests: fix typo 2018-04-13 13:23:08 +01:00
Nick Craig-Wood e82452ce9a drive: check Open calls for google error messages
This should also enable Open calls to retry properly
2018-04-11 20:55:58 +01:00