Commit Graph

7584 Commits

Author SHA1 Message Date
Nick Craig-Wood 863b4125c3 Version v1.65.1 2024-01-08 10:58:32 +00:00
Vincent Murphy 576ecf559d docs: Fix broken test_proxy.py link again
The previous fix fixed the auto generated output - this fixes the source.
2024-01-08 10:57:13 +00:00
Nick Craig-Wood cfd581a986 operations: fix files moved by rclone move not being counted as transfers
Before this change we were only counting moves as checks. This means
that when using `rclone move` the `Transfers` stat did not count up
like it should do.

This changes introduces a new primitive operations.MoveTransfers which
counts moves as Transfers for use where that is appropriate, such as
rclone move/moveto. Otherwise moves are counted as checks and their
bytes are not accounted.

See: #7183
See: https://forum.rclone.org/t/stats-one-line-date-broken-in-1-64-0-and-later/43263/
2024-01-07 11:29:12 +00:00
Nick Craig-Wood ad8bde69b3 accounting: fix stats to show server side transfers
Before this fix we were not counting transferred files nor transferred
bytes for server side moves/copies.

If the server side move/copy has been marked as a transfer and not a
checker then this accounts transferred files and transferred bytes.

The transferred bytes are not accounted to the network though so this
should not affect the network stats.
2024-01-07 11:29:11 +00:00
Nick Craig-Wood 771ec943f2 onedrive: fix "unauthenticated: Unauthenticated" errors when uploading
Before this change, sometimes when uploading files the onedrive
servers return 401 Unauthorized errors with the text "unauthenticated:
Unauthenticated".

This is because we are sending the Authorization header with the
request and it says in the docs that we shouldn't.

https://learn.microsoft.com/en-us/graph/api/driveitem-createuploadsession?view=graph-rest-1.0#remarks

> If you include the Authorization header when issuing the PUT call,
> it may result in an HTTP 401 Unauthorized response. Only send the
> Authorization header and bearer token when issuing the POST during
> the first step. Don't include it when you issue the PUT call.

This patch fixes the problem by doing the PUT request with an
unauthenticated client.

Fixes #7405
See: https://forum.rclone.org/t/onedrive-unauthenticated-when-trying-to-copy-sync-but-can-use-lsd/41149/
See: https://forum.rclone.org/t/onedrive-unauthenticated-issue/43792/
2024-01-07 11:24:55 +00:00
Nick Craig-Wood 4a297b35e5 Revert "mount: fix macOS not noticing errors with --daemon"
Unfortunately this does not compile on all platforms and the fix is
too big for the point release.

This reverts commit 5a22dad9a7.
2024-01-07 11:24:55 +00:00
Nick Craig-Wood 6b61967507 s3: fix crash if no UploadId in multipart upload
Before this change if the S3 API returned a multipart upload with no
UploadId then rclone would crash.

This detects the problem and attempts to retry the multipart upload
creation.

See: https://forum.rclone.org/t/panic-runtime-error-invalid-memory-address-or-nil-pointer-dereference/43425
2024-01-05 16:19:19 +00:00
Nick Craig-Wood e174c8f822 serve s3: fix listing oddities
Before this change, listing a subdirectory gave errors like this:

    Entry doesn't belong in directory "" (contains subdir) - ignoring

It also did full recursive listings when it didn't need to.

This was caused by the code using the underlying Fs to do recursive
listings on bucket based backends.

Using both the VFS and the underlying Fs is a mistake so this patch
removes the code which uses the underlying Fs and just uses the VFS.

Fixes #7500
2024-01-05 16:19:19 +00:00
Nick Craig-Wood bff56d0b24 protondrive: fix CVE-2023-45286 / GHSA-xwh9-gc39-5298
A race condition in go-resty can result in HTTP request body
disclosure across requests.

See: https://pkg.go.dev/vuln/GO-2023-2328
Fixes: #7491
2024-01-05 16:19:19 +00:00
Nick Craig-Wood 59ff59e45a build: fix docker build on arm/v6
Unexpectedly the team which runs the Go docker images have removed the
arm/v6 image which means that the rclone docker images no longer
build.

One of the recommended fixes is what we've done here - switch to the
alpine builder. This has the advantage that it actually builds arm/v6
architecture unlike the previous builder which build arm/v5.

See: https://github.com/docker-library/golang/issues/502
2024-01-05 16:19:19 +00:00
dependabot[bot] c27ab0211c build(deps): bump golang.org/x/crypto to fix ssh terrapin CVE-2023-48795
Fixes SSH terrapin attack: see https://terrapin-attack.com.

Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.14.0 to 0.17.0.
- [Commits](https://github.com/golang/crypto/compare/v0.14.0...v0.17.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-05 16:19:19 +00:00
rkonfj 9979b9d082 oauthutil: avoid panic when `*token` and `*ts.token` are the same
the field `raw` of `oauth2.Token` may be an uncomparable type(often map[string]interface{}), causing `*token != *ts.token` expression to panic(comparing uncomparable type ...).

the semantics of comparing whether two tokens are the same can be achieved by comparing accessToken, refreshToken and expire to avoid panic.
2024-01-05 16:19:19 +00:00
WeidiDeng 2be627aa56 ftp: fix multi-thread copy
Before this change multi-thread copies using the FTP backend used to error with

    551 Error reading file

This was caused by a spurious error being reported which this code silences.

Fixes #7532
See #3942
2024-01-05 16:19:19 +00:00
Nick Craig-Wood 3f7abd278d googlephotos: fix nil pointer exception when batch failed
This was a simple error check that was missing. Interestingly the
errcheck linter did not spot this.

See: https://forum.rclone.org/t/invalid-memory-address-or-nil-pointer-dereference-error-when-copy-to-google-photos/43634/
2024-01-05 16:19:19 +00:00
nielash 489c36b101 hasher: fix invalid memory address error when MaxAge == 0
When f.opt.MaxAge == 0, f.db is never set, however several methods later assume
it is set and attempt to access it, causing an invalid memory address error.
This change fixes the issue in a few spots (there may still be others I haven't
yet encountered.)
2024-01-05 16:19:19 +00:00
albertony df65aced2e docs/librclone: the newer and recommended ucrt64 subsystem of msys2 can now be used for building on windows 2024-01-05 16:19:19 +00:00
rarspace01 141e97edb8 docs: fix broken link in serve webdav 2024-01-05 16:19:19 +00:00
Oksana 8571eaf425 azure-files: fix storage base url
Documented in https://learn.microsoft.com/en-us/azure/storage/common/storage-account-overview
2024-01-05 16:19:19 +00:00
Manoj Ghosh 6ccbebd903 oracle object storage: fix object storage endpoint for custom endpoints 2024-01-05 16:19:19 +00:00
Nick Craig-Wood 8b8156f7c3 chunker,compress,crypt,hasher,union: fix rclone move a file over itself deleting the file
This fixes the Root() returned by the backend when it has returned
fs.ErrorIsFile.

Before this change it returned a root which included the file path.

Because Root() was wrong this caused the detection of the file being
moved over itself check to fail.

This adds an integration test to check it for all backends.

See: https://forum.rclone.org/t/rclone-move-chunker-dir-file-chunker-dir-deletes-all-file-chunks/43333/
2024-01-05 16:19:19 +00:00
keongalvin a0b19fefdf docs: fix broken link 2024-01-05 16:19:19 +00:00
Nick Craig-Wood d0e68480be dropbox: fix used space on dropbox team accounts
Before this change we were not using the used space from the team
stats.

This patch uses that as the used space if available as it seems to
include the user stats in it.

See: https://forum.rclone.org/t/rclone-about-with-dropbox-reporte-size-incorrectly/43269/
2024-01-05 16:19:19 +00:00
Nick Craig-Wood ab6c5252f1 vfs: note that --vfs-refresh runs in the background #6830 2024-01-05 16:19:19 +00:00
emyarod 29a23c5e18 docs: update contributor email 2024-01-05 16:19:19 +00:00
dependabot[bot] caacf55b69 build(deps): bump actions/setup-go from 4 to 5
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 4 to 5.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-01-05 16:19:19 +00:00
Eli Orzitzer f62ae71b4c Doc change: Add the CreateBucket permission requirement for AWS S3 2024-01-05 16:19:19 +00:00
Nick Craig-Wood 4245a042c0 nfsmount: compile for all unix oses, add --sudo and fix error/option handling
- make compile on all unix OSes - this will make the docs appear on linux and rclone.org!
- add --sudo flag for using with mount
- improve error reporting
- fix option handling
2024-01-05 16:19:19 +00:00
Nick Craig-Wood 3f3245fcd4 serve nfs: Mark as experimental 2024-01-05 16:19:19 +00:00
Nick Craig-Wood 5742a61d23 onedrive: fix error listing: unknown object type <nil>
This error was introduced in this commit when refactoring the list
routine.

b8591b230d onedrive: implement ListR method which gives --fast-list support

The error was caused by OneNote files not being skipped properly.
2024-01-05 16:19:19 +00:00
ben-ba 768c57c1ba docs: fix typo in docs.md
- OpenChunkedWriter
+ OpenChunkWriter
2024-01-05 16:19:19 +00:00
Manoj Ghosh 9f42ed3380 multipart copy create bucket if it doesn't exist. 2024-01-05 16:19:19 +00:00
halms 40a7edab2d smb: fix shares not listed by updating go-smb2
Before this change the IP address of the server was used in the SMB
connect request (see CloudSoda/go-smb2#18).
The updated library now can pass the hostname instead.

The update requires a small change in the dial method call.

Fixes rclone#6672
2024-01-05 16:19:19 +00:00
Nick Craig-Wood 5a22dad9a7 mount: fix macOS not noticing errors with --daemon
See: https://forum.rclone.org/t/rclone-mount-daemon-exits-successfully-even-when-mount-fails/43146
2024-01-05 16:19:19 +00:00
Nick Craig-Wood b3c2985544 install.sh: fix harmless error message on install
This was caused by trying to write to a non existent file, and
changing the order of the cleanup fixed it.

https://forum.rclone.org/t/rclone-v1-65-0-release/43100/18
2024-01-05 16:19:19 +00:00
Nick Craig-Wood 938753ddc3 Start v1.65.1-DEV development 2024-01-05 16:03:50 +00:00
Nick Craig-Wood 82b963e372 Version v1.65.0 2023-11-26 16:07:39 +00:00
Nick Craig-Wood 74d5477fad onedrive: add --onedrive-delta flag to enable ListR
Before this change ListR was unconditionally enabled on onedrive.

This caused performance problems for some uses, so now the
--onedrive-delta flag has to be supplied.

Fixes #7362
2023-11-26 16:06:49 +00:00
Nick Craig-Wood b5857f0bf8 smb: fix modtime of multithread uploads by setting PartialUploads
Before this change PartialUploads was not set. This is clearly wrong
since incoming files are visible on the smb server.

Setting PartialUploads fixes the multithread upload modtime problem as
it uses the PartialUploads flag as an indication that it needs to set
the modtime explicitly.

This problem was detected by the new TestMultithreadCopy integration
tests

Fixes #7411
2023-11-25 18:46:48 +00:00
Nick Craig-Wood edb5ccdd0b smb: fix about size wrong by switching to github.com/cloudsoda/go-smb2/ fork
Before this change smb drives sometimes showed a fraction of the
correct size using `rclone about`.

This fixes the problem by switching the upstream library from
github.com/hirochachacha/go-smb2 to github.com/cloudsoda/go-smb2 which
has a fix for the problem.

The new library passes the integration tests.

Fixes #6733
2023-11-25 18:45:41 +00:00
Nick Craig-Wood 0244caf13a serve s3: fix overwrite of files with 0 length file
Before this change overwriting an existing file with a 0 length file
didn't update the file size.

This change corrects the issue and makes sure the file is truncated
properly.

This was discovered by the full integration tests.
2023-11-24 20:47:06 +00:00
Nick Craig-Wood aaa897337d serve s3: fix error handling for listing non-existent prefix - fixes #7455
Before this change serve s3 would return NoSuchKey errors when a non
existent prefix was listed.

This change fixes it to return an empty list like AWS does.

This was discovered by the full integration tests.
2023-11-24 20:47:06 +00:00
Nick Craig-Wood e7c002adef test_all: make integration test for serve s3 2023-11-24 20:47:06 +00:00
Nick Craig-Wood 9e62a74a23 Add Abhinav Dhiman to contributors 2023-11-24 20:47:06 +00:00
Nick Craig-Wood a10abf9934 Add 你知道未来吗 to contributors 2023-11-24 20:47:06 +00:00
Abhinav Dhiman 36eb3cd660
imagekit: Added ImageKit backend 2023-11-24 18:18:01 +00:00
你知道未来吗 fd2322cb41
fs/fshttp: fix --contimeout being ignored
The following command will block for 60s(default) when the network is slow or unavailable:

```
rclone  --contimeout 10s --low-level-retries 0 lsd dropbox:
```

This change will make it timeout after the expected 10s.

Signed-off-by: rkonfj <rkonfj@gmail.com>
2023-11-24 17:53:33 +00:00
Nick Craig-Wood 4eed3ae99a s3: ensure we can set upload cutoff that we use for Rclone provider
This is a workaround to make the new multipart upload integration
tests pass.
2023-11-24 16:32:06 +00:00
Nick Craig-Wood d8855b21eb serve s3: document multipart copy doesn't work #7454
This puts in a workaround for the tests also
2023-11-24 15:49:33 +00:00
Nick Craig-Wood 8f47b6746d b2: fix streaming chunked files an exact multiple of chunk size
Before this change, streaming files an exact multiple of the chunk
size would cause rclone to attempt to stream a 0 sized chunk which was
rejected by the b2 servers.

This bug was noticed by the new integration tests for chunked streaming.
2023-11-24 14:32:01 +00:00
Nick Craig-Wood cc2a4c2e20 fstest: factor chunked streaming tests from b2 and use in all backends 2023-11-24 12:58:40 +00:00