Commit Graph

7616 Commits

Author SHA1 Message Date
Nick Craig-Wood dd0e5b9a7f operations: use built in io.OffsetWriter for go1.20 2024-01-15 16:22:07 +00:00
Nick Craig-Wood da244a3709 ssh: shorten wait delay for external ssh binaries now that we are using go1.20
Now we are guaranteed to have go1.20 or later we can use the WaitDelay
flag when running external ssh binaries.
2024-01-15 16:22:07 +00:00
Nick Craig-Wood 938b43c26c build: remove random.Seed since random generator is seeded automatically in go1.20
Now that the minimum version is go1.20 we can stop seeding the random
number generator.
2024-01-15 16:22:07 +00:00
Nick Craig-Wood 13fb2fb2ec build: update to go1.22rc1 and make go1.20 the minimum required version 2024-01-15 16:22:07 +00:00
Nick Craig-Wood 43cc2435c3 build: update indirect dependencies where possible 2024-01-15 16:18:42 +00:00
Nick Craig-Wood 1b1e43074f build: update direct dependencies and fix serve nfs
This updates the direct dependencies.

The latest github.com/willscott/go-nfs has changed the interface
slightly so this implements a dummy InvalidateHandle method in order
to satisfy it.
2024-01-15 16:18:42 +00:00
Nick Craig-Wood cacfc100de docs: add warp.dev sponsorship to github home page 2024-01-15 11:57:27 +00:00
Nick Craig-Wood f8c5695aed docs: add warp.dev as a sponsor 2024-01-15 11:55:38 +00:00
Nick Craig-Wood a5972fe0d1 docs: update website footer 2024-01-15 11:55:38 +00:00
Nick Craig-Wood 184459ba8f vfs: fix stale data when using --vfs-cache-mode full
Before this change the VFS cache could get into a state where when an
object was updated remotely, the fingerprint of the item was correct
for the new object but the data in the VFS cache was for the old
object.

This fixes the problem by updating the fingerprint of the item at the
point we remove the stale data. The empty cache item now represents
the new item even though it has no data in.

This stops the fallback code for an empty fingerprint running (used
when we are writing items to the cache instead of reading them) which
was causing the problem.

Fixes #6053
See: https://forum.rclone.org/t/cached-webdav-mount-fingerprints-get-nuked-on-ls/43974/
2024-01-15 11:12:59 +00:00
Nick Craig-Wood 519fe98e6e azureblob: implement --azureblob-delete-snapshots
This flag controls what happens when we try to delete a blob with a
snapshot. The UI follows the azcopy tool.

See: https://forum.rclone.org/t/how-to-delete-undeleted-blobs-on-azure/43911/
2024-01-13 14:27:54 +00:00
Nick Craig-Wood 3df6518006 Add Nikhil Ahuja to contributors 2024-01-13 14:27:54 +00:00
Nikhil Ahuja 1045f54128 oracleobjectstorage: Support "backend restore" command - fixes #7371 2024-01-09 09:43:36 +00:00
dependabot[bot] 0563cc6314 build(deps): bump github.com/cloudflare/circl from 1.3.6 to 1.3.7
Bumps [github.com/cloudflare/circl](https://github.com/cloudflare/circl) from 1.3.6 to 1.3.7.
- [Release notes](https://github.com/cloudflare/circl/releases)
- [Commits](https://github.com/cloudflare/circl/compare/v1.3.6...v1.3.7)

---
updated-dependencies:
- dependency-name: github.com/cloudflare/circl
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-01-08 17:38:09 +00:00
Nick Craig-Wood e20f2eee59 Changelog updates from Version v1.65.1 2024-01-08 11:54:02 +00:00
Vincent Murphy 41b8935a6c docs: Fix broken test_proxy.py link again
The previous fix fixed the auto generated output - this fixes the source.
2024-01-08 11:54:02 +00:00
Nick Craig-Wood fbdf71ab64 operations: fix files moved by rclone move not being counted as transfers
Before this change we were only counting moves as checks. This means
that when using `rclone move` the `Transfers` stat did not count up
like it should do.

This changes introduces a new primitive operations.MoveTransfers which
counts moves as Transfers for use where that is appropriate, such as
rclone move/moveto. Otherwise moves are counted as checks and their
bytes are not accounted.

See: #7183
See: https://forum.rclone.org/t/stats-one-line-date-broken-in-1-64-0-and-later/43263/
2024-01-07 11:26:09 +00:00
Nick Craig-Wood d392f9fcd8 accounting: fix stats to show server side transfers
Before this fix we were not counting transferred files nor transferred
bytes for server side moves/copies.

If the server side move/copy has been marked as a transfer and not a
checker then this accounts transferred files and transferred bytes.

The transferred bytes are not accounted to the network though so this
should not affect the network stats.
2024-01-07 11:26:09 +00:00
Nick Craig-Wood dedad9f071 onedrive: fix "unauthenticated: Unauthenticated" errors when uploading
Before this change, sometimes when uploading files the onedrive
servers return 401 Unauthorized errors with the text "unauthenticated:
Unauthenticated".

This is because we are sending the Authorization header with the
request and it says in the docs that we shouldn't.

https://learn.microsoft.com/en-us/graph/api/driveitem-createuploadsession?view=graph-rest-1.0#remarks

> If you include the Authorization header when issuing the PUT call,
> it may result in an HTTP 401 Unauthorized response. Only send the
> Authorization header and bearer token when issuing the POST during
> the first step. Don't include it when you issue the PUT call.

This patch fixes the problem by doing the PUT request with an
unauthenticated client.

Fixes #7405
See: https://forum.rclone.org/t/onedrive-unauthenticated-when-trying-to-copy-sync-but-can-use-lsd/41149/
See: https://forum.rclone.org/t/onedrive-unauthenticated-issue/43792/
2024-01-07 11:14:08 +00:00
Nick Craig-Wood 1f6271fa15 s3: copy parts in parallel when doing chunked server side copy
Before this change rclone copied each chunk serially.

After this change it does --s3-upload-concurrency at once.

See: https://forum.rclone.org/t/transfer-big-files-50gb-from-s3-bucket-to-another-s3-bucket-doesnt-starts/43209
2024-01-05 15:54:52 +00:00
Nick Craig-Wood c16c22d6e1 s3: fix crash if no UploadId in multipart upload
Before this change if the S3 API returned a multipart upload with no
UploadId then rclone would crash.

This detects the problem and attempts to retry the multipart upload
creation.

See: https://forum.rclone.org/t/panic-runtime-error-invalid-memory-address-or-nil-pointer-dereference/43425
2024-01-05 15:52:52 +00:00
Nick Craig-Wood 486a10bec5 serve s3: fix listing oddities
Before this change, listing a subdirectory gave errors like this:

    Entry doesn't belong in directory "" (contains subdir) - ignoring

It also did full recursive listings when it didn't need to.

This was caused by the code using the underlying Fs to do recursive
listings on bucket based backends.

Using both the VFS and the underlying Fs is a mistake so this patch
removes the code which uses the underlying Fs and just uses the VFS.

Fixes #7500
2024-01-05 15:51:13 +00:00
Nick Craig-Wood 5fa13e3e31 protondrive: fix CVE-2023-45286 / GHSA-xwh9-gc39-5298
A race condition in go-resty can result in HTTP request body
disclosure across requests.

See: https://pkg.go.dev/vuln/GO-2023-2328
Fixes: #7491
2024-01-04 17:14:53 +00:00
Nick Craig-Wood 0e746f25a3 amazonclouddrive: remove Amazon Drive backend code and docs #7539
The Amazon Drive backend is closed from 2023-12-31.

See: https://www.amazon.com/b?ie=UTF8&node=23943055011
2024-01-04 17:05:54 +00:00
Nick Craig-Wood 578b9df6ea build: fix docker build on arm/v6
Unexpectedly the team which runs the Go docker images have removed the
arm/v6 image which means that the rclone docker images no longer
build.

One of the recommended fixes is what we've done here - switch to the
alpine builder. This has the advantage that it actually builds arm/v6
architecture unlike the previous builder which build arm/v5.

See: https://github.com/docker-library/golang/issues/502
2024-01-03 17:43:23 +00:00
Nick Craig-Wood 208e49ce4b fs: update use of math/rand to modern practice 2024-01-03 16:14:40 +00:00
Nick Craig-Wood 7aa066cff8 Add Paul Stern to contributors 2024-01-03 16:14:40 +00:00
dependabot[bot] 64df4cf2db
build(deps): bump golang.org/x/crypto to fix ssh terrapin CVE-2023-48795
Fixes SSH terrapin attack: see https://terrapin-attack.com.

Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.14.0 to 0.17.0.
- [Commits](https://github.com/golang/crypto/compare/v0.14.0...v0.17.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-03 15:47:34 +00:00
rkonfj 451d7badf7 oauthutil: avoid panic when `*token` and `*ts.token` are the same
the field `raw` of `oauth2.Token` may be an uncomparable type(often map[string]interface{}), causing `*token != *ts.token` expression to panic(comparing uncomparable type ...).

the semantics of comparing whether two tokens are the same can be achieved by comparing accessToken, refreshToken and expire to avoid panic.
2024-01-03 15:15:14 +00:00
WeidiDeng d977fa25fa
ftp: fix multi-thread copy
Before this change multi-thread copies using the FTP backend used to error with

    551 Error reading file

This was caused by a spurious error being reported which this code silences.

Fixes #7532
See #3942
2024-01-03 12:21:08 +00:00
Paul Stern bb679a9def backend: add description field for all backends
Fixes #4391
2024-01-03 10:57:59 +00:00
Nick Craig-Wood a3d19942bd googlephotos: fix nil pointer exception when batch failed
This was a simple error check that was missing. Interestingly the
errcheck linter did not spot this.

See: https://forum.rclone.org/t/invalid-memory-address-or-nil-pointer-dereference-error-when-copy-to-google-photos/43634/
2024-01-03 10:57:59 +00:00
Nick Craig-Wood 394195cfdf Add rarspace01 to contributors 2024-01-03 10:57:59 +00:00
nielash 3ca766b2f1 hasher: fix invalid memory address error when MaxAge == 0
When f.opt.MaxAge == 0, f.db is never set, however several methods later assume
it is set and attempt to access it, causing an invalid memory address error.
This change fixes the issue in a few spots (there may still be others I haven't
yet encountered.)
2024-01-02 18:14:01 +00:00
albertony 3bf8c877c3 docs/librclone: the newer and recommended ucrt64 subsystem of msys2 can now be used for building on windows 2024-01-01 21:56:45 +01:00
rarspace01 fba2d4c4a7
docs: fix broken link in serve webdav 2023-12-30 18:10:27 +01:00
Oksana 8503282a5a
azure-files: fix storage base url
Documented in https://learn.microsoft.com/en-us/azure/storage/common/storage-account-overview
2023-12-18 14:15:13 +00:00
Manoj Ghosh 743ea6ac26
oracle object storage: fix object storage endpoint for custom endpoints 2023-12-15 10:13:35 +00:00
Nick Craig-Wood c69eb84573 chunker,compress,crypt,hasher,union: fix rclone move a file over itself deleting the file
This fixes the Root() returned by the backend when it has returned
fs.ErrorIsFile.

Before this change it returned a root which included the file path.

Because Root() was wrong this caused the detection of the file being
moved over itself check to fail.

This adds an integration test to check it for all backends.

See: https://forum.rclone.org/t/rclone-move-chunker-dir-file-chunker-dir-deletes-all-file-chunks/43333/
2023-12-10 22:29:57 +00:00
Nick Craig-Wood f98e672f37 selfupdate: fix crash in tests if beta not found 2023-12-10 22:29:57 +00:00
Nick Craig-Wood 242fe96b18 Add keongalvin to contributors 2023-12-10 22:29:57 +00:00
rkonfj 3f159bac16 backend: fs implements the `Shutdowner` interface
Since `tokenRenewer` adds a Shutdown method, we should call it to
clean up resources.

changes backends:
onedrive,box,pcloud,amazonclouddrive,hidrive,jottacloud,sharefile
,premiumizeme

Signed-off-by: rkonfj <rkonfj@gmail.com>
2023-12-09 11:44:50 +00:00
rkonfj 6c58e9976c oauthutil: add Shutdown method
Before this change, calling the `oauthutil.NewRenew` func may
cause goroutine leaks.

This change adds a `Shutdown` method to allow the caller to exit
the goroutine to avoid leaks.

Signed-off-by: rkonfj <rkonfj@gmail.com>
2023-12-09 11:44:50 +00:00
keongalvin 110d07548f docs: fix broken link 2023-12-08 16:21:09 +00:00
Nick Craig-Wood f45cee831f dropbox: fix used space on dropbox team accounts
Before this change we were not using the used space from the team
stats.

This patch uses that as the used space if available as it seems to
include the user stats in it.

See: https://forum.rclone.org/t/rclone-about-with-dropbox-reporte-size-incorrectly/43269/
2023-12-08 14:26:46 +00:00
Nick Craig-Wood ef0f3020e4 vfs: note that --vfs-refresh runs in the background #6830 2023-12-08 14:26:46 +00:00
Nick Craig-Wood 113b2b648c Add emyarod to contributors 2023-12-08 14:26:46 +00:00
Nick Craig-Wood 57ab4d279e Add Anthony Metzidis to contributors 2023-12-08 14:26:46 +00:00
Nick Craig-Wood 8e21c77ead Add Eli Orzitzer to contributors 2023-12-08 14:26:46 +00:00
emyarod 4751980659 docs: update contributor email 2023-12-08 11:21:26 +00:00