5065c422b4
lib/random: unify random string generation into random.String
...
This was factored from fstest as we were including the testing
enviroment into the main binary because of it.
This was causing opening the browser to fail because of 8243ff8bc8
2019-08-06 12:44:08 +01:00
57d5de6fba
build: fix up package paths after repo move
...
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
2019-07-28 18:47:38 +01:00
f78cd1e043
Add context propagation to rclone
...
- Change rclone/fs interfaces to accept context.Context
- Update interface implementations to use context.Context
- Change top level usage to propagate context to lover level functions
Context propagation is needed for stopping transfers and passing other
request-scoped values.
2019-06-19 11:59:46 +01:00
8ee6034b23
Look for Fs in the cache rather than calling NewFs directly
...
This will save operations when rclone is used in remote control mode
or with the same remote multiple times in the command line.
2019-05-27 16:14:55 +01:00
7c4fe3eb75
local: define OpenWriterAt interface and test and implement it #2252
...
This will enable multipart downloads in future commits
2019-05-11 23:35:19 +01:00
aba15f11d8
cache: note unsupported optional methods
2019-04-16 13:34:06 +01:00
47099466c0
cache: Use walk.ListR for listing the temporary Fs.
2019-03-16 17:41:12 +00:00
a0d4c04687
backend: fix misspellings
2019-02-07 19:51:03 +01:00
d966cef14c
build: fix problems found with unconvert
2019-02-02 11:45:07 +00:00
97752ca8fb
build: fix problems found with ineffasign linter
2019-02-02 11:45:07 +00:00
8d5d332daf
build: fix problems found with golint
2019-02-02 11:45:07 +00:00
26b5f55cba
Update after goimports change
2018-12-04 10:11:57 +00:00
0b80d1481a
cache: make tests not start an rc but use the internal framework
2018-11-04 15:11:51 +00:00
0bfa9811f7
rc: factor server code into rcserver and implement serving objects
...
If a GET or HEAD request is receivied with a URL parameter of fs then
it will be served from that remote.
2018-11-03 11:32:00 +00:00
696e7b2833
backend/cache: Print correct info about Cache Writes
2018-10-27 00:27:47 +01:00
ade252f13b
build: fixup code formatting after goimports change
2018-10-13 22:47:12 +01:00
1a40bceb1d
backend: unify NewFs path handling for wrapping remotes
...
Use the same function to join the root paths for the wrapping remotes
alias, cache and crypt.
The new function fspath.JoinRootPath is equivalent to path.Join, but if
the first non empty element starts with "//", this is preserved to allow
Windows network path to be used in these remotes.
2018-10-10 17:50:27 +01:00
a9273c5da5
docs: move documentation for options from docs/content into backends
...
In the following commit, the documentation will be autogenerated.
2018-10-06 11:47:46 +01:00
c0084f43dd
cache: Remove entries that no longer exist in the source
...
list directory with 25k files
before(1.43.1)
5m24s
after
3m21s
2018-10-06 11:23:33 +01:00
1eec59e091
fs: update ChangeNotifier interface
...
This introduces a channel to the ChangeNotify function, which can be
used to update the poll-interval and cleanly exit the polling function.
2018-09-25 14:01:13 +02:00
3c44ef788a
cache: add plex_insecure option to skip certificate validation
...
Fixes #2215
2018-09-10 21:19:25 +01:00
f6ee0795ac
cache: preserve leading / in wrapped remote path
...
When combining the remote value and the root path, preserve the absence
or presence of the / at the beginning of the wrapped remote path.
e.g. a remote "cloud:" and root path "dir" becomes "cloud:dir" instead
of "cloud:/dir".
Fixes #2553
2018-09-10 20:35:50 +01:00
5a6750e1cd
cache: documentation fix for cache-chunk-total-size - Fixes #2519
...
Signed-off-by: Anagh Kumar Baranwal <anaghk.dos@gmail.com>
2018-09-04 16:16:35 +01:00
deda093637
cache: fix error return value of cache/fetch rc method
2018-09-03 17:32:11 +02:00
a4c4019032
cache: improve performance by not sending info requests for cached chunks
2018-09-03 15:41:06 +01:00
a0b3fd3a33
cache: fix worker scale down
...
Ensure that calling scaleWorkers will create/destroy the right amount of
workers.
2018-09-03 12:29:35 +02:00
cdbe3691b7
cache: add cache/fetch rc function
2018-09-03 12:29:35 +02:00
76450c01f3
cache: remove accidentally committed files
...
* Delete cache_upload_test.go.orig
* Delete cache_upload_test.go.rej
2018-09-02 15:51:22 +01:00
6156bc5806
cache: fix nil pointer deref - fixes #2448
2018-08-07 21:33:13 +01:00
eb6aafbd14
cache: implement fs.ObjectUnWrapper
2018-08-07 09:28:55 +01:00
ffd11662ba
cache: fix nil pointer deref when using lsjson on cached directory
...
This stops embedding the fs.Directory into the cache.Directory because
it can be nil and implements an ID method which checks the nil status.
See: https://forum.rclone.org/t/runtime-error-with-cache-remote-and-lsjson/6305
2018-08-05 09:42:31 +01:00
f3f48d7d49
Implement new backend config system
...
This unifies the 3 methods of reading config
* command line
* environment variable
* config file
And allows them all to be configured in all places. This is done by
making the []fs.Option in the backend registration be the master
source of what the backend options are.
The backend changes are:
* Use the new configmap.Mapper parameter
* Use configstruct to parse it into an Options struct
* Add all config to []fs.Option including defaults and help
* Remove all uses of pflag
* Remove all uses of config.FileGet
2018-07-16 21:20:47 +01:00
aaa8591661
cache: add non cached dirs on notifications - #2155
2018-06-13 23:57:26 +03:00
4df1794932
cache: fix panic when running without plex configs
2018-06-13 15:06:14 +03:00
339fbf0df5
cache: reconnect plex websocket on failures
2018-06-12 22:58:15 +03:00
13ccb39819
cache: allow root to be expired from rc - #2237
2018-06-12 22:19:03 +03:00
f9a1a7e700
cache: fix root folder caching
2018-06-10 21:54:20 +03:00
4e8e5fed7d
cache: clean remaining empty folders from temp upload path
2018-06-09 14:52:31 +03:00
c0f772bc14
cache: update internal tests and small fixes
2018-06-08 23:34:38 +03:00
8a84975993
cache: cache lists using batch writes
2018-06-04 21:04:45 +03:00
71b4f1ccab
cache: use secure websockets for HTTPS Plex addresses
2018-05-22 11:47:57 +01:00
e5ff375948
Use config.FileGet instead of fs.ConfigFileGet
2018-05-22 09:43:24 +01:00
1320e84bc2
build: remove unused code spotted by the deadcode linter
2018-05-05 17:32:41 +01:00
cb5bd47e61
build: fix errors spotted by ineffassign linter
...
These were mostly caused by shadowing err and a good fraction of them
will have caused errors not to be propagated properly.
2018-05-05 17:32:41 +01:00
9ab2521ef2
rc: autogenerate and tidy the docs and commands
...
* Rename rc/pid -> core/pid
* Sort the output of `rc list`
* Make a script to autogenerate the docs
* Tidy docs
2018-04-23 20:57:17 +01:00
1ac6dacf0f
about: complete other providers and re-work internals
...
* Implement about for:
* local, crypt, cache, drive, swift, hubic, onedrive, pcloud, dropbox
* Implement `--json` and `---full` flag for `rclone about`
* change About interface to return a Usage structure
* Remove operations.About as it is too thin an interface
* Implement Integration test
Relates to #1138 and #1564
2018-04-17 21:53:27 +01:00
94e277d759
about: add new command 'about' to get quota info from a remote
...
Implemented for drive only.
Relates to #1138 and #1564 .
2018-04-17 21:50:14 +01:00
0fa700b3cf
Make integration tests use go1.7+ nested tests #2154
...
* Removed generated code and code generator
* Updated docs on how to write integration tests
* Tidied up the actual integration tests
2018-04-08 15:04:14 +01:00
be54fd8f70
Remove builds conditional on go1.7 since that is now guaranteed #2154
...
Old fallback code was deleted and the go1.7 style code inlined where
appropriate.
2018-04-07 11:42:55 +01:00
e5be471ce0
Use io.SeekStart/End/Current constants now for go1.7+ #2154
2018-04-07 11:42:36 +01:00
80588a5a6b
Replace "golang.org/x/net/context" with "context" for go1.7+ #2154
2018-04-07 11:42:08 +01:00
6e11a25df5
cache: flush the memory cache after close
2018-04-04 23:25:53 +03:00
1dea99ab20
cache: purge file data on notification
2018-04-03 23:24:45 +03:00
e42cee5e02
cache: always forget parent dir for notifications - for #2117
2018-03-31 12:44:09 +03:00
a8267d1628
link: allow creating public link to files and folders - closes #1562
2018-03-29 09:10:19 +02:00
b52e34ef5e
cache: add info log on notification - for #2150
2018-03-23 22:41:01 +02:00
0ed0d9a7bc
cache: integrate with Plex websocket
2018-03-22 21:21:03 +02:00
d9c13bff83
add rc cache/stats
2018-03-22 21:16:16 +02:00
4924ac2f17
cache: reduce log level for plex api - for #2102
2018-03-17 11:57:36 +02:00
078d705dbe
cache: notify vfs and support crypt in rpc - #2111
2018-03-15 11:39:16 +02:00
677971643c
cache: add support for rc
2018-03-14 22:58:20 +02:00
d76da1f5fd
cache: fix dir cache issue - #2117
2018-03-14 11:08:30 +02:00
a7dbf32c53
cache: Implement --cache-db-wait-time flag
...
This can be used to make the cache wait for other running cache
backends to finish rather than erroring after 1 second.
2018-03-08 21:14:09 +00:00
70f07fd3ac
fs: add ChangeNotify and backend support for it ( #2094 )
...
* fs: rename DirChangeNotify to ChangeNotify
* cache: switch to ChangeNotify
* ChangeNotify: keep order of notifications
2018-03-08 22:03:34 +02:00
fe8eeec5b5
cache: improve efficiency with RangeOption and RangeSeek #1825
...
* All remotes now support RangeOption so remove SeekOption
* Correct off by one error as RangeOption arguments are inclusive.
* Use RangeSeek in preference to Seek if available
2018-03-02 17:10:56 +00:00
aeefa34f62
fstests: add TestInternal ( #2085 )
...
TestInternal allows to perform a custom test on the backend using the
optional InternalTester interface.
2018-02-25 10:58:06 +01:00
9a34fd984c
cache: fix dirmove with temp fs enabled
2018-02-14 23:47:45 +02:00
54724a1362
cache: notify vfs when using temp fs - fixes #2051
2018-02-11 22:30:58 +02:00
b33e3f779c
cache: add support for polling
2018-02-10 22:01:05 +02:00
b3d8b7e22e
cache: use atexit for cleanup
2018-01-30 22:35:53 +02:00
29d428040c
cache: clean root path ( #2023 )
...
Trim "/" from the root path to fix "slice bounds out of range" panic
in cache.go:1272.
Fixes #1945
2018-01-30 14:35:40 +01:00
40af98b0b3
cache: offline uploading
2018-01-30 00:05:04 +02:00
fe52502f19
fs: Adjust RangeOption.Decode to return -1 for read to end
...
A Range request can never request 0 bytes however this change was made
to make a clearer signal that the limit means read to the end.
Add test and more documentation and fixup uses
2018-01-27 14:31:29 +00:00
88e0770f2d
cache: Implement RangeOption #1825
2018-01-22 19:44:55 +00:00
8653944a6d
Make RangeOption manadatory for Open - #1825
...
Add an integration test to make sure all backends implement
RangeOption correctly.
2018-01-21 17:09:12 +00:00
71722b5b95
config: factor Obscure and Reveal into its own package
2018-01-18 20:19:55 +00:00
11da2a6c9b
Break the fs package up into smaller parts.
...
The purpose of this is to make it easier to maintain and eventually to
allow the rclone backends to be re-used in other projects without
having to use the rclone configuration system.
The new code layout is documented in CONTRIBUTING.
2018-01-15 17:51:14 +00:00
b8b620f5c2
Move all backends into backend directory
2018-01-12 17:07:38 +00:00
Nick Craig-Wood
Aleksandar Jankovic
Fabian Möller
Florian Gamboeck
dcpu
Anagh Kumar Baranwal
dcpu
Dr. Tobias Quathamer
remusb
John Clayton
a-roussos
Stefan
seuffert