Commit Graph

187 Commits

Author SHA1 Message Date
Stefan Breunig 52b042971a keep file permissions and try to keep user/group on supported systems (fixes #1467) 2017-06-25 09:05:24 +02:00
Stefan Breunig 2d2778eabf don't delete remote if name does not change while renaming (fixes #1495) 2017-06-25 08:55:54 +02:00
Nick Craig-Wood 50928a5027 Implement --fast-list flag.
This is supported remotes which can do a recursive listing.  It will
use more memory.

This is related to #1277 but doesn't fix that issue yet.
2017-06-14 16:49:40 +01:00
Nick Craig-Wood 740b3f6ae2 Fix problems found with ineffassign 2017-06-13 11:52:36 +01:00
Nick Craig-Wood ee14efd3c2 config: fix menu selection when no remotes 2017-06-10 11:39:40 +01:00
Nick Craig-Wood 52e1bfae2a oauth: Allow auth_url and token_url to be set in the config file
If set in the config file, these override the ones configured into the
remote.  This enables alternative oauth servers to be used for all
oauth remotes.  This can only be altered by editing the config file
for the moment.
2017-06-08 20:35:32 +01:00
Nick Craig-Wood b047402294 config: Fix save of temp file under Windows - fixes #1458 2017-06-01 16:38:19 +01:00
Stefan 9cede6b372 fully write new config file before moving to target location (fixes #1287)
* fully write new config file before moving to target location (fixes #1287)
* do not fail if there is no previous config; print temporary config path on failure
2017-06-01 08:57:10 +01:00
Nick Craig-Wood 88ef475629 config: allow keys to be deleted from the config file 2017-05-25 10:15:22 +01:00
Hraban Luyat 66e8c1600e Print password prompts to stderr
This makes rclone with encrypted config better suited for use in
pipelines. E.g.:

$ rclone lsl mydrive:Some/Dir | sort -k 4

If the password prompt ("Enter configuration password") is printed to
stdout, it will be swallowed by sort. By printing it to stderr, you
still see the prompt, without sacrificing compatibility with the unix
pipeline.
2017-04-22 09:38:39 +01:00
Nick Craig-Wood 493da54113 Add --buffer-size parameter to control buffer size for copy 2017-02-14 22:36:37 +00:00
Nick Craig-Wood ac1c041377 Redo log level flags
* -vv or --log-level DEBUG
  * -v or --log-level INFO
  * --log-level NOTICE (default)
  * -q --log-level ERROR

Replace Config.Verbose and Config.Quiet with Config.LogLevel

Fixes #739 Fixes #1108 Fixes #1000
2017-02-11 20:22:42 +00:00
Nick Craig-Wood 80f53176d9 Rename log functions and factor into own file 2017-02-11 17:54:50 +00:00
Nick Craig-Wood 7c6cd3a9e1 Make --delete-after the default and refactor --delete-{before,during,after} parsing 2017-02-04 10:30:25 +00:00
Nick Craig-Wood 9d331ce04b Implement --ignore-checksum flag
Fixes #793 Fixes #863 Fixes #981
2017-02-03 08:11:10 +00:00
Nick Craig-Wood 2192805360 rclone config: when choosing from a list, allow the value to be entered 2017-01-29 15:51:26 +00:00
Nick Craig-Wood db0b93c0ad rclone config: allow rename and copy of remotes - fixes #641 2017-01-29 15:37:44 +00:00
Dario Giovannetti 9d36258923 Comply with XDG Base Directory specification
Fixes #868
2017-01-26 10:22:08 +00:00
Nick Craig-Wood e2bf9eb8e9 Implement --suffix for use with --backup-dir only #98
This also makes sure we remove files we are about to override in the
--backup-dir properly.
2017-01-19 20:11:17 +00:00
ncw 3745c526f1 Implement --backup-dir - fixes #98
The parameter of backup-dir specifies a remote that all deleted or
overwritten files will be copied to.
2017-01-16 17:33:25 +00:00
Nick Craig-Wood f73ee5eade Make all config file variables be settable in the environment
These are set in the form RCLONE_CONFIG_remote_option where remote is
the uppercased remote name and option is the uppercased config file
option name.  Note that RCLONE_CONFIG_remote_TYPE must be set if
defining a new remote.

Fixes #616
2017-01-03 22:42:47 +00:00
Nick Craig-Wood 0d75d2585f Allow all options to be set from environment variables
The option names are munged changing - to _ making upper case and
prepending RCLONE_.  The values are as parsed by pflag.
2017-01-03 22:42:47 +00:00
Marco Paganini 3b0f944e23 Add time-based bandwidth limits.
- Change the --bwlimit command line parameter to accept both a limit (as
  before) or a full timetable (formatted as "hh:mm,limit
  hh:mm,limit...")
- The timetable is checked once a minute by a ticker function. A new
  tokenBucket is created every time a bandwidth change is necessary.
- This change is compatible with the SIGUSR2 change to toggle bandwidth
  limits.

This resolves #221.
2017-01-03 21:00:38 +00:00
Bjørn Erik Pedersen 47d3a450a4 sync: Track and perform server-side renames
This commits adds support for tracking of file renames if `track-renames` flag is set,
and it then performs server-side renames for remotes that support it, i.e.
remotes that implement either the `Mover` or the `Copier` interface.

Fixes #888
2017-01-03 20:35:05 +00:00
Nick Craig-Wood 1b2dda8c4c oauthutil: Reload config file off disk before updating token
This fixes the config file being overwritten when two rclones are running.

Fixes #682
2016-12-19 15:04:07 +00:00
Scott McGillivray f9df545e3c add --stats-unit option and improve alignment for --stats output 2016-11-28 12:18:30 +00:00
Nick Craig-Wood 318e335137 Remove `Authorization:` headers from `--dump-headers` output
Add in `--dump-auth` flag to put it back.
2016-11-02 15:53:43 +00:00
Nick Craig-Wood 83849e0a36 Don't show encrypted password to stop confusion - fixes #656 2016-10-08 11:26:14 +01:00
Nick Craig-Wood 0cb9bb3b54 Redo http Transport code
* Insert User-Agent in Transport - fixes #199
  * Update timeouts to use Context
  * Modernise transport
2016-09-12 17:50:19 +01:00
Nick Craig-Wood 9985fc40f4 Make Password parameters obey Optional flag and offer to generate random ones 2016-08-23 17:45:37 +01:00
Nick Craig-Wood b1de4c8cba Implement password Option and re-implement editing
Editing now shows all the options for the fs and asks one at a time
whether they should be changed.
2016-08-23 17:45:37 +01:00
Nick Craig-Wood 6a4e424630 Re-implement Obscure/Reveal so they use AES-CTR encryption 2016-08-23 17:45:37 +01:00
Nick Craig-Wood ae56df7d4f Add --dedupe-mode only to dedupe command 2016-08-04 21:47:14 +01:00
Nick Craig-Wood b9479cf7ab Implement --no-update-modtime flag - fixes #511 2016-07-12 10:46:45 +01:00
Nick Craig-Wood a20d80565b Tidy stats output - fixes #541 2016-07-11 13:04:30 +01:00
Nick Craig-Wood af4ef8ad8d Implement --no-traverse flag to stop copy traversing the destination remote.
Refactor sync/copy/move
  * Don't load the src listing unless doing a sync and --delete-before
  * Don't load the dst listing if doing copy/move and --no-traverse is set

`rclone --no-traverse copy src dst` now won't load either of the
listings into memory so will use the minimum amount of memory.

This change will reduce the amount of memory rclone uses dramatically
too as in normal operations (copy without --notraverse or sync) as it
no longer loads the source file listing into memory at all.

Fixes #8
Fixes #544
Fixes #546
2016-06-29 17:38:50 +01:00
Nick Craig-Wood 46135d830e Add --ignore-size flag - fixes #399 2016-06-17 17:20:08 +01:00
Nick Craig-Wood 4c5b2833b3 Convert to using github.com/pkg/errors everywhere 2016-06-13 17:43:03 +01:00
Nick Craig-Wood 67d0375b98 Audit use of log.Print and change to Debug, Log, or ErrorLog as appropriate 2016-06-06 21:23:54 +01:00
Nick Craig-Wood 772f99fd74 Make SizeSuffix output without `b` suffix for more useful printouts 2016-06-03 22:49:14 +01:00
Nick Craig-Wood e818b7c206 Represent -1 as "off" for SIZE values 2016-06-03 21:51:39 +01:00
Nick Craig-Wood 5723d788a4 Add `b` suffix so we can specify bytes in --bwlimit, --min-size etc
Fixes #449
2016-06-03 21:16:48 +01:00
Nick Craig-Wood ccdd1ea6c4 Add --max-depth parameter
This will apply to ls/lsd/sync/copy etc

Fixes #412
Fixes #213
2016-06-03 17:05:39 +01:00
Klaus Post 6c0c9abd57 Use "password:" instead of "password>" prompt
Fixes #410
2016-04-21 19:39:46 +01:00
Klaus Post 8f4d6973fb Fix missing "quit" option when there are no remotes. 2016-03-23 16:57:56 +00:00
Nick Craig-Wood ced3a4bc19 Implement -I, --ignore-times for unconditional upload - fixes #311 2016-03-22 17:02:27 +00:00
Nick Craig-Wood 8ef551bf9c Make dedupe remove identical copies without asking and add non interactive mode - fixes #338
* Now removes identical copies without asking
  * Now obeys `--dry-run`
  * Implement `--dedupe-mode` for non interactive running
    * `--dedupe-mode interactive` - interactive the default.
    * `--dedupe-mode skip` - removes identical files then skips anything left.
    * `--dedupe-mode first` - removes identical files then keeps the first one.
    * `--dedupe-mode newest` - removes identical files then keeps the newest one.
    * `--dedupe-mode oldest` - removes identical files then keeps the oldest one.
    * `--dedupe-mode rename` - removes identical files then renames the rest to be different.
  * Add tests which will only run on Google Drive.
2016-03-06 18:15:01 +00:00
Nick Craig-Wood dfc7cd97a3 Optionally disable gzip compression on downloads with `--no-gzip-encoding` - fixes #353 2016-02-29 19:48:54 +00:00
Nick Craig-Wood 280ac26464 Implement -u/--update so creation times can be used on all remotes - #226 2016-02-29 17:46:40 +00:00
Nick Craig-Wood 867a26fe4f Implement --low-level-retries flag - fixes #266 2016-02-25 22:58:21 +00:00
Nick Craig-Wood 95e4d837ef Make config chooser easier to understand 2016-02-21 13:40:16 +00:00
Nick Craig-Wood e08e35984c Add help to remote chooser in rclone config - fixes #43 2016-02-21 13:40:16 +00:00
Nick Craig-Wood 700e47d6e2 Stub out ReadPassword on plan9 and solaris to fix compilation 2016-02-21 10:31:53 +00:00
klauspost 84b00b362f Change back to original goconfig package.
Add documentation for `--ask-password`.
2016-02-17 11:45:05 +01:00
klauspost bfd7601cf9 Add configuration file encryption
See #317 for details.

Use `rclone config` to add/change/remove password.

Tests that loads the default configuration will now fail with a better error message, and add a switch that makes it possible to disable password prompts and fail instead.

Make it possible to use the "RCLONE_CONFIG_PASS" environment variable as password for configuration.
2016-02-16 16:32:05 +01:00
Nick Craig-Wood 0f73129ab7 dedupe command to deduplicate a remote. Useful with google drive - fixes #41 2016-01-31 16:09:42 +00:00
klauspost 14069fd8e6 Implement --delete-before, --delete-during, --delete-after - fixes #252. 2016-01-26 16:57:09 +00:00
klauspost 25f22ec561 Add "--ignore-existing" flag.
Add option to completely ignore existing files and not consider them for transfer.

Fixes #274
2016-01-08 08:20:04 +00:00
Nick Craig-Wood 5189231a34 Tweaks to rclone authorize
* Document the headless / remote setup procedure
  * Move Config constants into fs
  * Parse arguments in main for Authorize
2016-01-07 20:31:23 +00:00
klauspost bcbd30bb8a Add easier headless configuration.
This will allow setting up a remote with copy&paste of values to a headless machine. It will allow copy+pasting a token into the configuration.

This requires rclone to be on a machine with a proper browser. Custom client id and secrets are supported.

To test token generation, use `rclone auth "fs type"`.
2016-01-07 20:31:23 +00:00
klauspost b85ddc4e4f Extend CI tests to include formatting checks.
CI tests now tests 'go vet', 'go fmt' (via goimports) and golint.

Adds Travis experimental OSX support.
2015-11-03 13:50:29 +01:00
klauspost b872ff0237 Add option to disable server certificate verification.
The option name mirrors the 'wget' option (also `--no-check-certificate`). The cURL equivalent is called `--insecure`, which is a bit unclear.

Put in the "developers" section in documentation with proper warnings.

Fixes #168
2015-10-29 16:42:25 +01:00
Nick Craig-Wood 4e952af614 Allow spaces in remotes and check remote names for validity at creation time - fixes #171 2015-10-12 17:54:09 +01:00
Nick Craig-Wood a91bcaaeb0 Implement rsync like include and exclude - fixes #27
* Implement include/exclude
  * Implement rsync compatible file globbing
  * Implement command line filtering flags
    * --delete-excluded - Delete files on dest excluded from sync
    * --filter - Add a file-filtering rule
    * --filter-from - Read filtering patterns from a file
    * --exclude - Exclude files matching pattern
    * --exclude-from - Read exclude patterns from file
    * --include - Include files matching pattern
    * --include-from - Read include patterns from file
    * --files-from - Read list of source-file nam
    * --min-size - Don't transfer any file smaller than this in k or suffix k|M|G
    * --max-size - Don't transfer any file larger than this in k or suffix k|M|G
  * Document
2015-09-28 19:18:21 +01:00
Nick Craig-Wood e9c915e6fe Fix golint warnings 2015-09-22 18:47:16 +01:00
Nick Craig-Wood dd8717797e Implement --dump-headers and --dump-bodies debug flags 2015-09-08 21:02:48 +01:00
Nick Craig-Wood c98a51b26c Lightly obscure secrets 2015-09-01 22:33:34 +01:00
Nick Craig-Wood 0168f55f3e Switch to spf13 fork of pflag - fixes #116
This supports --long value as well as --long=value which is as
expected for a unix utility.
2015-08-29 18:14:24 +01:00
Nick Craig-Wood 771e60bd07 Show errors when reading the config file 2015-08-15 17:15:02 +01:00
Nick Craig-Wood e22b445cff Implement --size-only flag to sync on size not checksum & modtime - fixes #75 2015-06-06 08:49:01 +01:00
Alex Couper 7af1282375 Add --checksum flag to only discard transfers by MD5SUM - fixes #61
Useful for copying between backends where checksum fetching is fast,
ie any of s3, swift, drive or googlecloudstorage.
2015-06-05 19:46:03 +01:00
Nick Craig-Wood 1d254a3674 Implement --timeout and --conntimeout - fixes #54
NB dropbox still to do
2015-05-10 11:29:55 +01:00
Nick Craig-Wood 09d71239b6 Make file size render more neatly and prevent from being < 0 2015-03-15 15:27:55 +00:00
Nick Craig-Wood fe68737268 Fix niggles found by go vet 2015-02-28 15:35:54 +00:00
Nick Craig-Wood a287e3ced7 Implement --bwlimit to limit data transfer bandwidth 2015-02-27 15:03:47 +00:00
Nick Craig-Wood 8d1c0ad07c Fix config loop - fixes #25 2015-02-10 16:48:04 +00:00
Nick Craig-Wood 20ad96f3cd windows: Stop drive letters (eg C:) getting mixed up with remotes (eg drive:)
This was done by stopping the user configuring single letter remotes
and making sure we don't treat single letter remotes as a remote name,
but as a drive letter.
2015-02-07 22:32:51 +00:00
Nick Craig-Wood af9c447146 Fix "Couldn't find home directory" on OSX - fixes #15 2014-12-12 19:18:23 +00:00
Nick Craig-Wood 7ebf48ef42 Fix --dry-run not working and add tests for it - fixes #3 2014-06-26 15:33:06 +01:00
Nick Craig-Wood 1d67b014cb Make compatible with go 1.1 - fixes #1 2014-06-26 15:18:48 +01:00
Nick Craig-Wood 92ec29fe3f Factor the generic code into fs and add some more intefaces 2014-03-28 22:26:42 +00:00
Nick Craig-Wood a628bef9c2 Use pflag for gnu style flags 2014-03-27 16:55:29 +00:00
Nick Craig-Wood 1b3a49929b Add ability for FS to have a Config helper function run 2014-03-16 13:54:43 +00:00
Nick Craig-Wood 1ea9972be7 Set strict permissions on the config file 2014-03-16 13:53:51 +00:00
Nick Craig-Wood 7d786204b4 Add -config option to specify a config file 2014-03-15 17:01:13 +00:00
Nick Craig-Wood b3f1a45bbf Config fixes
* Fix empty config configuration
  * Alter menus when no remotes
  * Save config file after delete remote
2014-03-15 16:52:51 +00:00
Nick Craig-Wood 0a108832e2 Remove filesystem flags and put in config file with editor 2014-03-15 16:06:11 +00:00