diff --git a/README.md b/README.md index e9280204d..cc3ecdecb 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,13 @@ Rclone ====== +[![Logo](http://rclone.org/rclone-120x120.png)](http://rclone.org/) + Sync files and directories to and from - * Openstack Swift / Rackspace cloud files / Memset Memstore - * Amazon S3 * Google Drive + * Amazon S3 + * Openstack Swift / Rackspace cloud files / Memset Memstore * The local filesystem Features @@ -16,6 +18,11 @@ Features * Copy mode to just copy new/changed files * Sync mode to make a directory identical * Check mode to check all MD5SUMs + * Can sync to and from network, eg two different Drive accounts + +Home page + + * http://rclone.org/ Install ------- @@ -30,8 +37,9 @@ Or alternatively if you have Go installed use go get github.com/ncw/rclone -and this will build the binary in `$GOPATH/bin`. You can then modify -the source and submit patches. +and this will build the binary in `$GOPATH/bin`. + +You can then modify the source and submit patches. Configure --------- @@ -75,9 +83,6 @@ Choose a number from below, or type in your own value * US Region, Northern Virginia only. * Leave location constraint empty. 2) https://s3-external-1.amazonaws.com - * US West (Oregon) Region - * Needs location constraint us-west-2. - 3) https://s3-us-west-2.amazonaws.com [snip] * South America (Sao Paulo) Region * Needs location constraint sa-east-1. @@ -89,8 +94,6 @@ Choose a number from below, or type in your own value 1) * US West (Oregon) Region. 2) us-west-2 - * US West (Northern California) Region. - 3) us-west-1 [snip] * South America (Sao Paulo) Region. 9) sa-east-1 @@ -240,11 +243,57 @@ Google drive Paths are specified as drive://path Drive paths may be as deep as required. -FIXME describe how to set up initially +The initial setup for drive involves getting a token from Google drive +which you need to do in your browser. The `rclone config` walks you +through it. -So to copy a local directory to a drive directory called backup +Here is an example of how to make a remote called `drv` -rclone sync /home/source s3://backup +``` +$ ./rclone config +n) New remote +d) Delete remote +q) Quit config +e/n/d/q> n +name> drv +What type of source is it? +Choose a number from below + 1) swift + 2) s3 + 3) local + 4) drive +type> 4 +Google Application Client Id - leave blank to use rclone's. +client_id> +Google Application Client Secret - leave blank to use rclone's. +client_secret> +Remote config +Go to the following link in your browser +https://accounts.google.com/o/oauth2/auth?access_type=&approval_prompt=&client_id=XXXXXXXXXXXX.apps.googleusercontent.com&redirect_uri=urn%3XXXXX%3Awg%3Aoauth%3XX.0%3Aoob&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive&state=state +Log in, then type paste the token that is returned in the browser here +Enter verification code> X/XXXXXXXXXXXXXXXXXX-XXXXXXXXX.XXXXXXXXX-XXXXX_XXXXXXX_XXXXXXX +-------------------- +[drv] +client_id = +client_secret = +token = {"AccessToken":"xxxx.xxxxxxx_xxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"1/xxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxx","Expiry":"2014-03-16T13:57:58.955387075Z","Extra":null} +-------------------- +y) Yes this is OK +e) Edit this remote +d) Delete this remote +y/e/d> y +``` + +You can then use it like this + + rclone lsd drv:// + rclone ls drv:// + +To copy a local directory to a drive directory called backup + + rclone copy /home/source drv://backup + +Google drive stores modification times accurate to 1 ms. License ------- @@ -255,25 +304,27 @@ COPYING file included in this package). Bugs ---- -Save the google drive auth in this config file too! - -Describe how to do the google auth. + * Doesn't sync individual files yet, only directories. + * Drive: Sometimes get: Failed to copy: Upload failed: googleapi: Error 403: Rate Limit Exceeded + * quota is 100.0 requests/second/user + * Empty directories left behind with Local and Drive + * eg purging a local directory with subdirectories doesn't work Contact and support ------------------- The project website is at: -- https://github.com/ncw/rclone + * https://github.com/ncw/rclone There you can file bug reports, ask for help or contribute patches. Authors ------- -- Nick Craig-Wood + * Nick Craig-Wood Contributors ------------ -- Your name goes here! + * Your name goes here! diff --git a/drive/drive.go b/drive/drive.go index 68f5bb3dc..10e36de2a 100644 --- a/drive/drive.go +++ b/drive/drive.go @@ -9,12 +9,6 @@ package drive // FIXME list directory should list to channel for concurrency not // append to array -// FIXME perhaps have a drive setup mode where we ask for all the -// params interactively and store them all in one file -// - don't need to store client* apparently - -// NB permissions of token file is too open - // FIXME need to deal with some corner cases // * multiple files with the same name // * files can be in multiple directories @@ -22,14 +16,13 @@ package drive // * files with / in name import ( - "errors" + "encoding/json" "flag" "fmt" "io" "log" "mime" "net/http" - "os" "path" "strings" "sync" @@ -40,36 +33,106 @@ import ( "github.com/ncw/rclone/fs" ) +// Constants +const ( + rcloneClientId = "202264815644.apps.googleusercontent.com" + rcloneClientSecret = "X4Z3ca8xfWDb1Voo-F9a7ZxJ" + driveFolderType = "application/vnd.google-apps.folder" +) + +// Globals +var ( + // Flags + driveFullList = flag.Bool("drive-full-list", true, "Use a full listing for directory list. More data but usually quicker.") +) + // Register with Fs func init() { fs.Register(&fs.FsInfo{ - Name: "drive", - NewFs: NewFs, + Name: "drive", + NewFs: NewFs, + Config: Config, Options: []fs.Option{{ Name: "client_id", - Help: "Google Application Client Id.", - Examples: []fs.OptionExample{{ - Value: "202264815644.apps.googleusercontent.com", - Help: "rclone's client id - use this or your own if you want", - }}, + Help: "Google Application Client Id - leave blank to use rclone's.", }, { Name: "client_secret", - Help: "Google Application Client Secret.", - Examples: []fs.OptionExample{{ - Value: "X4Z3ca8xfWDb1Voo-F9a7ZxJ", - Help: "rclone's client secret - use this or your own if you want", - }}, - }, { - Name: "token_file", - Help: "Path to store token file.", - Examples: []fs.OptionExample{{ - Value: path.Join(fs.HomeDir, ".gdrive-token-file"), - Help: "Suggested path for token file", - }}, + Help: "Google Application Client Secret - leave blank to use rclone's.", }}, }) } +// Configuration helper - called after the user has put in the defaults +func Config(name string) { + // See if already have a token + tokenString := fs.ConfigFile.MustValue(name, "token") + if tokenString != "" { + fmt.Printf("Already have a drive token - refresh?\n") + if !fs.Confirm() { + return + } + } + + // Get a drive transport + t, err := newDriveTransport(name) + if err != nil { + log.Fatalf("Couldn't make drive transport: %v", err) + } + + // Generate a URL for the user to visit for authorization. + authUrl := t.Config.AuthCodeURL("state") + fmt.Printf("Go to the following link in your browser\n") + fmt.Printf("%s\n", authUrl) + fmt.Printf("Log in, then type paste the token that is returned in the browser here\n") + + // Read the code, and exchange it for a token. + fmt.Printf("Enter verification code> ") + authCode := fs.ReadLine() + _, err = t.Exchange(authCode) + if err != nil { + log.Fatalf("Failed to get token: %v", err) + } + +} + +// A token cache to save the token in the config file section named +type tokenCache string + +// Get the token from the config file - returns an error if it isn't present +func (name tokenCache) Token() (*oauth.Token, error) { + tokenString, err := fs.ConfigFile.GetValue(string(name), "token") + if err != nil { + return nil, err + } + if tokenString == "" { + return nil, fmt.Errorf("Empty token found - please reconfigure") + } + token := new(oauth.Token) + err = json.Unmarshal([]byte(tokenString), token) + if err != nil { + return nil, err + } + return token, nil + +} + +// Save the token to the config file +// +// This saves the config file if it changes +func (name tokenCache) PutToken(token *oauth.Token) error { + tokenBytes, err := json.Marshal(token) + if err != nil { + return err + } + tokenString := string(tokenBytes) + old := fs.ConfigFile.MustValue(string(name), "token") + if tokenString != old { + fs.ConfigFile.SetValue(string(name), "token", tokenString) + fs.SaveConfig() + } + return nil +} + // FsDrive represents a remote drive server type FsDrive struct { svc *drive.Service // the connection to the drive server @@ -141,19 +204,6 @@ func (m *dirCache) Flush() { // ------------------------------------------------------------ -// Constants -const ( - // defaultDriveTokenFile = ".google-drive-token" // FIXME root in home directory somehow - driveFolderType = "application/vnd.google-apps.folder" -) - -// Globals -var ( - // Flags - driveAuthCode = flag.String("drive-auth-code", "", "Pass in when requested to make the drive token file.") - driveFullList = flag.Bool("drive-full-list", true, "Use a full listing for directory list. More data but usually quicker.") -) - // String converts this FsDrive to a string func (f *FsDrive) String() string { return fmt.Sprintf("Google drive root '%s'", f.root) @@ -214,39 +264,15 @@ OUTER: return } -// Ask the user for a new auth -func MakeNewToken(t *oauth.Transport) error { - if *driveAuthCode == "" { - // Generate a URL to visit for authorization. - authUrl := t.Config.AuthCodeURL("state") - fmt.Fprintf(os.Stderr, "Go to the following link in your browser\n") - fmt.Fprintf(os.Stderr, "%s\n", authUrl) - fmt.Fprintf(os.Stderr, "Log in, then re-run this program with the -drive-auth-code parameter\n") - fmt.Fprintf(os.Stderr, "You only need this parameter once until the drive token file has been created\n") - return errors.New("Re-run with --drive-auth-code") - } - - // Read the code, and exchange it for a token. - //fmt.Printf("Enter verification code: ") - //var code string - //fmt.Scanln(&code) - _, err := t.Exchange(*driveAuthCode) - return err -} - -// NewFs contstructs an FsDrive from the path, container:path -func NewFs(name, path string) (fs.Fs, error) { +// Makes a new drive transport from the config +func newDriveTransport(name string) (*oauth.Transport, error) { clientId := fs.ConfigFile.MustValue(name, "client_id") if clientId == "" { - return nil, errors.New("client_id not found") + clientId = rcloneClientId } clientSecret := fs.ConfigFile.MustValue(name, "client_secret") if clientSecret == "" { - return nil, errors.New("client_secret not found") - } - tokenFile := fs.ConfigFile.MustValue(name, "token_file") - if tokenFile == "" { - return nil, errors.New("token-file not found") + clientSecret = rcloneClientSecret } // Settings for authorization. @@ -257,7 +283,22 @@ func NewFs(name, path string) (fs.Fs, error) { RedirectURL: "urn:ietf:wg:oauth:2.0:oob", AuthURL: "https://accounts.google.com/o/oauth2/auth", TokenURL: "https://accounts.google.com/o/oauth2/token", - TokenCache: oauth.CacheFile(tokenFile), + TokenCache: tokenCache(name), + } + + t := &oauth.Transport{ + Config: driveConfig, + Transport: http.DefaultTransport, + } + + return t, nil +} + +// NewFs contstructs an FsDrive from the path, container:path +func NewFs(name, path string) (fs.Fs, error) { + t, err := newDriveTransport(name) + if err != nil { + return nil, err } root, err := parseDrivePath(path) @@ -266,22 +307,10 @@ func NewFs(name, path string) (fs.Fs, error) { } f := &FsDrive{root: root, dirCache: newDirCache()} - t := &oauth.Transport{ - Config: driveConfig, - Transport: http.DefaultTransport, - } - // Try to pull the token from the cache; if this fails, we need to get one. - token, err := driveConfig.TokenCache.Token() + token, err := t.Config.TokenCache.Token() if err != nil { - err := MakeNewToken(t) - if err != nil { - return nil, fmt.Errorf("Failed to authorise: %s", err) - } - } else { - if *driveAuthCode != "" { - return nil, fmt.Errorf("Only supply -drive-auth-code once") - } + return nil, fmt.Errorf("Failed to get token: %s", err) } t.Token = token diff --git a/notes.txt b/notes.txt index ee1e84547..2c4334de5 100644 --- a/notes.txt +++ b/notes.txt @@ -4,7 +4,7 @@ Todo * FIXME: ls without an argument for buckets/containers? * FIXME: More -dry-run checks for object transfer * Might be quicker to check md5sums first? for swift <-> swift certainly, and maybe for small files - * Ignoring the pseudo directories + * swift: Ignoring the pseudo directories * if object.PseudoDirectory { * fmt.Printf("%9s %19s %s\n", "Directory", "-", fs.Remote()) * Make Account wrapper @@ -20,20 +20,6 @@ Todo * Add max object size to fs metadata - 5GB for swift, infinite for local, ? for s3 * tie into -max-size flag -Drive - * Do we need the secrets or just the code? If just the code then - can make a web service which does the request on the clients - behalf so don't need to expose the client secrets - * Apparently we don't need -drive-client-id or -drive-client-secret once we have a token - * Make a cgi which we send the user to - * It has the client secrets - * It gets google to authenticate - * It receives the token back - * It displays the token to the user to paste in to the code - * Should be https really - * Sometimes get: Failed to copy: Upload failed: googleapi: Error 403: Rate Limit Exceeded - * quota is 100.0 requests/second/user - Ideas * could do encryption - put IV into metadata? * optimise remote copy container to another container using remote @@ -45,7 +31,6 @@ Ideas * Google cloud storage: https://developers.google.com/storage/ * rsync over ssh * dropbox: https://github.com/nickoneill/go-dropbox (no MD5s) - * grive seems to have its secrets in the source code which would make things easier! Need to make directory objects otherwise can't upload an empty directory * Or could upload empty directories only? @@ -57,17 +42,6 @@ s3 * Otherwise can set metadata * Returns etag and last modified in bucket list - Bugs -local & drive need to delete directories - -2013/01/18 16:31:32 Waiting for deletions to finish -2013/01/18 16:31:32 z3: FIXME Skipping directory -2013/01/18 16:31:32 z3/x: Deleted -2013/01/18 16:31:32 Deleting path -2013/01/18 16:31:32 Rmdir failed: remove z3: directory not empty - ------------------------------------------------------------- - Non verbose - not sure number transferred got counted up? CHECK