community
/
rclone
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

drive: Rework token aquisition into config framework and store token in config file

This commit is contained in:
Nick Craig-Wood 2014-03-16 14:01:17 +00:00
parent 1b3a49929b
commit b4dd693d23
3 changed files with 182 additions and 128 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

87
README.md
View File

@ -1,11 +1,13 @@
Rclone Rclone
====== ======
[![Logo](http://rclone.org/rclone-120x120.png)](http://rclone.org/)
Sync files and directories to and from Sync files and directories to and from
* Openstack Swift / Rackspace cloud files / Memset Memstore
* Amazon S3
* Google Drive * Google Drive
* Amazon S3
* Openstack Swift / Rackspace cloud files / Memset Memstore
* The local filesystem * The local filesystem
Features Features
@ -16,6 +18,11 @@ Features
* Copy mode to just copy new/changed files * Copy mode to just copy new/changed files
* Sync mode to make a directory identical * Sync mode to make a directory identical
* Check mode to check all MD5SUMs * Check mode to check all MD5SUMs
* Can sync to and from network, eg two different Drive accounts
Home page
* http://rclone.org/
Install Install
------- -------
@ -30,8 +37,9 @@ Or alternatively if you have Go installed use
go get github.com/ncw/rclone go get github.com/ncw/rclone
and this will build the binary in `$GOPATH/bin`. You can then modify and this will build the binary in `$GOPATH/bin`.
the source and submit patches.
You can then modify the source and submit patches.
Configure Configure
--------- ---------
@ -75,9 +83,6 @@ Choose a number from below, or type in your own value
* US Region, Northern Virginia only. * US Region, Northern Virginia only.
* Leave location constraint empty. * Leave location constraint empty.
2) https://s3-external-1.amazonaws.com 2) https://s3-external-1.amazonaws.com
* US West (Oregon) Region
* Needs location constraint us-west-2.
3) https://s3-us-west-2.amazonaws.com
[snip] [snip]
* South America (Sao Paulo) Region * South America (Sao Paulo) Region
* Needs location constraint sa-east-1. * Needs location constraint sa-east-1.
@ -89,8 +94,6 @@ Choose a number from below, or type in your own value
1) 1)
* US West (Oregon) Region. * US West (Oregon) Region.
2) us-west-2 2) us-west-2
* US West (Northern California) Region.
3) us-west-1
[snip] [snip]
* South America (Sao Paulo) Region. * South America (Sao Paulo) Region.
9) sa-east-1 9) sa-east-1
@ -240,11 +243,57 @@ Google drive
Paths are specified as drive://path Drive paths may be as deep as required. Paths are specified as drive://path Drive paths may be as deep as required.
FIXME describe how to set up initially The initial setup for drive involves getting a token from Google drive
which you need to do in your browser. The `rclone config` walks you
through it.
So to copy a local directory to a drive directory called backup Here is an example of how to make a remote called `drv`
rclone sync /home/source s3://backup ```
$ ./rclone config
n) New remote
d) Delete remote
q) Quit config
e/n/d/q> n
name> drv
What type of source is it?
Choose a number from below
1) swift
2) s3
3) local
4) drive
type> 4
Google Application Client Id - leave blank to use rclone's.
client_id>
Google Application Client Secret - leave blank to use rclone's.
client_secret>
Remote config
Go to the following link in your browser
https://accounts.google.com/o/oauth2/auth?access_type=&approval_prompt=&client_id=XXXXXXXXXXXX.apps.googleusercontent.com&redirect_uri=urn%3XXXXX%3Awg%3Aoauth%3XX.0%3Aoob&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive&state=state
Log in, then type paste the token that is returned in the browser here
Enter verification code> X/XXXXXXXXXXXXXXXXXX-XXXXXXXXX.XXXXXXXXX-XXXXX_XXXXXXX_XXXXXXX
--------------------
[drv]
client_id =
client_secret =
token = {"AccessToken":"xxxx.xxxxxxx_xxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"1/xxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxx","Expiry":"2014-03-16T13:57:58.955387075Z","Extra":null}
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
```
You can then use it like this
rclone lsd drv://
rclone ls drv://
To copy a local directory to a drive directory called backup
rclone copy /home/source drv://backup
Google drive stores modification times accurate to 1 ms.
License License
------- -------
@ -255,25 +304,27 @@ COPYING file included in this package).
Bugs Bugs
---- ----
Save the google drive auth in this config file too! * Doesn't sync individual files yet, only directories.
* Drive: Sometimes get: Failed to copy: Upload failed: googleapi: Error 403: Rate Limit Exceeded
Describe how to do the google auth. * quota is 100.0 requests/second/user
* Empty directories left behind with Local and Drive
* eg purging a local directory with subdirectories doesn't work
Contact and support Contact and support
------------------- -------------------
The project website is at: The project website is at:
- https://github.com/ncw/rclone * https://github.com/ncw/rclone
There you can file bug reports, ask for help or contribute patches. There you can file bug reports, ask for help or contribute patches.
Authors Authors
------- -------
- Nick Craig-Wood <nick@craig-wood.com> * Nick Craig-Wood <nick@craig-wood.com>
Contributors Contributors
------------ ------------
- Your name goes here! * Your name goes here!

195
drive/drive.go
View File

@ -9,12 +9,6 @@ package drive
// FIXME list directory should list to channel for concurrency not // FIXME list directory should list to channel for concurrency not
// append to array // append to array
// FIXME perhaps have a drive setup mode where we ask for all the
// params interactively and store them all in one file
// - don't need to store client* apparently
// NB permissions of token file is too open
// FIXME need to deal with some corner cases // FIXME need to deal with some corner cases
// * multiple files with the same name // * multiple files with the same name
// * files can be in multiple directories // * files can be in multiple directories
@ -22,14 +16,13 @@ package drive
// * files with / in name // * files with / in name
import ( import (
"errors" "encoding/json"
"flag" "flag"
"fmt" "fmt"
"io" "io"
"log" "log"
"mime" "mime"
"net/http" "net/http"
"os"
"path" "path"
"strings" "strings"
"sync" "sync"
@ -40,36 +33,106 @@ import (
"github.com/ncw/rclone/fs" "github.com/ncw/rclone/fs"
) )
// Constants
const (
rcloneClientId = "202264815644.apps.googleusercontent.com"
rcloneClientSecret = "X4Z3ca8xfWDb1Voo-F9a7ZxJ"
driveFolderType = "application/vnd.google-apps.folder"
)
// Globals
var (
// Flags
driveFullList = flag.Bool("drive-full-list", true, "Use a full listing for directory list. More data but usually quicker.")
)
// Register with Fs // Register with Fs
func init() { func init() {
fs.Register(&fs.FsInfo{ fs.Register(&fs.FsInfo{
Name: "drive", Name: "drive",
NewFs: NewFs, NewFs: NewFs,
Config: Config,
Options: []fs.Option{{ Options: []fs.Option{{
Name: "client_id", Name: "client_id",
Help: "Google Application Client Id.", Help: "Google Application Client Id - leave blank to use rclone's.",
Examples: []fs.OptionExample{{
Value: "202264815644.apps.googleusercontent.com",
Help: "rclone's client id - use this or your own if you want",
}},
}, { }, {
Name: "client_secret", Name: "client_secret",
Help: "Google Application Client Secret.", Help: "Google Application Client Secret - leave blank to use rclone's.",
Examples: []fs.OptionExample{{
Value: "X4Z3ca8xfWDb1Voo-F9a7ZxJ",
Help: "rclone's client secret - use this or your own if you want",
}},
}, {
Name: "token_file",
Help: "Path to store token file.",
Examples: []fs.OptionExample{{
Value: path.Join(fs.HomeDir, ".gdrive-token-file"),
Help: "Suggested path for token file",
}},
}}, }},
}) })
} }
// Configuration helper - called after the user has put in the defaults
func Config(name string) {
// See if already have a token
tokenString := fs.ConfigFile.MustValue(name, "token")
if tokenString != "" {
fmt.Printf("Already have a drive token - refresh?\n")
if !fs.Confirm() {
return
}
}
// Get a drive transport
t, err := newDriveTransport(name)
if err != nil {
log.Fatalf("Couldn't make drive transport: %v", err)
}
// Generate a URL for the user to visit for authorization.
authUrl := t.Config.AuthCodeURL("state")
fmt.Printf("Go to the following link in your browser\n")
fmt.Printf("%s\n", authUrl)
fmt.Printf("Log in, then type paste the token that is returned in the browser here\n")
// Read the code, and exchange it for a token.
fmt.Printf("Enter verification code> ")
authCode := fs.ReadLine()
_, err = t.Exchange(authCode)
if err != nil {
log.Fatalf("Failed to get token: %v", err)
}
}
// A token cache to save the token in the config file section named
type tokenCache string
// Get the token from the config file - returns an error if it isn't present
func (name tokenCache) Token() (*oauth.Token, error) {
tokenString, err := fs.ConfigFile.GetValue(string(name), "token")
if err != nil {
return nil, err
}
if tokenString == "" {
return nil, fmt.Errorf("Empty token found - please reconfigure")
}
token := new(oauth.Token)
err = json.Unmarshal([]byte(tokenString), token)
if err != nil {
return nil, err
}
return token, nil
}
// Save the token to the config file
//
// This saves the config file if it changes
func (name tokenCache) PutToken(token *oauth.Token) error {
tokenBytes, err := json.Marshal(token)
if err != nil {
return err
}
tokenString := string(tokenBytes)
old := fs.ConfigFile.MustValue(string(name), "token")
if tokenString != old {
fs.ConfigFile.SetValue(string(name), "token", tokenString)
fs.SaveConfig()
}
return nil
}
// FsDrive represents a remote drive server // FsDrive represents a remote drive server
type FsDrive struct { type FsDrive struct {
svc *drive.Service // the connection to the drive server svc *drive.Service // the connection to the drive server
@ -141,19 +204,6 @@ func (m *dirCache) Flush() {
// ------------------------------------------------------------ // ------------------------------------------------------------
// Constants
const (
// defaultDriveTokenFile = ".google-drive-token" // FIXME root in home directory somehow
driveFolderType = "application/vnd.google-apps.folder"
)
// Globals
var (
// Flags
driveAuthCode = flag.String("drive-auth-code", "", "Pass in when requested to make the drive token file.")
driveFullList = flag.Bool("drive-full-list", true, "Use a full listing for directory list. More data but usually quicker.")
)
// String converts this FsDrive to a string // String converts this FsDrive to a string
func (f *FsDrive) String() string { func (f *FsDrive) String() string {
return fmt.Sprintf("Google drive root '%s'", f.root) return fmt.Sprintf("Google drive root '%s'", f.root)
@ -214,39 +264,15 @@ OUTER:
return return
} }
// Ask the user for a new auth // Makes a new drive transport from the config
func MakeNewToken(t *oauth.Transport) error { func newDriveTransport(name string) (*oauth.Transport, error) {
if *driveAuthCode == "" {
// Generate a URL to visit for authorization.
authUrl := t.Config.AuthCodeURL("state")
fmt.Fprintf(os.Stderr, "Go to the following link in your browser\n")
fmt.Fprintf(os.Stderr, "%s\n", authUrl)
fmt.Fprintf(os.Stderr, "Log in, then re-run this program with the -drive-auth-code parameter\n")
fmt.Fprintf(os.Stderr, "You only need this parameter once until the drive token file has been created\n")
return errors.New("Re-run with --drive-auth-code")
}
// Read the code, and exchange it for a token.
//fmt.Printf("Enter verification code: ")
//var code string
//fmt.Scanln(&code)
_, err := t.Exchange(*driveAuthCode)
return err
}
// NewFs contstructs an FsDrive from the path, container:path
func NewFs(name, path string) (fs.Fs, error) {
clientId := fs.ConfigFile.MustValue(name, "client_id") clientId := fs.ConfigFile.MustValue(name, "client_id")
if clientId == "" { if clientId == "" {
return nil, errors.New("client_id not found") clientId = rcloneClientId
} }
clientSecret := fs.ConfigFile.MustValue(name, "client_secret") clientSecret := fs.ConfigFile.MustValue(name, "client_secret")
if clientSecret == "" { if clientSecret == "" {
return nil, errors.New("client_secret not found") clientSecret = rcloneClientSecret
}
tokenFile := fs.ConfigFile.MustValue(name, "token_file")
if tokenFile == "" {
return nil, errors.New("token-file not found")
} }
// Settings for authorization. // Settings for authorization.
@ -257,7 +283,22 @@ func NewFs(name, path string) (fs.Fs, error) {
RedirectURL: "urn:ietf:wg:oauth:2.0:oob", RedirectURL: "urn:ietf:wg:oauth:2.0:oob",
AuthURL: "https://accounts.google.com/o/oauth2/auth", AuthURL: "https://accounts.google.com/o/oauth2/auth",
TokenURL: "https://accounts.google.com/o/oauth2/token", TokenURL: "https://accounts.google.com/o/oauth2/token",
TokenCache: oauth.CacheFile(tokenFile), TokenCache: tokenCache(name),
}
t := &oauth.Transport{
Config: driveConfig,
Transport: http.DefaultTransport,
}
return t, nil
}
// NewFs contstructs an FsDrive from the path, container:path
func NewFs(name, path string) (fs.Fs, error) {
t, err := newDriveTransport(name)
if err != nil {
return nil, err
} }
root, err := parseDrivePath(path) root, err := parseDrivePath(path)
@ -266,22 +307,10 @@ func NewFs(name, path string) (fs.Fs, error) {
} }
f := &FsDrive{root: root, dirCache: newDirCache()} f := &FsDrive{root: root, dirCache: newDirCache()}
t := &oauth.Transport{
Config: driveConfig,
Transport: http.DefaultTransport,
}
// Try to pull the token from the cache; if this fails, we need to get one. // Try to pull the token from the cache; if this fails, we need to get one.
token, err := driveConfig.TokenCache.Token() token, err := t.Config.TokenCache.Token()
if err != nil { if err != nil {
err := MakeNewToken(t) return nil, fmt.Errorf("Failed to get token: %s", err)
if err != nil {
return nil, fmt.Errorf("Failed to authorise: %s", err)
}
} else {
if *driveAuthCode != "" {
return nil, fmt.Errorf("Only supply -drive-auth-code once")
}
} }
t.Token = token t.Token = token

28
notes.txt
View File

@ -4,7 +4,7 @@ Todo
* FIXME: ls without an argument for buckets/containers? * FIXME: ls without an argument for buckets/containers?
* FIXME: More -dry-run checks for object transfer * FIXME: More -dry-run checks for object transfer
* Might be quicker to check md5sums first? for swift <-> swift certainly, and maybe for small files * Might be quicker to check md5sums first? for swift <-> swift certainly, and maybe for small files
* Ignoring the pseudo directories * swift: Ignoring the pseudo directories
* if object.PseudoDirectory { * if object.PseudoDirectory {
* fmt.Printf("%9s %19s %s\n", "Directory", "-", fs.Remote()) * fmt.Printf("%9s %19s %s\n", "Directory", "-", fs.Remote())
* Make Account wrapper * Make Account wrapper
@ -20,20 +20,6 @@ Todo
* Add max object size to fs metadata - 5GB for swift, infinite for local, ? for s3 * Add max object size to fs metadata - 5GB for swift, infinite for local, ? for s3
* tie into -max-size flag * tie into -max-size flag
Drive
* Do we need the secrets or just the code? If just the code then
can make a web service which does the request on the clients
behalf so don't need to expose the client secrets
* Apparently we don't need -drive-client-id or -drive-client-secret once we have a token
* Make a cgi which we send the user to
* It has the client secrets
* It gets google to authenticate
* It receives the token back
* It displays the token to the user to paste in to the code
* Should be https really
* Sometimes get: Failed to copy: Upload failed: googleapi: Error 403: Rate Limit Exceeded
* quota is 100.0 requests/second/user
Ideas Ideas
* could do encryption - put IV into metadata? * could do encryption - put IV into metadata?
* optimise remote copy container to another container using remote * optimise remote copy container to another container using remote
@ -45,7 +31,6 @@ Ideas
* Google cloud storage: https://developers.google.com/storage/ * Google cloud storage: https://developers.google.com/storage/
* rsync over ssh * rsync over ssh
* dropbox: https://github.com/nickoneill/go-dropbox (no MD5s) * dropbox: https://github.com/nickoneill/go-dropbox (no MD5s)
* grive seems to have its secrets in the source code which would make things easier!
Need to make directory objects otherwise can't upload an empty directory Need to make directory objects otherwise can't upload an empty directory
* Or could upload empty directories only? * Or could upload empty directories only?
@ -57,17 +42,6 @@ s3
* Otherwise can set metadata * Otherwise can set metadata
* Returns etag and last modified in bucket list * Returns etag and last modified in bucket list
Bugs Bugs
local & drive need to delete directories
2013/01/18 16:31:32 Waiting for deletions to finish
2013/01/18 16:31:32 z3: FIXME Skipping directory
2013/01/18 16:31:32 z3/x: Deleted
2013/01/18 16:31:32 Deleting path
2013/01/18 16:31:32 Rmdir failed: remove z3: directory not empty
------------------------------------------------------------
Non verbose - not sure number transferred got counted up? CHECK Non verbose - not sure number transferred got counted up? CHECK