s3: add docs on how to add a new provider

This commit is contained in:
Nick Craig-Wood 2023-09-21 12:35:23 +01:00
parent eb3082a1eb
commit 9e80d48b03
2 changed files with 52 additions and 7 deletions

View File

@ -484,6 +484,46 @@ alphabetical order of full name of remote (e.g. `drive` is ordered as
Once you've written the docs, run `make serve` and check they look OK Once you've written the docs, run `make serve` and check they look OK
in the web browser and the links (internal and external) all work. in the web browser and the links (internal and external) all work.
## Adding a new s3 provider
It is quite easy to add a new S3 provider to rclone.
You'll need to modify the following files
- `backend/s3/s3.go`
- Add the provider to `providerOption` at the top of the file
- Add endpoints and other config for your provider gated on the provider in `fs.RegInfo`.
- Exclude your provider from genric config questions (eg `region` and `endpoint).
- Add the provider to the `setQuirks` function - see the documentation there.
- `docs/content/s3.md`
- Add the provider at the top of the page.
- Add a section about the provider linked from there.
- Add a transcript of a trial `rclone config` session
- Edit the transcript to remove things which might change in subsequent versions
- **Do not** alter or add to the autogenerated parts of `s3.md`
- **Do not** run `make backenddocs` or `bin/make_backend_docs.py s3`
- `README.md` - this is the home page in github
- Add the provider and a link to the section you wrote in `docs/contents/s3.md`
- `docs/content/_index.md` - this is the home page of rclone.org
- Add the provider and a link to the section you wrote in `docs/contents/s3.md`
When adding the provider, endpoints, quirks, docs etc keep them in
alphabetical order by `Provider` name, but with `AWS` first and
`Other` last.
Once you've written the docs, run `make serve` and check they look OK
in the web browser and the links (internal and external) all work.
Once you've written the code, test `rclone config` works to your
satisfaction, and check the integration tests work `go test -v -remote
NewS3Provider:`. You may need to adjust the quirks to get them to
pass. Some providers just can't pass the tests with control characters
in the names so if these fail and the provider doesn't support
`urlEncodeListings` in the quirks then ignore them. Note that the
`SetTier` test may also fail on non AWS providers.
For an example of adding an s3 provider see [eb3082a1](https://github.com/rclone/rclone/commit/eb3082a1ebdb76d5625f14cedec3f5154a5e7b10).
## Writing a plugin ## ## Writing a plugin ##
New features (backends, commands) can also be added "out-of-tree", through Go plugins. New features (backends, commands) can also be added "out-of-tree", through Go plugins.

View File

@ -70,6 +70,9 @@ import (
// NB if you add a new provider here, then add it in the setQuirks // NB if you add a new provider here, then add it in the setQuirks
// function and set the correct quirks. Test the quirks are correct by // function and set the correct quirks. Test the quirks are correct by
// running the integration tests "go test -v -remote NewS3Provider:". // running the integration tests "go test -v -remote NewS3Provider:".
//
// See https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#adding-a-new-s3-provider
// for full information about how to add a new s3 provider.
var providerOption = fs.Option{ var providerOption = fs.Option{
Name: fs.ConfigProvider, Name: fs.ConfigProvider,
Help: "Choose your S3 provider.", Help: "Choose your S3 provider.",
@ -2998,15 +3001,17 @@ func setEndpointValueForIDriveE2(m configmap.Mapper) (err error) {
// There should be no testing against opt.Provider anywhere in the // There should be no testing against opt.Provider anywhere in the
// code except in here to localise the setting of the quirks. // code except in here to localise the setting of the quirks.
// //
// These should be differences from AWS S3 // Run the integration tests to check you have the quirks correct.
//
// go test -v -remote NewS3Provider:
func setQuirks(opt *Options) { func setQuirks(opt *Options) {
var ( var (
listObjectsV2 = true listObjectsV2 = true // Always use ListObjectsV2 instead of ListObjects
virtualHostStyle = true virtualHostStyle = true // Use bucket.provider.com instead of putting the bucket in the URL
urlEncodeListings = true urlEncodeListings = true // URL encode the listings to help with control characters
useMultipartEtag = true useMultipartEtag = true // Set if Etags for multpart uploads are compatible with AWS
useAcceptEncodingGzip = true useAcceptEncodingGzip = true // Set Accept-Encoding: gzip
mightGzip = true // assume all providers might gzip until proven otherwise mightGzip = true // assume all providers might use content encoding gzip until proven otherwise
) )
switch opt.Provider { switch opt.Provider {
case "AWS": case "AWS":