Commit Graph

22 Commits

Author SHA1 Message Date
Lunny Xiao fb7b743bd0
Azure blob storage support (#30995)
This PR implemented object storages(LFS/Packages/Attachments and etc.)
for Azure Blob Storage. It depends on azure official golang SDK and can
support both the azure blob storage cloud service and azurite mock
server.

Replace #25458
Fix #22527 

- [x] CI Tests
- [x] integration test, MSSQL integration tests will now based on
azureblob
  - [x] unit test 
- [x] CLI Migrate Storage
- [x] Documentation for configuration added

------

TODO (other PRs):
- [ ] Improve performance of `blob download`.

---------

Co-authored-by: yp05327 <576951401@qq.com>
2024-05-30 07:33:50 +00:00
dicarne d0d6aad85f
Supports forced use of S3 virtual-hosted style (#30969)
Add a configuration item to enable S3 virtual-hosted style (V2) to solve
the problem caused by some S3 service providers not supporting path
style (V1).
2024-05-15 08:56:17 -05:00
Chongyi Zheng e80466f734
Resolve lint for unused parameter and unnecessary type arguments (#30750)
Resolve all cases for `unused parameter` and `unnecessary type
arguments`

Related: #30729

---------

Co-authored-by: Giteabot <teabot@gitea.io>
2024-04-29 08:47:56 +00:00
wxiaoguang bbe5cd7c92
Refactor startup deprecation messages (#30305)
It doesn't change logic, it only does:

1. Rename the variable and function names
2. Use more consistent format when mentioning config section&key
3. Improve some messages
2024-04-07 01:11:25 +00:00
wxiaoguang 6d34ce25b1
Do not allow different storage configurations to point to the same directory (#30169)
Replace #29171
2024-03-31 03:03:24 +00:00
Lunny Xiao 92fda9c5a2
Disallow duplicate storage paths (#26484)
Replace #26380
2024-02-09 14:06:03 +00:00
Lunny Xiao d519a39302
Support storage base path as prefix (#27827)
This PR adds a prefix path for all minio storage and override base path
will override the path.
The previous behavior is undefined officially, so it will be marked as
breaking.
2023-11-01 19:17:18 +08:00
Lunny Xiao 389fb754a2
Fix storage path logic especially for relative paths (#26441)
This PR rewrites the function `getStorage` and make it more clear.

Include tests from #26435, thanks @earl-warren

---------

Co-authored-by: Earl Warren <contact@earl-warren.org>
2023-08-13 22:09:25 +02:00
Lunny Xiao 96f151392f
Fix the wrong derive path (#26271)
This PR will fix #26264, caused by #23911.

The package configuration derive is totally wrong when storage type is
local in that PR.

This PR fixed the inherit logic when storage type is local with some
unit tests.

---------

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2023-08-04 03:41:16 +00:00
Lunny Xiao d6dd6d641b
Fix all possible setting error related storages and added some tests (#23911)
Follow up #22405

Fix #20703 

This PR rewrites storage configuration read sequences with some breaks
and tests. It becomes more strict than before and also fixed some
inherit problems.

- Move storage's MinioConfig struct into setting, so after the
configuration loading, the values will be stored into the struct but not
still on some section.
- All storages configurations should be stored on one section,
configuration items cannot be overrided by multiple sections. The
prioioty of configuration is `[attachment]` > `[storage.attachments]` |
`[storage.customized]` > `[storage]` > `default`
- For extra override configuration items, currently are `SERVE_DIRECT`,
`MINIO_BASE_PATH`, `MINIO_BUCKET`, which could be configured in another
section. The prioioty of the override configuration is `[attachment]` >
`[storage.attachments]` > `default`.
- Add more tests for storages configurations.
- Update the storage documentations.

---------

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2023-06-14 11:42:38 +08:00
Lunny Xiao 5cf7da63ee
Refactor config provider (#24245)
This PR introduces more abstract about `ConfigProvider` and hides more `ini` references.

---------

Co-authored-by: delvh <dev.lh@web.de>
2023-04-25 23:06:39 +08:00
wxiaoguang 5727056ea1
Make minio package support legacy MD5 checksum (#23768)
A feedback from discord:
https://discord.com/channels/322538954119184384/561007778139734027/1090185427115319386

Some storages like:

 * https://developers.cloudflare.com/r2/api/s3/api/
 * https://www.backblaze.com/b2/docs/s3_compatible_api.html

They do not support "x-amz-checksum-algorithm" header

But minio recently uses that header with CRC32C by default. So we have
to tell minio to use legacy MD5 checksum.

I guess this needs to be backported because IIRC we 1.19 and 1.20 are
using similar minio package.


The minio package code for SendContentMD5 looks like this:

<details>

<img width="755" alt="image"
src="https://user-images.githubusercontent.com/2114189/228186768-4f2f6f67-62b9-4aee-9251-5af714ad9674.png">

</details>
2023-03-28 11:10:24 -04:00
Yarden Shoham 0e7bec1849
Add InsecureSkipVerify to Minio Client for Storage (#23166)
Allows using Minio with untrusted certificates

Closes #23128

Signed-off-by: Yarden Shoham <hrsi88@gmail.com>
2023-02-27 16:26:13 +00:00
Lunny Xiao c53ad052d8
Refactor the setting to make unit test easier (#22405)
Some bugs caused by less unit tests in fundamental packages. This PR
refactor `setting` package so that create a unit test will be easier
than before.

- All `LoadFromXXX` files has been splited as two functions, one is
`InitProviderFromXXX` and `LoadCommonSettings`. The first functions will
only include the code to create or new a ini file. The second function
will load common settings.
- It also renames all functions in setting from `newXXXService` to
`loadXXXSetting` or `loadXXXFrom` to make the function name less
confusing.
- Move `XORMLog` to `SQLLog` because it's a better name for that.

Maybe we should finally move these `loadXXXSetting` into the `XXXInit`
function? Any idea?

---------

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: delvh <dev.lh@web.de>
2023-02-20 00:12:01 +08:00
flynnnnnnnnnn e81ccc406b
Implement FSFE REUSE for golang files (#21840)
Change all license headers to comply with REUSE specification.

Fix #16132

Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
2022-11-27 18:20:29 +00:00
Lunny Xiao b223d36195
Rework repository archive (#14723)
* Use storage to store archive files

* Fix backend lint

* Add archiver table on database

* Finish archive download

* Fix test

* Add database migrations

* Add status for archiver

* Fix lint

* Add queue

* Add doctor to check and delete old archives

* Improve archive queue

* Fix tests

* improve archive storage

* Delete repo archives

* Add missing fixture

* fix fixture

* Fix fixture

* Fix test

* Fix archiver cleaning

* Fix bug

* Add docs for repository archive storage

* remove repo-archive configuration

* Fix test

* Fix test

* Fix lint

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-23 17:12:38 -04:00
胡玮文 c21a399646
more test case for STORAGE_TYPE overrides (and fixes) (#14096)
Signed-off-by: 胡玮文 <huww98@outlook.com>
2020-12-22 01:03:18 +02:00
胡玮文 addd4248da
Fix storage config implementation (#14091)
The design is very flexible, but not implemented correctly.
This commit fixes several issues:
* Costom storage type stated in https://docs.gitea.io/en-us/config-cheat-sheet/#storage-storage
  not working
* [storage.attachments], [storage.minio] section not respected

Signed-off-by: 胡玮文 <huww98@outlook.com>
2020-12-21 18:59:18 +01:00
Lunny Xiao e4e85a3e51
Storage configuration support `[storage]` (#13314)
* Fix minio bug

* Add tests for storage configuration

* Change the Seek flag to keep compitable minio?

* Fix test when first-byte-pos of all ranges is greater than the resource length

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-10-31 16:51:48 -04:00
zeripath 172e7ee87f
Fix Storage mapping (#13297)
This PR fixes several bugs in setting storage

* The default STORAGE_TYPE should be the provided type.
* The Storage config should be passed in to NewStorage as a pointer - otherwise the Mappable interface function MapTo will not be found
* There was a bug in the MapTo function.

Fix #13286

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-10-25 17:19:06 +00:00
zeripath 6b1266b6b3
Provide self-registering storage system (#12978)
* Provide self-registering storage system

Signed-off-by: Andrew Thornton <art27@cantab.net>

* More simplification

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Remove old strings from setting

Signed-off-by: Andrew Thornton <art27@cantab.net>

* oops attachments not attachment

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-10-12 23:58:34 -04:00
Lunny Xiao 3878e985b6
Add default storage configurations (#12813)
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: zeripath <art27@cantab.net>
2020-09-29 12:05:13 +03:00