Commit Graph

4880 Commits

Author SHA1 Message Date
Milos Gajdos 8cc58797e8
Merge pull request #3794 from AdamKorcz/fuzz1
Fuzzing: Rewrite existing fuzzers to native go fuzzers
2022-11-29 09:57:09 +00:00
Sebastiaan van Stijn 0b4311d5ce
manifest: improve test output and use const
Use consts to make clear these values are fixed, and improve the output
to make it clearer which part is the expected output, and which part
the actual.

Before this:

    === RUN   TestManifest
        manifest_test.go:87: manifest bytes not equal: "{\n   \"schemaVersion\": 2,\n   \"mediaType\": \"application/vnd.oci.image.manifest.v1+json\",\n   \"config\": {\n      \"mediaType\": \"application/vnd.oci.image.config.v1+json\",\n      \"size\": 985,\n      \"digest\": \"sha256:1a9ec845ee94c202b2d5da74a24f0ed2058318bfa9879fa541efaecba272e86b\",\n      \"annotations\": {\n         \"apple\": \"orange\"\n      }\n   },\n   \"layers\": [\n      {\n         \"mediaType\": \"application/vnd.oci.image.layer.v1.tar+gzip\",\n         \"size\": 153263,\n         \"digest\": \"sha256:62d8908bee94c202b2d35224a221aaa2058318bfa9879fa541efaecba272331b\",\n         \"annotations\": {\n            \"lettuce\": \"wrap\"\n         }\n      }\n   ],\n   \"annotations\": {\n      \"hot\": \"potato\"\n   }\n}" != "{\n   \"schemaVersion\": 2,\n   \"mediaType\": \"application/vnd.oci.image.manifest.v1+json\",\n   \"config\": {\n      \"mediaType\": \"application/vnd.oci.image.config.v1+json\",\n      \"size\": 985,\n      \"digest\": \"sha256:1a9ec845ee94c202b2d5da74a24f0ed2058318bfa9879fa541efaecba272e86b\",\n      \"annotations\": {\n         \"apple\": \"orange\"\n      }\n   },\n   \"layers\": [\n      {\n         \"mediaType\": \"application/vnd.oci.image.layer.v1.tar+gzip\",\n         \"size\": 153263,\n         \"digest\": \"sha256:62d8908bee94c202b2d35224a221aaa2058318bfa9879fa541efaecba272331b\",\n         \"annotations\": {\n            \"lettuce\": \"wrap\"\n         }\n      }\n   ],\n   \"annotations\": {\n      \"hot\": \"potato\"\n   }\n}"
    --- FAIL: TestManifest (0.00s)

After this:

    === RUN   TestManifest
        manifest_test.go:72: manifest bytes not equal:
            expected:
            {
               "schemaVersion": 2,
               "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
               "config": {
                  "mediaType": "application/vnd.docker.container.image.v1+json",
                  "size": 985,
                  "digest": "sha256:1a9ec845ee94c202b2d5da74a24f0ed2058318bfa9879fa541efaecba272e86b"
               },
               "layers": [
                  {
                     "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                     "size": 153263,
                     "digest": "sha256:62d8908bee94c202b2d35224a221aaa2058318bfa9879fa541efaecba272331b"
                  }
               ]
            }
            actual:
            {
               "schemaVersion": 2,
               "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
               "config": {
                  "mediaType": "application/vnd.docker.container.image.v1+json",
                  "size": 985,
                  "digest": "sha256:1a9ec845ee94c202b2d5da74a24f0ed2058318bfa9879fa541efaecba272e86b"
               },
               "layers": [
                  {
                     "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                     "size": 153263,
                     "digest": "sha256:62d8908bee94c202b2d35224a221aaa2058318bfa9879fa541efaecba272331b"
                  }
               ]
            }
    --- FAIL: TestManifest (0.00s)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-28 10:08:05 +01:00
Sebastiaan van Stijn f2db7faa2f
registry/storage: rename variables that collided with imports
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-28 10:08:05 +01:00
Sebastiaan van Stijn 030489ca66
testutil: rename variables that collided with imports
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-28 10:08:05 +01:00
Sebastiaan van Stijn 0e3efe749b
manifest: rename variables that collided with imports
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-28 10:08:02 +01:00
Milos Gajdos b9f9073d1a
Merge pull request #3805 from crazy-max/update-gha
ci: update github actions
2022-11-27 10:56:15 +00:00
CrazyMax b91c9a22f4
ci: add concurrency check
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2022-11-26 16:09:46 +01:00
CrazyMax 2400718d81
ci: update github actions
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2022-11-26 16:09:46 +01:00
Sebastiaan van Stijn ff2bce2731
manifest/schema1: mark docker manifest v2, schema 1 deprecated
Docker Image manifest v2, schema version 1 is deprecated since 2015, when
manifest v2, schema version 2 was introduced (2e3f4934a7).

Users should no longer use this specification other than for backward
compatibility.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-26 13:21:06 +01:00
Milos Gajdos ede90db01c
Merge pull request #3789 from thaJeztah/reference_consts
reference: clean up regular expressions
2022-11-25 12:38:43 +00:00
Sebastiaan van Stijn 02e88c0f15
reference: move exported regexes to separate block
This makes them easier to find between the non-exported ones, and puts
them as separate sections in the generated docs. While updating, also
extended documentation for some to be more descriptive.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-23 00:01:57 +01:00
Sebastiaan van Stijn a4cec8ca82
reference: introduce const for "localhost"
Localhost is treated special when parsing references, and always considered
to be a domain, despite not having a "." nor a ":port". Adding a const for
this, to allow documenting this special case (making it more visible).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-22 23:45:26 +01:00
Sebastiaan van Stijn bbd41f40bb
reference: introduce remoteName variable
This pattern was used in two places, so adding an intermediate variable allows
documenting its purpose. The "remote-name" grammer (which is interchangably
used with "path") also seemed to be missing from the grammar, so adding it.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-22 23:45:23 +01:00
Sebastiaan van Stijn 71a0666398
reference: optional repeated == any number of times
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-22 23:41:56 +01:00
Sebastiaan van Stijn 919bd8ab09
reference: add const for (optional) port, and rename "domain" variable
The `domain` variable didn't make it clear that this could include port-numbers
as well, so renaming it makes that more visible.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-20 15:11:33 +01:00
Sebastiaan van Stijn f0c7c97e73
reference: remove remaining uses of "expression()"
The remaining uses of "expression()" were quite trivial; probably goes without
saying, but just using string-concatenating for these is more performant as well,
and removing the extra abstraction may make it easier to read;

    pkg: github.com/distribution/distribution/v3/reference
    BenchmarkExpression
    BenchmarkExpression-10    27260877        43.10 ns/op      24 B/op       1 allocs/op
    BenchmarkConcat
    BenchmarkConcat-10      1000000000         0.3154 ns/op     0 B/op       0 allocs/op
    PASS
    ok  	github.com/distribution/distribution/v3/reference	1.762s

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-20 15:11:33 +01:00
Sebastiaan van Stijn 04d6592df1
reference: remove "literal()" utility
With the exception of ".", none of the literals used required escaping, which made
the function rather redundant (and the extra abstraction made it harder to read).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-20 15:11:27 +01:00
Sebastiaan van Stijn c786a2bd3e
reference: inline "group()"
It was only used in a couple of places, and more transparent to just
inline it.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-20 15:07:25 +01:00
Sebastiaan van Stijn 1d4917d4fb
reference: expression(): use strings.Join()
It's easier to read, and more performant:

    pkg: github.com/distribution/distribution/v3/reference
    BenchmarkExpression
    BenchmarkExpression-10    	10474380	        97.32 ns/op	      64 B/op	       4 allocs/op
    BenchmarkJoin
    BenchmarkJoin-10          	27722588	        42.71 ns/op	      24 B/op	       1 allocs/op
    PASS

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-20 15:07:25 +01:00
Sebastiaan van Stijn a7e7ff933c
reference: align docs and variables with grammar
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-20 15:07:25 +01:00
Sebastiaan van Stijn 32a4d8e39f
reference: fix docs for NameRegexp
NameRegexp does not have capturing groups, so updating the documentation
to reflect that.

To verify if this was an unintentional regression, I looked up the commit
that introduced this regex (31a448a628), and
it looks like it never had capturing groups, so this was just a mistake in
the docs.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-20 15:07:25 +01:00
Sebastiaan van Stijn 226b21beb6
reference: make some regexp vars a const, remove intermediate vars
This patch:

- makes regexp strings that are constant a const
- moves some variables closer to where they're used
- removes some intermediate vars
- un-wraps some lines; they're lengthy, but probably more readable than having
  them wrapped over multiple lines.
- touches-up some docs.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-20 15:07:24 +01:00
Sebastiaan van Stijn 10eace9a53
reference: document consts for normalizing and legacy domain
These const could use some documentation, as it won't be clear why
they exist.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-20 15:07:24 +01:00
Sebastiaan van Stijn 53757ea337
reference: ParseDockerRef: slight refactor, and update docs
- improve documentation
- remove redundant error-check
- simplify interface matching, which slightly improves readability
- touch-up some docs

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-20 15:07:24 +01:00
Sebastiaan van Stijn 2bf5e1879e
reference: remove left-over occurrences of "short-identifier"
This was removed in 6d4f62d7fd, which forgot
to remove it from the grammar.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-20 15:07:24 +01:00
Sebastiaan van Stijn 8163e20c6f
reference: splitDockerDomain: remove incorrect "TODO"
My mistake; I added this TODO in 552b1526c6, but it
only applies to familiarizeName.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-20 15:07:21 +01:00
Milos Gajdos 85d4039064
Merge pull request #3786 from thaJeztah/reference_subtests 2022-11-19 09:36:43 +00:00
Sebastiaan van Stijn b6a040faf4
reference: run tests with t.Parallel()
Not all tests have been rewritten to use sub-tests; for those
I enabled t.Parallel() for the parent test only.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-18 21:17:56 +01:00
Sebastiaan van Stijn 5703bcf17d
reference: use subtests for regex tests
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-18 21:16:51 +01:00
Sebastiaan van Stijn f481f877f1
reference: use subtests for reference tests
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-18 21:13:49 +01:00
Milos Gajdos 21622ca699
Merge pull request #3787 from thaJeztah/simplify_mocks 2022-11-18 08:53:26 +00:00
Milos Gajdos 5357c45703
Merge pull request #3788 from thaJeztah/deprecate_ReadSeekCloser 2022-11-18 08:53:15 +00:00
AdamKorcz 9337b8df66 Fuzzing: Rewrite existing fuzzers to native go fuzzers
Signed-off-by: AdamKorcz <adam@adalogics.com>
2022-11-12 17:30:10 +00:00
Milos Gajdos 3b8fbf9752
Merge pull request #3686 from m5i-work/exp 2022-11-11 17:07:14 +00:00
Wei Meng 35cae1099e Realloc slice exponentially in mfs
`registry/storage/driver/inmemory/driver_test.go` times out after ~10min. The slow test is `testsuites.go:TestWriteReadLargeStreams()` which writes a 5GB file.
Root cause is inefficient slice reallocation algorithm. The slice holding file bytes grows only 32K on each allocation. To fix it, this PR grows slice exponentially.

Signed-off-by: Wei Meng <wemeng@microsoft.com>
2022-11-11 18:18:08 +08:00
Milos Gajdos e67e40bfd2
Merge pull request #3769 from thaJeztah/strings_cut 2022-11-11 08:37:28 +00:00
Sebastiaan van Stijn 1d8cd5e443
registry/client: use struct literals
Remove some intermediate variables, and use struct literals instead.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 23:11:12 +01:00
Sebastiaan van Stijn d71ad5b3a6
transport.NewHTTPReadSeeker: return concrete type, deprecate ReadSeekCloser
General convention is to define interfaces on the receiver side, and
to return concrete types.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 23:10:39 +01:00
Sebastiaan van Stijn 019ead86f5
deprecate ReadSeekCloser in favor of io.ReadSeekCloser
Go's io package in stdlib now defines this interface, so we can switch
to using that instead.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 23:10:32 +01:00
Sebastiaan van Stijn 842d4c04f5
cloudfront: use strings.Equalfold()
Minor optimization :)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 22:38:16 +01:00
Sebastiaan van Stijn 3b391d3290
replace strings.Split(N) for strings.Cut() or alternatives
Go 1.18 and up now provides a strings.Cut() which is better suited for
splitting key/value pairs (and similar constructs), and performs better:

```go
func BenchmarkSplit(b *testing.B) {
	b.ReportAllocs()
	data := []string{"12hello=world", "12hello=", "12=hello", "12hello"}
	for i := 0; i < b.N; i++ {
		for _, s := range data {
			_ = strings.SplitN(s, "=", 2)[0]
		}
	}
}

func BenchmarkCut(b *testing.B) {
	b.ReportAllocs()
	data := []string{"12hello=world", "12hello=", "12=hello", "12hello"}
	for i := 0; i < b.N; i++ {
		for _, s := range data {
			_, _, _ = strings.Cut(s, "=")
		}
	}
}
```

    BenchmarkSplit
    BenchmarkSplit-10    	 8244206	       128.0 ns/op	     128 B/op	       4 allocs/op
    BenchmarkCut
    BenchmarkCut-10      	54411998	        21.80 ns/op	       0 B/op	       0 allocs/op

While looking at occurrences of `strings.Split()`, I also updated some for alternatives,
or added some constraints;

- for cases where an specific number of items is expected, I used `strings.SplitN()`
  with a suitable limit. This prevents (theoretical) unlimited splits.
- in some cases it we were using `strings.Split()`, but _actually_ were trying to match
  a prefix; for those I replaced the code to just match (and/or strip) the prefix.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 22:38:12 +01:00
Sebastiaan van Stijn 2cd52d5c0c
simplify mocks
Embed the interface that we're mocking; calling any of it's methods
that are not implemented will panic, so should give the same result
as before.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 17:09:24 +01:00
Sebastiaan van Stijn 552b1526c6
reference: check for prefix instead of splitting, and use consts
- use strings.HasPrefix() to check for the prefix we're interested in instead
  of doing a strings.Split() without limits. This makes the code both easier
  to read, and prevents potential situations where we end up with a long slice.
- use consts for defaults; these should never be modified, so better to use
  consts for them to indicate they're fixed values.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-09 15:21:47 +01:00
Wang Yan 9d38ed78d2
Merge pull request #3764 from thaJeztah/bump_cobra
go.mod: github.com/spf13/cobra v1.6.1
2022-11-09 11:19:22 +08:00
Wang Yan 0071e46eee
Merge pull request #3781 from thaJeztah/fix_usage_of_deprecated_funcs
Remove uses of deprecated go-digest.NewDigestFromHex, go-digest.Digest.Hex
2022-11-09 11:03:33 +08:00
Milos Gajdos 9bb63e6e46
Merge pull request #3775 from thaJeztah/remove_digestset 2022-11-08 22:48:25 +00:00
Milos Gajdos 47c3610bff
Merge pull request #3679 from Jamstah/building-doc-updates 2022-11-08 22:46:09 +00:00
Milos Gajdos 75093e0aab
Merge pull request #3777 from thaJeztah/reference_add_sorting 2022-11-08 22:45:19 +00:00
Sebastiaan van Stijn 7b651a9692
digestset: deprecate package in favor of go-digest/digestset
This package was only used for the deprecated "shortid" syntax. Now that
support for this syntax was removed, we can also remove this package.

This patch deprecates and removes the package, adding temporary aliases pointing
to the new location to ease migration from docker/distribution to the new
distribution/distribution/v3. We should remove those aliases in a future update.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-08 23:17:10 +01:00
Derek McGowan 1052518d9f
reference: implement Sort()
This upstreams `Sort()` as originally implemented in containerd in
https://github.com/containerd/containerd/0886ceaea2470edc7339dfc5ebe0e3257ae84d06

From that commit:

> Fix reference ordering in CRI image store
>
> Currently image references end up being stored in a
> random order due to the way maps are iterated through
> in Go. This leads to inconsistent identifiers being
> resolved when a single reference is needed to identify
> an image and the ordering of the references is used for
> the selection.
>
> Sort references in a consistent and ranked manner,
> from higher information formats to lower.
>
> Note: A `name + tag` reference is considered higher
> information than a `name + digest` reference since a
> registry may be used to resolve the digest from a
> `name + tag` reference.

Co-Authored-by: Derek McGowan <derek@mcg.dev>
Co-Authored-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-08 22:38:47 +01:00