gitea/modules/git/foreachref/parser_test.go

228 lines
6.7 KiB
Go
Raw Normal View History

Improve sync performance for pull-mirrors (#19125) This addresses https://github.com/go-gitea/gitea/issues/18352 It aims to improve performance (and resource use) of the `SyncReleasesWithTags` operation for pull-mirrors. For large repositories with many tags, `SyncReleasesWithTags` can be a costly operation (taking several minutes to complete). The reason is two-fold: 1. on sync, every upstream repo tag is compared (for changes) against existing local entries in the release table to ensure that they are up-to-date. 2. the procedure for getting _each tag_ involves a series of git operations ```bash git show-ref --tags -- v8.2.4477 git cat-file -t 29ab6ce9f36660cffaad3c8789e71162e5db5d2f git cat-file -p 29ab6ce9f36660cffaad3c8789e71162e5db5d2f git rev-list --count 29ab6ce9f36660cffaad3c8789e71162e5db5d2f ``` of which the `git rev-list --count` can be particularly heavy. This PR optimizes performance for pull-mirrors. We utilize the fact that a pull-mirror is always identical to its upstream and rebuild the entire release table on every sync and use a batch `git for-each-ref .. refs/tags` call to retrieve all tags in one go. For large mirror repos, with hundreds of annotated tags, this brings down the duration of the sync operation from several minutes to a few seconds. A few unscientific examples run on my local machine: - https://github.com/spring-projects/spring-boot (223 tags) - before: `0m28,673s` - after: `0m2,244s` - https://github.com/kubernetes/kubernetes (890 tags) - before: `8m00s` - after: `0m8,520s` - https://github.com/vim/vim (13954 tags) - before: `14m20,383s` - after: `0m35,467s` I added a `foreachref` package which contains a flexible way of specifying which reference fields are of interest (`git-for-each-ref(1)`) and to produce a parser for the expected output. These could be reused in other places where `for-each-ref` is used. I'll add unit tests for those if the overall PR looks promising.
2022-03-31 23:30:40 +11:00
// Copyright 2022 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
Improve sync performance for pull-mirrors (#19125) This addresses https://github.com/go-gitea/gitea/issues/18352 It aims to improve performance (and resource use) of the `SyncReleasesWithTags` operation for pull-mirrors. For large repositories with many tags, `SyncReleasesWithTags` can be a costly operation (taking several minutes to complete). The reason is two-fold: 1. on sync, every upstream repo tag is compared (for changes) against existing local entries in the release table to ensure that they are up-to-date. 2. the procedure for getting _each tag_ involves a series of git operations ```bash git show-ref --tags -- v8.2.4477 git cat-file -t 29ab6ce9f36660cffaad3c8789e71162e5db5d2f git cat-file -p 29ab6ce9f36660cffaad3c8789e71162e5db5d2f git rev-list --count 29ab6ce9f36660cffaad3c8789e71162e5db5d2f ``` of which the `git rev-list --count` can be particularly heavy. This PR optimizes performance for pull-mirrors. We utilize the fact that a pull-mirror is always identical to its upstream and rebuild the entire release table on every sync and use a batch `git for-each-ref .. refs/tags` call to retrieve all tags in one go. For large mirror repos, with hundreds of annotated tags, this brings down the duration of the sync operation from several minutes to a few seconds. A few unscientific examples run on my local machine: - https://github.com/spring-projects/spring-boot (223 tags) - before: `0m28,673s` - after: `0m2,244s` - https://github.com/kubernetes/kubernetes (890 tags) - before: `8m00s` - after: `0m8,520s` - https://github.com/vim/vim (13954 tags) - before: `14m20,383s` - after: `0m35,467s` I added a `foreachref` package which contains a flexible way of specifying which reference fields are of interest (`git-for-each-ref(1)`) and to produce a parser for the expected output. These could be reused in other places where `for-each-ref` is used. I'll add unit tests for those if the overall PR looks promising.
2022-03-31 23:30:40 +11:00
package foreachref_test
import (
"errors"
"fmt"
"io"
"strings"
"testing"
"code.gitea.io/gitea/modules/git/foreachref"
"code.gitea.io/gitea/modules/json"
"github.com/stretchr/testify/require"
)
type refSlice = []map[string]string
func TestParser(t *testing.T) {
tests := []struct {
name string
givenFormat foreachref.Format
givenInput io.Reader
wantRefs refSlice
wantErr bool
expectedErr error
}{
// this would, for example, be the result when running `git
// for-each-ref refs/tags` on a repo without tags.
{
name: "no references on empty input",
givenFormat: foreachref.NewFormat("refname:short"),
givenInput: strings.NewReader(``),
wantRefs: []map[string]string{},
},
// note: `git for-each-ref` will add a newline between every
// reference (in addition to the ref-delimiter we've chosen)
{
name: "single field requested, single reference in output",
givenFormat: foreachref.NewFormat("refname:short"),
givenInput: strings.NewReader("refname:short v0.0.1\x00\x00" + "\n"),
wantRefs: []map[string]string{
{"refname:short": "v0.0.1"},
},
},
{
name: "single field requested, multiple references in output",
givenFormat: foreachref.NewFormat("refname:short"),
givenInput: strings.NewReader(
"refname:short v0.0.1\x00\x00" + "\n" +
"refname:short v0.0.2\x00\x00" + "\n" +
"refname:short v0.0.3\x00\x00" + "\n"),
wantRefs: []map[string]string{
{"refname:short": "v0.0.1"},
{"refname:short": "v0.0.2"},
{"refname:short": "v0.0.3"},
},
},
{
name: "multiple fields requested for each reference",
givenFormat: foreachref.NewFormat("refname:short", "objecttype", "objectname"),
givenInput: strings.NewReader(
"refname:short v0.0.1\x00objecttype commit\x00objectname 7b2c5ac9fc04fc5efafb60700713d4fa609b777b\x00\x00" + "\n" +
"refname:short v0.0.2\x00objecttype commit\x00objectname a1f051bc3eba734da4772d60e2d677f47cf93ef4\x00\x00" + "\n" +
"refname:short v0.0.3\x00objecttype commit\x00objectname ef82de70bb3f60c65fb8eebacbb2d122ef517385\x00\x00" + "\n",
),
wantRefs: []map[string]string{
{
"refname:short": "v0.0.1",
"objecttype": "commit",
"objectname": "7b2c5ac9fc04fc5efafb60700713d4fa609b777b",
},
{
"refname:short": "v0.0.2",
"objecttype": "commit",
"objectname": "a1f051bc3eba734da4772d60e2d677f47cf93ef4",
},
{
"refname:short": "v0.0.3",
"objecttype": "commit",
"objectname": "ef82de70bb3f60c65fb8eebacbb2d122ef517385",
},
},
},
{
name: "must handle multi-line fields such as 'content'",
givenFormat: foreachref.NewFormat("refname:short", "contents", "author"),
givenInput: strings.NewReader(
"refname:short v0.0.1\x00contents Create new buffer if not present yet (#549)\n\nFixes a nil dereference when ProcessFoo is used\nwith multiple commands.\x00author Foo Bar <foo@bar.com> 1507832733 +0200\x00\x00" + "\n" +
"refname:short v0.0.2\x00contents Update CI config (#651)\n\n\x00author John Doe <john.doe@foo.com> 1521643174 +0000\x00\x00" + "\n" +
"refname:short v0.0.3\x00contents Fixed code sample for bash completion (#687)\n\n\x00author Foo Baz <foo@baz.com> 1524836750 +0200\x00\x00" + "\n",
),
wantRefs: []map[string]string{
{
"refname:short": "v0.0.1",
"contents": "Create new buffer if not present yet (#549)\n\nFixes a nil dereference when ProcessFoo is used\nwith multiple commands.",
"author": "Foo Bar <foo@bar.com> 1507832733 +0200",
},
{
"refname:short": "v0.0.2",
"contents": "Update CI config (#651)",
"author": "John Doe <john.doe@foo.com> 1521643174 +0000",
},
{
"refname:short": "v0.0.3",
"contents": "Fixed code sample for bash completion (#687)",
"author": "Foo Baz <foo@baz.com> 1524836750 +0200",
},
},
},
{
name: "must handle fields without values",
givenFormat: foreachref.NewFormat("refname:short", "object", "objecttype"),
givenInput: strings.NewReader(
"refname:short v0.0.1\x00object \x00objecttype commit\x00\x00" + "\n" +
"refname:short v0.0.2\x00object \x00objecttype commit\x00\x00" + "\n" +
"refname:short v0.0.3\x00object \x00objecttype commit\x00\x00" + "\n",
),
wantRefs: []map[string]string{
{
"refname:short": "v0.0.1",
"object": "",
"objecttype": "commit",
},
{
"refname:short": "v0.0.2",
"object": "",
"objecttype": "commit",
},
{
"refname:short": "v0.0.3",
"object": "",
"objecttype": "commit",
},
},
},
{
name: "must fail when the number of fields in the input doesn't match expected format",
givenFormat: foreachref.NewFormat("refname:short", "objecttype", "objectname"),
givenInput: strings.NewReader(
"refname:short v0.0.1\x00objecttype commit\x00\x00" + "\n" +
"refname:short v0.0.2\x00objecttype commit\x00\x00" + "\n" +
"refname:short v0.0.3\x00objecttype commit\x00\x00" + "\n",
),
wantErr: true,
expectedErr: errors.New("unexpected number of reference fields: wanted 2, was 3"),
},
{
name: "must fail input fields don't match expected format",
givenFormat: foreachref.NewFormat("refname:short", "objectname"),
givenInput: strings.NewReader(
"refname:short v0.0.1\x00objecttype commit\x00\x00" + "\n" +
"refname:short v0.0.2\x00objecttype commit\x00\x00" + "\n" +
"refname:short v0.0.3\x00objecttype commit\x00\x00" + "\n",
),
wantErr: true,
expectedErr: errors.New("unexpected field name at position 1: wanted: 'objectname', was: 'objecttype'"),
},
}
for _, test := range tests {
tc := test // don't close over loop variable
t.Run(tc.name, func(t *testing.T) {
parser := tc.givenFormat.Parser(tc.givenInput)
//
// parse references from input
//
gotRefs := make([]map[string]string, 0)
for {
ref := parser.Next()
if ref == nil {
break
}
gotRefs = append(gotRefs, ref)
}
err := parser.Err()
//
// verify expectations
//
if tc.wantErr {
require.Error(t, err)
require.EqualError(t, err, tc.expectedErr.Error())
} else {
require.NoError(t, err, "for-each-ref parser unexpectedly failed with: %v", err)
require.Equal(t, tc.wantRefs, gotRefs, "for-each-ref parser produced unexpected reference set. wanted: %v, got: %v", pretty(tc.wantRefs), pretty(gotRefs))
}
})
}
}
func pretty(v any) string {
Improve sync performance for pull-mirrors (#19125) This addresses https://github.com/go-gitea/gitea/issues/18352 It aims to improve performance (and resource use) of the `SyncReleasesWithTags` operation for pull-mirrors. For large repositories with many tags, `SyncReleasesWithTags` can be a costly operation (taking several minutes to complete). The reason is two-fold: 1. on sync, every upstream repo tag is compared (for changes) against existing local entries in the release table to ensure that they are up-to-date. 2. the procedure for getting _each tag_ involves a series of git operations ```bash git show-ref --tags -- v8.2.4477 git cat-file -t 29ab6ce9f36660cffaad3c8789e71162e5db5d2f git cat-file -p 29ab6ce9f36660cffaad3c8789e71162e5db5d2f git rev-list --count 29ab6ce9f36660cffaad3c8789e71162e5db5d2f ``` of which the `git rev-list --count` can be particularly heavy. This PR optimizes performance for pull-mirrors. We utilize the fact that a pull-mirror is always identical to its upstream and rebuild the entire release table on every sync and use a batch `git for-each-ref .. refs/tags` call to retrieve all tags in one go. For large mirror repos, with hundreds of annotated tags, this brings down the duration of the sync operation from several minutes to a few seconds. A few unscientific examples run on my local machine: - https://github.com/spring-projects/spring-boot (223 tags) - before: `0m28,673s` - after: `0m2,244s` - https://github.com/kubernetes/kubernetes (890 tags) - before: `8m00s` - after: `0m8,520s` - https://github.com/vim/vim (13954 tags) - before: `14m20,383s` - after: `0m35,467s` I added a `foreachref` package which contains a flexible way of specifying which reference fields are of interest (`git-for-each-ref(1)`) and to produce a parser for the expected output. These could be reused in other places where `for-each-ref` is used. I'll add unit tests for those if the overall PR looks promising.
2022-03-31 23:30:40 +11:00
data, err := json.MarshalIndent(v, "", " ")
if err != nil {
// shouldn't happen
panic(fmt.Sprintf("json-marshalling failed: %v", err))
}
return string(data)
}