- Deep Dive
- Beginner’s guide
- Developing new Git features
- Gitaly-Related Test Failures
TooManyInvocationsError
errors- Request counts
- Running tests with a locally modified version of Gitaly
- Wrapping RPCs in feature flags
- Using Praefect in test
- Git references used by Gitaly
Gitaly development guidelines
In May 2019, Bob Van Landuyt
hosted a Deep Dive (GitLab team members only: https://gitlab.com/gitlab-org/create-stage/-/issues/1
)
on the . It included how to contribute to it as a
Ruby developer, and shared domain-specific knowledge with anyone who may work in this part of the
codebase in the future.
You can find the recording on YouTube, and the slides on Google Slides and in Beginner’s guide
Start by reading the Gitaly repository’s
.
It describes how to set up Gitaly, the various components of Gitaly and what
they do, and how to run its test suites.
To read or write Git data, a request has to be made to Gitaly. This means that
if you’re developing a new feature where you need data that’s not yet available
in There should be no new code that touches Git repositories via disk access
anywhere in the It’s often easier to develop a new feature in Gitaly if you make the changes to
GitLab that intends to use the new feature in a separate merge request, to be merged
immediately after the Gitaly one. This allows you to test your changes before
they are merged.
If your test-suite is failing with Gitaly issues, as a first step, try running:
During RSpec tests, the Gitaly instance writes logs to During development and testing, you may experience As a temporary measure, export Raise an issue in the GitLab CE or EE repositories to report the issue. Include the labels ~Gitaly
~performance ~"technical debt". Ensure that the issue contains the full stack trace and error message of the
Isolate the source of the n+1 problem. This is usually a loop that results in Gitaly being called for each
element in an array. If you are unable to isolate the problem, contact a member
of the for assistance.
After the source has been found, wrap it in an After the code is wrapped in this block, this code path is excluded from n+1 detection.
Commits and other Git data, is now fetched through Gitaly. These fetches can,
much like with a database, be batched. This improves performance for the client
and for Gitaly itself and therefore for the users too. To keep performance stable
and guard performance regressions, Gitaly calls can be counted and the call count
can be tested against. This requires the Usually, GitLab CE/EE tests use a local clone of Gitaly in
If you want to run tests locally against a modified version of Gitaly you
can replace Make sure this directory contains the files Make sure you run If you make changes to your local Gitaly in between test runs you need
to manually run CI tests do not use your locally modified version of
Gitaly. To use a custom Gitaly version in CI, you must update
To use a different Gitaly repository, such as if your changes are present
on a fork, you can specify a If your fork of Gitaly is private, you can generate a Deploy Token
and specify it in the URL:
To use a custom Gitaly repository in CI/CD, for instance if you want your
GitLab fork to always use your own Gitaly fork, set If you are making changes to the RPC client, such as adding a new endpoint or adding a new
parameter to an existing endpoint, follow the guide for
. Then:
Build the RPC client gem from the root directory of Gitaly:
In the Change the Re-run steps 2-5 each time you want to try out new changes.
Return to Development documentation
Here are the steps to gate a new feature in Gitaly behind a feature flag.
Create a package scoped flag name:
Create a switch in the code using the Create Prometheus metrics:
Set headers in tests:
Test in a Rails console by setting the feature flag:
Pay attention to the name of the flag and the one used in the Rails console. There is a difference
between them (dashes replaced by underscores and name prefix is changed). Make sure to prefix all
flags with To be sure that the flag is set correctly and it goes into Gitaly, you can check
the integration by using GDK:
Check that the list of current metrics has the new counter for the feature flag:
Start a Rails console:
Check the list of feature flags:
It should be disabled Enable it:
Verify the feature is on by observing the metrics for it:
By default Praefect in test uses an in-memory election strategy. This strategy
is deprecated and no longer used in production. It mainly is kept for
unit-testing purposes.
A more modern election strategy requires a connection with a PostgreSQL
database. This behavior is disabled by default when running tests, but you can
enable it by setting This requires you have PostgreSQL running, and you have the database created.
When you are using GDK, you can set it up with:
Gitaly uses many Git references (refs) to provide Git services to GitLab.
These standard Git references are used by GitLab (through Gitaly) in any Git repository:
These GitLab-specific references are used exclusively by GitLab (through Gitaly):
Developing new Git features
lib/gitlab/git
changes have to be made to Gitaly.
gitlab
repository. Anything that
needs direct access to the Git repository must be implemented in Gitaly, and
exposed via an RPC.
gdk install
and restart GDK using gdk restart
to use a locally modified Gitaly version for development
Gitaly-Related Test Failures
rm -rf tmp/tests/gitaly
gitlab/log/gitaly-test.log
.
TooManyInvocationsError
errors
Gitlab::GitalyClient::TooManyInvocationsError
failures.
The GitalyClient
attempts to block against potential n+1 issues by raising this error
when Gitaly is called more than 30 times in a single Rails request or Sidekiq execution.
GITALY_DISABLE_REQUEST_LIMITS=1
to suppress the error. This disables the n+1 detection
in your development environment.
TooManyInvocationsError
. Also include any known failing tests if possible.
allow_n_plus_1_calls
block, as follows:
# n+1: link to n+1 issue
Gitlab::GitalyClient.allow_n_plus_1_calls do
# original code
commits.each { |commit| ... }
end
Request counts
:request_store
flag to be set.
describe 'Gitaly Request count tests' do
context 'when the request store is activated', :request_store do
it 'correctly counts the gitaly requests made' do
expect { subject }.to change { Gitlab::GitalyClient.get_request_count }.by(10)
end
end
end
Running tests with a locally modified version of Gitaly
tmp/tests/gitaly
pinned at the version specified in
GITALY_SERVER_VERSION
. The GITALY_SERVER_VERSION
file supports also
branches and SHA to use a custom commit in .
GITALY_SERVER_VERSION
was aligned with Omnibus syntax.
It no longer supports =revision
, it evaluates the file content as a Git
reference (branch or SHA). Only if it matches a semantic version does it prepend a v
.tmp/tests/gitaly
with a symlink. This is much faster
because it avoids a Gitaly re-install each time you run rspec
.
config.toml
and praefect.config.toml
.
You can copy config.toml
from config.toml.example
, and praefect.config.toml
from config.praefect.toml.example
.
After copying, make sure to edit them so everything points to the correct paths.
rm -rf tmp/tests/gitaly
ln -s /path/to/gitaly tmp/tests/gitaly
make
in your local Gitaly directory before running
tests. Otherwise, Gitaly fails to boot.
make
again.
GITALY_SERVER_VERSION
as described at the beginning of this section.
GITALY_REPO_URL
environment variable when
running tests:
GITALY_REPO_URL=https://gitlab.com/nick.thomas/gitaly bundle exec rspec spec/lib/gitlab/git/repository_spec.rb
GITALY_REPO_URL=https://gitlab+deploy-token-1000:token-here@gitlab.com/nick.thomas/gitaly bundle exec rspec spec/lib/gitlab/git/repository_spec.rb
GITALY_REPO_URL
as a CI/CD variable.
Use a locally modified version of Gitaly RPC client
bundle install
in the tools/protogem
directory of Gitaly.
BUILD_GEM_OPTIONS=--skip-verify-tag make build-proto-gem
_build
directory of Gitaly, unpack the newly created .gem
file and create a gemspec
:
gem unpack gitaly.gem &&
gem spec gitaly.gem > gitaly/gitaly.gemspec
gitaly
line in the Rails’ Gemfile
to:
gem 'gitaly', path: '../gitaly/_build'
bundle install
to use the modified RPC client.
Wrapping RPCs in feature flags
Gitaly
var findAllTagsFeatureFlag = "go-find-all-tags"
featureflag
package:
if featureflag.IsEnabled(ctx, findAllTagsFeatureFlag) {
// go implementation
} else {
// ruby implementation
}
var findAllTagsRequests = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "gitaly_find_all_tags_requests_total",
Help: "Counter of go vs ruby implementation of FindAllTags",
},
[]string{"implementation"},
)
func init() {
prometheus.Register(findAllTagsRequests)
}
if featureflag.IsEnabled(ctx, findAllTagsFeatureFlag) {
findAllTagsRequests.WithLabelValues("go").Inc()
// go implementation
} else {
findAllTagsRequests.WithLabelValues("ruby").Inc()
// ruby implementation
}
import (
"google.golang.org/grpc/metadata"
"gitlab.com/gitlab-org/gitaly/internal/featureflag"
)
//...
md := metadata.New(map[string]string{featureflag.HeaderKey(findAllTagsFeatureFlag): "true"})
ctx = metadata.NewOutgoingContext(context.Background(), md)
c, err = client.FindAllTags(ctx, rpcRequest)
require.NoError(t, err)
GitLab Rails
Feature.enable('gitaly_go_find_all_tags')
gitaly_
.
Testing with GDK
make gitaly-setup
and restart the service with gdk restart gitaly
.
gdk status | grep praefect
.
cat ./services/praefect/run | grep praefect
value of the -config
flag
prometheus_listen_addr
in the configuration file and run gdk restart gitaly
.
curl --silent "/proxy/http://localhost:9236/metrics" | grep go_find_all_tags
bundle install && bundle exec rails console
Feature::Gitaly.server_feature_flags
"gitaly-feature-go-find-all-tags"=>"false"
.
Feature.enable('gitaly_go_find_all_tags')
curl --silent "/proxy/http://localhost:9236/metrics" | grep go_find_all_tags
Using Praefect in test
GITALY_PRAEFECT_WITH_DB=1
in your environment.
gdk start db
eval $(cd ../gitaly && gdk env)
createdb --encoding=UTF8 --locale=C --echo praefect_test
Git references used by Gitaly
Standard Git references
refs/heads/
. Used for branches. See the git branch
documentation.
refs/tags/
. Used for tags. See the git tag
documentation.
GitLab-specific references
refs/keep-around/<object-id>
. References to commits that have pipeline jobs or merge requests. The object-id
points to the commit the pipeline was run on.
refs/merge-requests/<merge-request-iid>/
. Merges merge two histories together. This ref namespace tracks information about a
merge using the following refs under it:
head
. Current HEAD
of the merge request.
merge
. Commit for the merge request. Every merge request creates a commit object under refs/keep-around
.
train
. Commit for the merge train.
refs/pipelines/<pipeline-iid>
. References to pipelines. Temporarily used to store the pipeline commit object ID.
refs/environments/<environment-slug>
. References to commits where deployments to environments were performed.
refs/heads/revert-<source-commit-short-object-id>
. References to the commit’s object ID created when reverting changes.