Go to file
John Cai 8acada46b2 Merge branch 'sh-update-msgpack' into 'master'
Update msgpack to 1.3.0

See merge request gitlab-org/gitaly!1381
2019-07-25 17:59:22 +00:00
_support Generate embedded Go and Ruby proto stubs 2019-07-25 15:11:52 +00:00
.gitlab/issue_templates Add roll out issue template 2019-06-12 14:33:08 +02:00
auth Replace net/context with context package 2019-01-14 12:52:18 +01:00
changelogs/unreleased Merge branch 'sh-update-msgpack' into 'master' 2019-07-25 17:59:22 +00:00
client Close client connections in client.TestDial 2019-06-25 17:43:36 +00:00
cmd Merge branch 'jv-http-progress' into 'master' 2019-07-24 21:30:51 +00:00
danger MR labels to be inherited from issues it closes 2019-06-18 12:53:23 +02:00
doc Document answers to some questions 2019-07-24 21:29:00 +00:00
internal Unable to extract target repo when nested in oneOf field 2019-07-25 15:56:46 +00:00
proto Generate embedded Go and Ruby proto stubs 2019-07-25 15:11:52 +00:00
ruby Merge branch 'sh-update-msgpack' into 'master' 2019-07-25 17:59:22 +00:00
streamio Add environment variable for streamio buffer size 2017-07-13 12:20:53 +02:00
.codeclimate.yml Use gitlab-styles 2017-09-28 17:43:07 +00:00
.codecov.yml Use codecov for better code coverage reporting 2017-10-09 09:08:50 +00:00
.gitattributes Use danger-gitlab to auto review merge requests 2018-04-23 14:59:58 +02:00
.gitignore Replace govendor with 'go mod' 2019-06-05 17:06:46 +00:00
.gitlab-ci.yml Generate embedded Go and Ruby proto stubs 2019-07-25 15:11:52 +00:00
.mailmap Add a mailmap file 2017-10-20 12:02:52 +00:00
.ruby-version Upgrade to Ruby 2.6 2019-05-08 06:18:23 -07:00
CHANGELOG.md Update CHANGELOG.md for 1.55.0 2019-07-18 21:41:28 +00:00
config.praefect.toml.example Handle failover by catching SIGUSR1 signal AND other refactors 2019-07-19 05:22:14 +00:00
config.toml.example Pass down log config through env vars 2019-06-27 12:32:18 -07:00
CONTRIBUTING.md Tweak doc/beginners_guide.md 2019-07-02 11:49:37 +02:00
Dangerfile Have GitalyBot apply the milestone if it can 2019-03-20 15:32:28 +01:00
Dockerfile Update Git version and remove GitLab-Shell clone 2019-07-03 14:14:22 +02:00
Dockerfile.praefect Adding docker-compose for praefect testing cluster 2019-07-05 14:46:49 -07:00
go.mod Handle failover by catching SIGUSR1 signal AND other refactors 2019-07-19 05:22:14 +00:00
go.sum Handle failover by catching SIGUSR1 signal AND other refactors 2019-07-19 05:22:14 +00:00
LICENSE Update license and contributing.md 2017-11-21 12:02:59 +01:00
Makefile Generate embedded Go and Ruby proto stubs 2019-07-25 15:11:52 +00:00
NOTICE Merge branch 'jc-filesystem-uuid' into 'master' 2019-06-28 12:51:29 +00:00
README.md Merge branch 'zj-bump-git-version-test-matrix' into 'master' 2019-07-16 13:03:07 +00:00
STYLE.md Refer to prometheus best practices in STYLE.md 2019-07-10 16:31:02 +00:00
VERSION Version 1.53.0 2019-07-11 12:11:31 +02:00

Gitaly

Quick Links: Roadmap | Want to Contribute? | GitLab Gitaly Issues | GitLab Gitaly Merge Requests |


Gitaly is a Git RPC service for handling all the git calls made by GitLab.

To see where it fits in please look at GitLab's architecture.

Project Goals

Fault-tolerant horizontal scaling of Git storage in GitLab, and particularly, on gitlab.com.

This will be achieved by focusing on two areas (in this order):

  1. Migrate from repository access via NFS to gitaly-proto, GitLab's new Git RPC protocol
  2. Evolve from large Gitaly servers managed as "pets" to smaller Gitaly servers that are "cattle"

Current Status

As of GitLab 11.5, almost all application code accesses Git repositories through Gitaly instead of direct disk access. GitLab.com production no longer uses direct disk access to touch Git repositories; the NFS mounts have been removed.

For performance reasons some RPCs can be performed through NFS still. An effort is made to mitigate performance issues by removing Gitaly N+1. Once that is no longer neccesairy we can conclude the migration project by removing the Git repository storage paths from gitlab-rails's configuration.

In the meantime we are building features according to our roadmap.

If you're interested in seeing how well Gitaly is performing on GitLab.com, read about our observibiltiy story!

Overall

image

By Feature

image

Installation

Most users won't install Gitaly on its own. It is already included in your GitLab installation.

Gitaly requires Go 1.11.4 or newer and Ruby 2.5. Run make to download and compile Ruby dependencies, and to compile the Gitaly Go executable.

Gitaly uses git. Versions 2.21.0 and 2.22.0 are supported.

Configuration

See configuration documentation.

Contributing

See CONTRIBUTING.md.

Name

Gitaly is a tribute to git and the town of Aly. Where the town of Aly has zero inhabitants most of the year we would like to reduce the number of disk operations to zero for most actions. It doesn't hurt that it sounds like Italy, the capital of which is the destination of all roads. All git actions in GitLab end up in Gitaly.

Design

High-level architecture overview:

Gitaly Architecture

Edit this diagram directly in Google Drawings

Gitaly clients

As of Q4 2018, the following GitLab components act as Gitaly clients:

  • gitlab-rails: the main GitLab Rails application.
  • gitlab-shell: for git clone, git push etc. via SSH.
  • gitlab-workhorse: for git clone via HTTPS and for slow requests that serve raw Git data. (example)
  • gitaly-ssh: for internal Git data transfers between Gitaly servers.
  • gitaly-ruby: for RPC's that interact with more than one repository, such as merging a branch.

The clients written in Go (gitlab-shell, gitlab-workhorse, gitaly-ssh) use library code from the gitlab.com/gitlab-org/gitaly/client package.

Further reading

More about the project and its processes is detailed in the docs.

Distributed Tracing

Gitaly supports distributed tracing through LabKit using OpenTracing APIs.

By default, no tracing implementation is linked into the binary, but different OpenTracing providers can be linked in using build tags/build constraints. This can be done by setting the BUILD_TAGS make variable.

For more details of the supported providers, see LabKit, but as an example, for Jaeger tracing support, include the tags: BUILD_TAGS="tracer_static tracer_static_jaeger".

$ make BUILD_TAGS="tracer_static tracer_static_jaeger"

Once Gitaly is compiled with an opentracing provider, the tracing configuration is configured via the GITLAB_TRACING environment variable.

For example, to configure Jaeger, you could use the following command:

GITLAB_TRACING=opentracing://jaeger ./gitaly config.toml

Presentations