Gitlab in Production – Part 4: Gitlab Installation

Welcome to part 4 of my Gitlab series – in this part you’ll see how Gitlab can be installed and how you do the initial configuration.

Note: this part is still work in progress so let me know if I need to be more elaborate with my explanations

Gitlab Installation

The gitlab installation manual recommends using their Omnibus packages for the setup:

We recommend installing the Omnibus package instead of installing GitLab from source. Omnibus GitLab takes just 2 minutes to install and is packaged in the popular deb and rpm formats. Compared to an installation from source, the Omnibus package is faster to install and upgrade, more reliable to upgrade and maintain, and it shortens the response time for our subscribers’ issues. A package contains GitLab and all its depencies (Ruby, PostgreSQL, Redis, Nginx, Unicorn, etc.), it can be installed without an internet connection. For troubleshooting and configuration options please see the Omnibus GitLab readme.

In addition to the Omnibus packages they provide bash scripts that take care of everything for you (chef, puppet and bundler configurations are available as well) and that’s what we’re going to use. In our case, since we’re using Debian, we run the deb installation script like this:

curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash

What’s happening behind the scenes is this:

  • Detect OS (and its version)
  • Install required packages for the actual Gitlab installation
  • Add Gitlab Package Repositories to /etc/apt/sources.list.d/
  • Add PGP Keys for Gitlab Package Repositories

It adds the following packages…

apt-get install -q -y curl
apt-get install -y debian-archive-keyring &> /dev/null
apt-get install -y apt-transport-https

… then adds the Gitlab Repositories…

ladmin@srvapp037:~/scripts$ cat /etc/apt/sources.list.d/gitlab_gitlab-ce.list
# this file was generated by packages.gitlab.com for
# the repository at https://packages.gitlab.com/gitlab/gitlab-ce

deb https://packages.gitlab.com/gitlab/gitlab-ce/debian/ jessie main
deb-src https://packages.gitlab.com/gitlab/gitlab-ce/debian/ jessie main

… and finally installs the GPG key:

curl https://packages.gitlab.com/gpg.key 2> /dev/null | apt-key add - &>/dev/null

So now that we’ve run the script which has taken care of all the prerequisites for us we can finally install Gitlab CE:

sudo apt-get update
sudo apt-get install gitlab-ce

It’ll take a bit and you’ll see the output from their chef recipes before it shows you that all Gitlab services have been started.

Gitlab Configuration

SSL Configuration

I’ve created a SHA2 certificate for the Gitlab web server with the SAN field (Subject Alternative Name) containing the alias name of the server name that we’re going to use from now on.

common name: srvapp037.mydomain.internal
SAN: srvapp037, git, git.mydomain.internal

Next we create the certificate folder on our git server, apply proper permissions and place our certificates in there:

sudo mkdir /etc/gitlab/ssl
sudo chmod 700 /etc/gitlab/ssl
sudo vim /etc/gitlab/ssl/git.mydomain.internal.crt
sudo vim /etc/gitlab/ssl/git.mydomain.internal.key

Note: I’ve pasted the certificate contents from the clipboard, that’s why I’m using vim here

After copying my new cert to the server I have to configure our Gitlab webserver by editing the Gitlab configuration file /etc/gitlab/gitlab.rb:

The most important line here is

nginx['ssl_ciphers'] = "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256"

It contains two cipher suites that make use of elliptic curve key exchanges. These are TLS 1.2 only which is fine since we don’t have to ensure backwards compatibility (IE10 doesn’t support TLS 1.2 but isn’t supported anymore anyway).

Then restart Gitlab with:

sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart

So far so good, right? Nope, nginx doesn’t start – browsing to https://git.mydomain.internal gives me a server error message. So what’s the problem?

Well, it turns out that nginx doesn’t know how to deal with the passphrase on my certificate key. You can see that by looking at the nginx logs under /var/log/gitlab/nginx/:

ladmin@srvapp037:~$ sudo grep –iR "error" /var/log/gitlab/nginx/
2015/11/03 17:38:04 [emerg] 21995#0: SSL_CTX_use_PrivateKey_file("/etc/gitlab/ssl/git.mydomain.internal.key") failed (SSL: error:0906406D:PEM routines:PEM_def_callback:problems getting password error:0906A068:PEM routines:PEM_do_header:bad password read error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib)

You can remove the passphrase from the key with OpenSSL like this:

sudo openssl rsa -in /etc/gitlab/ssl/git.mydomain.internal.key -out /etc/gitlab/ssl/git.mydomain.internal.nocrypt.key

Since we now have a key file without passphrase we need to change one line in the gitlab nginx configuration as follows:

# old:
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/git.mydomain.internal.key"
# new:
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/git.mydomain.internal.nocrypt.key"

LDAP Authentication

Note: we’re in the process of setting up a PKI which is why we still connect via LDAP (TCP\389) instead of LDAPS (TCP\636) – the required steps for LDAPS will be added once the PKI is in place.

Again we edit our global Gitlab configuration file /etc/gitlab/gitlab.rb:

I believe that is fairly self-explanatory but I’ll cover the settings really quick anyway. The easiest way to get the bind_dn path is to use PowerShell with the Active Directory module:

Import-Module ActiveDirectory
Get-AdUser -Identity <ldap-service-account-samaccountname>

Base – will be used as searchbase to find your LDAP users
active_directory: true – just means that the LDAP server is a Microsoft AD
allow_username_or_email_login: false – we want users to login with their samaccountname and not their email address so we disable this
block_auto_created_users: true – if an LDAP user authenticates against Gitlab for the first time it is imported into Gitlab but needs to be unblocked by a Gitlab administrator before the user can actually log in

So now that this is in place we can apply the configuration by running

sudo gitlab-ctl reconfigure

SMTP Configuration

In our case the SMTP configuration is fairly simple:

I enable SMTP, specify our internal relay with its SMTP port and domain and don’t use authentication (Note: I know but the Gitlab server and relay are on the management network that has additional security measures in place). As you can see I set the Reply-To field to my public email address so that I can respond to accidental replies to Gitlab emails.

Verify Webserver Security

In order to verify that the webserver is secure I use the testssl shell script from https://github.com/drwetter/testssl.sh. Just download the archive and extract it:

testssl-structure

 

What we have now is a fully usable Gitlab Server. In the next part I’m going to cover sensible user defaults and the steps to configure your git client for use with our server.

Advertisements

Gitlab in Production – Part 3: Server Setup

Welcome to part 3 of my Gitlab series. I’m going to cover the pitfalls I encountered while trying to get Gitlab up and running as well as the steps required to set it up yourself.

Our CIO has defined a set of rules and guidelines through a system called Enterprise Architecture Management (EAM) that dictates which software and operating systems we should use in our corporate IT environment.

EAM Compliance

We use SUSE Linux Enterprise Server (SLES) 11 SP4 as standard Linux OS. SLES is not listed as supported OS for Gitlab but I’d really like to use it in order to keep the number of different OS’ I have to support to a minimum (see EAM). The setup itself is simple, just download the Gitlab omnibus installer and run the setup, right? Nope, even though SLES uses rpm the installation with the CentOS rpm installation script fails. No biggie, then I just download the rpm and install it manually. Again, no sucsess. During the troubleshooting process I see that for Gitlab glibc version 2.14 is required but SLES 11 SP4 ships with 2.11. Oh well, it was worth a try.

But wait, this is when I found the Bitnami Gitlab installer that bundles all of the required components in an all-in-one installation. I download the setup, make it executable with chmod +x /tmp/bitnami-gitlab-7.14.3-0-linux-x64-installer.run and run it. During the setup I answer a few configuration options and we’re done. Gitlab is up and running. Phew, easier than expected. Everything works until…. the system is rebooted. The redis service won’t start no matter what I do. Reading up on the Bitnami Gitlab installer feedback made me realize that its upgrades are error-prone and time consuming. This is where I stopped wasting my time just to be compliant with our corporate policy.

Fresh start

I had stopped trying to get Gitlab running on SLES at this point as the maintenance costs of running Gitlab on an unsupported OS would’ve been a lot more than what we have to invest in supporting an additional server OS.

I discussed the OS choices with my colleagues and we didn’t really have a preference so we went with Debian 8. 2 (Jessie). I provision a new VM in vSphere, grab a free IP address from our Microsoft IPAM (IP Address Management) service (tracking IPs in spreadsheets nowadays is a big no-no) and run the Debian setup.

Firewall rules

In order for the Debian server to reach its update servers from our internal network the following domains had to be whitelisted:

# outgoing traffic (http/https)
security.debian.org
ftp.de.debian.org
ftp2.de.debian.org
# the following domain is required for the Gitlab installation
packages.gitlab.com

Additional Packages

Since I’m more familiar with Ubuntu I was missing the sudo command on Debian. Switch to root and add our new local user (linux admin = “ladmin”) to the sudo group to grant it admin privileges and set his password (log out and back in to apply them):

su
apt-get install sudo
adduser ladmin sudo
passwd ladmin
install sudo as root and set up user with admin privileges

I need a few packages to be productive on the machine so let’s install the following packages:

sudo apt-get update && sudo apt-get upgrade
sudo apt-get install vim screen tree open-vm-tools unattended-upgrades apt-listchanges curl ca-certificates
Packages excluding the ones required for Gitlab

Welcome screen

If you’ve ever used Ubuntu you’ve probably come across the output of their landscape monitoring tool:ubuntu-landscape

A while ago I’ve written a shell script to output the same information on our SLES boxes. I had to change it a little to make it work on Debian. [Link to article will be added later]

I store the script as /etc/profile.d/post_login_banner_ourcompanyname.sh so that it will be displayed everytime a user logs in:

Using username "ladmin".
LEGAL NOTICE: Use of this device restricted to authorized persons. 
This device is subject to monitoring at all times, use of this device 
constitutes consent to monitoring.
Authenticating with public key "MyDomain Key - ladmin srvapp037 - megamorf" from agent
No mail.
Last login: Fri Nov 6 17:57:08 2015 from xpc001.mydomain.internal


Welcome to Debian GNU/Linux 8.2 (jessie)
 (GNU/Linux 3.16.0-4-amd64 x86_64)

 * motd made by: megamorf

 System information as of Fr 6. Nov 18:00:01 CET 2015
 You're connected from: 10.10.9.14

 System load: 0.02 Memory usage:        751/1000MB(75%)
 Processes:   127  IP address for eth0: 10.10.4.104
 User count:  1

 Disk Utilization:
 Usage of / 25% of 8,2G
 Usage of /home 1% of 26G
 Usage of /var 32% of 2,7G
 Usage of /tmp 24% of 360M

 Connected Users:
 UserLogin         timeSource
 ladmin 2015-11-05 16:13(xpc001:S.0)
 ladmin 2015-11-05 16:13(xpc001:S.1)
 ladmin 2015-11-06 17:57(xpc001.mydomain.internal)

ladmin@srvapp037:~$
Custom Debian Login Message

Custom gnu screen configuration

I like to run screen to be able to use multiple persistent terminals on the target machine that survive a disconnected SSH session.

Now that screen is installed it’s time to customize the global screen configuration /etc/screenrc. Add the following lines at the end of the configuration file. This will add the computername, the current time and tabs to the screen terminal.

Hardening SSH

Sources:

So the first thing I do is define the order of SSH hostkeys from secure to less secure. Root Login and X11 forwarding is not permitted. Finally I define the allowed SSH ciphers,  MACs and key exchange algorithms from secure to less secure.

root@srvapp037:/tmp# egrep -v "^#|^$" /etc/ssh/sshd_config

# Supported HostKey algorithms by order of preference for protocol version 2.
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key

# Use kernel sandbox mechanisms where possible in unprivileged processes
UsePrivilegeSeparation yes

PermitRootLogin no
StrictModes yes
RSAAuthentication yes
PubkeyAuthentication yes
IgnoreRhosts yes
RhostsRSAAuthentication no
HostbasedAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
X11Forwarding no
PrintMotd no

# Corporate Security Settings:
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,umac-128@openssh.com
KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256
relevant lines in /etc/ssh/sshd_config colored in red

Don’t forget to apply the new configuration with sudo service ssh restart

Enable automatic installation of security updates

Since this is the only non-SLES machine that isn’t a vendor-hardened virtual appliance in our environment we need to ensure that security updates are installed on a regular basis. But we’re lazy so we let the server take care of that. We take the vanishingly small risk of the system breaking due to a security update in exchange for the time savings from automatic update installation (the system is backed up on  a daily basis anyway)

1) uncomment two lines in /etc/apt/apt.conf.d/50unattended-upgrades to enable unattended security updates and notify you of update problems:

# --- output omitted ---
Unattended-Upgrade::Origins-Pattern {
        // Codename based matching:
        // This will follow the migration of a release through different
        // archives (e.g. from testing to stable and later oldstable).
//      "o=Debian,n=jessie";
//      "o=Debian,n=jessie-updates";
//      "o=Debian,n=jessie-proposed-updates";
//      "o=Debian,n=jessie,l=Debian-Security";

        // Archive or Suite based matching:
        // Note that this will silently match a different release after
        // migration to the specified archive (e.g. testing becomes the
        // new stable).
//      "o=Debian,a=stable";
//      "o=Debian,a=stable-updates";
//      "o=Debian,a=proposed-updates";
        "origin=Debian,codename=${distro_codename},label=Debian-Security";
};
# --- output omitted ---
// Send email to this address for problems or packages upgrades
// If empty or unset then no email is sent, make sure that you
// have a working mail setup on your system. A package that provides
// 'mailx' must be installed. E.g. "user@example.com"
Unattended-Upgrade::Mail "root";
# --- output omitted ---
Uncomment the red lines

2) run sudo dpkg-reconfigure -plow unattended-upgrades to create the apt 20auto-upgrades configuration file

3) verify the /etc/apt/apt.conf.d/20auto-upgrades file looks like this:

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
Set both entries to 1 in /etc/apt/apt.conf.d/20auto-upgrades

You can then do a test run to see that everything is working as expected (Note: it’s unattended-upgrade in this case, without a trailing “s”):

sudo unattended-upgrade --debug --dry-run

Details can be found in the log file /var/log/unattended-upgrades/unattended-upgrades.log

This concludes our preparation for the actual Gitlab installation. Stay tuned for part 4 and don’t hesitate to give constructive feedback 😉

Gitlab in Production – Part 2: Design Goals

Welcome to part 2 of my Gitlab in Production series where I’ll guide you through the process of getting Gitlab set up yourself.

These are my design goals:

Mandatory

  • run Gitlab CE on-premise
  • server-side OS is Linux
  • client-side OS is Windows
  • create basic operating manual for my team (git & gitlab)
  • hardening (even though the server is not going to be exposed to the internet it should be secured appropriately)
  • using gitlab should be as easy and accessible as possible (we have a number of mouse jockeys on our team, go figure :/ )
  • document Gitlab upgrade procedure
  • create required documentation (network diagram, recovery scenarios, etc.)

Optional

  • create concept for Gitlab CI (Continuous Integration) which is included since version 8
  • use Gitlab CI to run test and build tasks for our projects

As you can see these requirements are not out of the ordinary. Implementation should be a breeze but as always you’ll see that things can be more complicated than you thought.

Stay tuned for part 3 where I cover the Linux setup and basic hardening steps.

Gitlab in Production – Part 1: Introduction

Most of you are probably somewhat familiar with github or have at least heard of the source control system git. But what is Gitlab exactly? Wikipedia describes Gitlab as follows:

GitLab, the software, is a web-based Git repository manager with wiki and issue tracking features.

And indeed, that is Gitlab in a nutshell. You can either host your projects on the official and publicly hosted Gitlab servers or run your own instance in the cloud/on-premise.

prkjrko
Gitlab dashboard

Gitlab itself comes in two flavours: as free Community Edition (CE) or paid Enterprise Edition (EE). The main difference between these two, as the name implies, is that Gitlab EE is under a proprietary license, and contains features not present in the CE version.

The Gitlab software is based on ruby and supports the following operating systems (as of 2016.02.15):

  • Ubuntu 12.04
  • Ubuntu 14.04 (recommended, for 15.04 select and download the package manually)
  • Debian 7
  • Debian 8
  • CentOS 6 (and RedHat/Oracle/Scientific Linux 6)
  • CentOS 7 (and RedHat/Oracle/Scientific Linux 7)
  • Raspberry PI 2 on Raspbian Wheezy

In the course of this blog series I’m going to cover hosting your own Gitlab CE instance with the necessary hardening measures required for production use.

Stay tuned for part 2.