Gitlab in Production – Part 3: Server Setup

Welcome to part 3 of my Gitlab series. I’m going to cover the pitfalls I encountered while trying to get Gitlab up and running as well as the steps required to set it up yourself.

Our CIO has defined a set of rules and guidelines through a system called Enterprise Architecture Management (EAM) that dictates which software and operating systems we should use in our corporate IT environment.

EAM Compliance

We use SUSE Linux Enterprise Server (SLES) 11 SP4 as standard Linux OS. SLES is not listed as supported OS for Gitlab but I’d really like to use it in order to keep the number of different OS’ I have to support to a minimum (see EAM). The setup itself is simple, just download the Gitlab omnibus installer and run the setup, right? Nope, even though SLES uses rpm the installation with the CentOS rpm installation script fails. No biggie, then I just download the rpm and install it manually. Again, no sucsess. During the troubleshooting process I see that for Gitlab glibc version 2.14 is required but SLES 11 SP4 ships with 2.11. Oh well, it was worth a try.

But wait, this is when I found the Bitnami Gitlab installer that bundles all of the required components in an all-in-one installation. I download the setup, make it executable with chmod +x /tmp/bitnami-gitlab-7.14.3-0-linux-x64-installer.run and run it. During the setup I answer a few configuration options and we’re done. Gitlab is up and running. Phew, easier than expected. Everything works until…. the system is rebooted. The redis service won’t start no matter what I do. Reading up on the Bitnami Gitlab installer feedback made me realize that its upgrades are error-prone and time consuming. This is where I stopped wasting my time just to be compliant with our corporate policy.

Fresh start

I had stopped trying to get Gitlab running on SLES at this point as the maintenance costs of running Gitlab on an unsupported OS would’ve been a lot more than what we have to invest in supporting an additional server OS.

I discussed the OS choices with my colleagues and we didn’t really have a preference so we went with Debian 8. 2 (Jessie). I provision a new VM in vSphere, grab a free IP address from our Microsoft IPAM (IP Address Management) service (tracking IPs in spreadsheets nowadays is a big no-no) and run the Debian setup.

Firewall rules

In order for the Debian server to reach its update servers from our internal network the following domains had to be whitelisted:

# outgoing traffic (http/https)
security.debian.org
ftp.de.debian.org
ftp2.de.debian.org
# the following domain is required for the Gitlab installation
packages.gitlab.com

Additional Packages

Since I’m more familiar with Ubuntu I was missing the sudo command on Debian. Switch to root and add our new local user (linux admin = “ladmin”) to the sudo group to grant it admin privileges and set his password (log out and back in to apply them):

su
apt-get install sudo
adduser ladmin sudo
passwd ladmin
install sudo as root and set up user with admin privileges

I need a few packages to be productive on the machine so let’s install the following packages:

sudo apt-get update && sudo apt-get upgrade
sudo apt-get install vim screen tree open-vm-tools unattended-upgrades apt-listchanges curl ca-certificates
Packages excluding the ones required for Gitlab

Welcome screen

If you’ve ever used Ubuntu you’ve probably come across the output of their landscape monitoring tool:ubuntu-landscape

A while ago I’ve written a shell script to output the same information on our SLES boxes. I had to change it a little to make it work on Debian. [Link to article will be added later]

I store the script as /etc/profile.d/post_login_banner_ourcompanyname.sh so that it will be displayed everytime a user logs in:

Using username "ladmin".
LEGAL NOTICE: Use of this device restricted to authorized persons. 
This device is subject to monitoring at all times, use of this device 
constitutes consent to monitoring.
Authenticating with public key "MyDomain Key - ladmin srvapp037 - megamorf" from agent
No mail.
Last login: Fri Nov 6 17:57:08 2015 from xpc001.mydomain.internal


Welcome to Debian GNU/Linux 8.2 (jessie)
 (GNU/Linux 3.16.0-4-amd64 x86_64)

 * motd made by: megamorf

 System information as of Fr 6. Nov 18:00:01 CET 2015
 You're connected from: 10.10.9.14

 System load: 0.02 Memory usage:        751/1000MB(75%)
 Processes:   127  IP address for eth0: 10.10.4.104
 User count:  1

 Disk Utilization:
 Usage of / 25% of 8,2G
 Usage of /home 1% of 26G
 Usage of /var 32% of 2,7G
 Usage of /tmp 24% of 360M

 Connected Users:
 UserLogin         timeSource
 ladmin 2015-11-05 16:13(xpc001:S.0)
 ladmin 2015-11-05 16:13(xpc001:S.1)
 ladmin 2015-11-06 17:57(xpc001.mydomain.internal)

ladmin@srvapp037:~$
Custom Debian Login Message

Custom gnu screen configuration

I like to run screen to be able to use multiple persistent terminals on the target machine that survive a disconnected SSH session.

Now that screen is installed it’s time to customize the global screen configuration /etc/screenrc. Add the following lines at the end of the configuration file. This will add the computername, the current time and tabs to the screen terminal.

Hardening SSH

Sources:

So the first thing I do is define the order of SSH hostkeys from secure to less secure. Root Login and X11 forwarding is not permitted. Finally I define the allowed SSH ciphers,  MACs and key exchange algorithms from secure to less secure.

root@srvapp037:/tmp# egrep -v "^#|^$" /etc/ssh/sshd_config

# Supported HostKey algorithms by order of preference for protocol version 2.
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key

# Use kernel sandbox mechanisms where possible in unprivileged processes
UsePrivilegeSeparation yes

PermitRootLogin no
StrictModes yes
RSAAuthentication yes
PubkeyAuthentication yes
IgnoreRhosts yes
RhostsRSAAuthentication no
HostbasedAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
X11Forwarding no
PrintMotd no

# Corporate Security Settings:
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,umac-128@openssh.com
KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256
relevant lines in /etc/ssh/sshd_config colored in red

Don’t forget to apply the new configuration with sudo service ssh restart

Enable automatic installation of security updates

Since this is the only non-SLES machine that isn’t a vendor-hardened virtual appliance in our environment we need to ensure that security updates are installed on a regular basis. But we’re lazy so we let the server take care of that. We take the vanishingly small risk of the system breaking due to a security update in exchange for the time savings from automatic update installation (the system is backed up on  a daily basis anyway)

1) uncomment two lines in /etc/apt/apt.conf.d/50unattended-upgrades to enable unattended security updates and notify you of update problems:

# --- output omitted ---
Unattended-Upgrade::Origins-Pattern {
        // Codename based matching:
        // This will follow the migration of a release through different
        // archives (e.g. from testing to stable and later oldstable).
//      "o=Debian,n=jessie";
//      "o=Debian,n=jessie-updates";
//      "o=Debian,n=jessie-proposed-updates";
//      "o=Debian,n=jessie,l=Debian-Security";

        // Archive or Suite based matching:
        // Note that this will silently match a different release after
        // migration to the specified archive (e.g. testing becomes the
        // new stable).
//      "o=Debian,a=stable";
//      "o=Debian,a=stable-updates";
//      "o=Debian,a=proposed-updates";
        "origin=Debian,codename=${distro_codename},label=Debian-Security";
};
# --- output omitted ---
// Send email to this address for problems or packages upgrades
// If empty or unset then no email is sent, make sure that you
// have a working mail setup on your system. A package that provides
// 'mailx' must be installed. E.g. "user@example.com"
Unattended-Upgrade::Mail "root";
# --- output omitted ---
Uncomment the red lines

2) run sudo dpkg-reconfigure -plow unattended-upgrades to create the apt 20auto-upgrades configuration file

3) verify the /etc/apt/apt.conf.d/20auto-upgrades file looks like this:

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
Set both entries to 1 in /etc/apt/apt.conf.d/20auto-upgrades

You can then do a test run to see that everything is working as expected (Note: it’s unattended-upgrade in this case, without a trailing “s”):

sudo unattended-upgrade --debug --dry-run

Details can be found in the log file /var/log/unattended-upgrades/unattended-upgrades.log

This concludes our preparation for the actual Gitlab installation. Stay tuned for part 4 and don’t hesitate to give constructive feedback 😉

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s