Essential security for Linux servers

Pixelated scene from the movie Sneakers with Whistler hacking through his visor

In this post we are going to secure a freshly provisioned Linux server to protect it against the most common attack vectors. The aim of this approach is to create an acceptable security baseline while keeping the server easy to manage.

My starting point is a newly created DigitalOcean droplet running Ubuntu 23.04 x64 but the following steps will apply to most Linux distros.

Let’s get started by connecting to the droplet via ssh using the root account:

# connect to the server
ssh root@[IP-ADDRESS]

Once connected, you will be prompted to change the root password. Make sure you select a password with high entropy and store it in your password manager.

The next step is setting the correct timezone and enabling network time sync:

# set the timezone
dpkg-reconfigure tzdata

# enable network time sync
timedatectl set-ntp true

Now we are going to update the OS:

# update the repositories
apt update

# upgrade all the packages
apt upgrade

Using the root account for ssh and managing the server will increase our security risk due to its elevated privileges. To mitigate some of the risk we are going to create a new non-privileged user and add it to the sudoers to allow it to perform administrative tasks when required:

# create the new user
useradd --create-home --shell /bin/bash [USERNAME]

# add the user to the sudoers
usermod -aG sudo [USERNAME]

# change the password
passwd [USERNAME]

Now that we have a new user we can tweak the sshd config to lock down the openssh server:

# edit /etc/ssh/sshd_config
Port 3392
PermitRootLogin no
PubkeyAuthentication yes
PasswordAuthentication no
AllowUsers [USERNAME]

A brief explaination of the changes above (in order):

Change the default port to reduce some of the noise caused by bots trying to brute force our openssh server on port 22
Disable login via ssh using the root account
Enable public key authentication (we are going to generate a key pair to log into the server)
Disable password authentication
Permit login via ssh only to the newly created user

As an additional security measure, if you only log into the server from a specific IP address, you can lock down the openssh service even more by changing the last line to this:

# edit /etc/ssh/sshd_config
...
AllowUsers [USERNAME]@[IP-ADDRESS]

Just make sure you have a static address or you won’t be able to login into the server if your IP changes.

Now that our ssh service is secured, we need to add our public key to the server to be able to log in. If you don’t already have a key go ahead and create one by following the steps on github help page.

With our public key ready we need to create the file structure, add the key to the authorized_keys file and assign the correct permissions:

# create the .ssh folder in the new user home
mkdir /home/[USERNAME]/.ssh

# restric access to it
chmod 700 /home/[USERNAME]/.ssh

# paste your public key into the authorized_keys file
nano /home/[USERNAME]/.ssh/authorized_keys

# lock down access to the key file
chmod 400 /home/[USERNAME]/.ssh/authorized_keys

# change ownership of the .ssh folder recursively
chown [USERNAME]:[USERNAME] -R /home/[USERNAME]/.ssh

Next step we are enabling the firewall to block any unsolicited incoming traffic.

We are going to use the ufw package, an easy to use front-end to iptables.

The only open port required at this stage is port 3392 to ssh into the server:

# open port 3392 on the firewall
ufw allow 3392/tcp

# enable the firewall at boot
ufw enable

# check the firewall status
ufw status

We are now ready to reload the ssh daemon to test our configuration:

systemctl reload sshd

At this stage it’s good practice to fire up a new terminal window (while keeping the old ssh session open) to check that we are able to connect using our new user and public key:

# using a new terminal window
ssh -p 3392 [USERNAME]@[IP-ADDRESS]

If the login fails we can still use the previous terminal session to debug the issue. If the connection works fine we can disconnect and go back to our old session.

In the next step we are going to install and configure two new packages.

The first one, called fail2ban, is a monitoring service that constantly scans the system logs and temporarily bans users via the system firewall if it detects suspicious activity (for instance too many failed ssh log in attempts from a specific IP address over a specified time range).

The second package, called ssmtp, is a simple smtp client that will be used by fail2ban to notify us via email.

You can read more about the two packages on their respective pages fail2ban and ssmtp of the arch linux wiki.

Let’s install the required packages:

apt install fail2ban ssmtp

To send emails from the server you can use any smtp relay you have access to.

Edit the ssmtp config file to add your smtp details:

# /etc/ssmtp/ssmtp.conf
root=postmaster
mailhub=mail
hostname=[HOSTNAME]
FromLineOverride=YES

# smtp server
mailhub=[SMTP]:[PORT]
UseSTARTTLS=yes
UseTLS=yes
AuthUser=[EMAIL]
AuthPass=[PASSWORD]

To allow ssmtp to send out emails from real users on the server we will need to edit the aliases:

# edit /etc/ssmtp/revaliases
root:[EMAIL]:[SMTP]:[PORT]
postmaster:[EMAIL]:[SMTP]:[PORT]
[USERNAME]:[EMAIL]:[SMTP]:[PORT]

Fail2ban uses jail files to configure its behaviour so we need to copy the sample file to create our configuration:

cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Now that we have a new jail file we can tweak the configuration, it’s always a good idea to check the fail2ban man page for a detailed explaination of the settings:

# edit /etc/fail2ban/jail.local

# to email address
destemail = [EMAIL]

# from email address
sender = [EMAIL]

mta = sendmail

# tweak the action to get additional info in the email
action = %(action_mwl)s

# enable the sshd jail to monitor ssh connections
[sshd]
enabled = true

Finally we can start the fail2ban service:

systemctl start fail2ban

Last step is to setup automatic updates to avoid security breaches caused by vulnerabilities in out-of-date packages.

We start by configuring the schedule:

# /etc/apt/apt.conf.d/10periodic
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";

Then we enable the unattended upgrades:

# /etc/apt/apt.conf.d/50unattended-upgrades
Unattended-Upgrade::Allowed-Origins {
...
# comment-in the updates
"${distro_id}:${distro_codename}-updates";
...
}

# comment-in and enable automatic reboot
Unattended-Upgrade::Automatic-Reboot "true";

# comment-in the reboot time
Unattended-Upgrade::Automatic-Reboot-Time "02:00";

That’s all folks 🙌 the droplet is now ready and slightly more secure than when we started.

Time for a reboot, logging-in using the new user and configuring the server for its intended purpose.

And don’t forget to enjoy your newly-acquired sense of security 😜