The Random Adventure That Is Life (RATIL)

Server Migration

Trevor Steen -

I had a need to migrate this server to a different provider for cost reasons. This was also a good chance to document my process since I did NOT do that last time.

I ended up going with Digital Ocean.  Amazon, Linode, and Digital Ocean all had the same specs for the same price.  Honestly the thing that got me to pick Digital Ocean was simply the number of times I had used their documentation in the past to do server maintenance tasks. Their documentation is always up-to-date, easy to follow, and exactly what I am looking for at the time.

To migrate, I had a handful of tasks I needed to complete:

  • Spin up, do basic configuration, and secure the server
  • Install/Configure NGINX (reverse proxy)
  • Install/Configure MySQL (database)
  • Install/Configure Node.JS (Ghost JS engine)
  • Install/Configure/Restore Ghost
  • Install/Configure/Restore Confluence
  • Update DNS for the new server
  • Update my backup scripts

Spinning Up The New Server

Spinning up the server in Digital Ocean was super easy.  The only thing that I did not like was that if you add an SSH key at deployment, it is for the root user.  I would have liked to be able to kick the server with a non-admin user as well.

I also configured Digital Ocean monitoring for:

  • Inbound/Outbound bandwidth over 70Mbps for 5 minutes
  • CPU over 70% utilization for 5 minutes
  • Disk I/O over 70% utilization for 5 minutes

Updates

First thing was to get everything updated: apt-get update && apt-get upgrade -y

Then set up automatic updates.  I followed this article. In Ubuntu 18, unattended-upgrades is installed by default.  In the /etc/apt/apt.conf.d/50unattended-upgrades file I also uncommented the updates line. In /etc/apt/apt.conf.d/20auto-upgrades I just added the two missing lines from their example into my file.

Create Low Privilege User

useradd -m -s /bin/bash trevor
# Copy the key I uploaded at creation
cp .ssh/authorized_keys /home/trevor/.ssh/
chown trevor:trevor /home/trevor/.ssh/authorized_keys
passwd
usermod -aG sudo trevor
su trevor

Update SSH Config

Port 22 gets scanned and probed like it is going out of style. So I like to avoid that, updating these lines in /etc/ssh/sshd_config

# My super secret SSH port (not really)
Port 12345
PermitRootLogin no
PubkeyAuthentication yes

Firewall

UFW is installed by default in Ubuntu 18 so I just needed to add some rules. Specifically I want to allow SSH on my port and since I am going to be using Cloudflare, I only want to allow HTTP(S) traffic from their IPs. There is a good script here to populate the Cloudflare IPs.

ufw allow 12345
sudo ufw default deny incoming
sudo ufw default allow outgoing

sudo ./cloudflare-ufw.sh

# Add this line to crontab
0 0     * * 1   root    /home/trevor/cloudflare-ufw/cloudflare-ufw.sh > /dev/null 2>&1

Fail2Ban

Followed there two guides:
https://www.digitalocean.com/community/tutorials/how-to-protect-ssh-with-fail2ban-on-ubuntu-14-04
https://www.tricksofthetrades.net/2018/05/18/fail2ban-installing-bionic/

The only considerations were to update the SSH port to my custom port, add my home IP (kind of static) to the ignoreip directive, and change the ban actions to use ufw.conf.

SSMTP and Sendmail

I had to install both in Ubuntu 18 with apt-get install. Then I copied over my config file from the older server in /etc/ssmtp/ssmtp.conf.

Installing Required Software

NGINX

Because I had already done this on my old server, this part was fairly simple. 

sudo apt-get install nginx

Copy config from old server in /etc/nginx/sites-available/ghost.conf

sudo ln -s /etc/nginx/sites-available/ghost.conf /etc/nginx/sites-enabled/ghost.conf

Generate new cert from Cloudflare Origin Cert under the Crypto tab to enable proper (strict) TLS between Cloudflare and my server.  Update NGINX config to use this cert and key by updating any ssl_certificate or ssl_certificate_key lines in /etc/nginx/sites-available/ghost.conf with the appropriate files.

To ensure ONLY Cloudflare can access the server, I also installed the client certificate available here.  Place that certificate in /etc/ssl/certs/cloudflare.crt and add these lines to the appropriate server blocks in /etc/nginx/sites-available/ghost.conf

ssl_client_certificate /etc/ssl/certs/cloudflare.crt;
ssl_verify_client on;

Edit /etc/nginx/nginx.conf and /etc/nginx/sites-available/ghost.conf to remove TLSv1 and TLSv1.1.

MySQL

Just followed this guide.

Node.JS

Just follow their guide being sure to follow the steps to install the Build Tools.

Ghost

Follow the Ghost Install Guide.  I did have some issues with files in my home directory not having proper permissions, I think due to installing Node with sudo. Running chown -R trevor:trevor /home/trevor did the trick.

In the guide, I skipped the UFW section since I already did that. Once the install is complete, login to the Ghost web portal to finish initial configuration.  Due to my firewall rules and NGINX config, I had to set up a local SSH tunnel to the internal Ghost instance.

From here I exported the blog content using the native export function in the Labs tab. This export does NOT include images, so that have to be copied separately. 

# Copy image to local host
scp -r ghost:/var/www/ghost/content/images ./images
# Copy inmages from local host to new server
scp -r images/ ghost2:~
sudo mv ~/images/* /var/www/ghost/content/images
# Reset permissions
chown ghost:ghost /var/www/ghost/content

Update /var/www/ghost/config.production.json with the URL of your blog so that any internal links do not point to localhost.

Import the exported JSON content from the old blog and check any warnings that are produced.  All of my warnings were benign and all data was properly transferred.

Delete the default Ghost posts.

Under the design tab, ensure that the links for any sections that are configured get updated to the public link instead of the internal link.

Ghost Tweaks

I also did a couple tweeks to the base installation.

  • Added PrismJS code syntax highlighting per this guide.
  • Added Disqus comments per this guide
  • Added Table Of Contents to all my posts per this guide. This one required some additional tweaking. In the toc.js file I had to make these changes:
    • Line 10: Change value to 0. This seems to go along with including H1 and just makes the output format look correct
    • Line 57: Per your comment, added h1 to the list.
    • My full section from post.hbs looks slightly different:
<section class="post-full-content">
<div id="toc"></div>

<div class="post-content">
{{content}}
</div>

{{!-- Table of Contents --}}
<script type="text/javascript">
var toc = document.getElementById('toc');
toc.innerHTML = getTocMarkup(document);
</script>
</section>

Confluence

In the old Confluence installation, login as a system admin and create a new backup in the General Configuration > Backup and Restore section and download that to your local host. During the install process, you will be asked for this.

Get the binary download link here.

wget https://www.atlassian.com/software/confluence/downloads/binary/atlassian-confluence-6.11.2-x64.bin
chmod +x atlassian-confluence-6.11.2-x64.bin
sudo ./atlassian-confluence-6.11.2-x64.bin

During install, all defaults can safely be accepted. Once the installer is complete,  install the MySQL drivers. And also take care of Step 2 here.

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.47.tar.gz
sudo cp mysql-connector-java-5.1.47/mysql-connector-java-5.1.47-bin.jar /opt/atlassian/confluence/confluence/WEB-INF/lib
sudo /etc/init.d/confluence restart

Now browse to the web interface (again needing an SSH tunnel) and follow the setup instructions. On the database configuration page, select MySQL and use the "Connection String" method. Your connection string will look something like this (input proper DB name): jdbc:mysql://127.0.0.1:3306/confluence?sessionVariables=tx_isolation='READ-COMMITTED'

At the step to import from a backup, select the file downloaded earlier.

Login using the old user/pass and go to General Configuration to update the "Base URL".  Also follow this article to update the Confluence proxy settings as well.

In /opt/atlassian/confluence/conf/server.xml I needed to add address="ip6-localhost" to the Connector block to force Confluence to only listen on localhost.

Finally, for my backups I needed to add my standard user to the Confluence group: sudo usermod -aG confluence trevor.

OSSEC

Follow the guide here to install OSSEC-HIDS as a local installation.

Follow this guide to configure POSTFIX to send OSSEC emails.  I also changed inet_interfaces in main.cf to 127.0.0.1 so that I did not have an SMTP server listening to the world.

DNS

Now that everything was set up and running, time to cut over my DNS configuration.  This was amazingly easy: log in to Cloudflare, update IPs...done!  The changes were almost immediate.

Backups

At this point I needed to move my backup scripts over to the new server, update my NAS rsync jobs to point to the new IP, and copy over my cronjobs. This also required a new SSH key pair for my NAS.