Migration of Wekan from local to AWS

I migrated Wekan from my local Mac to the Amazon cloud.

In the picture on the right side of the page, you may see how complicated my setup was before.

I took Vagrant and put all the components into one virtual machine. In the end, Vagrant for me is just a wrapper around VirtualBox. At this point, I had Wekan on local, but I wanted to use it when I'm not at home. The next step was to expose it to the wild Internet.

I created a subdomain in the domain I already own and associated this subdomain with the public IP address of my router at home.

Wekan hosted on my Mac at home.

This setup worked well only for a day or so because my internet provider changes the IP address of my router every 24 hours. I could use a service like DynDNS to have a stable name for my Wekan installation, but it is not free. To work-around that I wrote a python script which checked every minute the public IP address of my router, and made changes in the DNS when the address gets updated.

The next thing I set up was the Nginx reverse-proxy and Certbot from Let's Encrypt to have SSL/TLS certificate for the Wekan. Both apps I run in Docker containers.

I put all my tasks into Wekan to have only one source of truth about the work I need to do. The backup of the database is a must for me. Unfortunately, Wekan itself doesn't have "backup" feature, but it wasn't hard to craft another script to pull data out of Wekan's MongoDB regularly. This data I then stored on the external drive and in the cloud, using two different software solution.

As for me, this setup has only one significant advantage. It was free, except for electricity. Everything else was a downside.

When my internet provider interrupts the service (oh, well... we are in the 21st century only), I cannot access my boards. Because I have no other places where I manage my tasks, at those moments when it happened, I was slightly angry.

The router software lives its own life, and sometimes the port forwarding was broken. Again, no access to my boards.

When you run a docker on Mac for about 2 or 3 months, it might crash. Then the proxy doesn't work, and I don't have access to the board.

And don't forget Chinese hackers. Every day somebody tried to scan or hack my machine. I saw thousands of attempts in my Nginx's logs. However, the way Nginx works in a docker on Mac is so unique, so it wasn't possible to set up GeoIP blocks.

One day I thought, OK, that's enough from me. Let's do it the right way.

I launched two smallest AWS EC2 instances. Today it is t2.nano. On the first instance, I installed MongoDB and restored all my data into it. On the second instance, I run three docker containers with Wekan, Nginx, and Certbot. I opened only one port on the instance with MongoDB and allowed connections initiated only from the second instance. This means, no-one, except Wekan, can connect to the database. The IP address of the instance with Wekan I put into DNS, and I don't need to update this record regularly, because the IP address is static. In a couple of minutes, I also configured snapshots of the whole disk of the instance with MongoDB, so in case of a crash, I can restore not only DB data but also database itself.

The same Wekan app, but now it is on AWS.

Just for the record, the next section is a chain of commands I executed to migrate from Wekan on my Mac to the Wekan in the AWS.

# First I need to install MongoDB. I ssh into EC2 instance
ssh -i ~/.ssh/wekan.pem ubuntu@wekan.mydomainname.com

# update software packages list
sudo apt-get update

# check that Mongo isn't installed (shouldn't be, it's fresh EC2 instance)
sudo apt list --installed | grep mongodb

# then just follow installation instructions from MongoDB docs
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
sudo apt-get update
sudo apt-get install -y mongodb-org

# prevents MongoDB uncontrolled auto-updates
echo "mongodb-org hold" | sudo dpkg --set-selections
echo "mongodb-org-server hold" | sudo dpkg --set-selections
echo "mongodb-org-shell hold" | sudo dpkg --set-selections
echo "mongodb-org-mongos hold" | sudo dpkg --set-selections
echo "mongodb-org-tools hold" | sudo dpkg --set-selections

# start the service
sudo service mongod start

# then I bind mongo to all network interfaces (replace 127.0.0.1 to 0.0.0.0 in /etc/mongod.conf)
sudo vim /etc/mongod.conf

# restart and status check
sudo service mongod restart
sudo service mongod status
exit

# Now I'm back on my Mac.
# I do not have Mongo and its utils on my Mac, so I run a docker container with Mongo.
docker run -ti \
   --network host \
   --entrypoint bash \
   -v `PWD`/backups:/backups:ro \
   mongo:4.0.10-xenial

# Inside Mongo container
# refreshes the list of available software
apt-get update

# installs 'unzip' utility                            
apt-get install unzip

# unpacks certain file with backup into /backup folder
unzip backups/2019-06-29_14.39.03.zip -d /backup

# restores data into MongoDB on AWS
mongorestore -d wekan /backup/2019-06-29_14.39.03/dump/wekan/ --host 52.59.236.34:27017

Here is a configuration of my docker containers (I use docker-compose):

version: '3.7'
services:
  nginx:
    image: nginx
    restart: always
    ports:
      - "80:80"
      - "443:443"
    networks:
      - nginx-wekan-network
    volumes:
      - ./data/nginx:/etc/nginx/conf.d
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
    # This makes nginx reload its configuration (and certificates) every six hours in the background and launches nginx in the foreground.
    command: "/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"'"
  certbot:
    image: certbot/certbot
    volumes:
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
    # This will check if your certificate is up for renewal every 12 hours as recommended by Let’s Encrypt.
    entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"
  wekan:
    image: quay.io/wekan/wekan
    restart: always
    ports:
      - "8080:8080"
    networks:
      - nginx-wekan-network
    environment:
      - MONGO_URL=mongodb://ip-172-31-17-195.eu-central-1.compute.internal:27017/wekan
      - ROOT_URL=http://wekan.mydomainname.com
      - MAIL_URL='smtp://<mail_url>:25/?ignoreTLS=true&tls={rejectUnauthorized:false}'
      - MAIL_FROM='Wekan Notifications <noreply.wekan@wekan.mydomainname.com>'
      - WITH_API=true
      - BROWSER_POLICY_ENABLED=true
networks:
  nginx-wekan-network:

Now everything works on AWS, way faster than it was before. So far, I have no idea how much AWS will charge me, but I expect it to be less than $10 / month.

Jun 30, 2019. Berlin