Dylan Smith

Dropbox Based Static Site

How I'm using Dropbox to Sync Content and Deploy Hugo

This site is built using Hugo, a blazing-fast static site generator based on the Go programming language. I’ve put together a writing and publishing workflow that can be used across devices, including mobile, that I’m really happy with. It’s not for everyone, but if you favor Markdown as I do, and want to have all your content in a single managed place, it could be a good solution for you. This post outlines how I went about setting it all up.

Docker all of it

I’m using docker-hugo to run all Hugo commands, both locally and on the server. This way I don’t have to manage Go or any dependencies. It just works.

Local development

To start a new Hugo project using the Docker image, run:

docker run --rm --name hugo-init -v $(pwd):/src jojomi/hugo hugo new site src

Be sure to do this in a Dropbox sub-directory so that your content syncs across devices and, most importantly, on the server (we’ll get to that part).

Now you can start watching the site in development mode using:

docker run --rm -it --name hugo-watch -e HUGO_WATCH=1 -p "1313:1313" -v "$(PWD)/src":/src -v "$(PWD)/output":/output jojomi/hugo

This will start the Hugo server on port 1313, which is why we map that port to the host.

(If you are following along and just created a new Hugo site, at this point you will only see a blank page. You will have to create or download a Hugo theme to get started. This overview will not go into the details of customizing and configuring Hugo itself. I’ll save that for another essay.)

At this point, the root of your project will have two directories:

ProjectDirectory/
+ output/
+ src/

The Docker Hugo container should always be run from the base of the project since it will be linking volumes in the container to the src and output directories.

On the server

Make sure Docker and docker-compose are available on the server. I won’t go into Docker installation in this post. DigitalOcean has a great post outlining the process.

Also required for the next few steps is Python 2.7 and inotify-tools. The latter can be installed with:

sudo apt-get install inotify-tools

I run Dropbox with a customized Dropbox Docker image. To set up Dropbox on Ubuntu, run the following command, which will automatically pull the latest image:

docker run -d --restart=always --name=dropbox -v ~/dropbox:/home/user/Dropbox -v ~/.dropbox:/home/user/.dropbox dylansm/dropbox

This will link the running container to two directories, dropbox, which will hold your Dropbox-synced files, and .dropbox, which will hold the Dropbox configuration files.

In order to link to your Dropbox account you will need to view the Docker Dropbox container logs. An authorization URL should be visible by issuing:

docker logs dropbox

Copy and paste the URL in a browser and log in to your Dropbox account to associate the container with your Dropbox account.

Exclusions

If you are an active Dropbox user, you may wish to use the selective sync feature to exclude all files and folders that are not related to your website.

Do this by issuing the exclude command:

docker exec -it dropbox dropbox-cli exclude add <dir or file name(s)>

You can list as many directories and files as you like in a single command. After a few moments, excluded files should disappear from your Dropbox directory.

Build your site by saving a file

The following steps will set up a watcher that rebuilds the site when a text file has been modified.

  1. In the root of the src directory of your project, either on your local development box or on the server, create a file called rebuild.txt (you can call it whatever you want as long as you reference it correctly in the following steps). This will be the file that, on save, will sync to the server and trigger a site build.

  2. On the server, create an empty file in the home directory called .websitename_last_build (replace websitename with the name your website). This file will store the last modified date of the dropbox-synced rebuild.txt file.

  3. Now visit the Dropbox for Developers Apps page and create a new application. Grant it full Dropbox permissions and generate an access token by clicking Generate button under the OAuth 2 section. Copy the access code.

  4. Create a directory called bin in your server home directory. The first file we will add is a Python script to check the status of the rebuild.txt file and, if it changes, store the last modification date to the file created in step two .websitename_last_build . Save the following as ~/bin/dropbox.py and be sure to paste your access code into the headers dict at the top:

import json
import os
import requests
import subprocess

url = "https://api.dropboxapi.com/2/files/list_revisions"

headers = {
    "Authorization": "Bearer YOUR-ACCESS-CODE-HERE",
    "Content-Type": "application/json"
}

data = {
    "path": "/Sites/servername.com/src/rebuild.txt",
}

request = requests.post(url, headers=headers, data=json.dumps(data))

created = json.loads(request.text)

last_modified = created['entries'][0]['server_modified']

with open("/home/serveruser/.servername.com_last_build", 'r') as f:
    last_build = f.readline()
with open("/home/serveruser/.servername.com_last_build", 'w') as f:
    if last_build != last_modified:
        f.write(last_modified)
        subprocess.call("/home/serveruser/bin/hugo_build.sh sitename.com www", shell=True)
    else:
        f.write(last_build)

Don’t forget to replace the access key in the headers dictionary near the top. Also, change the server user from ‘serveruser’ to the correct user name.

  1. Next, create the file ~/bin/hugo_build:
#!/bin/bash

echo ""
echo "Building $2.$1..."

START_DATE_WITH_TIME=`date +"%Y-%m-%d %H-%M-%S"`
echo "Start: $START_DATE_WITH_TIME"

production_path="/home/serveruser/dropbox/Sites/$1/src"
production_build="/var/www/www.$1/public"
docker run --rm --name hugo-production -e HUGO_BASEURL=https://www.$1/ -v "$production_path":/src -v "$production_build":/output jojomi/hugo
chown -R serveruser:serveruser "$production_build"

END_DATE_WITH_TIME=`date +"%Y-%m-%d %H-%M-%S"`
echo "End: $END_DATE_WITH_TIME"
  1. Create a third file ~/bin/watch_build that has the following:
#!/bin/bash

function watch {
  FILE=/home/serveruser/dropbox/Sites/websitename.com/src/rebuild.txt

  if [[ -e "$FILE" ]]; then

    while inotifywait -qe MOVE_SELF $FILE; do
      /usr/bin/python /home/serveruser/bin/dropbox.py >> /home/serveruser/logs/build.log 2>&1
      break
    done
  fi

  watch
}
  1. After creating these files make sure they are executable:
chmod +x ~/bin/dropbox.py
chmod +x ~/bin/hugo_build
chmod +x ~/bin/watch_build
  1. Now we will set up the server directory so that when Hugo builds the site it will be generated at /var/www/www.websitename/public
sudo mkdir -p /var/www/www.websitename
sudo chown -R serveruser:serveruser /var/www

Be sure to use the correct server user name.

  1. Create a docker-compose.yml file with the following content:
nginx-proxy:
  image: jwilder/nginx-proxy:alpine
  volumes:
    - /var/run/docker.sock:/tmp/docker.sock:ro
    - /etc/letsencrypt/live/websitename.com:/etc/nginx/certs:ro
    - /etc/nginx/vhost.d
    - /usr/share/nginx/html
  ports:
    - 443:443
    - 80:80
  restart: always

letsencrypt:
  image: jrcs/letsencrypt-nginx-proxy-companion:latest
  environment:
    - DEBUG=true
  restart: always
  volumes_from:
    - nginx-proxy
  volumes:
    - /etc/letsencrypt/live/websitename.com:/etc/nginx/certs:rw
    - /var/run/docker.sock:/var/run/docker.sock:ro
web:
  image: nginx:alpine
  environment:
    - VIRTUAL_HOST=websitename.com, www.websitename.com
    - LETSENCRYPT_HOST=websitename.com, www.websitename.com
    - LETSENCRYPT_EMAIL=your@email.com
  restart: always
  volumes:
    - /var/www/www.websitename.com/public:/usr/share/nginx/html
    - /var/www/www.websitename.com/etc/nginx/conf.d/:/etc/nginx/conf.d

This will start up an nginx-proxy and another instance of nginx to host the static site. It will also set up a letsencrypt SSL certificate, which is super cool, too.

Important note: if you do not have a domain resolving at the address your server is using do not include the letsencrypt block or the volume link to the letsencrypt certificates, or any of the environment variables that reference letsencrypt. If you do, the site will error-out until the service can verify that the domain is yours and correctly set up.

  1. Make sure you build the server manually at least once before automating the process with the scripts above. This will deploy Hugo to the public directory under our website directory:
docker run --rm --name hugo-production -e HUGO_BASEURL=https://www.websitename.com/ -v /home/serveruser/dropbox/Sites/mysite/src:/src -v /var/www/www.websitename.com/public:/output jojomi/hugo
  1. Start up the web server by issuing the following in the same directory where you created the above docker-compose.yml file:
docker-compose up -d

At this point, you should be able to bring up your website but modifying rebuild.txt will not yet trigger a build because our watcher isn’t running. We’ll use Ubuntu’s systemd to load ~/bin/watch_build at system start time and continue watching for changes.

  1. Create the following under /etc/systemd/system/watch-build.service:
[Unit]
After=docker.service

[Service]
ExecStart=/home/serveruser/bin/watch_build

[Install]
WantedBy=default.target
  1. Modify permissions and enable service:
chmod 744 /home/serveruser/bin/watch_build
chmod 664 /etc/systemd/system/watch-build.service
sudo systemctl daemon-reload
sudo systemctl enable watch-build.service
  1. Start the service with:
sudo systemctl start watch-build.service

If all went according to plan, you can view the status of the loaded service with:

sudo systemctl list-units | grep watch-build

You should see “loaded active running” next to the service name.

That was a lot. With any luck, you can now build your Hugo site by editing src/rebuild.txt in any text editor and you should see the changes live on production within moments. Since your content is stored in Dropbox things will stay in sync, as well, across all your devices.