Linux, Python, devops, etc

Blogging with Dropbox and S3

10 May 2012

I've done away that clumsy old database thing. Webserver too. My blog is now statically generated. What follows is a quick brain dump of my experiences getting it up and running. First up, here's the various moving parts that make the magic happen: jilcrow, Dropbox, Amazon S3, s3cmd, incron, and Disqus.


A static blog is only feasible if you're not handcrafting the html for every post, which means we need something to generate it for you. I started off looking at Jekyll and then Octopress, but jekyll is written in ruby, and octopress is built up around jekyll. I wanted something written in python. I wanted something I could hack on. I did start writing my own, a combination of flask and the flask-freezer plugin, but thought there was a better way to do it (though I do have an idea for where my original code might be useful). That's about when I stumbled upon jilcrow. It uses jinja2 templates and markdown for the content, perfect! It didn't quite do everything I wanted, or in the way I wanted, but I fixed that (I later found pelican but by that point I was too invested in jilcrow to consider changing).


I assume you know what dropbox is. You don't? Go look it up. It's awesome.

Amazon S3 and s3cmd

S3 recently (as in, over a year ago) got the ability to host actual web content on users' own urls. For a blog such as mine, I could triple the number of users I get, and still not pay more than a dollar a month hosting. And I don't see S3 going away any time soon.

S3cmd is a tool (also written in python!) for interacting with S3 via the command line - perfect for scripting S3 like we need.


Incron is essentially cron for file events. You tell it to watch a file or directory, tell it what events you're waiting for, and lastly, what command to run when the event is triggered. You could use inotify directly I suppose, but I didn't, this time.

EDIT: incron isn't quite the solution I'd hoped for. It doesn't recursively monitor subtrees, which means it doesn't do what I needed it to do. Watcher might be a suitable alternative. Will update once I've had a play with it.


Disqus is a global comment system. It implements everything I've ever tried to cram into my own comments system, and happens in javascript. That means it's trivial to embed into a static site, and allow for a real, interactive blog with no database to back it.

Bringing it all together

I have a small virtual machine that's set up for "local" development work. The main idea being, doesn't matter what device is in front of me, as long as I have an internet connection, I can work. (It works great, too - once you add something like iremap jj <Esc> to your vimrc. Darned virtual keyboard and their lack of an escape key...) Point is, this server is connected to my dropbox account. I've lost the specific instructions I used, but plenty of people have posted howtos.

In a virtualenv, I have installed both jilcrow and s3cmd:

 sudo apt-get install python-dev python-setuptools inotify-tools incron
 sudo easy_install pip virtualenv virtualenvwrapper.sh
 # Add these to your .bashrc too
 export WORKON=/home/jls/.virtualenvs
 source /usr/local/bin/virtualenvwrapper.sh
 mkvirtualenv autoblog
 # Download jilcrow from git, and s3cmd from the s3tools site
 # and install them with python setup.py install

Cool. That's all the tools installed. Might as well copy the example-site into your dropbox folter too. The template is rather ugly, but I'm sure you can customize it a little. Your jinja templates go in /templates, your CSS, javascript, and images go in /files, and your content goes in, well, /content - Your folder structure will determine your URLS, so think about that as you write.

The following script is part of the auto update magic (update the paths and urls to match your own environment!):


# activate the virtualenv, so we can use s3cmd and jilcrow
source /usr/local/bin/virtualenvwrapper.sh
export WORKON_HOME=$HOME/.virtualenvs
workon jilcrow

# Stop any current build attempts...
pkill jilcrow
pkill s3cmd

# Rebuild and push
jilcrow /home/jls/Dropbox/autoblog/
s3cmd sync -P --delete-removed /home/jls/Dropbox/autoblog/deploy/ s3://www.justinsteward.com

Save a copy of it somewhere on your system as update-blog, and make it executable.

And the other part of the update magic is to trigger the script when your content files get updated. Run incrontab -e and add the following line:

/home/jls/Dropbox/autoblog/content IN_CLOSE_WRITE /home/jls/bin/update-blog.sh

Remember to fix the paths to match your setup. What does this do? Well, every time a CLOSE_WRITE event is triggered, the update script will run, and sync our site with S3.

Set up a disqus account for your site if you haven't done so already, and add disqus: yoursiteshortname to the site.yaml file. Assuming you haven't changed the entry template too much, that's all you need to do for disqus to work.

What else?

That's really all there was to setting it up. It's been running for a few months, live for only a few days, and so far it's working great. No regrets here.