*Update 29/12/2013* – I’ve written a new post on this topic with updated instructions here. I’m leaving this page as is for posterity, and I recommend you use the new post since it has (I think) better handling of routing.
I run two servers in two different locations which are connected via a permanent OpenSSH VPN, complete with routing between the two. The servers both sit on independent internet connections and I’ve implemented it so that if one side or the other drops, the connection will be automatically re-established. This means I can have some devices which route through the secure connection and access the internet through the remote end point, which can be quite useful in some situations.
Last week, one of the servers suffered severe disk corruption when power failed (no, these boxes aren’t important enough to run on a UPS), and I had to rebuild it from scratch and re-configure the permanent VPN from scratch. This was a bit of a pain the first time, and just as much the second, so I have decided to document it for posterity and (if nothing else) future reference.