Devious Fish
DCloud Instructions

DCloud Instructions

Overview

Setting up DCloud will involve at least 2 installations:

  • A Subversion or Mercurial repository on a host available via the Internet, along with supporting software.
  • One or more client machines.

Dependencies

Clients:

  • Korn shell
  • Subversion or Mercurial

Server/clients with web publisher installed:

  • Korn shell
  • Subversion or Mercurial
  • Python3 (3.5 or later) with pytz & ics modules (installed via pip/pip3).
  • Make
  • m4 macro processor

Subversion

Setting up a Subversion Server

Before setting up a Subversion repository for your cloud, create a new user svn on the desired node. Make sure svn has access to Subversion (svn, svnadmin, svnserve), Python 3 (`python3'), Korn Shell (ksh), and their own $HOME/bin (aka ~/bin).

As that user, download and extract the DCloud tarball. Then, in resulting directory:

$ make install_svn

This will install:

  • Some utilities and a post-commit hook in ~/bin.
  • A repository named cloud, in ~/Repositories.
  • DCloud website template code in ~/Sites/Cloud.

Configure the repository (Subversion)

Next, configure repository users by editing editing 3 configuration files in Repositories/cloud/config.

In passwd, add something like:

christine = some_password
publisher = another_password

In authz, add something like:

[cloud:/]
christine = rw
publisher = r
# Prevent publication of private stuff
[cloud:/Private]
publisher =

Revise svnserve.conf with these options:

anon-access = none
password-db = passwd
authz-db = authz
realm = Name Your Cloud

Configure repository access (Subversion)

If you want to set up svnserve to run as a service, or have it invoked by launchd/systemd/inetd on-demand, refer to your system and Subversion documentation. These methods do not use encryption; that requires Apache.

To use Subversion over SSL (svn+ssl), use ssh-keygen (or the appropriate utility on your system) to create a public/private key pair. In ~/.ssh, retrieve the public key just constructed (named id_something.pub). Then, add or edit the authorized_keys file in the same directory.

Repeating that as a cookbook:

$ mkdir ~/.ssh
$ chmod 700 .ssh
$ cd .ssh
$ ssh-keygen

Follow ssh-keygen’s prompts; refer to man pages as necessary.

$ cat id_rsa.pub
ssh-rsa AAASeveralLinesOfAlphabetAndNumbersSoupHereAQ32 perette@lenora

Select and copy the key, including the ssh-rsa and user@host portions.

$ touch authorized_keys
$ chmod 600 authorized_keys
$ your_favorite_editor authorized_keys

Append to the file:

command="svnserve -t tunnel-user=publisher -r Repositories" [pasted key]

Save, quit, and test:

$ ssh localhost
( success ( 2 2 ( ) ( edit-pipeline svndiff1 absent-entries commit-revprops depth log-revprops partial-replay ) ) )

Use Control-D to terminate the session. If you don’t get something that looks vaguely lisp-like, diagnose and fix. If svnserve is not in /bin or /usr/bin, you may need to provide a full path to it in authorized_keys.

Checkout the repository (Subversion)

Create a working copy of the repository. After changes are submitted, this will be used to extract a copy to generate a web-accessible version of your site. If you set up Subversion with the suggested svn+ssl setup:

$ cd ~/Documents
$ svn co svn+ssh://localhost/cloud Cloud
Checked out revision 0.

Configuring Subversion Clients

To set up a client, download and extract the DCloud tarball. In the resulting directory:

$ make install_client

This will install a few utilities in ~/bin.

Follow the procedure “Configure repository access”, above, to create a private/public key pair for this node. Add the public key (in id_something.pub) to the Subversion user’s authorized_keys on the server, similar to before but using the user’s assigned Subversion name:

command="svnserve -t tunnel-user=christine -r Repositories" [pasted key]

You can now:

$ ssh ssh@the_svn_server
( success ( 2 2 ( ) ( edit-pipeline svndiff1 absent-entries commit-revprops depth log-revprops partial-replay ) ) )
^D
$ cd ~/Documents
$ svn co svn+ssh://svn@the_svn_server/cloud Cloud
Checked out revision 0.

You can now put documents into your cloud. You can move documents or directories there, then use svn add to add them; or you can use cloud edit -c to add new files. Use cloud -? for more information on that utility.

Mercurial

Setting up a Mercurial Server

Before setting up a Mercurial repository for your cloud, create a new user hg on the desired node. Make sure hg has access to Mercurial (hg), Python 2 (python), Python 3 (`python3'), Korn Shell (ksh), and their own $HOME/bin (aka ~/bin).

As that user, download and extract the DCloud tarball. Then, in resulting directory:

$ make install_hg

This will:

  • Install some utilities and the post-commit hook in ~/bin.
  • Create a repository named cloud, in ~/Repositories.
  • Configure the commit hook to the repository’s hgrc configuration file.
  • Clone the repository in ~/Documents/Cloud
  • Install DCloud website template code in ~/Sites/Cloud.

Configure the repository (Mercurial)

The install Makefile sets up a usable single-repository, single-user configuration. If you want multiple repositories or multi-user access to your cloud, you will need to research adding users to Mercurial. Look into hg-ssh, hg-ssh2, and hg-ssh3, which seem to be variations on a theme.

If you want to improve this manual by writing up and sharing instructions on how to do make this all work, that would be lovely.

Configure repository access (Mercurial)

To enable remote access to your repository, follow the same “Configure repository access” procedure as for Subversion, but with the command= string in authorized_keys:

command="hg -R Repositories/cloud serve --stdio" [pasted key]

It can be tested by making a clone, below.

Configuring Mercurial Clients

To set up a client, download and extract the DCloud tarball. In resulting directory:

$ make install_client

This will install a few utilities in ~/bin.

Checkout the repository (Mercurial)

Once access has been set up, configure a client by:

$ hg clone ssh://the_hg_host/ ~/Documents/Cloud
requesting all changes
adding changesets
adding manifests
adding file changes
added 4 changesets with 13 changes to 7 files
updating to branch default
7 files updated, 0 files merged, 0 files removed, 0 files unresolved

You can now put documents into your cloud. You can move documents or directories there, then use hg add to add them; or you can use cloud edit -c to add new files. Use cloud -? for more information on that utility.

Web View Support

Set up web access

DCloud’s post-commit hander generates a set of simple, friendly, static web pages in ~/Sites/Cloud. Afterwards, if it exists, upload is invoked with the parameter ‘cloud’. Implement this script to transfer updates to a web server.

For example, use your hosting company’s CPanel to create cloud.mydomain.com. Log into your hosting account, and add the repository user’s public key to authorized_keys (do not include any command=... stuff). Your upload script might then look something like this:

#!/bin/ksh
rsync -lptz -r -H -e "ssh -p 22" ~/Documents/Cloud/ my_server.hostingcompany.com:cloud.mydomain.com

Remember to chmod u+x ~/bin/upload. You can test it by invoking it on the command line.

Configure behavior

You can set pathnames, calendar categories, magic filenames, and case of output filenames, and other settings in ~/.dcloudrc. Available settings are listed at the top of the post-commit (dcloud-post-commit on clients) script.

Restrict access

Configure restricted access in your hosting provider’s CPanel. This usually results in a .htaccess file in the served directory, which must be copied to ~/Documents/Cloud to prevent it being deleted by rsync.

Formats vary; consult your hosting provider’s documentation for details if you need to create or customize .htaccess.

Test well to ensure security.

File Formats

The web publisher recognizes text documents, markdown, RTF, PDF, and various graphics formats. For some formats, you need additional utilities:

  • Markdown (.md): Install multimarkdown from http://fletcherpenney.net/multimarkdown/ If not available, Markdown is converted as plain-text.
  • RTF (.rtf): Install unrtf, available via apt, MacPorts, and other fine package managers. If not available, RTF files are omitted from web access.
  • Unidentified (no extension): the document is probed for Markdown formatting; if found, it is processed with Multimarkdown (if available). Otherwise, it’s treated as plain-text.

Customizing

Handling transient server unavailability

You may want to add a periodic cloud sync to your clients' crontab.

Cleaning and archiving of completed to-do tasks

I have a nightly job that extracts completed tasks from the to-do list, and puts them in a completed-tasks list:

#!/bin/ksh
# Extract done stuff from my to-do list, and keep a log of it.
if cd ~/Documents/Cloud
then
        cloud sync
        tasks.py -f markdown -c -b -m "Completed $(date '+%Y-%m-%d %H:%M')" -s tasks.txt >> completed.txt
        cloud sync
fi

Helpful aliases

You’ll probably want to add some aliases to ease use. Here’s a few of mine:

alias tasks='cloud edit tasks.txt'
    alias queue='cloud edit queue.txt'
alias grocery='cloud edit Purchasing/grocery.txt'
alias brain='cloud edit brain.txt'

tasks.txt is the to-do list. Tasks are put in a markdown multi-level bulleted list:

- [ ] Incomplete task
- [x] Completed task

queue.txt is a text-file based calendar.

Both tasks.txt and queue.txt are converted to HTML and put on the index page of the cloud website. queue.txt is also converted to a calendar.

brain.txt is filled with assorted notes, much of it system-administration related but not entirely. It’s a sort of swapfile for my brain; it’s got all the stuff I use rarely enough that I need to look it up every time: Vim modelines, public wifi passwords, part numbers for wood chipper blades. Mac boot key-chords, and which services to enable after a fresh install to make a Mac behave correctly. A list of packages to install on a fresh LinuxMint. Dimensions for making fetish items. How to spell the word ‘tschochke’. You get the idea.

Improving reponsiveness

The Subversion repository is configured with the post-commit hook run in the foreground, so that errors are reported to clients in real-time. Commits can be sped up by deferring web conversion, with the peril that errors may go unnoticed. Modify the invokation of dcloud-post-commit in Repositories/cloud/hooks/post-commit:

batch << EOF
~/bin/dcloud-post-commit "$1" "$2" > $HOME/Logs/post-commit.log 2>&1
EOF

Box pictures instead of checkboxes in to-do lists

Have a Samsung phone that shows a picture of a ballot going in a box instead of a box with/without a check? I consider this a defect by some literalists in Samsung’s font department.

A workaround is to add this CSS to custom.css to use a crossed or uncrossed circle:

UL.todolist LI:before {
        content: '\25ef  ';
}
UL.todolist LI.completed:before {
        content: '\2bbe  ';
}

However, these may pose problems on iPhones. Another choice is a checkmark and a bullet:

UL.todolist LI:before {
        content: '\2219  ';
}
UL.todolist LI.completed:before {
        content: '\2713  ';
        color: green;
        text-decoration: none;
}