Devious Fish
DCloud: a cloud with the UNIX philosophy
profile for Perette at Stack Overflow, Q&A for professional and enthusiast programmers

DCloud Instructions

Overview

Usually, setting up DCloud will involve at least 2 installations:

Alternately, it can be use local documents without a repository, although this defeats the cloud-like characteristics.

Dependencies

Clients:

Server/clients with web publisher installed:

Subversion

Setting up a Subversion Server

Before setting up a Subversion repository for your cloud, create a new user svn on the desired node. Make sure svn has access to Subversion (svn, svnadmin, svnserve), Python 3 (`python3'), Korn Shell (ksh), and their own $HOME/bin (aka ~/bin).

As that user, download and extract the DCloud tarball. Then, in resulting directory:

$ make install_svn

This will install:

Configure the repository (Subversion)

Next, configure repository users by editing editing 3 configuration files in Repositories/cloud/config.

In passwd, add something like:

christine = some_password
publisher = another_password

In authz, add something like:

[cloud:/]
christine = rw
publisher = r
# Prevent publication of private stuff
[cloud:/Private]
publisher =

Revise svnserve.conf with these options:

anon-access = none
password-db = passwd
authz-db = authz
realm = Name Your Cloud

Configure repository access (Subversion)

If you want to set up svnserve to run as a service, or have it invoked by launchd/systemd/inetd on-demand, refer to your system and Subversion documentation. These methods do not use encryption; that requires Apache.

To use Subversion over SSL (svn+ssl), use ssh-keygen (or the appropriate utility on your system) to create a public/private key pair. In ~/.ssh, retrieve the public key just constructed (named id_something.pub). Then, add or edit the authorized_keys file in the same directory.

Repeating that as a cookbook:

$ mkdir ~/.ssh
$ chmod 700 .ssh
$ cd .ssh
$ ssh-keygen

Follow ssh-keygen’s prompts; refer to man pages as necessary.

$ cat id_rsa.pub
ssh-rsa AAASeveralLinesOfAlphabetAndNumbersSoupHereAQ32 perette@lenora

Select and copy the key, including the ssh-rsa and user@host portions.

$ touch authorized_keys
$ chmod 600 authorized_keys
$ your_favorite_editor authorized_keys

Append to the file:

command="svnserve -t tunnel-user=publisher -r Repositories" [pasted key]

Save, quit, and test:

$ ssh localhost
( success ( 2 2 ( ) ( edit-pipeline svndiff1 absent-entries commit-revprops depth log-revprops partial-replay ) ) )

Use Control-D to terminate the session. If you don’t get something that looks vaguely lisp-like, diagnose and fix. If svnserve is not in /bin or /usr/bin, you may need to provide a full path to it in authorized_keys.

Checkout the repository (Subversion)

Create a working copy of the repository. After changes are submitted, this will be used to extract a copy to generate a web-accessible version of your site. If you set up Subversion with the suggested svn+ssl setup:

$ cd ~/Documents
$ svn co svn+ssh://localhost/cloud Cloud
Checked out revision 0.

Configuring Subversion Clients

To set up a client, download and extract the DCloud tarball. In the resulting directory:

$ make install_client

This will install a few utilities in ~/bin.

Follow the procedure “Configure repository access”, above, to create a private/public key pair for this node. Add the public key (in id_something.pub) to the Subversion user’s authorized_keys on the server, similar to before but using the user’s assigned Subversion name:

command="svnserve -t tunnel-user=christine -r Repositories" [pasted key]

You can now:

$ ssh ssh@the_svn_server
( success ( 2 2 ( ) ( edit-pipeline svndiff1 absent-entries commit-revprops depth log-revprops partial-replay ) ) )
^D
$ cd ~/Documents
$ svn co svn+ssh://svn@the_svn_server/cloud Cloud
Checked out revision 0.

You can now put documents into your cloud. You can move documents or directories there, then use svn add to add them; or you can use cloud edit -c to add new files. Use cloud -? for more information on that utility.

Mercurial

Setting up a Mercurial Server

Before setting up a Mercurial repository for your cloud, create a new user hg on the desired node. Make sure hg has access to Mercurial (hg), Python 2 (python), Python 3 (`python3'), Korn Shell (ksh), and their own $HOME/bin (aka ~/bin).

As that user, download and extract the DCloud tarball. Then, in resulting directory:

$ make install_hg

This will:

Configure the repository (Mercurial)

The install Makefile sets up a usable single-repository, single-user configuration. If you want multiple repositories or multi-user access to your cloud, you will need to research adding users to Mercurial. Look into hg-ssh, hg-ssh2, and hg-ssh3, which seem to be variations on a theme.

If you want to improve this manual by writing up and sharing instructions on how to do make this all work, that would be lovely.

Configure repository access (Mercurial)

To enable remote access to your repository, follow the same “Configure repository access” procedure as for Subversion, but with the command= string in authorized_keys:

command="hg -R Repositories/cloud serve --stdio" [pasted key]

It can be tested by making a clone, below.

Configuring Mercurial Clients

To set up a client, download and extract the DCloud tarball. In resulting directory:

$ make install_client

This will install a few utilities in ~/bin.

Checkout the repository (Mercurial)

Once access has been set up, configure a client by:

$ hg clone ssh://the_hg_host/ ~/Documents/Cloud
requesting all changes
adding changesets
adding manifests
adding file changes
added 4 changesets with 13 changes to 7 files
updating to branch default
7 files updated, 0 files merged, 0 files removed, 0 files unresolved

You can now put documents into your cloud. You can move documents or directories there, then use hg add to add them; or you can use cloud edit -c to add new files. Use cloud -? for more information on that utility.

Standalone

To install DCloud without a repository back-end, make sure your node has Python 3 (`python3') and Korn Shell (ksh). Download and extract the DCloud tarball. Then, in resulting directory:

$ make install_standalone

This will:

Add to your .kshrc and/or .bashrc:

export DCLOUD_BACKEND=none

And create your (not-so) cloud document directory:

mkdir ~/Documents/Cloud

Web View Support

Set up web access

DCloud’s post-commit hander generates a set of simple, friendly, static web pages in ~/Sites/Cloud. Afterwards, if it exists, upload is invoked with the parameter ‘cloud’. Implement this script to transfer updates to a web server.

For example, use your hosting company’s CPanel to create cloud.mydomain.com. Log into your hosting account, and add the repository user’s public key to authorized_keys (do not include any command=... stuff). Your upload script might then look something like this:

#!/bin/ksh
rsync -lptz -r -H -e "ssh -p 22" ~/Documents/Cloud/ my_server.hostingcompany.com:cloud.mydomain.com

Remember to chmod u+x ~/bin/upload. You can test it by invoking it on the command line.

Configure behavior

You can set pathnames, calendar categories, magic filenames, and case of output filenames, and other settings in ~/.dcloudrc. Available settings are listed at the top of the dcloud-post-commit script.

Restrict access

Configure restricted access in your hosting provider’s CPanel. This usually results in a .htaccess file in the served directory, which must be copied to ~/Documents/Cloud to prevent it being deleted by rsync.

Formats vary; consult your hosting provider’s documentation for details if you need to create or customize .htaccess.

Test well to ensure security.

File Formats

The web publisher recognizes text documents, markdown, RTF, PDF, and various graphics formats. For some formats, you need additional utilities:

Customizing

Handling transient server unavailability

You may want to add a periodic cloud sync to your clients' crontab.

Cleaning and archiving of completed to-do tasks

I have a nightly job that extracts completed tasks from the to-do list, and puts them in a completed-tasks list:

#!/bin/ksh
# Extract done stuff from my to-do list, and keep a log of it.
if cd ~/Documents/Cloud
then
        cloud sync
        tasks.py -f markdown -c -b -m "Completed $(date '+%Y-%m-%d %H:%M')" -s tasks.txt >> completed.txt
        cloud sync
fi

Helpful aliases

You’ll probably want to add some aliases to ease use. Here’s a few of mine:

alias tasks='cloud edit tasks.txt'
    alias queue='cloud edit queue.txt'
alias grocery='cloud edit Purchasing/grocery.txt'
alias brain='cloud edit brain.txt'

tasks.txt is the to-do list. Tasks are put in a markdown multi-level bulleted list:

- [ ] Incomplete task
- [x] Completed task

queue.txt is a text-file based calendar.

Both tasks.txt and queue.txt are converted to HTML and put on the index page of the cloud website. queue.txt is also converted to a calendar.

brain.txt is filled with assorted notes, much of it system-administration related but not entirely. It’s a sort of swapfile for my brain; it’s got all the stuff I use rarely enough that I need to look it up every time: Vim modelines, public wifi passwords, part numbers for wood chipper blades. Mac boot key-chords, and which services to enable after a fresh install to make a Mac behave correctly. A list of packages to install on a fresh LinuxMint. Dimensions for making fetish items. How to spell the word ‘tschochke’. You get the idea.

Improving reponsiveness

The Subversion repository is configured with the post-commit hook run in the foreground, so that errors are reported to clients in real-time. Commits can be sped up by deferring web conversion, with the peril that errors may go unnoticed. Modify the invokation of dcloud-post-commit in Repositories/cloud/hooks/post-commit:

batch << EOF
~/bin/dcloud-post-commit "$1" "$2" > $HOME/Logs/post-commit.log 2>&1
EOF

Box pictures instead of checkboxes in to-do lists

Have a Samsung phone that shows a picture of a ballot going in a box instead of a box with/without a check? I consider this a defect by some literalists in Samsung’s font department.

A workaround is to add this CSS to custom.css to use a crossed or uncrossed circle:

UL.todolist LI:before {
        content: '\25ef  ';
}
UL.todolist LI.completed:before {
        content: '\2bbe  ';
}

However, these may pose problems on iPhones. Another choice is a checkmark and a bullet:

UL.todolist LI:before {
        content: '\2219  ';
}
UL.todolist LI.completed:before {
        content: '\2713  ';
        color: green;
        text-decoration: none;
}