Deploying Lamson And OneShotBlog

These instructions will teach you how to setup a completely clean Python 2.6 installation, a virtualenv with lamson, and all the gear needed to run the oneshotblog.com software on your machine. It then leads you through the nitty-gritty details of getting oneshotblog.com running on your machine and testing it.

Most of these instructions could be easily turned into an automated script, which may happen in the future. For now it is meant to teach you about the typically dirty details involved in setting up a system for the first time. It also tries to avoid various problems with different operating systems, so let me know how it works for you.

A Warning

Deploying server software is a notoriously nasty process, especially the first 10 or 20 times. Most operating systems do their best to enforce completely arbitrary restrictions on your file layouts and configurations, and every system has different arbitrary restrictions.

When you go through these instructions, make sure you stay awake and be ready to delve into why a particular step might not work on your system. There’s a good chance you missed something or that there’s something just slightly different about your system that makes the step not work.

For example, in the parts of this document where I setup oneshotblog.com I ran into a problem with SpamBayes dying because it couldn’t iterate a bsddb. Problem is this works just fine in the exact same setup on a CentOS machine and was only dying on a MacOSX machine that I later tested. For whatever reason, the exact same setup can’t run SpamBayes on OSX, even though it can run with the stock Python 2.5 in OSX.

To solve the problem I just had to show you how to disable SpamBayes in the oneshotblog.com code so you could test it. That’s just how deployment goes. You get on a machine and start setting things up and then 2/3 of the way through the configuration you find out that something doesn’t work.

Only choices are to work around the problem (like I did) or try to figure out why your machine is different and fix it.

Step 0: Setup A Workplace

You’ll want a directory to do this in so that you don’t screw up your machine. Here’s what I did:

$ mkdir deploy
$ cd deploy
$ export DEPLOY=$PWD

That last bit is so you can refer to this deployment directory with $DEPLOY (which I’ll be using in the instructions from now on).

Step 1: Get Python

Many operating systems have old versions of Python, and even though Lamson works with 2.6 or 2.5, you’ll probably want to get 2.6 for your deployment. If your OS has 2.6 available then go ahead and install it.

If it doesn’t have the right Python version, then here’s how you can install it from source and use it as your default Python. To do this, just punch in these commands:

$ mkdir source
$ cd source
$ wget http://www.python.org/ftp/python/2.6.2/Python-2.6.2.tgz
$ tar -xvf Python-2.6.2.tgz
$ cd Python-2.6.2
$ ./configure --prefix=$DEPLOY
$ make
$ make install

After this you will have a bunch of new directories in $DEPLOY:

$ ls $DEPLOY
bin     include lib     share   source

Finally, you just have to put this new bin directory into your $PATH:

$ export PATH=$DEPLOY/bin:$PATH

... then you just try it out to make sure that you have the right one:

Python-2.6.2 $ which python
$DEPLOY/deploy/bin/python

Python-2.6.2 $ python 
Python 2.6.2 (r262:71600, Jun  8 2009, 00:44:56) 
[GCC 4.0.1 (Apple Inc. build 5490)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D

That’s it, you’ll now be able to use this Python when you need to run your Lamson server, and setup a virtualenv (coming next) so that you’re walled off from the rest of the system.

Operating system fanatics will scoff at putting the python install in this directory, so if you want you can just install it to the default /usr/local on your system and deal with all the various clashes and conflicts you’ll have, especially if you are on an MacOSX machine.

Step 2: Install VirtualEnv

Now we need to create a “virtual environment” to install all your software. To do this we’ll need easy_install installed to your $DEPLOY directory:

$ cd $DEPLOY/source
$ wget http://peak.telecommunity.com/dist/ez_setup.py
$ python ez_setup.py 
$ which easy_install
$DEPLOY/bin/easy_install

As you can see, you now have a clean install of easy_install in your fresh $DEPLOY/bin directory for you to use. Now you need to install virtualenv:

$ easy_install --prefix $DEPLOY virtualenv
$ which virtualenv
$DEPLOY/bin/virtualenv

Make sure you use --prefix $DEPLOY above or you’ll install things into the default system setup even though easy_install is clearly and obviously running from a Python in a totally different location so easy_install should know that.

Step 3: Create Your VirtualEnv

With that you are ready to setup your virtual environment which will house your Lamson setup and fill it with the gear you need.

First up is getting your virtualenv created and activated:

$ cd $DEPLOY
$ virtualenv LAMSON
New python executable in LAMSON/bin/python
Installing setuptools............done.
$ cd LAMSON
$ . bin/activate

That’s pretty simple, and it tells you clearly that you are using the LAMSON virtualenv. It prepends that to your currently prompt, so your prompt may look different.

After that we can use easy_install to install our packages to this LAMSON virtual env. Keep in mind that these packages will be in $DEPLOY/LAMSON, so they won’t infect your regular $DEPLOY setup.

$ cd $DEPLOY/LAMSON
$ easy_install lamson

After that, you have lamson installed and ready to go, and you can install anything you want, but there is one catch:

You MUST be in the $DEPLOY/LAMSON directory or easy_install barfs complaining that the package is not there.

Step 4: Making Sure It Works

All of this setup is pointless if you can’t get back to it later, so exit your terminal completely and start a new one so you can do this:

$ cd projects/lamson/deploy/
$ export DEPLOY=$PWD
$ export PATH=$DEPLOY/bin:$PATH
$ cd $DEPLOY/LAMSON
$ . bin/activate
(LAMSON) $ which python
$DEPLOY/deploy/LAMSON/bin/python
(LAMSON) $ which easy_install
$DEPLOY/deploy/LAMSON/bin/easy_install
(LAMSON) $ which lamson
$DEPLOY/deploy/LAMSON/bin/lamson
(LAMSON) $ cd $DEPLOY
(LAMSON) $ lamson help

If you can do all that, then you know you’ve got the setup going, now you just need a little shell script to kick this all into gear automatically:

#!/bin/sh

export DEPLOY=$PWD
export PATH=$DEPLOY/bin:$PATH
cd $1
source bin/activate
cd $DEPLOY

To use this script, you just do this:

$ cd projects/lamson/deploy
$ . activate LAMSON

With that you have a fully ready to go setup that’s not using your normal system’s Python at all, has Python 2.6 installed, a fully virtualenv, and the start of your lamson setup.

Step 5: Setting Up The OneShotBlog

Let’s see if we can setup the OneShotBlog example from the Lamson source the way it is on the oneshotblog.com site. We’ll need a few more modules installed with easy_install:

$ cd $DEPLOY/LAMSON
$ easy_install markdown
$ easy_install mock
$ easy_install spambayes

Let’s grab the 0.9.3 source from PyPI so we can get at the OSB example source:

$ cd $DEPLOY/source
$ wget http://pypi.python.org/packages/source/l/lamson/lamson-0.9.3.tar.gz
$ tar -xzf lamson-0.9.3.tar.gz
$ cd lamson-0.9.3/examples/osb

Now we hit a slight snag. OSB is using SpamBayes to do spam filtering, but you probably will have a broken setup and would need to configure a ton of stuff to get it working. For now we’re, just going to cheat, since it looks like SpamBayes has problems with trying to iterate the keys in a bsddb under some Python builds. To avoid the problem, we’re just going to edit app/handlers/comment.py to remove the line with spam_filter:

@route("(user_id)-AT-(domain)-(post_name)-comment@(host)")
# DELETE THIS LINE IN app/handlers/comments.py
@spam_filter(SPAM['db'], SPAM['rc'], SPAM['queue'], next_state=SPAMMING)
def START(message, user_id=None, post_name=None, host=None, domain=None):
    comment.attach_headers(message, user_id, post_name, domain) 
    confirmation.send(relay, "comment", message, "mail/comment_confirm.msg", locals())
    return CONFIRMING

The spam filtering does work, but SpamBayes is difficult to get working in such a small test run.

You are now sitting in the OSB example code, so you can fire up the logger server and run the unit tests to make sure everything is working:

$ mkdir logs
$ mkdir run
$ mkdir app/data/posts
$ lamson log
$ nosetests

You should get two errors you can ignore for now:

..............
======================================================================
FAIL: handlers.comments_tests.test_spam_sent_by_unconfirmed_user
----------------------------------------------------------------------
Traceback (most recent call last):
    ...
-------------------- >> begin captured logging << --------------------
root: WARNING: Attempt to post to user 'spamtester@somehost.com' but user doesn't exist.
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: handlers.comments_tests.test_spam_sent_by_confirmed_user
----------------------------------------------------------------------
Traceback (most recent call last):
    ...
-------------------- >> begin captured stdout << ---------------------
run/posts count after dever 1
run/posts count after dever 2

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
root: WARNING: Attempt to post to user 'spamtester@somehost.com' but user doesn't exist.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 25 tests in 1.363s

Those are just fine since you don’t have PyEnchant installed and aren’t using the spam filtering.

Step 6: Run OneShotBlog Example

Now you’re running the logger server and have your unit tests going, and hopefully you can fix anything that you run into by now. All you need now is to run the whole setup and try it out:

$ lamson start
$ lamson start -pid run/queue.pid -boot config.queue
$ lamson start -pid run/forward.pid -boot config.forward

With all this gear running you should be able to look in the logs/lamson.log and logs/logger.log to see what’s going on. You’ll see the following activity:

If you want to see the configuration for these two other servers look in config/queue.py and config/forward.py or better yet, diff them against config/boot.py to see what’s really different.

You should also check to see that they are really running:

$ ps ax | grep lamson
29438   ??  S 0:05.78 python lamson log
29605   ??  S 0:00.63 python lamson start
29612   ??  S 0:00.19 python lamson start -pid run/queue.pid -boot config.queue
29617   ??  S 0:00.34 python lamson start -pid run/forward.pid -boot config.forward

Step 7: Playing With OneShotBlog

Now we get to play with it. Lamson comes with a web server that you can run to do simple testing so start up a second window/terminal and do this:

$ cd projects/lamson/deploy/
$ . activate LAMSON
(LAMSON) $ cd source/lamson-0.9.3/examples/osb/
(LAMSON) $ lamson web -basedir app/data
Starting server on 127.0.0.1:8888 out of directory 'app/data'

Now hit http://localhost:8888/ with your browser and see the junk left over from your test runs. Most of those posts won’t actually exist, so let’s make a fake one for now.

You can forget about this web server window for now, and go back to your LAMSON window to do this with mutt:

  1. mutt -F muttrc to get it going with a fake setup.
  2. Send an email to first.blog@oneshotblog.com (m is the key).
  3. You’ll get a confirmation back, reply to it.
  4. You’ll get a welcome message, but this message isn’t in the index yet. You can go look at it directly though.
  5. Send another email, this time to my.new.post@oneshotblog.com.
  6. No confirmation this time, just a message saying it was completed.
  7. Now go look at the index (might take a few seconds, up to 10).
  8. Click on the post title and go look at it.
  9. Right click on the [send comment] link and copy the email address.
  10. Go back to mutt and send an email to that address, this will post a comment to that post.
  11. Reply to the comment confirmation email, this should be the only one you get.
  12. In about 10 seconds you’ll see your comment show up.

With that you have fully tested out the OneShotBlog example. All that remains would be a full deployment in a for-real situation, which is what we’ll do next.

Step 8: Running On Port 25 For Real

The only problem with testing out the OneShotBlog with your own email client is that you need to trick your computer into thinking your localhost address is also oneshotblog.com. To do that, open your /etc/hosts file and make whatever changes you need to have localhost be oneshotblog.com also.

You’ll know you’ve got it right when you can point your browser at oneshotblog.com and see your lamson web window display log messages showing you click around.

Remember to undo this or you might be annoyed later.

Next, you’ll need to stop lamson and restart it to use port 25. This will be a problem if you have another server running on port 25, so make sure you turn that server off for now.

I hope you aren’t doing this on a live site where that’s a problem.

$ lamson stop
$ vim config/settings.py

At this point you’ll want to change the receiver_config to look like this in config/settings.py:

receiver_config = {'host': 'localhost', 'port': 25}

Then you’ll want to restart lamson so that it drops privilege to your user after grabbing that port. Easiest way to find out what your user id (uid) and group id (gid) are is to use Python:

Python 2.6.2 (r262:71600, Jun  8 2009, 00:44:56) 
[GCC 4.0.1 (Apple Inc. build 5490)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.getuid()
500
>>> os.getgid()
500
>>> ^D

This shows that I’m uid 500 and gid 500, so now I can start my server:

$ sudo lamson start -uid 500 -gid 500
$ sudo chown -R zedshaw ./

That last command just makes sure that lamson didn’t accidentally change the permission of a stray file or queue to root.

With that you should be running your lamson server as your user rather than as root, but still bound to port 25. Here’s how to check:

$ ps aux | grep lamson | grep root

If you don’t see anything printed out, then you’re safe. If you see something running as root, then you’ve got some work to do.

Conclusion

I’ll leave it to you to actually get your mail client to talk to this OneShotBlog and working. If you have problems, look at all the files in logs/ and also use the lamson queue command to inspect the different queues in the run/ directory.

At this point, you should know enough about setting up a lamson server and configuring a real application (warts and all). You should also have learned how to get a clean Python installation that you can use no matter what your native OS does to Python, even if it’s retarded and renames python to Python for no apparent reason.

This document is very fresh, so send me feedback on your experience with running through it. Make sure you tell me what system you are on and that you ran each command exactly when you do.