Getting Started With Lamson

Lamson is designed to work like modern web application frameworks like Django, TurboGears, ASP.NET, Ruby on Rails, and whatever PHP is using these days. At every design decision Lamson tries to emulate terminology and features found in these frameworks. This Getting Started document will help you get through that terminology, get you started running your first lamson application, and walk you through the code you should read.

In total it should take you about 30 minutes to an hour to complete. If you just want to try Lamson, at least go through the 30 second introduction given first.

The 30 Second Introduction

If you have Python and easy_install already, then try this out:

$ easy_install lamson
$ lamson gen -project mymailserver
$ cd mymailserver
$ lamson syncdb
$ lamson start
$ lamson log -queue run/queue -port 8899
$ nosetests
$ lamson help -for send
$ lamson send -sender me@mydomain.com -to test@test.com \
        -subject "My test." -body "Hi there." -port 8823
$ less logs/lamson.log
$ mutt -F muttrc
$ lamson stop -ALL run

You now have a working base Lamson setup ready for you to work on with the following installed:

  • Lamson and all dependencies (SQLAlchemy, Mako, nosetests)
  • Code for your project in mymailserver. Look in app/handlers and config/settings.py
  • Two initial tests that verify your server is not an open relay and forwards mail in tests/handlers/open_relay_tests.py
  • A “logger” server running on port 8899 that dumps all of its mail into run/queue
  • A config script for mutt (muttrc) that you can use to inspect the run/queue and also send mail using Lamson’s send command.

Important Terminology

If you are an old SMTP guru and/or you’ve never written a web application with a modern web framework, then some of the terminology used in Lamson may seem confusing. Other terms may just confuse you or scare you because they sound complicated. I tried my best to make the concepts used in Lamson understandable and the code that implements them easy to read. In fact, you could probably read the code to Lamson in an evening and understand how everything works.

Experience though has taught me that nobody reads the code, even if it is small. Therefore, here are the most important concepts you should know to get a grasp of Lamson and how it works.

  • MVC — Model View Controller is a design methodology used in web application frameworks
where the data (model), presentation (view), and logic (controller) layers of the application are strictly separated.
  • FSM — Finite State Machine is a special Lamson handler that stores it’s state
between executions. Each time it runs it will perform an action based on what it is send and what it was doing last. FSM in computer science class are overly complex, but in Lamson they are as easy to use as a return statement.
  • Template — Lamson generates the bodies of it’s messages using Templates, which are text files
that have parts that get replaced with variables you pass in. Templates are converted to their final form with a process called rendering.
  • Relay — The relay for a Lamson server is where Lamson delivers it’s messages. Usually the Relay
is a smart tougher server that’s not as smart, but very good at delivering mail. Lamson can also be run as a Relay for testing purposes.
  • Receiver — Lamson typically runs as the Receiver of email. If you are familiar with a web application setup,
then Lamson is the inverse. Instead of Lamson runing “behind” an Apache or Nginx server, Lamson runs “in front” of an SMTP server like Postfix. It listens on port 25, handles the mail it should, and forwards the rest to the Relay. This makes Lamson much more of a Proxy or filter server.
  • Queue — Lamson can also do all of its processing off a queue. In this setup you would have your normal
mail server dump all mail to a maildir queue, and then tell Lamson to process messages out of there. This can be combined with the usual Receiver+Relay configuration for processing messages that might take a long time.
  • Maildir — A standard created for the qmail project with stores mail in a directory such that you can access
the mail atomically and store it on a shared disk without conflicts or locking.

Managing Your Server

Your Lamson application is now running inside the Lamson Python server. This is a very simple server based on Python’s smtpd and asyncore libraries.

If you want to know more about how it operates, take a look at the lamson/server.py file in the source distribution.

You’ll need to use a few Lamson commands to manage the server. You already experienced them in the 30 second introduction, and you can review them all or see them by using the lamson help command.

Right now you have Lamson running on port 8823 and a “lamson logger” running on 8899. Before we learn how to manage them and what they do, open up the config/settings.py file and take a look at the following lines:

relay = { "host": "localhost", "port": 8899, "debug": 1}
receiver = { "host": "localhost", "port": 8823}

Your file probably has some comments telling you what these do, but it’s important to understand how they work.

The receiver setting is used by the lamson start command to figure out where to listen for incoming SMTP connections. In a real installation this would be port 25 on your external IP address. It’s where the internet talks to your server.

The relay setting is used by Lamson to figure out where to forward message replies (responses) for real delivery. Normally this would be a “smart host” running a more established server like Postfix or Exim to do the grunt work of delivering to the final recipients.

Lamson can deliver directly, but that is a waste of resources. In the same way you wouldn’t have Python or Ruby deliver video, static text, or images over HTTP, you don’t want Lamson bothering with the nasty business of delivering mail to people.

You probably don’t have another SMTP server running, and even if you did, it’d be a pain to configure it for development purposes. You’d have to setup aliases, new mail boxes, restart it all the time, and other annoyances.

For development, what we want is our own little private SMTP relay, and since Lamson can also deliver mail, that is what we get with the command:

$ lamson log -queue run/queue -port 8899

This tells Lamson to run as a “logging server”, which doesn’t actually deliver any mail. If you don’t give a -queue option, then it will log to logs/lamson.log and print out any emails it receives. In this case we actually want the logger to dump mail it receives into the run/queue Maildir. That’s what -queue does in this command.

Lamson uses Maildir by default since it is the most reliable and fastest mail queue format available. It could also store mail messages to any queue supported by Python’s mailbox library. If you were adventurous you could also use a RDBMS, but that’s just silly.

Stopping Lamson

The PID files are stored in the run directory. Here’s a sample session where I stop all the running servers:

$ ls -l run/*.pid
-rw-r—r—  1 zedshaw  staff  5 May 16 16:41 run/log.pid
-rw-r—r—  1 zedshaw  staff  5 May 16 16:41 run/smtp.pid

$ lamson stop -ALL run Stopping processes with the following PID files: ['run/log.pid’, 'run/smtp.pid’] Attempting to stop lamson at pid 1693 Attempting to stop lamson at pid 1689

You can also pass other options to the stop command to just stop one server. Use lamson help -for stop to see all the options.

Starting Lamson Again

Hopefully you’ve been paying attention and have figured out how to restart lamson and the logging server. Just in case, here it is again:

$ lamson start
$ lamson log -queue run/queue -port 8899

You should also look in the logs/lamson.log file to see that it actually started. The other files in the logs directory contain messages dumped to various output methods (like Python’s stdout and stderr). Periodically, if the information you want is not in logs/lamson.log then it is probably in the other files.

You can change your logging configuration by eddin the logging line your config/settings.py file.

Other Useful Commands

You should read the available commands documentation to get an overview, and you can also use lamson help to see them at any time.

send

The first useful command is lamson send, which lets you send mail to SMTP servers (not just Lamson) and watch the full SMTP protocol chatter. Here’s a sample:

$ lamson send -port 25 -host zedshaw.com -debug 1 \
    -sender tester@test.com -to zedshaw@zedshaw.com \
    -subject "Hi there" -body "Test body."
send: 'ehlo zed-shaws-macbook.local\r\n'
reply: '502 Error: command "EHLO" not implemented\r\n'
reply: retcode (502); Msg: Error: command "EHLO" not implemented
send: 'helo zed-shaws-macbook.local\r\n'
reply: '250 localhost.localdomain\r\n'
reply: retcode (250); Msg: localhost.localdomain
send: 'mail FROM:<tester@test.com>\r\n'
reply: '250 Ok\r\n'
reply: retcode (250); Msg: Ok
send: 'rcpt TO:<zedshaw@zedshaw.com>\r\n'
reply: '250 Ok\r\n'
reply: retcode (250); Msg: Ok
send: 'data\r\n'
reply: '354 End data with <CR><LF>.<CR><LF>\r\n'
reply: retcode (354); Msg: End data with <CR><LF>.<CR><LF>
data: (354, 'End data with <CR><LF>.<CR><LF>')
send: 'Content-Type: text/plain; charset="us-ascii"\r\nMIME-Version: 1.0\r\nContent-Transfer-Encoding: 7bit\r\nSubject: Hi there\r\nFrom: tester@test.com\r\nTo: zedshaw@zedshaw.com\r\n\r\n.\r\n'
reply: '250 Ok\r\n'
reply: retcode (250); Msg: Ok
data: (250, 'Ok')
send: 'quit\r\n'
reply: '221 Bye\r\n'
reply: retcode (221); Msg: Bye

Using this helps you debug your Lamson server by showing you the exact protocol sent between you and the server. It is also a useful SMTP server debug command by itself.

When you use the supplied muttrc you’ll be configured to use Lamson’s sendmail (not *send) command as your delivery command. This lets you use mutt as a complete development tool with minimal configuration.

queue

The lamson queue command lets you investigate and manipulate the run/queue (or any maildir). You can pop a message off, get a message by its key, remove a message by its key, count the messages,clear the queue, list keys in the queue. It gives you a lower level view of the queue than mutt would, and lets you manipulate it behind the scenes.

restart

Lamson does reload the code of your project when it receives a new request (probably too frequently), but if you change the config/settings.py file then you need to restart. Easiest way to do that is with the restart command.

deq

A more advanced configuration is to have Lamson or another mail server deliver pending messages to the run/queue, and then you run a “dequeue server” that processes the messages in it. A dequeue server is great for processing that would take too long or needs to be distributed. Using it is outside the scope of this manual, but you can check its help and look at the mailinglist sample to see how you might do this.

syncdb

Lamson uses the SQLAlchemy library as its ORM layer, and also needs a working database to process any requests. This is because the FSM system needs to store status information in a table in the database.

To get your database start use the lamson syncdb command, and if you make changes to your model rerun it.

Testing It With Mutt

Mutt is a very handy testing tool, and with our development Lamson setup we can use the included muttrc to fake Mutt out. Here’s the muttrc:

set mbox_type=Maildir
set folder="run/queue"
set mask="!^\\.[^.]"
set mbox="run/queue"
set record="+.Sent"
set postponed="+.Drafts"
set spoolfile="run/queue"
set sendmail="/usr/local/bin/lamson sendmail -port 8823 -host 127.0.0.1"

This tells Mutt to use the run/queue directory as a Maildir queue for its inbox. It also configures it to use the /usr/local/bin/lamson command to do the delivery. Using the lamson sendmail command (NOT *send) lets Mutt deliver to different ports and servers in a development situation.

Another example of how difficult SMTP systems are to use is the fact that you have to reconfigure the postfix main.cf to make the sendmail command use a different port or server. There are no command line options to specify it.

Walking Through The Code

Coming soon….

Writing A Unit Test

Coming soon….

Making It Do Something

Coming soon….

Other Examples

Next you’ll want to sink your teeth in a bigger example. Go grab the source distribution .tar.gz and extract it so you can get at the examples:

$ tar -xzvf lamson-VERSION.tar.gz
$ cd lamson-VERSION
$ cd examples/mailinglist

You are now in the mailinglist example. Using what you’ve learned so far you can start reviewing the code and finding out how a working example operates.

Getting Help

As you work through this documentation, send your questions to me and I’ll try to help you.