SQLObject and Database Programming in Python

Ian Bicking

Imaginary Landscape

The Ongoing Example

Address book:

person:
- first_name
- last_name
- email (many emails for one person)
email:
- type
- address

The DB API

Database programming in Python is based on the DB-API. Some supported databases:

MySQL
PostgreSQL (multiple drivers)
SQLite
Firebird (Interbase)
MAXDB (SAP DB)
Oracle, Sybase, MS SQL Server

DB API Example

Example:

import MySQLdb
conn = MySQLdb.connect(
    db='test', username='ianb')
cur = conn.cursor()
cur.execute(
    "SELECT id, first_name, last_name FROM person")
result = cur.fetchall()
for id, first_name, last_name in result:
    print "%2i %s, %s" % (id, last_name, first_name)

DB API

Import the database module (MySQLdb, psycopg, sqlite, etc)
Use module.connect(...) to create a connection.
Use connection.cursor() to get a cursor. Cursors do all the work.
Use cursor.execute(sql_query) to run something.
Use cursor.fetchall() to get results.

DB API problems

Connections and cursors are tedious.
The databases don't all work the same; multiple SQL dialects.
Lots of time spent writing SQL.
Database updates can effect all the SQL you've written.
Unless you make abstractions...

Abstractions

Every significant database application has a database abstraction layer.
You write one whether you mean to or not.

Object-Relational Mappers

The table:

person
id	first_name	last_name
1	John	Doe
2	Tom	Jones
...

ORMs: Classes

The class:

person
id	first_name	last_name
1	John	Doe
2	Tom	Jones
...

class Person(SQLObject):
    # id is implicit
    first_name = StringCol()
    last_name = StringCol()

ORMs: Instances

An instance:

person
id	first_name	last_name
1	John	Doe
2	Tom	Jones
...

>>> john = Person.get(1)
>>> john.first_name
'John'

ORM: summary

Every table is a class
Every row is an instance
Columns become attributes

Usage

__connection__ = 'postgres://pgsql@localhost/test'
class Person(SQLObject):
    first_name = StringCol()
    last_name = StringCol()
    emails = MultipleJoin('Email')

Usage (classes)

class Email(SQLObject):
    person = ForeignKey('Person')
    type = EnumCol(['home', 'work'])
    address = StringCol()

Usage (creating tables)

def createTables():
    for table in (Person, Email):
        table.createTable(ifNotExists=True)

Usage (instances)

Using your classes:

>>> john = Person(first_name='John', last_name='Doe')
>>> email = Email(person=john, type='home', 
...     address='john@work.com')
>>> john.emails
[]

Usage (instances)

>>> tom = Person(first_name='Tom', last_name='Jones')
>>> tom is Person.get(tom.id)
True
>>> list(Person.selectBy(first_name='John'))
[]

Defining Classes

The class name matches the table name
The attributes match the column names
(kind of...)

Defining Classes...

You can add extra methods:

class Person(SQLObject):
    ...
    def _get_name(self):
        return self.first_name + ' ' + self.last_name

>>> tom.name
'Tom Jones'

Automatic properties

_get_attr() methods are called whenever the obj.attr attribute is called
_set_attr() methods are called whenever the obj.attr = value statement is run
_set_attr() is optional

Defining classes...

You can override columns:

class Person(SQLObject):
    ...
    last_name_searchable = StringCol()
    def _set_last_name(self, value):
        self._SO_set_last_name(self, value)
        self.last_name_lower = re.sub(r'[^a-zA-Z]', '', value).lower()

Defining classes...

You can fiddle with the naming:

class Person(SQLObject):
    _table = "people"
    first_name = StringCol(dbName="fname")
    ...

Connecting classes

Foreign Keys:

class Email(SQLObject):
    person = ForeignKey('Person')
    ...

Note we use a string for 'Person'.

Backreferences

The other side of one-to-many:

class Person(SQLObject):
    ...
    emails = MultipleJoin('Email')

Many-to-many

Many to many relationships imply a "hidden" table:

class Address(SQLObject):
    people = RelatedJoin('Person')
    ...
class Person(SQLObject):
    addresses = RelatedJoin('Address')

Many-to-many...

The intermediate table created:

CREATE TABLE address_person (
  address_id INT NOT NULL,
  person_id INT NOT NULL
);

Lazy classes

SQLObject can use reflection to figure out what columns your table has:

class Person(SQLObject):
    _fromDatabase = True

You can start with _fromDatabase = True and add in explicit columns, overriding defaults.

Instances

Use Person.get(id) to retrieve a row
Use Person(first_name=...) to insert a row (the new object is returned)
Use aPerson.destroySelf() to delete a row

Updating

aPerson.first_name = "new_name"
UPDATE is executed immediately
Update multiple columns at once:
aPerson.set(first_name="new_fname", last_name="new_lname")

Selecting

You can use Python expressions (kind of):

query = Person.q.first_name == "Joe"
Person.select(query)

Selecting...

Class.q.column_name creates a magical object that creates queries.
sqlrepr() turns these query objects into SQL (mostly hidden)
sqlrepr(Person.q.first_name == "Joe", 'mysql') creates the SQL person.first_name = 'Joe'

Selecting...

Complicated joins are possible:

Person.select((Person.q.id == Email.q.personID)
              & (Email.q.address.startswith('joe')))

Becomes:

SELECT person.id, person.first_name, person.last_name
FROM person, email
WHERE person.id = email.person_id
      AND email.address LIKE 'joe%'

SelectResult

Select results are more than just lists:

You can slice them, and LIMIT and OFFSET statements are added to your query
You can iterate through them, and avoid loading all objects into memory
You can do things like select_result.count() to run aggregate functions
To get a real list, you must do list(select_result)

Caching

SQLObject caches everything
Maybe too much (bad if you have multiple updaters)
Needs to because there's no joins
Has potential to be faster than ad hoc queries
Thread safety or thread confusion?

Performance

Not good if you are dealing with lots and lots of rows
Inserts and selects substantially slower
But for many applications you'll pay that price sooner or later anyway

The problem with ORMs

ORMs live in the world between Object Oriented programmers (and programs) and Relational programmers (and programs). Neither will be happy.

ORM Mismatch

Python classes aren't tables
Python instances aren't rows

ORM Mismatch: Classes

Databases don't have inheritance
Even when they do, no one uses it
Some tables are too boring (e.g., intermediate tables for many-to-many relationships), or don't warrant the overhead

ORM Mismatch: Instances

Instances get garbage collected, rows are forever
Rows exist even when instances don't
Instances may still exist when the row is gone
In the relational calculus, rows don't have real identity

ORM Mismatch: Relations

Things like joins don't make sense for objects, but are central to relations
Views and stored procedures also don't make sense
Aggregate functions and set operations aren't natural in Python (the for loop is natural)

Solving the question by avoidance

SQLObject doesn't let you forget SQL
You still have to think relationally
But many relational concepts don't work either
Advantage: it's easy

Solving the Mismatch

SQLObject's answer: don't try too hard.

There exist database restrictions; not every database schema maps well. (But most are fine)
Sometimes you have to do things more slowly or more imperatively, compared to a purely relational technique
There exist object restrictions; not every object-oriented concept maps well; e.g., no inheritance
Inheritance isn't planned; this isn't transparent persistence

Some other alternatives

Several Python ORMs exist; contrasts:

SQLObject doesn't try to be too OO, so it works well with normal database schemas
SQLObject doesn't have a compile step
SQLObject isn't just a SQL wrapper
SQLObject is reasonably complete

Other alternatives...

See the Python Wiki at: python.org/moin
The page is HigherLevelDatabaseProgramming.

ChiPy, 11 November 2004