Basic module usage

The basic Psycopg usage is common to all the database adapters implementing the DB API 2.0 protocol. Here is an interactive session showing some of the basic commands:

>>> import psycopg2

# Connect to an existing database
>>> conn = psycopg2.connect("dbname=test user=postgres")

# Open a cursor to perform database operations
>>> cur = conn.cursor()

# Execute a command: this creates a new table
>>> cur.execute("CREATE TABLE test (id serial PRIMARY KEY, num integer, data varchar);")

# Pass data to fill a query placeholders and let Psycopg perform
# the correct conversion (no more SQL injections!)
>>> cur.execute("INSERT INTO test (num, data) VALUES (%s, %s)",
...      (100, "abc'def"))

# Query the database and obtain data as Python objects
>>> cur.execute("SELECT * FROM test;")
>>> cur.fetchone()
(1, 100, "abc'def")

# Make the changes to the database persistent
>>> conn.commit()

# Close communication with the database
>>> cur.close()
>>> conn.close()

The main entry point of Psycopg are:

Passing parameters to SQL queries

Psycopg casts Python variables to SQL literals by type. Many standard Python types are already adapted to the correct SQL representation.

Example: the Python function call:

>>> cur.execute(
...     """INSERT INTO some_table (an_int, a_date, a_string)
...         VALUES (%s, %s, %s);""",
...     (10, datetime.date(2005, 11, 18), "O'Reilly"))

is converted into the SQL command:

INSERT INTO some_table (an_int, a_date, a_string)
 VALUES (10, '2005-11-18', 'O''Reilly');

Named arguments are supported too using %(name)s placeholders. Using named arguments the values can be passed to the query in any order and many placeholder can use the same values:

>>> cur.execute(
...     """INSERT INTO some_table (an_int, a_date, another_date, a_string)
...         VALUES (%(int)s, %(date)s, %(date)s, %(str)s);""",
...     {'int': 10, 'str': "O'Reilly", 'date': datetime.date(2005, 11, 18)})

While the mechanism resembles regular Python strings manipulation, there are a few subtle differences you should care about when passing parameters to a query:

  • The Python string operator % is not used: the execute() method accepts a tuple or dictionary of values as second parameter. Never use % or + to merge values into queries.

  • The variables placeholder must always be a %s, even if a different placeholder (such as a %d for integers or %f for floats) may look more appropriate:

    >>> cur.execute("INSERT INTO numbers VALUES (%d)", (42,)) # WRONG
    >>> cur.execute("INSERT INTO numbers VALUES (%s)", (42,)) # correct
    
  • For positional variables binding, the second argument must always be a sequence, even if it contains a single variable. And remember that Python requires a comma to create a single element tuple:

    >>> cur.execute("INSERT INTO foo VALUES (%s)", "bar")    # WRONG
    >>> cur.execute("INSERT INTO foo VALUES (%s)", ("bar"))  # WRONG
    >>> cur.execute("INSERT INTO foo VALUES (%s)", ("bar",)) # correct
    >>> cur.execute("INSERT INTO foo VALUES (%s)", ["bar"])  # correct
    
  • Only variable values should be bound via this method: it shouldn’t be used to set table or field names. For these elements, ordinary string formatting should be used before running execute().

The problem with the query parameters

The SQL representation for many data types is often not the same of the Python string representation. The classic example is with single quotes in strings: SQL uses them as string constants bounds and requires them to be escaped, whereas in Python single quotes can be left unescaped in strings bounded by double quotes. For this reason a naïve approach to the composition of query strings, e.g. using string concatenation, is a recipe for terrible problems:

>>> SQL = "INSERT INTO authors (name) VALUES ('%s');" # NEVER DO THIS
>>> data = ("O'Reilly", )
>>> cur.execute(SQL % data) # THIS WILL FAIL MISERABLY
ProgrammingError: syntax error at or near "Reilly"
LINE 1: INSERT INTO authors (name) VALUES ('O'Reilly')
                                              ^

If the variable containing the data to be sent to the database comes from an untrusted source (e.g. a form published on a web site) an attacker could easily craft a malformed string, either gaining access to unauthorized data or performing destructive operations on the database. This form of attack is called SQL injection and is known to be one of the most widespread forms of attack to servers. Before continuing, please print this page as a memo and hang it onto your desk.

Psycopg can convert automatically Python objects into and from SQL literals: using this feature your code will result more robust and reliable. It is really the case to stress this point:

Warning

Never, never, NEVER use Python string concatenation (+) or string parameters interpolation (%) to pass variables to a SQL query string. Not even at gunpoint.

The correct way to pass variables in a SQL command is using the second argument of the execute() method:

>>> SQL = "INSERT INTO authors (name) VALUES (%s);" # Notice: no quotes
>>> data = ("O'Reilly", )
>>> cur.execute(SQL, data) # Notice: no % operator

Adaptation of Python values to SQL types

Many standards Python types are adapted into SQL and returned as Python objects when a query is executed.

If you need to convert other Python types to and from PostgreSQL data types, see Adapting new Python types to SQL syntax and Type casting of SQL types into Python objects. You can also find a few other specialized adapters in the psycopg2.extras module.

In the following examples the method mogrify() is used to show the SQL string that would be sent to the database.

  • Python None and boolean values True and False are converted into the proper SQL literals:

    >>> cur.mogrify("SELECT %s, %s, %s;", (None, True, False))
    >>> 'SELECT NULL, true, false;'
    
  • Numeric objects: int, long, float, decimal.Decimal are converted in the PostgreSQL numerical representation:

    >>> cur.mogrify("SELECT %s, %s, %s, %s;", (10, 10L, 10.0, Decimal("10.00")))
    >>> 'SELECT 10, 10, 10.0, 10.00;'
    
  • String types: str, unicode are converted in SQL string syntax. !unicode objects (!str in Python 3) are encoded in the connection encoding to be sent to the backend: trying to send a character not supported by the encoding will result in an error. Received data can be converted either as !str or !unicode: see Unicode handling.
  • Binary types: Python types representing binary objects are converted in PostgreSQL binary string syntax, suitable for bytea fields. Such types are buffer (only available in Python 2), memoryview (available from Python 2.7), bytearray (available from Python 2.6) and bytes (only form Python 3: the name is available from Python 2.6 but it’s only an alias for the type !str). Any object implementing the Revised Buffer Protocol should be usable as binary type where the protocol is supported (i.e. from Python 2.6). Received data is returned as !buffer (in Python 2) or !memoryview (in Python 3).

    Changed in version 2.4: only strings were supported before.

    Note

    In Python 2, if you have binary data in a !str object, you can pass them to a bytea field using the psycopg2.Binary wrapper:

    mypic = open('picture.png', 'rb').read()
    curs.execute("insert into blobs (file) values (%s)",
        (psycopg2.Binary(mypic),))
    

    Warning

    PostgreSQL 9 uses by default a new “hex” format to emit bytea fields. Unfortunately this format can’t be parsed by libpq versions before 9.0. This means that using a library client with version lesser than 9.0 to talk with a server 9.0 or later you may have problems receiving bytea data. To work around this problem you can set the bytea_output parameter to escape, either in the server configuration or in the client session using a query such as SET bytea_output TO escape; before trying to receive binary data.

    Starting from Psycopg 2.4 this condition is detected and signaled with a InterfaceError.

  • Date and time objects: builtin datetime.datetime, datetime.date, datetime.time, datetime.timedelta are converted into PostgreSQL’s timestamp, date, time, interval data types. Time zones are supported too. The Egenix mx.DateTime objects are adapted the same way:

    >>> dt = datetime.datetime.now()
    >>> dt
    datetime.datetime(2010, 2, 8, 1, 40, 27, 425337)
    
    >>> cur.mogrify("SELECT %s, %s, %s;", (dt, dt.date(), dt.time()))
    "SELECT '2010-02-08T01:40:27.425337', '2010-02-08', '01:40:27.425337';"
    
    >>> cur.mogrify("SELECT %s;", (dt - datetime.datetime(2010,1,1),))
    "SELECT '38 days 6027.425337 seconds';"
    
  • Python lists are converted into PostgreSQL ARRAYs:

    >>> cur.mogrify("SELECT %s;", ([10, 20, 30], ))
    'SELECT ARRAY[10, 20, 30];'
    
  • Python tuples are converted in a syntax suitable for the SQL IN operator and to represent a composite type:

    >>> cur.mogrify("SELECT %s IN %s;", (10, (10, 20, 30)))
    'SELECT 10 IN (10, 20, 30);'
    

    Note

    SQL doesn’t allow an empty list in the IN operator, so your code should guard against empty tuples.

    If you want PostgreSQL composite types to be converted into a Python tuple/namedtuple you can use the register_composite() function.

    New in version 2.0.6: the tuple IN adaptation.

    Changed in version 2.0.14: the tuple IN adapter is always active. In previous releases it was necessary to import the extensions module to have it registered.

    Changed in version 2.3: collections.namedtuple instances are adapted like regular tuples and can thus be used to represent composite types.

  • Python dictionaries are converted into the hstore data type. By default the adapter is not enabled: see register_hstore() for further details.

    New in version 2.3: the hstore adaptation.

Unicode handling

Psycopg can exchange Unicode data with a PostgreSQL database. Python !unicode objects are automatically encoded in the client encoding defined on the database connection (the PostgreSQL encoding, available in connection.encoding, is translated into a Python codec using the encodings mapping):

>>> print u, type(u)
àèìòù€ <type 'unicode'>

>>> cur.execute("INSERT INTO test (num, data) VALUES (%s,%s);", (74, u))

When reading data from the database, in Python 2 the strings returned are usually 8 bit !str objects encoded in the database client encoding:

>>> print conn.encoding
UTF8

>>> cur.execute("SELECT data FROM test WHERE num = 74")
>>> x = cur.fetchone()[0]
>>> print x, type(x), repr(x)
àèìòù€ <type 'str'> '\xc3\xa0\xc3\xa8\xc3\xac\xc3\xb2\xc3\xb9\xe2\x82\xac'

>>> conn.set_client_encoding('LATIN9')

>>> cur.execute("SELECT data FROM test WHERE num = 74")
>>> x = cur.fetchone()[0]
>>> print type(x), repr(x)
<type 'str'> '\xe0\xe8\xec\xf2\xf9\xa4'

In Python 3 instead the strings are automatically decoded in the connection encoding, as the !str object can represent Unicode characters. In Python 2 you must register a typecaster in order to receive !unicode objects:

>>> psycopg2.extensions.register_type(psycopg2.extensions.UNICODE, cur)

>>> cur.execute("SELECT data FROM test WHERE num = 74")
>>> x = cur.fetchone()[0]
>>> print x, type(x), repr(x)
àèìòù€ <type 'unicode'> u'\xe0\xe8\xec\xf2\xf9\u20ac'

In the above example, the UNICODE typecaster is registered only on the cursor. It is also possible to register typecasters on the connection or globally: see the function register_type() and Type casting of SQL types into Python objects for details.

Note

In Python 2, if you want to receive uniformly all your database input in Unicode, you can register the related typecasters globally as soon as Psycopg is imported:

import psycopg2
import psycopg2.extensions
psycopg2.extensions.register_type(psycopg2.extensions.UNICODE)
psycopg2.extensions.register_type(psycopg2.extensions.UNICODEARRAY)

and then forget about this story.

Time zones handling

The PostgreSQL type timestamp with time zone is converted into Python datetime.datetime objects with a datetime.datetime.tzinfo attribute set to a FixedOffsetTimezone instance.

>>> cur.execute("SET TIME ZONE 'Europe/Rome';")  # UTC + 1 hour
>>> cur.execute("SELECT '2010-01-01 10:30:45'::timestamptz;")
>>> cur.fetchone()[0].tzinfo
psycopg2.tz.FixedOffsetTimezone(offset=60, name=None)

Notice that only time zones with an integer number of minutes are supported: this is a limitation of the Python datetime module. A few historical time zones had seconds in the UTC offset: these time zones will have the offset rounded to the nearest minute, with an error of up to 30 seconds.

>>> cur.execute("SET TIME ZONE 'Asia/Calcutta';")  # offset was +5:53:20
>>> cur.execute("SELECT '1930-01-01 10:30:45'::timestamptz;")
>>> cur.fetchone()[0].tzinfo
psycopg2.tz.FixedOffsetTimezone(offset=353, name=None)

Changed in version 2.2.2: timezones with seconds are supported (with rounding). Previously such timezones raised an error. In order to deal with them in previous versions use psycopg2.extras.register_tstz_w_secs().

Transactions control

In Psycopg transactions are handled by the connection class. By default, the first time a command is sent to the database (using one of the cursors created by the connection), a new transaction is created. The following database commands will be executed in the context of the same transaction – not only the commands issued by the first cursor, but the ones issued by all the cursors created by the same connection. Should any command fail, the transaction will be aborted and no further command will be executed until a call to the connection.rollback() method.

The connection is responsible to terminate its transaction, calling either the commit() or rollback() method. Committed changes are immediately made persistent into the database. Closing the connection using the close() method or destroying the connection object (using !del or letting it fall out of scope) will result in an implicit !rollback() call.

It is possible to set the connection in autocommit mode: this way all the commands executed will be immediately committed and no rollback is possible. A few commands (e.g. CREATE DATABASE, VACUUM...) require to be run outside any transaction: in order to be able to run these commands from Psycopg, the session must be in autocommit mode. Read the documentation for connection.set_isolation_level() to know how to change the commit mode.

Server side cursors

When a database query is executed, the Psycopg cursor usually fetches all the records returned by the backend, transferring them to the client process. If the query returned an huge amount of data, a proportionally large amount of memory will be allocated by the client.

If the dataset is too large to be practically handled on the client side, it is possible to create a server side cursor. Using this kind of cursor it is possible to transfer to the client only a controlled amount of data, so that a large dataset can be examined without keeping it entirely in memory.

Server side cursor are created in PostgreSQL using the DECLARE command and subsequently handled using MOVE, FETCH and CLOSE commands.

Psycopg wraps the database server side cursor in named cursors. A named cursor is created using the cursor() method specifying the name parameter. Such cursor will behave mostly like a regular cursor, allowing the user to move in the dataset using the scroll() method and to read the data using fetchone() and fetchmany() methods.

Named cursors are also iterable like regular cursors. Notice however that before Psycopg 2.4 iteration was performed fetching one record at time from the backend, resulting in a large overhead. The attribute itersize now controls how many records are now fetched at time during the iteration: the default value of 2000 allows to fetch about 100KB per roundtrip assuming records of 10-20 columns of mixed number and strings; you may decrease this value if you are dealing with huge records.

Thread and process safety

The Psycopg module and the connection objects are thread-safe: many threads can access the same database either using separate sessions and creating a !connection per thread or using the same using the same connection and creating separate cursors. In DB API 2.0 parlance, Psycopg is level 2 thread safe.

The difference between the above two approaches is that, using different connections, the commands will be executed in different sessions and will be served by different server processes. On the other hand, using many cursors on the same connection, all the commands will be executed in the same session (and in the same transaction if the connection is not in autocommit mode), but they will be serialized.

The above observations are only valid for regular threads: they don’t apply to forked processes nor to green threads. libpq connections shouldn’t be used by a forked processes, so when using a module such as multiprocessing or a forking web deploy method such as FastCGI ensure to create the connections after the fork.

Connections shouldn’t be shared either by different green threads: doing so may result in a deadlock. See Support to coroutine libraries for further details.

Using COPY TO and COPY FROM

Psycopg cursor objects provide an interface to the efficient PostgreSQL COPY command to move data from files to tables and back. The methods exposed are:

copy_from()
Reads data from a file-like object appending them to a database table (COPY table FROM file syntax). The source file must have both !read() and !readline() method.
copy_to()
Writes the content of a table to a file-like object (COPY table TO file syntax). The target file must have a write() method.
copy_expert()
Allows to handle more specific cases and to use all the COPY features available in PostgreSQL.

Please refer to the documentation of the single methods for details and examples.

Access to PostgreSQL large objects

PostgreSQL offers support to large objects, which provide stream-style access to user data that is stored in a special large-object structure. They are useful with data values too large to be manipulated conveniently as a whole.

Psycopg allows access to the large object using the lobject class. Objects are generated using the connection.lobject() factory method. Data can be retrieved either as bytes or as Unicode strings.

Psycopg large object support efficient import/export with file system files using the !lo_import() and !lo_export() libpq functions.

Two-Phase Commit protocol support

New in version 2.3.

Psycopg exposes the two-phase commit features available since PostgreSQL 8.1 implementing the two-phase commit extensions proposed by the DB API 2.0.

The DB API 2.0 model of two-phase commit is inspired to the XA specification, according to which transaction IDs are formed from three components:

  • a format ID (non-negative 32 bit integer)
  • a global transaction ID (string not longer than 64 bytes)
  • a branch qualifier (string not longer than 64 bytes)

For a particular global transaction, the first two components will be the same for all the resources. Every resource will be assigned a different branch qualifier.

According to the DB API 2.0 specification, a transaction ID is created using the connection.xid() method. Once you have a transaction id, a distributed transaction can be started with connection.tpc_begin(), prepared using tpc_prepare() and completed using tpc_commit() or tpc_rollback(). Transaction IDs can also be retrieved from the database using tpc_recover() and completed using the above !tpc_commit() and !tpc_rollback().

PostgreSQL doesn’t follow the XA standard though, and the ID for a PostgreSQL prepared transaction can be any string up to 200 characters long. Psycopg’s Xid objects can represent both XA-style transactions IDs (such as the ones created by the !xid() method) and PostgreSQL transaction IDs identified by an unparsed string.

The format in which the Xids are converted into strings passed to the database is the same employed by the PostgreSQL JDBC driver: this should allow interoperation between tools written in Python and in Java. For example a recovery tool written in Python would be able to recognize the components of transactions produced by a Java program.

For further details see the documentation for the above methods.