Author: | Jacob Smullyan |
---|---|
Contact: | jsmullyan@gmail.com |
Date: | 2007-10-30 |
Revision: | 214 |
Copyright: | 2004, 2007 Jacob Smullyan |
skunk.web is the latest incarnation of the SkunkWeb application server, refactored into a set of libraries and WSGI applications.
Highlights include:
skunk.web uses setuptools, so you can install it with easy_install:
easy_install skunk.web
Or from a tarball or SVN checkout:
python setup.py install
If you want to follow development:
python setup.py develop
The skunk.cache package is a simple memoization facility for callables with in-memory, on-disk, and memcached backends.
Typical usage:
>>> from skunk.cache import * >>> mycache=DiskCache('/tmp/mycache') >>> cache=CacheDecorator(mycache, defaultPolicy=YES) >>> @cache(expiration="5m") ... def foo(x, y): ... print "actually calculating ...." ... return x + y >>> foo(5, 5) actually calculating .... 10 >>> foo(5, 5) 10
Normally you'd want to use the decorator, but you can also the same result by using the call() method of a cache object:
>>> from skunk.cache import * >>> mycache=MemoryCache() >>> def foo(x, y): ... print "actually calculating ...." ... return x+ y >>> entry=mycache.call(foo, (100, 50), FORCE) actually calculating .... >>> entry.value 150
When you call a function via the cache, you can specify one of several cache policies:
The cache expiration -- how long the cache entry should live -- can be specified either by the caller, by passing an expiration parameter, or by the callee, by setting an __expiration__ attribute on itself or on the value it returns. The expiration can be specified either as a duration, an absolute or relative expiration time, or a list of such values, expressed in a number of possible formats.
To specify a duration, a string may be used containing a sequence of integer-letter pairs specifying units of time:
You may use any combination of these integer-letter pairs, in any order, and the durations will be summed. For example, "3h2m8s" means "three hours, two minutes and eight seconds".
You may wish to cache a value until a particular (absolute) time. You can use a value that directly represents an absolute timestamp for this, such as a datetime.datetime or mx.DateTime.DateTimeType instance. Or you can use a string in the format yyyy-mm-dd[:hh:mm[:ss]].
You can also specify a relative time, with objects like mx.DateTime.RelativeDateTime and dateutil.relativedelta.relativedelta, and with strings like hh:mm[:ss] or :mm[:ss]
And finally, you can use a list or tuple of such values, freely intermixed; the nearest future time of those indicated will be the effective value.
For example, if the expiration is stated as:
(':00', ':30', '5m')
and the current time is 2:34 PM, the effective expiration will be 2:39 PM. But if the current time is 2:29 PM, the expiration will be 2:30 PM.
Many variables are configured in the skunk libraries by refering to attributes of a global configuration object, skunk.config.Configuration. This object manages default values of config attributes, user overrides of same, and conditional overrides, which are invoked when a certain predicate obtains.
Typical use:
>>> from skunk.config import Configuration, RegexMatcher >>> C.setDefaults(test1='foo') >>> C.test1 'foo' >>> C.load_kw(test1='hanky') # load user value >>> C.test1 'hanky' >>> C.addMatcher(RegexMatcher('HTTP_HOST', 'shch', test1='grape')) >>> C.scope({'HTTP_HOST', 'www.freshcheese.net'}) >>> C.test1 'grape' >>> C.trim() # discard scoping >>> C.test1 'hanky' >>> C.reset() # discard user configuration >>> C.test1 'foo'
Conditional overrides, like the HTTP_HOST example above, are accomplished by adding a ScopeMatcher, an object with the following significant attributes:
When the config object is "scoped" (by calling its scope() method), it is exposed to an "environment" to which its scope matchers may react, potentially rejiggering the object's effective values until it is "trimmed" (by calling trim()), at which all scope-related user configuration is dropped. The environment is expected to be a dictionary; as used in SkunkWeb, it is the WSGI environment, with a few additional keys for convenience ('url' and 'path', which otherwise matchers would need to calculate). Repeated calls to scope() with different data updates, but does not replace, the previous effective scope environment; the configuration after:
Configuration.trim() Configuration.scope({'bob' : 'present'}) Configuration.scope({'harry' : 'absent'})
is the same as after doing:
Configuration.trim() Configuration.scope({'bob' : 'present', 'harry' : 'absent'})
If you want to be sure that the values you are passing to scope are totally determinative of the resulting user configuration, you must call trim() first to wipe out any previous configuration state.
The most general matcher, PredicateMatcher, expects its predicate to be a function that takes one argument, the current scope environment, and returns a true or false value.
There are several other predefined matchers that are convenient for many tasks, performing tests (for equality, and various types of string matching) on specified keys within the scope environment.
User configuration can be added either from within Python modules via the load_kw method, as above, or loaded directly from a Python file or string that is executed in the namespace of the configuration object. In the latter case, several convenient functions are bound in that namespace:
Include: | includes another config file |
---|---|
Scope: | add scope matchers |
Predicate: | a PredicateMatcher |
Regex: | a RegexMatcher |
Glob: | a GlobMatcher |
Equal: | a StrictMatcher (tests for equality) |
A config file using these facilities might look like:
componentRoot='/var/www/roots/default' Scope(Glob('HTTP_HOST', 'www.pimppants.*', Equal('HTTP_PORT', '8080', componentRoot='/var/www/roots/8080'), componentRoot='/var/www/roots/pimppants')) Include('nighthats.conf')
The version of STML described here is largely but not completely backwards-compatible variant of the version implemented in SkunkWeb 3 (hereafter called "STML3"). The most important differences will be noted below.
STML is an extensible, tag-based templating language based on (and compiled to) Python, especially well-suited for dynamic generation of HTML or XML in web applications. It is tightly wedded to the skunk.components library, which permits code to be broken up into separate reusable components that can be invoked with arguments like functions, returning string output or arbitrary Python data, and the skunk.cache library, which permits the return values of such invocations to be cached in a highly flexible and easily controlled manner.
STML tags are delimited by the character sequences <: and :>. Between these delimiters, the first word is the tag name which identifies what tag is being called:
<:halt:>
The tag name may be followed by a number of tag attributes:
<:val expr=`3` fmt="plain":>
In the above, the first attribute has the name expr and the value `3`, and the second the name fmt and the value "plain". The value `3`, written between backticks, is a Python expression, and is equivalent to the Python value 3. Any valid Python expression (without backticks) may be used wherever STML accepts expressions, and are always written between backticks [1]. The value "plain" is a simply a string; it would be equivalent to write `"plain"`, `'plain'`, 'plain', or (since in this case the string between quotes contains no whitespace) simply the bare word plain.
[1] | Python itself uses backticks as a synonym for the repr builtin function, but this syntax is deprecated in any case and is not supported in STML. String literals inside Python expressions currently may not contain backticks either in STML, but this can be circumvented if necessary by used the hex escape '\x60'. |
STML tags usually expect attributes in a default order, and if when using the tag you write the attributes in that order, you can leave out the attribute names. The following is equivalent to the previous example:
<:val `3` plain:>
(Note that this is a bit different than markup languages like HTML and XML, in which attribute order, unlike element order, is not significant.) However, if you specify the attribute names, you can state them in any order:
<:val fmt=plain expr=`3`:>
Some attributes are optional and are defined as having default values; in this example, fmt is optional, and the default value is equivalent to plain, so you could simply write:
<:val `3`:>
Some tags accept arbitrary keyword arguments:
<:component foo.comp x=`y/2.0` d=`range(4)` p="this could go on and on...":>
Again along the lines of SGML-based markup like HTML and XML, STML tags are of two basic syntactical types: block and empty. Block tags open a block which will be parsed in the context of the tag, and must be closed by a matching tag with the same tagName, but preceded by a forward slash. For instance, the template below has one block tag and one empty tag:
<:filter:> This is some text inside a block. <:/filter:> <:break:>
<:set name value:>
The <:set:> tag, assigns a value to a variable:
<:set x "hello":>
This is exactly equivalent to the following Python:
x="hello"
<:default name value:>
The <:default:> tag assigns a value to a variable if the variable is currently undefined, and otherwise does nothing:
<:default x `myObj`:>
This is equivalent to the following Python:
try: x except NameError: x=myObj
STML3's ``<:default:>`` tag assigned a value to a variable either if the variable was undefined, or if it was equal to ``None``. This special treatment of ``None`` has been removed in skunk.stml.
<:del name:>
The <:del:> tag deletes the Python object with the name name. The above precisely equivalent to the Python statement:
del name
<:call expr:>
The <:call:> tag is an escape hatch which enables you to directly execute a Python expression:
<:call `x=4`:>
The Python equivalent of any given <:call:> tag is the expression being called.
<:val:> expr fmt=`None`:>
The <:val:> tag prints the value of the expression passed to the output stream, optionally passing it through a format function:
<:val `myVar` fmt=xml:>
The optional fmt attribute may be a callable that takes one argument and returns a string, or a string that is a key in the global dictionary skunk.stml.ValFormatRegistry. The default formatter is the "plain" formatter (str).
The output stream that <:val:> writes to is available in the STML namespace (actually, the global, not the local namespace that STML code is executed in) under the name OUTPUT.
STML3 does not accept arbitrary callables for ``fmt``, and has slightly different keys in its equivalent ValFormatRegistry. Also, while STML3 also has an output stream called OUTPUT, it is not directly available in the global namespace.
<:import module [names] [as=name]:>
The <:import:> tag is the equivalent of Python's import statement. The following table shows various forms of the tag with the corresponding Python code.
Python STML import M <:import M:> import M as C <:import M as=C:> from M import X <:import M X:> from M import X, Y <:import M "X, Y":> from M import X as C <:import M X as=C:> from M import * <:import M *:> import M1, M2 <:import "M1, M2":> import M1 as C, M2, M3 as C <:import "M1 as C, M2, M3 as C":>
<:filter name=`None` fmt=`None`:><:/filter:> <:spool name fmt=`None`:><:/spool:>
The previous tags are all empty tags; <:filter:> and <:spool:> are block tags.
<:filter:> enables you to filter the output resulting from a block of text, which may include other tags, through an arbitrary filter, and either directly output the filtered result, or save it in a variable. It takes two attributes, name and filter, both of which are optional. If you use neither, <:filter:> does nothing. If you use just filter, <:filter:> is essentially a block version of the <:val:> tag:
<:filter fmt=`str.upper`:> this will get output in upper case. <:/filter:>
What is output here (ignoring whitespace) will be "THIS WILL GET OUTPUT IN UPPER CASE.".
If you use name, with or without a filter, the block inside the tag outputs nothing, but the resulting string is stored in a variable with the name you specified.
<:spool:> is provided for convenience and backwards compatibility with STML3; it is equivalent to <:filter:>, except that the name attribute is required, and hence always assigns a string value to a variable.
In STML3, there is no ``<:filter:>`` tag, and ``<:spool:>`` has no ``filter`` attribute.
<:args [arg1 arg2...] [arg3=val3 arg4=val4 ...]:>
The <:args:> tag is a convenient way of accessing CGI parameters from a web request. It brings the CGI parameters you specify by name into the local namespace, optionally setting default values and converting them.
Each argument of this tag specifies a name and optionally a Python expression value, which may be a default value, a conversion function, or a 2-tuple of the form (default, converterFunc). The name of each argument indicates both the name of the CGI parameter and also of the local variable that will be bound.
If you just specify a parameter name, without a Python expression specifying a default and/or converter function, it will copy the CGI parameter of that name, if it exists, into a local variable of that name; if no such parameter was passed, the corresponding local variable will be initialized to None. If you specify a conversion function, the CGI parameter value will be converted by it prior to being bound in the local namespace; if an exception occurs, it will be silently swallowed and the resulting value will be the default.
<:args bopper nougat parsley=yum servings=`(int,5)`:>
If the above is passed the querystring nougat=pumpkin&servings=20, the following name/value pairs will be added to the template namespace:
bopper : None nougat : 'pumpkin' parsley : 'yum' servings : 20
If the querystring were bopper=frisbee&servings=cankersore, the template values would be:
bopper : 'frisbee' nougat : None parsley : 'yum' servings : 5
This tag only works when there is a webob.Request object in the current namespace under the name REQUEST. You would normally only use this tag in a top-level component into which a request is passed in.
The example above is equivalent to the following Python code:
from skunk.util.argextract import extra_args locals().update(extract_args(REQUEST.params.mixed(), 'bopper', 'nougat', parsley='yum', servings=(int, 5)))
<:if expr:> [<:elif expr:>] ... [<:else:>] <:/if:> <:try:> <:except [exc]:> [<:else:>] <:/try:> <:try:> <:finally:> <:/try:> <:raise [exc]:> <:for expr [name=sequence_item]:> [<:break:>] [<:continue:>] [<:else:>] <:/for:> <:while expr :> [<:break:>] [<:continue:>] [<:else:>] <:/while:>
STML's flow control facilities mirror those of Python itself almost exactly. The following Python:
try: v==0 while not v: for i in p: try: v=i.value except AttributeError: continue else: break else: break if v > m: do_bigger() elif v==m: do_the_same() else: do_smaller() finally: cleanup()
would be translated into STML as:
<:try:> <:set v `0`:> <:while `not v`:> <:for `p` i:> <:try:> <:set v `i.value`:> <:except `AttributeError`:> <:continue:> <:else:> <:break:> <:/try:> <:else:> <:break:> <:/for:> <:/while:> <:if `v>m`:> <:call `do_bigger`:> <:elif `v==m`:> <:call `do_the_same()`:> <:else:> <:call `do_smaller()`:> <:/if:> <:finally:> <:call `cleanup()`:> <:/try:>
The major differences to note are:
- indentation is not significant in STML (although, as in all languages, careful indentation helps legibility).
- <:try:>, <:for:>, and <:while:> are block tags, and must be closed like any other block tag.
- instead of Python's for sequence_item in sequence, STML drops the in, reverses the sequence_item and sequence, and makes sequence_item optional (it defaults to "sequence_item").
<:* ... *:> <:#:><:/#:> <:comment:><:/comment:>
The three comment tags are of two types. <:#:> and <:comment:> are synonymous; they are regular STML block tags that generate no code, but any stml within the tags will be parsed and must be well-formed. The <:* and *:> tags are not, strictly speaking, tag at all, but a special comment syntax supported directly by the STML lexer. Any text between the tags will be ignored. Therefore, <:* is better suited for temporarily commenting out blocks of STML that may be syntactically incorrect.
STML3 also included a ``<:doc:>`` tag for the purpose of adding documentation to templates.
<:component compname [arg1=value arg2=value ...] [__args__= argdict] [cache=no|yes|old|force] :> <:datacomp varname compname [arg1=value arg2=value ...] [__args__= argdict] [cache=no|yes|old|force] :> <:include compname:> <:compargs [arg1 arg2...] [arg3=val3 arg4=val4 ...]:> <:cache until="datespec"| duration="timespec":> <:halt:>
The <:component:> tag calls string components and outputs the results (to the output stream OUTPUT in the current namespace). The <:datacomp:> tag calls data components and places the returned value in varname. <:include:> calls an include component and outputs the results (to OUTPUT).
String and data components accept component arguments, and can be cached; includes cannot. Component arguments can be passed by keyword in the component tag, or as a dictionary with the reserved keyword __args__. The desired cache policy can be specified with the reserved keyword cache; acceptable values are skunk.cache.CachePolicy instances and the strings 'no', 'yes', 'old', 'force'.
To ensure that the correct component arguments have been passed to a component, and to set default values, the <:compargs:> tag may be used. This will compare the component arguments used in a particular call with a signature given in the tag.
The <:cache:> tag sets the expiration for the component, in case it is called with a CachePolicy that honors it. [add documentation of the formats accepted for this -- TO BE DONE.]
STML3 also includes a ``<:return:>`` tag which can be used to return values from data components written in STML. ``skunk.stml`` does not support writing data components in STML, and hence does not implement ``<:return:>``.
STML supports sharing common layouts through layout templates and slots, a very simple system that accomplishes much of what other templating engines achieve through template inheritance.
A layout template is an STML component that renders a complete document, but with empty slots in it that can be filled in various ways.
A slot is a named placeholder in such a template.
At render time, the templating engine will look for a dictionary named SLOTS in the current namespace to populate these slots; if not found, it will create a slot dictionary by looking at the slot configuration (see below).
If no data is found for a particular slot, or if the data is None or an empty string, no output will be produced for that slot. If the data is a string, that string will be output. If it is a callable, it will be called with any arguments you have specified in the template, and the output inserted.
This system is implemented through two tags: <:slot:> and <:calltemplate:>.
<:slot slotname [kwarg=val ...]:>
The slot tag is a placeholder for a slot in the layout template; it should be given a unique slotname. Any additional parameters passed in will be available to any code that runs to render the slot.
<:calltemplate [template=None] [slotmap=SLOTS] [kwarg=val ...]:>
The calltemplate tag is used for STML documents that want to invoke the layout template; in typical use it is called without arguments at the very end of the document.
To configure slots, create in the root directory of your site a file called slotconf.pydcmp, a Python data component that returns a slot dictionary. Every key you wish to be able to override should be present in this dictionary; all keys should be valid Python identifiers and be lowercase. Empty values are perfectly OK. This data component will be called with one parameter, path.
<:debug msg [arg1 args2 ...]:> <:info msg [arg1 args2 ...]:> <:warn msg [arg1 args2 ...]:> <:error msg [arg1 args2 ...]:> <:exception msg [arg1 args2 ...]:>
STML's log tags using Python's logging module to call the standard log methods (debug, info, warn, error, and exception) on the user logger, which by default is named USER (a different name can be assign by setting the configuration variable userLogger).
<:debug "attempting precarious function call":> <:info "received request from IP %s" `ip`:> <:warn "possible breakin attempt: %s %s" `ip` `user`:> <:error "that wasn't supposed to happen":> <:exception "exception handling order:":>
To access the logger programmatically:
from skunk.userlogger import getUserLogger logger=getUserLogger() logger.setLevel(logging.DEBUG) logger.debug("hello from Python")
Or as a shortcut for logging from Python components:
# imports debug, info, warn, error, exception from skunk.userlogger import * debug("I think I'm about to make a boo-boo....") error("Darn, I was right!")
<:use tagdict [prefix=name]:>
STML may be extended by other tag libraries. A tag library is simply a mapping of tag names to tag classes (including just those tags which should appear at the top level and that do not need to be nested in another tag).
The STML parser may be invoked with an extended set of tags that incorporates other libraries, but it may also be desirable to extend the vocabulary of available tags within an STML component; the <:use:> tag accomplishes this. Tags imported by this means are only available within the component, subsequent to the <:use:> call; they are not inherited by any nested components and do not affect the global tag dictionary.
For clarity, or to prevent name conflicts between tag names in different libraries, it is possible to specify a prefix when using a tag library. In use, the namespace will look like an xml namespace prefix:
<:use zapper.stmltags prefix=zapper:> <:zapper:mogrify foo=`1`:>
The <:use:> tag does not generate any code at runtime; the inclusion of the tag library happens at parse time. Therefore, the tagdict argument must be a string, not a Python expression (which only has meaning at runtime). Tag libraries must be defined in importable Python modules, and specified by their __name__ attribute.
STML3 does not have a ``<:use:>`` tag, and does not permit colons in tag names.
This is the application server proper, implemented as a series of WSGI components: an active page implementation for STML, a static file server (with X-Sendfile support), a controller framework, services for authorization and sessions, and bootstrapping code that can launch a server.
By default, skunk.web consists of four nested WSGI applications:
The ContextMiddleware initializes request and response attributes of a thread-local global object, skunk.web.Context. These are webob.Request and webob.Response objects, respectively.
RoutingMiddleware uses Routes to parse the request and populate the wsgi.routing_vars environment variable, as per the x-wsgiorg routing spec. If you don't want to use Routes, you can swap this out with another middleware that performs routing (such as Selector).
ControllerMiddleware is the controller framework per se. It employs the notion of a controller which contains one or more actions, which are servlets that return HTTP responses. For detail, see Controllers below.
Finally, DispatchingFileServer serves STML (and potentially other varieties of) active pages and static files.
A controller is any object, typically a module or class instance, with callable attributes exposed as actions (request handlers).
Since the routing framework may allow clients to choose the name of the action, we follow the example of CherryPy and require that action methods be explicitly marked as exposed, by setting their exposed attribute to a true value:
# the controller framework will execute this, # as it is exposed def an_action(): return webob.Response(body="Hello, World!") an_action.explosed=True # it will not deign to execute this, however def wannabe(): return "Greetings, universe!"
This is normally done by using the @expose() decorator:
from skunk.web import expose @expose() def an_action(): return "how convenient"
@expose() can also be used to set attributes on the response:
@expose(content_type='text/plain') def plain(): return "how dry I am"
The controller framework automatically tries to adapt return values to a WSGI application, most commonly a webob.Response. If a callable is directly returned, no adaptation is necessary; the callable is assumed to be a WSGI application and is invoked as such. Otherwise, the following types are adapted:
Anything else is turned in a string with str().
It is also possible to generate a response by raising a webob.exc.HTTPException. Note that if you do this, or if you explicitly return a WSGI application other than Context.response, the latter object will not be used to serve the request, and any state it may have will be irrelevant.
You can of course invoke a templating system manually from within a controller action, but as this is such a common thing to do, another decorator shortcut is provided:
@expose() @template() def hello_world(): return dict(message="Perhaps this isn't the best time to talk.")
More detail here -- @TBD.
@TBD
Discuss here:
skunk.web is available either under the GPL (v3 or later) or a BSD license. See COPYING and LICENSE in the source distribution for the exact wording of these licenses.