2.1   Download

You can find the source distribution here:

• Python Package Index -- http://pypi.python.org/pypi/generateDS/
• Source Forge -- http://sourceforge.net/projects/generateds/

• My Web site -- http://www.rexx.com/~dkuhlman/generateDS-2.0b.tar.gz

It is also available at:

• SourceForge: http://sourceforge.net/projects/generateds/
• The Python Package Index: http://pypi.python.org/pypi/generateDS/

2.2   Support and more information

There is a mailing list at SourceForge: generateds-users -- https://lists.sourceforge.net/lists/listinfo/generateds-users.

There is a tutorial in the distribution: tutorial/tutorial.html and at generateDS -- Introduction and Tutorial -- http://www.rexx.com/~dkuhlman/generateds_tutorial.html.

4.1   Running generateDS.py

Run generateDS.py with a single argument, the XML Schema file that defines the data structures. For example, the following will generate Python source code for data structures described in people.xsd and will write it to the file people.py. In addition, it will write subclass stubs to the file peoplesubs.py:

python generateDS.py -o people.py -s peoplesubs.py people.xsd

Here is the usage message displayed by generateDS.py:

Synopsis:
    Generate Python classes from XML Schema definition.
    Input is read from in_xsd_file or, if "-" (dash) arg, from stdin.
    Output is written to files named in "-o" and "-s" options.
Usage:
    python generateDS.py [ options ] <xsd_file>
    python generateDS.py [ options ] -
Options:
    -h, --help               Display this help information.
    -o <outfilename>         Output file name for data representation classes
    -s <subclassfilename>    Output file name for subclasses
    -p <prefix>              Prefix string to be pre-pended to the class names
    -f                       Force creation of output files.  Do not ask.
    -a <namespaceabbrev>     Namespace abbreviation, e.g. "xsd:".
                             Default = 'xs:'.
    -b <behaviorfilename>    Input file name for behaviors added to subclasses
    -m                       Generate properties for member variables
    --search-path="a:b:c:d"  Search these directories for additional
                             schema files.
    --subclass-suffix="XXX"  Append XXX to the generated subclass names.
                             Default="Sub".
    --root-element="XXX"     Assume XXX is root element of instance docs.
                             Default is first element defined in schema.
    --super="XXX"            Super module name in subclass module. Default="???"
    --validator-bodies=path  Path to a directory containing files that provide
                             bodies (implementations) of validator methods.
    --use-old-getter-setter  Name getters and setters getVar() and setVar(),
                             instead of get_var() and set_var().
    --user-methods= <module>,
    -u <module>              Optional module containing user methods.  See
                             section "User Methods" in the documentation.
    --no-dates               Do not include the current date in the generated
                             files. This is useful if you want to minimize
                             the amount of (no-operation) changes to the
                             generated python code.
    --no-versions            Do not include the current version in the generated
                             files. This is useful if you want to minimize
                             the amount of (no-operation) changes to the
                             generated python code.
    --no-process-includes    Do not process included XML Schema files.  By
                             default, generateDS.py will insert content
                             from files referenced by <include ... />
                             elements into the XML Schema to be processed.
    --silence                Normally, the code generated with generateDS
                             echoes the information being parsed. To prevent
                             the echo from occurring, use the --silence switch.
    --namespacedef='xmlns:abc="http://www.abc.com"'
                             Namespace definition to be passed in as the
                             value for the namespacedef_ parameter of
                             the export() method by the generated
                             parse() and parseString() functions.
                             Default=''.
    --external-encoding=<encoding>
                             Encode output written by the generated export
                             methods using this encoding.  Default, if omitted,
                             is the value returned by sys.getdefaultencoding().
                             Example: --external-encoding='utf-8'.
    --member-specs=list|dict
                             Generate member (type) specifications in each
                             class: a dictionary of instances of class
                             MemberSpec_ containing member name, type,
                             and array or not.  Allowed values are
                             "list" or "dict".  Default: None.
    --session=mysession.session
                             Load and use options from session file. You can
                             create session file in generateds_gui.py.
    --version                Print version and exit.

The following command line flags are recognized by generateDS.py:

o <filename>

Write the data representation classes to file filename.

s <filename>

Write the subclass stubs to file filename.

p <prefix>

Prepend prefix to the name of each generated data structure (class).

f

Force generation of output files even if they already exist. Do not ask before over-writing existing files.

a <namespaceabbrev>

Namespace abbreviation, for example "xsd:". The default is 'xs:'. If the <schema> element in your XML Schema, specifies something other than "xmlns:xs=", then you need to use this option. So, suppose you have the following at the beginning of your XSchema file:

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">

Then you can the following command line option:

-a "xsd:"

But, note that generateDS.py also tries to pick-up the namespace prefix used in the XMLSchema file automatically. If the <schema> element has an attribute "xmlns:xxx" whose value is "http://www.w3.org/2001/XMLSchema", then generateDS.py will use "xxx:" as the alias for the XMLSchema namespace in the XMLSchema document.

b <behaviorfilename>

Input file name for behaviors to be added to subclasses. Specifies is the name of an XML document containing descriptions of methods to be added to subclasses generated with the -s flag. The -b flag requires the -s flag. See the section on XMLBehaviors below.

m

Generate property members and new style classes. Causes generated classes to inherit from class object. Generates a call to the built-in property function for each pair of getters and setters. This is experimental.

subclass-suffix=<suffix>

Append suffix to the name of classes generated in the subclass file. The default, if omitted, is "Sub". For example, the following will append "_Action" to each generated subclass name:

generateDS.py --subclass-suffix="_Action" -s actions.py mydef.xsd

And the following will append nothing, making the superclass and subclass names the same:

generateDS.py --subclass-suffix="" -s actions.py mydef.xsd

root-element=<element_name>

Make element_name the assumed root of instance documents. The default is the name of the element whose definition is first in the XML Schema document. This flag effects the parsing functions (for example, parse(), parseString()).

super=<module_name>

Make module_name the name of the superclass module imported by the subclass module. If this flag is omitted, the following is generated near the top of the subclass file:

import ??? as supermod

and you will need to hand edit this so the correct superclass module is imported.

validator-bodies=<path>

Obtain the bodies (implementations) for validator methods for members defined as simpleType from files in directory specified by <path>. The name of the file in that directory should be the same as the simpleType name with an optional ".py" extension. If a file is not provided for a given type, an empty body (pass) is generated. In these files, lines with "##" in the first two columns are ignored and are not inserted.

use-old-getter-setter

generateDS.py now generates getter and setter methods (for variable "abc", for example) with the names get_abc() and set_abc(), which I believe is a more Pythonic style, instead of getAbc() and setAbc(), which was the old behavior. Use this flag to generate getters and setters in the old style (getAbc() and setAbc()).

u, user-methods=<module>

If specified, generateDS.py will add methods to generated classes as specified in the indicated module. For more information, see section User Methods.

no-dates

Do not include the current date in the generated files. This is useful if you want to minimize the amount of (no-operation) changes to the generated python code.

no-versions

Do not include the current version in the generated files. This is useful if you want to minimize the amount of (no-operation) changes to the generated python code.

no-process-includes

Do not process included XML Schema files. By default, generateDS.py will insert content from files referenced by <include ... /> elements into the XML Schema to be processed. See section Include file processing.

silence

Normally, the code generated with generateDS echoes the information being parsed. To prevent the echo from occurring, use the --silence switch. This switch causes generateDS.py, when it generates boiler-plate parsing functions, (parse(), parseString(), parseLiteral()), to generate code that does not print out output (export output to stdout).

namespacedef="<http://...>"

Namespace definition to be passed in as the value for the namespacedef_ parameter of the export() method by the generated parse() and parseString() functions. If this parameter is specified, then the export function will insert a namespace prefix definition attribute in the top-most (outer-most) element. (Actually, you can insert any attribute.) The default is an empty string.

external-encoding=<encoding>

If an XML instance document contains character data or attribute values that are not in the ASCII character set, then that data will not be written out correctly or will throw an exception. This flag enables the user to specify a character encoding into which character data will be encoded before it is written out by the export functions. The generated export methods encode data using this encoding. The default value, if this flag is omitted, is the value returned by sys.getdefaultencoding(). You can find a list of standard encodings here: http://docs.python.org/library/codecs.html#id3. Example use: --external-encoding='utf-8'.

member-specs

Generate member (type) specifications in each class: a dictionary of instances of class MemberSpec_ containing member name, type, and array or not. See User Methods section for more information about MemberSpec_. Allowed values are "list" or "dict". Default: do not generate member specifications (unless --user-methods specified).

session=mysession.session

Load and use options from session file. You can create a session file in generateds_gui.py, the graphical front-end for generateDS.py. Additional options on the command line can be used to override options in the session file. A session file is an XML document, so you can modify it with a text editor.

version

Print out the current version of generateDS.py and immediately exit.

4.2   Name conflicts

6.1   Namespace prefix mis-match

generateDS.py is not very intelligent about detecting what prefix is used in the schema file for the XML Schema namespace. When this problem occurs, you may see the following when running generateDS.py:

AttributeError: 'NoneType' object has no attribute 'annotate'

generateDS.py assumes that the XML Schema namespace prefix in your schema is "xs:".

So, if the XML Schema namespace prefix in your schema is not "xs:", you will need to use the "-a" command line option when you run generateDS.py. Here is an example:

generateDS.py -a "xsd:" --super=mylib -o mylib.py -s myapp.py someschema.xsd

7.1   Attributes + no nested children

Element definitions that contain attributes but no nested child elements provide access to their data content through getter and setter methods getValueOf_ and setValueOf_ and member variable valueOf_.

7.2   Mixed content

Elements that are defined to contain both text and nested child elements have "mixed content". generateDS.py provides access to mixed content, but the generated data structures (classes) are fundamentally different from that generated for other elements. See section Mixed content for more details.

Note that elements defined with attributes but with no nested sub-elements do not need to be declared as "mixed". For these elements, character data is captured in a member variable valueOf_, and can be accessed with member methods getValueOf_ and setValueOf_.

7.3   anyAttribute

generateDS.py supports anyAttribute. For example, if an element is defined as follows:

<xs:element name="Tool">
   <xs:complexType>
      <xs:attribute name="PartNumber" type="xs:string" />
      <xs:anyAttribute processContents="skip" />
   </xs:complexType>
</xs:element>

Then generateDS.py will generate a class with a member variable anyAttributes_ containing a dictionary. Any attributes found in the instance XML document that are not explicitly defined for this element will be stored in this dictionary. generateDS.py also generates getters and setters as well as code for parsing and export. generateDS.py ignores processContents. See section anyAttribute for more details.

7.4   Element extensions

generateDS.py now generates subclasses for extensions, that is when an element definition contains something like this:

<xs:extension base="sometag">

Limitation -- There is an important limitation, however: member names duplicated (overridden ?) in an extension generate erroneous code. Sigh. I guess I needed something more to do.

Several of the generated methods have been refactored so that subclasses can reuse the code in their superclasses. Take a look at the generated code to learn how to use it.

The Python compiler/interpreter requires that it has seen a superclass before it sees the subclass that uses it. Because of this, generateDS.py delays generating a subclass until after its superclass has been generated. Therefore, the order in which classes are generated may be different from what you expect.

7.5   Attribute groups

generateDS.py now handles definition and use of attribute groups. For example: the use of something like the following:

<xs:attributeGroup name="favorites">
    <xs:attribute name="fruit" />
    <xs:attribute name="vegetable" />
</xs:attributeGroup>

And, a reference or use like the following:

<xs:element name="person">
    <xs:complexType mixed="0">
        <xs:attributeGroup ref="favorites" />
        o
        o
        o

Results in generation of class person that contains members fruit and vegetable.

7.6   Substitution groups

generateDS.py now handles a limited range of substitution groups, but, there is an important limitation, in particular generateDS.py handles substitution groups that involve complex types, but does not handle those that involve (substitute for) simple types (for example, xs:string, xs:integer, etc). This is because the code generated for members defined as simple types does not provide the needed information to handle substitution groups.

7.7   Primitive types

generateDS.py supports some, but not all, simple types defined in "XML Schema Part 0: Primer Second Edition" ( http://www.w3.org/TR/xmlschema-0/. See section "Simple Types" and appendix B). Validation is performed for some simple types. When performed, validation is done while the XML document is being read and instances are created.

Here is a list of supported simple types:

• xs:string -- No validation.
• xs:token -- No validation. White space between tokens is coerced to a single blank between tokens.
• xs:integer, xs:short, xs:long. xs:int -- All treated the same. Checked for valid integer.
• xs:float, xs:double, xs:decimal -- All treated the same. Checked for valid float.
• xs:positiveInteger -- Checked for valid range (> 0).
• xs:nonPositiveInteger -- Checked for valid range (<= 0).
• xs:negativeInteger -- Checked for valid range (< 0).
• xs:nonNegativeInteger -- Checked for valid range (>= 0).
• xs:date, xs:dateTime -- All treated the same. No validation.
• xs:boolean -- Checked for one of 0, false, 1, true.

7.8   simpleType

generateDS.py generates minimal support for members defined as simpleType. However, the code generated by generateDS.py does not enforce restrictions. For notes on how to enforce restrictions, see section simpleType and validators.

A simpleType can be a restriction on a primitive type or on a defined element type. So, for example, the following will generate valid code:

<xs:element name="percent">
    <xs:simpleType>
        <xs:restriction base="xs:integer">
            <xs:minInclusive value="1"/>
            <xs:maxInclusive value="100"/>
        </xs:restriction>
    </xs:simpleType>
</xs:element>

And, the following will also generate valid code:

<xs:simpleType name="emptyString">
    <xs:restriction base="xs:string">
        <xs:whiteSpace value="collapse"/>
    </xs:restriction>
</xs:simpleType>

<xs:element name="merge">
    <xs:complexType>
        <xs:simpleContent>
            <xs:extension base="emptyString">
                <xs:attribute name="fromTag" type="xs:string"/>
                <xs:attribute name="toTag" type="xs:string"/>
            </xs:extension>
        </xs:simpleContent>
    </xs:complexType>
</xs:element>

7.9   List values, optional values, maxOccurs, etc.

For elements defined with maxOccurs="unbounded", generateDS.py generates code that processes a list of elements.

For elements defined with minOccurs="0" and maxOccurs="1", generateDS.py generates code that exports an element only if that element has a (non-None) value.

7.10   simpleType and validators

Here are a few notes that should help you use validator methods to enforce restrictions.

• Default behavior -- The generated code, by default, treats the value of a member whose type is a simpleType as if it were declared as type xs:string.

• Validator method stubs -- For a member variable name declared as a simpleType named X, a validator method validate_X is generated. Example -- from:

  <xs:simpleType name="tAnyName">
      <xs:restriction base="xs:string"/>
  </xs:simpleType>
  

  The class generated by generateDS.py will contain the following method definition:

  def validate_tAnyName(self, value):
      # Validate type tAnyName, a restriction on xs:string.
      pass
  

• Calls to validator methods -- For a member variable declared as a simpleType X, a call to validate X is added to the build method. Example -- from:

  <xs:element name="person">
      <xs:complexType mixed="0">
          <xs:sequence>
              <xs:element name="test2" type="tAnyName"/>
          </xs:sequence>
      </xs:complexType>
  </xs:element>
  

  generateDS.py produces the following call:

  self.validate_tAnyName(self.test2)    # validate type tAnyName
  

• Code bodies for validator methods can be added either (1) manually or (2) automatically from an external source. See command line option "--validator-bodies" and see below.

You can add code to the validator method stub to enforce the restriction for the base type and further restrictions imposed on that base type. This can be done in the following ways:

1. Add code manually after generation. I recommend that you use the "-s" command line option and override the validator method in the resulting subclass file.
2. Or, supply code bodies (implementations) in an external source and ask generateDS.py to insert those code bodies into generated validator methods. Here are notes on how to do this:
   • Use the "--validator-bodies=path" command line option to specify a directory.
   • In that directory, provide one file for each simpleType. The name of the file should be the same as the name of the simpleType with an optional extension ".py". generateDS.py looks for a file named type_name.py, first, and if not found, looks for a file named type_name.
   • If the "--validator-bodies" is not on the command line or neither type_name.py nor type_name is found, an empty body (a pass statement) is generated.
   • Lines from the file are inserted as is, except that lines containing "##" in the first two columns are omitted. Note that you will need to provide the correct indentation for a method in a class, specifically 8 spaces.

The support for simpleType in generateDS.py has the following limitations (among others, I'm sure):

• It only works for simpleType defined with and referenced through a name. It does not work for "in-line" definitions. So, for example, the following works:

  <xs:element name="person">
      <xs:complexType>
          <xs:sequence>
              <xs:element name="test3" type="tAnyName"/>
          </xs:sequence>
      </xs:complexType>
  </xs:element>
  
  <xs:simpleType name="tAnyName">
      <xs:restriction base="xs:string"/>
  </xs:simpleType>
  

  But, the following does not work:

  <xs:element name="person">
      <xs:complexType>
          <xs:sequence>
              <xs:element name="test3">
                  <xs:simpleType name="tAnyName">
                      <xs:restriction base="xs:string"/>
                  </xs:simpleType>
              </xs:element>
          </xs:sequence>
      </xs:complexType>
  </xs:element>
  

• Attributes defined as a simple type are not supported.

7.11   Include file processing

By default, generateDS.py will insert content from files referenced by include elements into the XML Schema to be processed. This behavior can be turned off by using the "--no-process-includes" command line option.

include elements are processed and the referenced content is inserted in the XML Schema by importing and using process_includes.py, which is included in the generateDS.py distribution.

process_includes.py will use either lxml or ElementTree, but its preference is lxml because lxml attempts to preserve namespace prefixes. So if your XML Schemas have <include ... /> elements in them, you might want to consider installing lxml, even though ElementTree is in the Python standard library for Python versions >= 2.5.

The include file processing is capable of retrieve included files via FTP and HTTP internet protocols as well as from the local file system.

Also see command line option "--search-path" (see Running generateDS.py), which can be used to specify a colon separated list of directories where include processing will look for included schemas.

7.12   Abstract types

generateDS.py has support for abstract types. For more on this, see: XML Schema Part 0: Primer Second Edition: Abstract Elements and Types -- http://www.w3.org/TR/xmlschema-0/#abstract.

8.1   Additional constructions

Here are a few additional constructions that generateDS.py understands.

<xs:element name="server-type">
    <xs:complexType>
        <xs:sequence>
            <xs:element name="server-name" type="xs:string"/>
            <xs:element name="server-description" type="xs:string"/>
        </xs:sequence>
    </xs:complexType>
</xs:element>

<xs:complexType name="server-type">
    <xs:sequence>
        <xs:element name="server-name" type="xs:string"/>
        <xs:element name="server-description" type="xs:string"/>
    </xs:sequence>
</xs:complexType>

<xs:element name="server-info">
    <xs:complexType>
        <xs:sequence>
            <xs:element name="server-comment" type="xs:string"/>
            <xs:element ref="server-type" />
        </xs:sequence>
    </xs:complexType>
</xs:element>
   in place of this:
<xs:element name="server-info">
    <xs:complexType>
        <xs:sequence>
            <xs:element name="server-comment" type="xs:string"/>
            <xs:element name="server-type" type="server-type"/>
        </xs:sequence>
    </xs:complexType>
</xs:element>

<xs:complexType name="BType">
    <xs:complexContent>
        <xs:extension base="AType">
            <xs:sequence>
                o
                o
                o

class BType(AType):
    o
    o
    o

9.1   The XMLBehaviors input file

This section describes the XMLBehavior XML document that is used as input to generateDS.py. The XMLBehavior XML document is an XML instance document (given as an argument to the "-b" command line flag) that describes behaviors (methods) to be added to class definitions in the subclass file (generated with the "-s" command line flag).

See file xmlbehavior_po.xml in the Demos/Xmlbehavior directory in the distribution for an example that you can use as a model.

The elements in the XMLBehavior document type are the following:

• <xb:xml-behavior> -- The base element in the document.
  • <xb:base-impl-url> -- The root (left-most portion) of URL containing implementation bodies. Implementation URLs are appended to this base URL.
  • <xb:behaviors> -- A list of behaviors.
    • <xb:behavior> -- Describes a single XMLBehavior.
      • <xb:class> -- The name of the class to which this behavior is to be added.
      • <xb:name> -- The name of the behavior/method. Must conform to Python name syntax.
      • <xb:args> -- A list of arguments to the behavior/method.
        • <xb:arg> -- A positional argument to the method.
          • <xb:name> -- The name of the argument.
          • <xb:data-type> -- The data-type of the argument.
      • <xb:return-type> -- The data-type of the value returned by the behavior/method.
      • <xb:impl-url> -- The URL of the implementation body. This value will be concatenated to the right-hand side of the base-impl-url.
      • <xb:ancillaries> -- A list of ancillary behaviors/methods. Each ancillary has a role, which defines how it is to be used.
        • <xb:ancillary> -- A specification of an ancillary behavior/method.
          • <xb:name> -- The name of the behavior/method. Must conform to Python name syntax.
          • <xb:role> -- The method's role. The following values are supported:
            • "DBC-precondition" -- A Design By Contract-style pre-condition check. This method will be called before the core behavior/method itself.
            • "DBC-postcondition" -- A Design By Contract-style post-condition check. This method will be called after the core behavior/method itself.
          • <xb:args> -- A list of arguments to the ancillary behavior/method. The element has the same content as the <xb:args> element for the core behavior/method.
          • <xb:return-type> -- The data-type of the value returned by the behavior/method.
          • <xb:impl-url> -- The URL of the implementation body. This value will be concatenated to the right-hand side of the base-impl-url.

9.2   Implementing other sources for implementation bodies

generateDS.py contains a function get_impl_body() that implements the ability to retrieve implementation bodies. The current implementation retrieves implementation bodies from an Internet Web URL. Other sources for implementation bodies can be implemented by modifying get_impl_body().

As an example, the version that follows first tries to retrieve an implementation body from a Web address and, if that fails, attempts to obtain the implementation body from a file in the local file system using the <xb:base-impl-url> as a path to a directory containing files, each of which contains one implementation body and <xb:impl-url> as the file name. This implementation of get_impl_body was provided by Colin Dembovsky of Systemsfusion Inc. Thanks, Colin. (I've included it in the generateDS.py script, but commented out, for those who want to use and possibly extend it.):

def get_impl_body(classBehavior, baseImplUrl, implUrl):
    impl = '        pass\n'
    if implUrl:
        trylocal = 0
        if baseImplUrl:
            implUrl = '%s%s' % (baseImplUrl, implUrl)
        try:
            implFile = urllib2.urlopen(implUrl)
            impl = implFile.read()
            implFile.close()
        except:
            trylocal = 1
        if trylocal:
            try:
                implFile = file(implUrl)
                impl = implFile.read()
                implFile.close()
            except:
                print '*** Implementation at %s not found.' % implUrl
    return impl

10.1   xsd:list element support

xsd:list elements can be used with a child xsd:simpleType which confuses the XschemaHandler stack unrolling. xsd:list element support should allow the following XML Schema definition to be supported in generateDS.py:

<xsd:attribute name="Foo">
    <xsd:simpleType>
        <xsd:list>
            <xsd:simpleType>
                ...
            </xsd:simpleType>
        </xsd:list>
    </xsd:simpleType>
</xsd:attribute>

10.2   xsd:enumeration support

The enumerated values for the parent element are resolved and made available through the instance attribute values.

10.3   xsd:union support

In order to properly resolve and query types which are unions in an XML Schema, an element's membership in an xsd:union is available through the instance attribute unionOf.

10.4   Extended xsd:choice support

When a parent xsd:choice is exists, an element's "maxOccurs" and "minOccurs" values can be inherited from the xsd:choice rather than the element itself. xsd:choice elements have been added to the child element via the choice instance attribute and are now used in the "maxOccurs" and "minOccurs" attribute resolution. This should allow the following XML Schema definition to be supported in generateDS.py:

<xsd:element name="Foo">
    <xsd:complexType>
        <xsd:choice maxOccurs="unbounded">
            <xsd:element ref="Bar"/>
            <xsd:element ref="Baz"/>
        </xsd:choice>
    </xsd:complexType>
</xsd:element>

10.5   minOccurs attribute support

Some applications require the availability of the "minOccurs" attribute in addition to the previous minimal support of "optionality". This is available through the getMinOccurs method (which follows the style of the existing API).

10.6   More thorough content type and base type resolution

The previous content type and base type resolution is insufficient for some needs. Basically it was unable to handle more complex and shared element and simpleType definitions. This support has been extended to more correctly resolve the base type and properly indicate the content type of the element. This should provide the ability to handle more complex XML Schema definitions in generateDS.py. Documentation on the algorithm for how this is achieved is available as comments in the source code of generateDS.py -- see comments in method resolve_type in class XschemaElement.

10.7   Making top level simpleTypes available from XschemaHandler

Some developers working to extend the analysis and code generation in generateDS.py may be helped by additional information collected during the parsing of the XML Schema file.

Some applications need all the top level simpleTypes to be available for further queries after the SAX parser has completed its work and after all types have been resolved. These types are available as an instance attribute topLevelSimpleTypes inside XschemaHandler.

10.8   Namespaces -- inserting namespace definition in exported documents

In some cases, the document produced by a call to an export method will contain elements that have namespace prefixes. For example, the following snippet contains namespace prefix "abc":

<abc:people >
    <abc:person>
    o
    o
    o
    </abc:person>
</abc:people>

A way is needed to insert a namespace prefix definition into the generated document. Here is how generateDS.py fills that need.

Each generated export method takes an optional argument namespacedef_. If provided, the value of that parameter is inserted in the exported element. So, for example, the following call:

people.export(sys.stdout, 0,
    namespacedef_='xmlns:abc="http://www.abc.com/namespace"')

might produce:

<abc:people xmlns:abc="http://www.abc.com/namespace">
    <abc:person>
    o
    o
    o
    </abc:person>
</abc:people>

If this is an issue for you, then you may also want to consider using the "--namespacedef" command line option when you run generateDS.py. The value of this option will be passed in to the export function in the generated parse functions. So, for example, running generateDS.py as follows:

generateDS.py --namespacedef='xmlns:abc="http://www.abc.com/namespace.xsd"'
    -o mylib.py -s myapp.py myschema.xsd

will generate parse methods that automatically add the namespacedef_ argument to the call to export.

11.1   The parsing functions

The simplest use is to call one of the parsing functions in the generated source file. You may be able to use one of these functions without change, or can modify one to fit your needs. generateDS.py generates the following parsing functions:

• parse -- Parse an XML document from a file.
• parseString -- Parse an XML document from a string.

These parsing functions are generated in both the superclass and the subclass files. Note the call to the export method. You may need to comment out or un-comment this call to export according to your needs.

For example, if the generated source is in people.py, then, from the command line, run something like the following:

python people.py people.xml

Or, from within other Python code, use something like the following:

import people
rootObject = people.parse('people.xml')

11.2   Recognizing the top level element

It might be that the generated module, when parsing an XML instance document, does not, by default, recognize the top level (root) element in an instance document. This might happen because generateDS.py does not detect the correct top level element from the XML schema or because you need to use the generated module to parse instance documents that have different top level elements. If this is the case, you might pick and use one of the following strategies:

1. In your schema, move the definition of the element type that defines the top level element in your instance documents to the top of the schema. By default, generateDS.py uses the first definition in the schema as the when constructing the generated parse function.

2. Use the "--root-element" command line option to specify top level element. But, be aware that this only works if the tag name and type name of the top level element are the same.

3. Modify the parse function in your generated module, replacing the class whose factory is called and the tag name passed in to the export method. For example, change:

   def parse(inFileName):
       doc = minidom.parse(inFileName)
       rootNode = doc.documentElement
       rootObj = type1.factory()
       rootObj.build(rootNode)
       # Enable Python to collect the space used by the DOM.
       doc = None
       sys.stdout.write('<?xml version="1.0" ?>\n')
       rootObj.export(sys.stdout, 0, name_="type1",
           namespacedef_='')
       return rootObj
   

   to:

   def parse(inFileName):
       doc = minidom.parse(inFileName)
       rootNode = doc.documentElement
       rootObj = type2.factory()
       rootObj.build(rootNode)
       # Enable Python to collect the space used by the DOM.
       doc = None
       sys.stdout.write('<?xml version="1.0" ?>\n')
       rootObj.export(sys.stdout, 0, name_="type2",
           namespacedef_='')
       return rootObj
   

   Notice that we've changed the two occurrences of "type1" to "type2".

4. Using the generated parse function as a model, create a separate module that imports your generated module. In the parse function in your module, make a change similar to that suggested above. And, of course, add any additional code needed by your application.

5. Write a separate module containing your own parse function that inspects the top level element of an input XML instance document and automatically determines which generated class should be used to parse it. Here is an example:

   #!/usr/bin/env python
   
   import sys
   from optparse import OptionParser
   from xml.dom import minidom
   import mygeneratedmodule as gendsmod
   
   def get_root_tag(node):
       tag = node.tagName
       tags = tag.split(':')
       if len(tags) > 1:
           tag = tags[-1]
       rootClass = None
       if hasattr(gendsmod, tag):
           rootClass = getattr(gendsmod, tag)
       return tag, rootClass
   
   def parse(inFilename, options):
       doc = minidom.parse(inFilename)
       rootNode = doc.documentElement
       rootTag, rootClass = get_root_tag(rootNode)
       rootObj = rootClass.factory()
       rootObj.build(rootNode)
       # Enable Python to collect the space used by the DOM.
       doc = None
       sys.stdout.write('<?xml version="1.0" ?>\n')
       rootObj.export(sys.stdout, 0, name_=rootTag,
           namespacedef_='')
       doc = None
       return rootObj
   
   USAGE_TEXT = """
       python %prog [options] <somefile.xml>"""
   
   def usage(parser):
       parser.print_help()
       sys.exit(1)
   
   def main():
       parser = OptionParser(USAGE_TEXT)
       (options, args) = parser.parse_args()
       if len(args) == 1:
           infilename = args[0]
           parse(infilename, options)
       else:
           usage(parser)
   
   if __name__ == "__main__":
       main()
   

   Notice the call to get_root_tag, which attempts to recognize the top level tag in the input XML document so that the parse function can parse and export it.

11.3   The export methods

The generated classes contain methods export and exportLiteral which can be called to export classes to several text formats, in particular to an XML instance document and a Python module containing Python literals. See the generated parse functions for examples showing how to call the export methods.

11.4   Building instances

If you have an instance of a minidom node that represents an element in an XML document, you can also use the 'build' member function to populate an instance of the corresponding class. Here is an example:

from xml.dom import minidom
from xml.dom import Node

doc = minidom.parse(inFileName)
rootNode = doc.childNodes[0]
people = []
for child in rootNode.childNodes:
    if child.nodeType == Node.ELEMENT_NODE and child.nodeName == 'person':
        obj = person()
        obj.build(child)
        people.append(obj)

11.5   Using the subclass module

If you choose to use the generated subclass module, and I encourage you to do so, you may need to edit and modify that file. Here are some of the things that you must do (look for "???"):

• Edit the import statement at the top of the file. It should import the generated superclass file. Note that you can also use the "--super" command line option to insert this automatically.
• Edit the USAGE_TEXT string so that it gives a help message appropriate for your use.
• Edit the main function toward the bottom of the file. It should call a method, that you have possibly added, to the root subclass.

You can also (and most likely will want to) add methods to the generated classes. See the section How to Modify the Generated Code for more on this.

The classes generated from each element definition provide getter and setter methods to access its attributes and child elements.

Elements that are referenced but not defined (i.e. that are simple, for example strings, integers, floats, and booleans) are accessed through getter and setter methods in the class in which they are referenced.

11.6   Elements with attributes but no nested children

Element definitions that contain attributes but no nested child elements provide access to their data content through getter and setter methods getValueOf_ and setValueOf_ and member variable valueOf_.

11.7   Mixed content

The goal of generateDS.py is to support data structures represented in XML as opposed to text mark-up. However, it does provides some support for mixed content. But, for mixed content, the data structures and code generated by generateDS.py are fundamentally different from those for elements that do not contain mixed content.

There are limitations, of course. A known limitation is related to extension elements. Specifically, if an element contains mixed content, and this element extends a base class, then the base class and any classes it extends must be defined to contain mixed content. This is due to the fact that generateDS.py generates a data structure (class) for elements containing mixed content that is fundamentally different from that generated for other elements.

Here is an example of mixed content:

<note>This is a <bold>nice</bold> comment.</note>

When an element is defined with something like the following:

<xs:complexType mixed="true">
    <xs:sequence>
        o
        o
        o

then, instead of generating a class whose named members refer to nested elements, a class containing a list of instances of class MixedContainer is generated. In order to process the content of a mixed content element, the code you write will need to walk this list of instances of MixedContainer and check the type of each item in that list. Basically, the structure becomes more DOM-like in the sense that it has a list of children, rather than named fields.

Instances of MixedContainer have the following methods:

• getCategory -- Returns one of the following, depending on the content:
  • CategoryText -- Text content.
  • CategorySimple -- Simple elements, that is, elements defined as xs:string, xs:integer, etc. For these, the member variable content_type, accessible through method getContenttype will contain one of TypeString, TypeInteger, TypeFloat, TypeDecimal, TypeDouble, or TypeBoolean.
  • CategoryComplex -- Complex elements represented by a generated class. For these, the member variable name, accessible through method getName will return the element/tag name and the member variable value, accessible through method getValue will return the instance.
• getContenttype -- Returns one of TypeString, TypeInteger, TypeFloat, TypeDecimal, TypeDouble, or TypeBoolean. Valid only when category is CategorySimple.
• getName -- For CategoryComplex, returns the name of the element.
• getValue -- Returns the value of this chunk of content. Its type depends on the value returned by getCategory and getContenttype.

Note that elements defined with attributes but with no nested sub-elements do not need to be declared as "mixed". For these elements, character data is captured in a member variable valueOf_, and can be accessed with member methods getValueOf_ and setValueOf_.

11.8   anyAttribute

For elements that specify anyAttributes, generateDS.py produces a class containing the following:

• A member variable anyAttributes_ containing a Python dictionary. After parsing an XML instance document, this dictionary will contain name-value pairs for any attributes in the instance document not explicitly defined for that element.
• The following getters and setters: getAnyAttributes_ and setAnyAttributes_.
• Code to export the attribute names and values stored in the dictionary.
• Code to parse attributes in addition to those explicitly defined for the element and store them in the dictionary.

Note: Attributes that are explicitly defined for an element are not stored in the dictionary anyAttributes_.

generateDS.py ignores the processContents attribute on the anyAttribute element in the XML Schema

11.9   User Methods

generateDS.py provides a mechanism that enables you to attach user defined methods to specific generated classes. In order to do so, create a Python module containing specifications of those methods and indicate that module on the command line with the "--user-methods" option. Example:

python generateDS.py -f --super=people_sup -o people_sup.py -s people_sub.py --user-methods=gends_user_methods people.xsd

The module named with this flag must be located where generateDS.py can import it. You might need to add the directory containing your user methods module to the PYTHONPATH environment variable.

The module specified with the "--user-methods" flag should define a variable METHOD_SPECS which contains a list of instances of a class that implements methods match_name and get_interpolated_source.

See file gends_user_methods.py for an example of this specification file and the definition of class MethodSpec. Read the comments in that file for more guidance.

The member_data_items_ class variable -- User methods, especially those attached to more than one class, are likely to need a list of the members in the current class. Each generated class has a class variable containing a list of specifications of the members in the class. Each item in this list is an instance of class MemberSpec_, which is defined near the top of your generated (super-class) file. Use the following to access the information in each member specification:

• m.get_name() -- Returns the name of the member variable (a string).

• m.get_data_type() -- Returns the data type of the member variable (a string). If the data type is a list, returns the terminal type, which is that last string in the list. (Also see get_data_type_chain().)

• m.get_data_type_chain() -- Returns the data type of the member variable (a string or list). When the data type is a simpleType that has another simpleType as it's base or is a complexType that extends a simpleType, then the data type is a list of strings, for example:

  ['RelationType', 'xs:string']
  

  The last string in the list is the terminal type, usually a built-in simple type. Note that m.get_data_type() returns the terminal (last) type.

• m.get_container() -- (an integer) Indicates whether the member variable is a single item or a list/container (i.e. generated from maxOccurs > 0): 0 indicates a single item; 1 indicates a list.

There are a number of things of interest in this sample file (gends_user_methods.py):

• Although, the MethodSpec class must be included in your user methods specification module, you can modify this class. For example, for special situations, it might be useful to modify either of the methods MethodSpec.match_name or MethodSpec.get_interpolated_source. These methods are called by generateDS.py. See comments on the definitions of these methods in gends_user_methods.py.

• A method set_up is attached to the root class. (This user method specification module is intended to be used with people.xsd/people.xml in the Demos/People directory.) It performs initialization, before the walk method is called. In particular, set_up initializes a counter and imports the types module (which saves us from having to modify the generated code).

• The walk_and_update and walk_and_show methods provide an example showing how to walk the entire document object tree.

• The method walk_and_update uses the member_data_items_ class variable to obtain a list of members of the class. It's a list of instances of class MemberSpec_, which support the m.get_name(), m.get_data_type()``, and m.get_container() methods described above.

• In method walk_and_show, note the use of getattr to retrieve the value of a member variable and the use of setattr to set the value of a member variable.

• The expression "%(class_name)s" is used to insert the class name into the generated source code.

• Notice how the types module is used to determine whether a member variable contains a simple type or an instance of a class. Example:

  obj1 = getattr(self, member[0])
  if type(obj1) == types.InstanceType:
      ...
  

• In string formatting operations, you will need to use double percent signs in order to "pass through" a single percent sign, for example:

  print '%%d. class: %(class_name)s  depth: %%d' %% (counter, depth, )
  

  where the single percent signs are interpolated ("%(class_name)s" is replace by the class name), and double percent signs are replace by single percent signs ("%%d" becomes "%d").

Suggestion -- How to begin:

1. Make a copy of gends_user_methods.py.
2. Modify the method specifications in that file. Replace the source code and the class_name pattern in each specification.
3. Run generateDS.py with the "--user-methods" (or "-u") flag.
4. Inspect the user methods in the generated classes.
5. Test your generated code.
6. Repeat as necessary.

11.10   Overridable methods

generateDS.py generates calls to several methods that each have a default implementation in a superclass. The default superclass with default implementations is included in the generated code. The user can replace this default superclass by implementing a module named generatedssuper.py containing a class named GeneratedsSuper.

What to look for in the generated code:

• In the generated superclass file (generated with command line option "-o"), look for the import of module generatedssuper.py and the definition of the (default) class GeneratedsSuper.
• Also look for calls to methods format_integer(), format_float(), format_double(), etc.

Currently the following methods are implemented:

class GeneratedsSuper:
    def format_string(self, input_data, input_name=''):
        return input_data
    def format_integer(self, input_data, input_name=''):
        return '%d' % input_data
    def format_float(self, input_data, input_name=''):
        return '%f' % input_data
    def format_double(self, input_data, input_name=''):
        return '%e' % input_data
    def format_boolean(self, input_data, input_name=''):
        return '%s' % input_data

Caution: Overriding any of the format_xxxx() methods enables you to export invalid XML. So, use at your own risk, test before using, etc.

How to modify the behavior of the default methods:

• Implement methods that override the default methods.
• Look at the definition of the default methods in class GeneratedsSuper in order to learn the signature of the methods in that class.
• Look at the definition of the default methods to determine what they do and what type of value they return, then do something similar in your overriding method.

Where to put (implement) methods that override the default methods -- You can place the implementations of methods that override the default methods in the following places:

• In a class named GeneratedsSuper in a separate module named generatedssuper. Since this class would replace the default implementations, you should provide implementations of all the default methods listed above in that class. The distribution contains a generatedssuper.py which you can modify for your specific needs.

• In individual generated (super) classes (the ones generated with the "-o" command line option) using the User Methods feature.

• In individual classes in a subclass module generated with the "-s" command line option.

  If you want to use the same method in more than one generated subclass, then you might consider putting that method in a "mix-in" class and inherited that method in the generated subclass. With this approach, you must put the mix-in class containing your methods before the regular superclass, so that Python will find your custom methods before the default ones. That is, you must use:

  class clientSub(MySpecialMethods, supermod.client):
  

  not:

  class clientSub(supermod.client, MySpecialMethods):
  

If you choose to implement module generatedssuper, here are a few suggestions:

• Implement a module generatedssuper.py containing definition of a class GeneratedsSuper. Note that a default version of this module is included in the distribution.

• Put this module in a location where it can be imported when your generated code is run.

• An easy way to begin is to copy the default definition of the class GeneratedsSuper from a module generated with the "-o" command line option into a module named generatedssuper.py. Then modify your (copied) implementation. Or, use the default module included in the distribution.

• To implement a method that does a task specific to particular class or a particular member of a class, do something like the following:

  def format_string(self, input_data, input_name=''):
      if self.__class__.__name__ == 'person':
          return '[[%s]]' % input_data
      else:
          return input_data
  

  or:

  def format_string(self, input_data, input_name=''):
      if self.__class__.__name__ == 'booster' and input_name == 'lastname':
          return '[[%s]]' % input_data
      else:
          return input_data
  

  Alternatively, to attach a method to a specific class, use the User Methods or a generated subclass module (command line option "-s"), as described above.

12.1   Adding features to class definitions

You can add new member definitions to a generated class. Look at the 'export' and 'exportLiteral' member functions for examples of how to access member variables and how to walk nested sub-elements.

Here are interesting places to look in each class definition:

• The 'export' and 'exportLiteral' methods -- These methods walk the object tree. You can consider copying and renaming them to produce other tree walking methods.
• The 'build' method -- These methods extract information from the minidom node. You can inspect the 'build' methods to learn how to extract information for other purposes.

And, if you need methods that are common to and shared by several of the generated subclasses, you can put them in a new class and add that class to the superclass list for each of your subclasses.

Although you can add your own methods to the generated superclasses, I'm recommeding that you add methods to the generated subclasses in the subclass module generated with the "-s" command line option, and then edit the subclass module in order to build your application. Why?

• The superclasses are cluttered with other code. Using the subclass file enables you to keep your application code separate.
• By putting your application code in the subclass file, you will be able to reuse the superclass file. You can generate multiple subclass files from the same XML Schema definition file. Each of these subclass files can import the same superclass file.

Here are some alternatives to using the subclass file:

• Add more than one method to each generated (super-)class. Each method implements a separate task or "application". If the number of tasks grows, this will create maintenance difficulties, however.
• Re-generate multiple (super-)class files. Add methods to the classes in these separate files to implement different tasks. This of course will not work well if you have had to modify the parser, for example, since generating the file.

14.1   Capturing xs:date elements as dates

The following extension employs a user method (see User Methods) in order to capture elements defined as xs:date as date objects.

Thanks to Lars Ericson for this code and explanation.

By default, generateDS.py treats elements declared as type xs:date as though they are strings.

To get xs:dates stored as dates, in your local copy, add the following user method (User Methods), a slight modification of the sample (in gends_user_methods.py):

method1 = MethodSpec(name='walk_and_update',
    source='''\
    def walk_and_update(self):
        members = %(class_name)s.member_data_items_
        for member in members:
            obj1 = getattr(self, member.get_name())
            if member.get_data_type() == 'xs:date':
                newvalue = date_calcs.date_from_string(obj1)
                setattr(self, member.get_name(), newvalue)
            elif member.get_container():
                for child in obj1:
                    if type(child) == types.InstanceType:
                        child.walk_and_update()
            else:
                obj1 = getattr(self, member.get_name())
                if type(obj1) == types.InstanceType:
                    obj1.walk_and_update()
''',
    class_names=r'^.*$',
    )

Then, define date_calcs.py as:

#!/usr/bin/env python
# -*- mode: pymode; coding: latin1; -*-

import datetime

# 2007-09-01

# test="2007-09-01"
# print test
# print date_from_string(test)

def date_from_string(str):
    year = int(str[:4])
    month = int(str[5:7])
    day = int(str[8:10])
    dt = datetime.date(year, month, day)
    return dt

And, add a "str" here in generateDS.py:

def quote_xml(inStr):
    s1 = str(inStr)
    s1 = s1.replace('&', '&')
    s1 = s1.replace('<', '&lt;')
    s1 = s1.replace('"', '&quot;')
    return s1

Also, add these imports to TEMPLATE_HEADER in generateDS.py:

import date_calcs
import types

15.1   XML Schema limitations

There are things in Xschema that are not supported. You will have to use a restricted sub-set of Xschema to define your data structures. See above for supported features. See people.xsd and people.xml for examples.

And, then, try it on your XML Schema, and let me know about what does not work.

15.2   Large documents

Warning -- This section describes an optional generated SAX parser which, I believe, is currently broken for all but the simplest schemas. Generation of a SAX parser has not been updated for the latest changes to generateDS.py. In particular, when names of elements are reused (in different parent elements), the SAX parser becomes confused. Until I've been able to figure out how to fix this, you are advised not to use the SAX parser.

generateDS.py generates two kinds of parsers: one kind is based on SAX and the other is build on minidom. See the generated functions saxParse, parse(), and parseString(). Using the SAX parser instead of the minidom parser should reduce memory requirements for large documents, since the minidom parser, but not the SAX parser, constructs a DOM tree for the entire document in memory.

However, both styles of parsers construct instances of the data structures generated by generateDS.py. This means that, even when the SAX parser is used, generateDS.py may not be well-suited for applications that read large XML documents, although what "large" means depends on your hardware. Notice that the minidom parsing functions (parse() and parseString()) over-write the variable doc so as to enable Python to reclaim the space occupied by the DOM tree, which may help alleviate the memory problem to some extent when the minidom parser is used.