Package cssutils :: Module tokenize2 :: Class Tokenizer
[hide private]
[frames] | no frames]

Class Tokenizer

source code

object --+
         |
        Tokenizer

generates a list of Token tuples:
(Tokenname, value, startline, startcolumn)
Instance Methods [hide private]
 
__init__(self, macros=None, productions=None)
inits tokenizer with given macros and productions which default to cssutils own macros and productions
source code
 
_expand_macros(self, macros, productions)
returns macro expanded productions, order of productions is kept
source code
 
_compile_productions(self, expanded_productions)
compile productions into callable match objects, order is kept
source code
 
tokenize(self, text, fullsheet=False)
Generator: Tokenize text and yield tokens, each token is a tuple of:
source code

Inherited from object: __delattr__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __str__

Class Variables [hide private]
  _atkeywords = {u'@font-face': 'FONT_FACE_SYM', u'@import': 'IM...
  _linesep = ...
Properties [hide private]

Inherited from object: __class__

Method Details [hide private]

__init__(self, macros=None, productions=None)
(Constructor)

source code 
inits tokenizer with given macros and productions which default to cssutils own macros and productions
Overrides: object.__init__

tokenize(self, text, fullsheet=False)

source code 

Generator: Tokenize text and yield tokens, each token is a tuple of:

(nname, value, line, col)

The token value will contain a normal string, meaning CSS unicode escapes have been resolved to normal characters. The serializer escapes needed characters back to unicode escapes depending on the stylesheet target encoding.

text
to be tokenized
fullsheet
if True appends EOF token as last one and completes incomplete COMMENT tokens

Class Variable Details [hide private]

_atkeywords

Value:
{u'@font-face': 'FONT_FACE_SYM',
 u'@import': 'IMPORT_SYM',
 u'@media': 'MEDIA_SYM',
 u'@namespace': 'NAMESPACE_SYM',
 u'@page': 'PAGE_SYM'}

_linesep

Value:
u'''
'''