1 """
2 This is a top level package, hosting the entire CSB test framework. It is divided
3 into several major parts:
4
5 - test cases, located under csb.test.cases
6 - test data, in C{/csb/test/data} (not a package)
7 - test console, in C{/csb/test/app.py}
8
9 This module, csb.test, contains all the glue-code functions, classes and
10 decorators you would need in order to write tests for CSB.
11
12 1. Configuration and Tree
13
14 L{Config<csb.test.Config>} is a common config object shared between CSB
15 tests. Each config instance contains properties like:
16
17 - data: the data folder, automatically discovered and loaded in
18 csb.test.Config.DATA at module import time
19 - temp: a default temp folder, which test cases can use
20
21 Each L{Config<csb.test.Config>} provides a convenient way to retrieve
22 files from C{/csb/test/data}. Be sure to check out L{Config.getTestFile}
23 and L{Config.getPickle}. In case you need a temp file, use
24 L{Config.getTempStream} or have a look at L{csb.io.TempFile} and
25 L{csb.io.TempFolder}.
26
27 All test data files should be placed in the C{data} folder. All test
28 modules must be placed in the root package: csb.test.cases. There is
29 a strict naming convention for test modules: the name of a test module
30 should be the same as the name of the CSB API package it tests. For
31 example, if you are writing tests for C{csb/bio/io/__init__.py}, the
32 test module must be C{csb/test/cases/bio/io/__init__.py}. C{csb.test.cases}
33 is the root package of all test modules in CSB.
34
35 2. Writing Tests
36
37 Writing a test is easy. All you need is to import csb.test and then
38 create your own test cases, derived from L{csb.test.Case}:
39
40 >>> import csb.test
41 >>> @csb.test.unit
42 class TestSomeClass(csb.test.Case):
43 def setUp(self):
44 super(TestSomeClass, self).setUp()
45 # do something with self.config here...
46
47 In this way your test case instance is automatically equipped with a
48 reference to the test config, so your test method can be:
49
50 >>> @csb.test.unit
51 class TestSomeClass(csb.test.Case):
52 def testSomeMethod(self):
53 myDataFile = self.config.getTestFile('some.file')
54 self.assert...
55
56 The "unit" decorator marks a test case as a collection of unit tests.
57 All possibilities are: L{csb.test.unit}, L{csb.test.functional}, L{csb.test.custom},
58 and L{csb.test.regression}.
59
60 Writing custom (a.k.a. "data", "slow", "dynamic") tests is a little bit
61 more work. Custom tests must be functions, not classes. Basically a
62 custom test is a function, which builds a unittest.TestSuite instance
63 and then returns it when called without arguments.
64
65 Regression tests are usually created in response to reported bugs. Therefore,
66 the best practice is to mark each test method with its relevant bug ID:
67
68 >>> @csb.test.regression
69 class SomeClassRegressions(csb.test.Case)
70 def testSomeFeature(self)
71 \"""
72 @see: [CSB 000XXXX]
73 \"""
74 # regression test body...
75
76 3. Style Guide:
77
78 - name test case packages as already described
79 - group tests in csb.test.Case-s and name them properly
80 - prefix test methods with "test", like "testParser" - very important
81 - use camelCase for methods and variables. This applies to all the
82 code under csb.test (including test) and does not apply to the rest
83 of the library!
84 - for functional tests it's okay to define just one test method: runTest
85 - for unit tests you should create more specific test names, for example:
86 "testParseFile" - a unit test for some method called "parse_file"
87 - use csb.test decorators to mark tests as unit, functional, regression, etc.
88 - make every test module executable::
89
90 if __name__ == '__main__':
91 csb.test.Console() # Discovers and runs all test cases in the module
92
93 4. Test Execution
94
95 Test discovery is handled by C{test builders} and a test runner
96 C{app}. Test builders are subclasses of L{AbstractTestBuilder}.
97 For every test type (unit, functional, regression, custom) there is a
98 corresponding test builder. L{AnyTestBuilder} is a special builder which
99 scans for unit, regression and functional tests at the same time.
100
101 Test builder classes inherit the following test discovery methods:
102
103 - C{loadTests} - load tests from a test namespace. Wildcard
104 namespaces are handled by C{loadAllTests}
105 - C{loadAllTests} - load tests from the given namespace, and
106 from all sub-packages (recursive)
107 - C{loadFromFile} - load tests from an absolute file name
108 - C{loadMultipleTests} - calls C{loadTests} for a list of
109 namespaces and combines all loaded tests in a single suite
110
111 Each of those return test suite objects, which can be directly executed
112 with python's unittest runner.
113
114 Much simpler way to execute a test suite is to use our test app
115 (C{csb/test/app.py}), which is simply an instance of L{csb.test.Console}::
116
117 $ python csb/test/app.py --help
118
119 The app has two main arguments:
120
121 - test type - tells the app which TestBuilder to use for test dicsovery
122 ("any" triggers L{AnyTestBuilder}, "unit" - L{UnitTestBuilder}, etc.)
123 - test namespaces - a list of "dotted" test modules, for example::
124
125 csb.test.cases.bio.io.* # io and sub-packages
126 csb.test.cases.bio.utils # only utils
127 . # current module
128
129 In addition to running the app from the command line, you can run it
130 also programmatically by instantiating L{csb.test.Console}. You can
131 construct a test console object by passing a list of test namespace(s)
132 and a test builder class to the Console's constructor.
133
134
135 5. Commit Policies
136
137 Follow these guidelines when making changes to the repository:
138
139 - B{no bugs in "trunk"}: after fixing a bug or implementing a new
140 feature, make sure at least the default test set passes by running
141 the test console without any arguments. This is equivalent to:
142 app.py -t any "csb.test.cases.*". (If no test case from this set covers
143 the affected code, create a test case first, as described in the other
144 policies)
145
146 - B{no recurrent issues}: when a bug is found, first write a regression
147 test with a proper "@see: BugID" tag in the docstring. Run the test
148 to make sure it fails. After fixing the bug, run the test again before
149 you commit, as required by the previous policy
150
151 - B{test all new features}: there should be a test case for every new feature
152 we implement. One possible approach is to write a test case first and
153 make sure it fails; when the new feature is ready, run the test again
154 to make sure it passes
155
156 @warning: for compatibility reasons do NOT import and use the unittest module
157 directly. Always import unittest from csb.test, which is guaranteed
158 to be python 2.7+ compatible. The standard unittest under python 2.6
159 is missing some features, that's why csb.test will take care of
160 replacing it with unittest2 instead.
161 """
162 from __future__ import print_function
163
164 import os
165 import sys
166 import imp
167 import types
168 import time
169 import getopt
170 import tempfile
171 import traceback
172
173 import csb.io
174 import csb.core
175
176 try:
177 from unittest import skip, skipIf
178 import unittest
179 except ImportError:
180 import unittest2 as unittest
181
182 from abc import ABCMeta, abstractproperty
191
193 """
194 General CSB Test Config. Config instances contain the following properties:
195
196 - data - path to the CSB Test Data directory. Default is L{Config.DATA}
197 - temp - path to the system's temp directory. Default is L{Config.TEMP}
198 - config - the L{Config} class
199 """
200
201 DATA = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'data')
202 """
203 @cvar: path to the default test data directory: <install dir>/csb/test/data
204 """
205 TEMP = os.path.abspath(tempfile.gettempdir())
206 """
207 @cvar: path to the default system's temp directory
208 """
209
215
216 @staticmethod
218 """
219 Override the default L{Config.DATA} with a new data root directory.
220
221 @param path: full directory path
222 @type path: str
223 """
224 if not os.path.isdir(path):
225 raise IOError('Path not found: {0}'.format(path))
226
227 Config.DATA = os.path.abspath(path)
228
229 @property
231 """
232 Test data directory
233 @rtype: str
234 """
235 return self.__data
236
237 @property
239 """
240 Test temp directory
241 @rtype: str
242 """
243 return self.__temp
244
246 """
247 Search for C{fileName} in the L{Config.DATA} directory.
248
249 @param fileName: the name of a test file to retrieve
250 @type fileName: str
251 @param subDir: scan a sub-directory of L{Config.DATA}
252 @type subDir: str
253
254 @return: full path to C{fileName}
255 @rtype: str
256
257 @raise IOError: if no such file is found
258 """
259 file = os.path.join(self.data, subDir, fileName)
260 if not os.path.isfile(file):
261 raise IOError('Test file not found: {0}'.format(file))
262 return file
263
265 """
266 Same as C{self.getTestFile}, but try to unpickle the data in the file.
267
268 @param fileName: the name of a test file to retrieve
269 @type fileName: str
270 @param subDir: scan a sub-directory of L{Config.DATA}
271 @type subDir: str
272 """
273 file = self.getTestFile(fileName, subDir)
274 return csb.io.Pickle.load(open(file, 'rb'))
275
276 - def getContent(self, fileName, subDir=''):
277 """
278 Same as C{self.getTestFile}, but also read and return the contents of
279 the file.
280
281 @param fileName: the name of a test file to retrieve
282 @type fileName: str
283 @param subDir: scan a sub-directory of L{Config.DATA}
284 @type subDir: str
285 """
286 with open(self.getTestFile(fileName, subDir)) as f:
287 return f.read()
288
290 """
291 Return a temporary file stream::
292
293 with self.getTempStream() as tmp:
294 tmp.write(something)
295 tmp.flush()
296 file_name = tmp.name
297
298 @param mode: file open mode (text, binary), default=t
299 @type mode: str
300 @rtype: file stream
301 """
302 return csb.io.TempFile(mode=mode)
303
305 """
306 Try to deserialize some pickled data files. Call L{Config.updateDataFiles}
307 if the pickles appeared incompatible with the current interpreter.
308 """
309 try:
310 self.getPickle('1nz9.model1.pickle')
311 except:
312 self.updateDataFiles()
313
315 """
316 Refresh the pickled structures in csb/test/data. This might be needed when
317 the internal representation of some classes has changed.
318 """
319 from csb.io import Pickle
320 from csb.bio.io.wwpdb import get
321 from csb.bio.structure import Ensemble, ChemElements
322
323 model1 = get('1nz9', model=1)
324 model2 = get('1nz9', model=2)
325
326 ensemble = Ensemble()
327 ensemble.models.append(model1)
328 ensemble.models.append(model2)
329 Pickle.dump(ensemble, open(os.path.join(self.data, '1nz9.full.pickle'), 'wb'))
330
331 mse = model1.chains['A'].find(164)
332 mse._pdb_name = 'MSE'
333 mse.atoms['SD']._element = ChemElements.Se
334 mse.atoms['SD']._full_name = 'SE '
335 Pickle.dump(model1, open(os.path.join(self.data, '1nz9.model1.pickle'), 'wb'))
336
337 -class Case(unittest.TestCase):
338 """
339 Base class, defining a CSB Test Case. Provides a default implementation
340 of C{unittest.TestCase.setUp} which grabs a reference to a L{Config}.
341 """
342
343 @property
345 """
346 Test config instance
347 @rtype: L{Config}
348 """
349 return self.__config
350
352 """
353 Provide a reference to the CSB Test Config in the C{self.config} property.
354 """
355 self.__config = Config()
356 assert hasattr(self.config, 'data'), 'The CSB Test Config must contain the data directory'
357 assert self.config.data, 'The CSB Test Config must contain the data directory'
358
360 """
361 Re-raise the last exception with its full traceback, but modify the
362 argument list with C{addArgs} and the original stack trace.
363
364 @param addArgs: additional arguments to append to the exception
365 @type addArgs: tuple
366 """
367 klass, ex, _tb = sys.exc_info()
368 ex.args = list(ex.args) + list(addArgs) + [''.join(traceback.format_exc())]
369
370 raise klass(ex.args)
371
373 """
374 Fail if it took more than C{duration} seconds to invoke C{callable}.
375
376 @param duration: maximum amount of seconds allowed
377 @type duration: float
378 """
379
380 start = time.time()
381 callable(*args, **kargs)
382 execution = time.time() - start
383
384 if execution > duration:
385 self.fail('{0}s is slower than {1}s)'.format(execution, duration))
386
388 """
389 Fail if the difference is larger than delta
390
391 @param value: input value
392 @type value: float
393
394 @param expected: expected value
395 @type expected: float
396
397 @param delta: allowed deviation
398 @type delta: float
399 """
400
401 if abs(value - expected) > delta:
402 self.fail('|{0} - {1}| > {2})'.format(value, expected, delta))
403
404 @classmethod
406 """
407 Run this test case.
408 """
409 suite = unittest.TestLoader().loadTestsFromTestCase(cls)
410 runner = unittest.TextTestRunner()
411
412 return runner.run(suite)
413
416
418 """
419 This is a base class, defining a test loader which exposes the C{loadTests}
420 method.
421
422 Subclasses must override the C{labels} abstract property, which controls
423 what kind of test cases are loaded by the test builder.
424 """
425
426 __metaclass__ = ABCMeta
427
428 @abstractproperty
431
433 """
434 Load L{csb.test.Case}s from a module file.
435
436 @param file: test module file name
437 @type file: str
438
439 @return: a C{unittest.TestSuite} ready for the test runner
440 @rtype: C{unittest.TestSuite}
441 """
442 mod = self._loadSource(file)
443 suite = unittest.TestLoader().loadTestsFromModule(mod)
444 return unittest.TestSuite(self._filter(suite))
445
447 """
448 Load L{csb.test.Case}s from the given CSB C{namespace}. If the namespace
449 ends with a wildcard, tests from sub-packages will be loaded as well.
450 If the namespace is '__main__' or '.', tests are loaded from __main__.
451
452 @param namespace: test module namespace, e.g. 'csb.test.cases.bio' will
453 load tests from '/csb/test/cases/bio/__init__.py'
454 @type namespace: str
455
456 @return: a C{unittest.TestSuite} ready for the test runner
457 @rtype: C{unittest.TestSuite}
458 """
459 if namespace.strip() == '.*':
460 namespace = '__main__.*'
461 elif namespace.strip() == '.':
462 namespace = '__main__'
463
464 if namespace.endswith('.*'):
465 return self.loadAllTests(namespace[:-2])
466 else:
467 loader = unittest.TestLoader()
468 tests = loader.loadTestsFromName(namespace)
469 return unittest.TestSuite(self._filter(tests))
470
472 """
473 Load L{csb.test.Case}s from a list of given CSB C{namespaces}.
474
475 @param namespaces: a list of test module namespaces, e.g.
476 ('csb.test.cases.bio', 'csb.test.cases.bio.io') will
477 load tests from '/csb/test/cases/bio.py' and
478 '/csb/test/cases/bio/io.py'
479 @type namespaces: tuple of str
480
481 @return: a C{unittest.TestSuite} ready for the test runner
482 @rtype: C{unittest.TestSuite}
483 """
484 if not csb.core.iterable(namespaces):
485 raise TypeError(namespaces)
486
487 return unittest.TestSuite(self.loadTests(n) for n in namespaces)
488
490 """
491 Load L{csb.test.Case}s recursively from the given CSB C{namespace} and
492 all of its sub-packages. Same as::
493
494 builder.loadTests('namespace.*')
495
496 @param namespace: test module namespace, e.g. 'csb.test.cases.bio' will
497 load tests from /csb/test/cases/bio/*'
498 @type namespace: str
499
500 @return: a C{unittest.TestSuite} ready for the test runner
501 @rtype: C{unittest.TestSuite}
502 """
503 suites = []
504
505 try:
506 base = __import__(namespace, level=0, fromlist=['']).__file__
507 except ImportError:
508 raise InvalidNamespaceError('Namespapce {0} is not importable'.format(namespace))
509
510 if os.path.splitext(os.path.basename(base))[0] != '__init__':
511 suites.append(self.loadTests(namespace))
512
513 else:
514
515 for entry in os.walk(os.path.dirname(base)):
516
517 for item in entry[2]:
518 file = os.path.join(entry[0], item)
519 if extension and item.endswith(extension):
520 suites.append(self.loadFromFile(file))
521
522 return unittest.TestSuite(suites)
523
525 """
526 Import and return the Python module identified by C{path}.
527
528 @note: Module objects behave as singletons. If you import two different
529 modules and give them the same name in imp.load_source(mn), this
530 counts for a redefinition of the module originally named mn, which
531 is basically the same as reload(mn). Therefore, you need to ensure
532 that for every call to imp.load_source(mn, src.py) the mn parameter
533 is a string that uniquely identifies the source file src.py.
534 """
535 name = os.path.splitext(os.path.abspath(path))[0]
536 name = name.replace('.', '-').rstrip('__init__').strip(os.path.sep)
537
538 return imp.load_source(name, path)
539
541 """
542 Extract test cases recursively from a test C{obj} container.
543 """
544 cases = []
545 if isinstance(obj, unittest.TestSuite) or csb.core.iterable(obj):
546 for item in obj:
547 cases.extend(self._recurse(item))
548 else:
549 cases.append(obj)
550 return cases
551
553 """
554 Filter a list of objects using C{self.labels}.
555 """
556 filtered = []
557
558 for test in self._recurse(tests):
559 for label in self.labels:
560 if hasattr(test, label) and getattr(test, label) is True:
561 filtered.append(test)
562
563 return filtered
564
566 """
567 Build a test suite of cases, marked as either unit, functional or regression
568 tests. For detailed documentation see L{AbstractTestBuilder}.
569 """
570 @property
573
575 """
576 Build a test suite of cases, marked as unit tests.
577 For detailed documentation see L{AbstractTestBuilder}.
578 """
579 @property
582
584 """
585 Build a test suite of cases, marked as functional tests.
586 For detailed documentation see L{AbstractTestBuilder}.
587 """
588 @property
591
593 """
594 Build a test suite of cases, marked as regression tests.
595 For detailed documentation see L{AbstractTestBuilder}.
596 """
597 @property
600
602 """
603 Build a test suite of cases, marked as custom tests. CustomTestBuilder will
604 search for functions, marked with the 'custom' test decorator, which return
605 a dynamically built C{unittest.TestSuite} object when called without
606 parameters. This is convenient when doing data-related tests, e.g.
607 instantiating a single type of a test case many times iteratively, for
608 each entry in a database.
609
610 For detailed documentation see L{AbstractTestBuilder}.
611 """
612 @property
615
617
618 mod = self._loadSource(file)
619 suites = self._inspect(mod)
620
621 return unittest.TestSuite(suites)
622
639
641
642 objects = map(lambda n: getattr(module, n), dir(module))
643 return self._filter(objects)
644
646 """
647 Filter a list of objects using C{self.labels}.
648 """
649 filtered = []
650
651 for obj in factories:
652 for label in self.labels:
653 if hasattr(obj, label) and getattr(obj, label) is True:
654 suite = obj()
655 if not isinstance(suite, unittest.TestSuite):
656 raise ValueError('Custom test function {0} must return a '
657 'unittest.TestSuite, not {1}'.format(obj.__name__, type(suite)))
658 filtered.append(suite)
659
660 return filtered
661
663 """
664 A class decorator, used to label unit test cases.
665
666 @param klass: a C{unittest.TestCase} class type
667 @type klass: type
668 """
669 if not isinstance(klass, type):
670 raise TypeError("Can't apply class decorator on {0}".format(type(klass)))
671
672 setattr(klass, Attributes.UNIT, True)
673 return klass
674
676 """
677 A class decorator, used to label functional test cases.
678
679 @param klass: a C{unittest.TestCase} class type
680 @type klass: type
681 """
682 if not isinstance(klass, type):
683 raise TypeError("Can't apply class decorator on {0}".format(type(klass)))
684
685 setattr(klass, Attributes.FUNCTIONAL, True)
686 return klass
687
689 """
690 A class decorator, used to label regression test cases.
691
692 @param klass: a C{unittest.TestCase} class type
693 @type klass: type
694 """
695 if not isinstance(klass, type):
696 raise TypeError("Can't apply class decorator on {0}".format(type(klass)))
697
698 setattr(klass, Attributes.REGRESSION, True)
699 return klass
700
702 """
703 A function decorator, used to mark functions which build custom (dynamic)
704 test suites when called.
705
706 @param function: a callable object, which returns a dynamically compiled
707 C{unittest.TestSuite}
708 @type function: callable
709 """
710 if isinstance(function, type):
711 raise TypeError("Can't apply function decorator on a class")
712 elif not hasattr(function, '__call__'):
713 raise TypeError("Can't apply function decorator on non-callable {0}".format(type(function)))
714
715 setattr(function, Attributes.CUSTOM, True)
716 return function
717
718 -def skip(reason, condition=None):
719 """
720 Mark a test case or method for skipping.
721
722 @param reason: message
723 @type reason: str
724 @param condition: skip only if the specified condition is True
725 @type condition: bool/expression
726 """
727 if isinstance(reason, types.FunctionType):
728 raise TypeError('skip: no reason specified')
729
730 if condition is None:
731 return unittest.skip(reason)
732 else:
733 return unittest.skipIf(condition, reason)
734
736 """
737 Build and run all tests of the specified namespace and kind.
738
739 @param namespace: a dotted name, which specifies the test module
740 (see L{csb.test.AbstractTestBuilder.loadTests})
741 @type namespace: str
742 @param builder: test builder to use
743 @type builder: any L{csb.test.AbstractTestBuilder} subclass
744 @param verbosity: verbosity level for C{unittest.TestRunner}
745 @type verbosity: int
746 @param update: if True, refresh all pickles in csb/test/data
747 @type update: bool
748 """
749
750 BUILDERS = {'unit': UnitTestBuilder, 'functional': FunctionalTestBuilder,
751 'custom': CustomTestBuilder, 'any': AnyTestBuilder,
752 'regression': RegressionTestBuilder}
753
754 USAGE = r"""
755 CSB Test Runner Console. Usage:
756
757 python {0.program} [-u] [-t type] [-v verbosity] namespace(s)
758
759 Options:
760 namespace(s) A list of CSB test dotted namespaces, from which to
761 load tests. '__main__' and '.' are interpreted as the
762 current module. If a namespace ends with an asterisk
763 '.*', all sub-packages will be scanned as well.
764
765 Examples:
766 "csb.test.cases.bio.*"
767 "csb.test.cases.bio.io" "csb.test.cases.bio.utils"
768 "."
769
770 -t type Type of tests to load from each namespace. Possible
771 values are:
772 {0.builders}
773
774 -v verbosity Verbosity level passed to unittest.TextTestRunner.
775
776 -u update-files Force update of the test pickles in csb/test/data.
777 """
778
779 - def __init__(self, namespace=('__main__',), builder=AnyTestBuilder, verbosity=1,
780 update=False, argv=None):
798
799 @property
801 return self._namespace
802 @namespace.setter
804 if csb.core.iterable(value):
805 self._namespace = list(value)
806 else:
807 self._namespace = [value]
808
809 @property
812 @builder.setter
814 self._builder = value
815
816 @property
818 return self._verbosity
819 @verbosity.setter
821 self._verbosity = value
822
823 @property
826
827 @property
830
831 @property
834 @update.setter
836 self._update = bool(value)
837
850
851 - def exit(self, message=None, code=0, usage=True):
859
861
862 try:
863
864 options, args = getopt.getopt(argv, 'hut:v:', ['help', 'update-files', 'type=', 'verbosity='])
865
866 for option, value in options:
867 if option in('-h', '--help'):
868 self.exit(message=None, code=0)
869 if option in('-t', '--type'):
870 try:
871 self.builder = Console.BUILDERS[value]
872 except KeyError:
873 self.exit(message='E: Invalid test type "{0}".'.format(value), code=2)
874 if option in('-v', '--verbosity'):
875 try:
876 self.verbosity = int(value)
877 except ValueError:
878 self.exit(message='E: Verbosity must be an integer.', code=3)
879 if option in('-u', '--update-files'):
880 self.update = True
881
882 if len(args) > 0:
883 self.namespace = list(args)
884
885 except getopt.GetoptError as oe:
886 self.exit(message='E: ' + str(oe), code=1)
887
888
889 if __name__ == '__main__':
890
891 Console()
892