This document describes main classes used for supervised learning of if-then rules. First two classes, RuleLearner and RuleBeamFinder , implement learning of rules by following separate-and-conquer strategy. This strategy consists of several replaceable functions, which are, in our classes, presented as components. As usual for all Orange classes, these components can be easily rewritten (in Python) and can replace the default ones. As there are many components here and to make the presentation of components a bit easier, the description of classes will start with the pseudo code of implemented algorithm to show where functions (components) do actually take place. After the algorithm, the actual description of individual components is given. The third class, class Rule, is a class used to represent a single rule. Examples at the end are used to illustrate how these three classes can be used to implement different ways of learning rules.
RuleLearner class
RuleLearner is a class for inductive learning rules from data. The algorithm follows separate-and-conquer strategy, which has its origins in the AQ family of algorithms (Fuernkranz J.; Separate-and-Conquer Rule Learning, Artificial Intelligence Review 13, 3-54, 1999). Basically, such algorithms search for the "best" possible rule in learning examples, remove covered data from learning examples (separate) and repeat the process (conquers) on the remaining examples. Follows the algorithm of separate-and-conquer rule learning:
RuleLearner's call function:
def __call__(self,examples,weightID=0):
ruleList = orange.RuleList()
allExamples = orange.ExampleTable(examples)
while not self.dataStopping(examples,weightID,self.targetClass):
newRule = self.ruleFinder(examples,weightID,self.targetClass,self.baseRules)
if self.ruleStopping(ruleList,newRule,examples,weightID):
break
examples,weightID = self.coverAndRemove(newRule,examples,weightID,self.targetClass)
ruleList.append(newRule)
return orange.RuleClassifier_firstRule(rules=ruleList,examples=allExamples)
Main functions of this algorithm are dataStopping, ruleFinder, coverAndRemove, and ruleStopping. Each of those functions corresponds to an callable attribute in the class and a component (callable class or function) needs to be set in order that method can work. By default, components that simulate CN2 algorithm will be used, but user can change it to any arbitrary function (component) that accepts and returns appropriate parameters. The instructions and requirements for writting such components is given at the description of attributes.
Methods
As ruleLearner is derived from orange Learner class, it has the classical call method that accepts examples and, if available, id number of weight attribute.
Attributes
- dataStopping
- This callable attribute accepts a component that checks from the examples whether there will be benefit from further learning, so basically checks if there is enough data to continue learning. The default component returns true if the set of examples is empty or, if targetClass is given, returns true if number of instances with given class is zero.
Default component: |
orange.RuleDataStoppingCriteria_NoPositives |
Derived from: |
orange.RuleDataStoppingCriteria |
Format: |
condition = dataStopping(examples,weightID,targetclass) |
- ruleStopping
- ruleStopping is a component that decides from the last rule learned, if it is worthwhile to use this rule and learn more rules. By default, this attribute is set to None - meaning that all rules are accepted.
Default component: |
None |
Derived from: |
orange.RuleStoppingCriteria |
Format: |
condition = ruleStopping(ruleList,rule,examples,weight) |
- coverAndRemove
- This component removes examples covered by the rule and returns remaining examples. If the targetClass is not given (targetClass = -1), default component does exactly this, and, if target class is given, removes only covered examples that are in the target class.
Default component: |
orange.RuleCovererAndRemover_Default |
Derived from: |
orange.RuleCovererAndRemover |
Format: |
(newExamples,newWeightID) = coverAndRemove(rule,examples,weightID,targetClass) |
- ruleFinder
- RuleFinder learns a single rule from examples. By default, RuleBeamFinder class is used, which is explained later on.
Default component: |
orange.RuleBeamFinder |
Derived from: |
orange.RuleFinder |
Format: |
rule = ruleFinder(examples,weightID,targetClass,baseRules) |
- storeExamples
- Set this to 1 if you want the rules to have stored examples.
- targetClass
- Learn rules with a specific target class
- baseRules
- Base rules are rules that we would like to use in ruleFinder to constrain the learning space. You can see it in the ruleFinder format specification that it takes also a set of base rules as a parameter. If attribute baseRules is not set, it will be set to a set containing only empty rule.
RuleBeamFinder class
RuleBeamFinder class performs beam search for the best rule. This is the default class used in RuleLearner to find the best rule. Pseudo code of the algorithm is shown here:
RuleBeamFinder's call function:
def __call__(self,examples,weightID,targetClass,baseRules):
prior = orange.Distribution(examples.domain.classVar,examples,weightID)
rulesStar,bestRule = self.initializer(examples,weightID,targetClass,baseRules,self.evaluator,prior)
# compute quality of rules in RuleStar and bestRule
...
while len(rulesStar)>0:
candidates, rulesStar = self.candidateSelector(rulesStar,examples,weightID)
for cand in candidates:
newRules = self.refiner(cand,examples,weightID,targetClass)
for newRule in newRules:
if self.ruleStoppingValidator(newRule,examples,weightID,targetClass,cand.classDistribution) newRule.quality = self.evaluator(newRule,examples,weightID,targetClass,prior)
rulesStar.append(newRule)
if self.validator(newRule,examples,weightID,targetClass,prior) and newRule.quality>bestRule.quality:
bestRule = newRule
rulesStar = self.ruleFilter(rulesStar,examples,weightID)
return bestRule
As you can see, there are several exchangeable components. Each of those is explained in the attributes section.
Attributes
- initializer
- This component is used to initialize rulesStar and for selecting the initial best rule. By default, baseRules are returned as starting rulesSet and the best from baseRules is set as bestRule.
If baseRules are not set, this class will return rulesStar with rule that covers all examples (has no selectors) and this rule will be also used as bestRule.
Default class: |
orange.RuleBeamInitializer_Default |
Derived from: |
orange.RuleBeamInitializer |
Format: |
(ruleList,bestRule) = initializer(examples,weigthID,targetClass,baseRules,evaluator,prior) |
- candidateSelector
- Separates a subset from the current rulesStar and returns it. These rules will be used in the next specification step. Default component takes all rules in rulesStar.
Default class: |
orange.RuleBeamCandidateSelector_TakeAll |
Derived from: |
orange.RuleBeamCandidateSelector |
Format: |
(candidates,ruleList) = candidateSelector(ruleList,examples,weigthID) |
- refiner
- Refines given rule. New rule should cover a strict subset of examples covered by given rule. Default component adds a a conjunctive selector (attribute = att_value) to selectors present in the rule.
Default class: |
orange.RuleBeamRefiner_Selector |
Derived from: |
orange.RuleBeamRefiner |
Format: |
newRule = candidateSelector(rule,examples,weigthID,targetClass) |
- ruleFilter
- Filters rules to keep beam relatively small to constrain search complexity. By default, it takes five best rules.
Default class: |
orange.RuleBeamFilter_Width(m=5) |
Derived from: |
orange.RuleBeamFilter |
Format: |
rules = ruleFilter(rules,examples,weigthID) |
- evaluator
- Evaluates rule from covered examples. By default, entropy is used as a measure.
Default class: |
orange.RuleEvaluator_Entropy |
Derived from: |
orange.RuleEvaluator |
Format: |
quality = evaluator(rule,examples,weigthID,targetClass,prior) |
- ruleStoppingValidator
- Validates whether the rule is specialized enough comparing it to its parent. The parent of a rule is a rule from which this rule was specialized.
Default class: |
None |
Derived from: |
orange.RuleValidator |
Format: |
ruleStoppingValidator(rule,examples,weigthID,targetClass,priorOfParentRule) |
- validator
- Validates, whether the rule is good enough to be considered as the best rule, i.e. the rule is good enough to be returned by rulefinder. By default, likelihood ratio statistics is used that gives an estimate if rule is statistically better than the default rule.
Default class: |
orange.RuleValidator_LRS(alpha=1.0) |
Derived from: |
orange.RuleValidator |
Format: |
ruleGoodEnough = validator(rule,examples,weigthID,targetClass,prior) |
Methods
- call operator(examples, weight, targetClass, baseRules);
- Learns a rule from examples
Rule class
Rule class is the base class for presentation of a single rule.
Attributes
- filter
- This attribute specifies the contents of the rule; it is the basis of the Rule class. Class Filter_values is set as a default, which can, together with default methods, be replaced with any class inherited from Filter.
- classifier
- Each rule can be used as a classical Orange like classificator. By default, Default classifier is used.
- learner
- Learner to be used for making a classifier. By default, MajorityLearner is used.
- classDistribution
- Distribution of class in examples covered by this rule.
- examples
- Covered examples by this rule.
- weightID
- Weight for the stored examples.
- quality
- Quality of the rule. Rules with higher quality are better.
- complexity
- Complexity of the rule. Complexity is used for selecting between rules with equal quality, where rules with lower complexity are preferred. Currently, complexity corresponds to the number of selectors in rule (actually to number of conditions in filter), but, obviously, any other measure can be applied.
Methods
- __init__(self)
- Creates and empty rule.
- operator (example)
- Returns True if Example if covered by rule and 0 otherwise.
- operator (examples, ref=True, negate=False)
- Filters examples using filter object. Returns examples that are covered by the rule.
- filterAndStore(examples, weightID=0, targetClass=-1)
- This method filters passed examples and stores them in the attribute examples. It also computes classDistribution, sets weight of stored examples and creates a new classifier using learner attribute.
Examples
This last section is used to give some examples how we can replace the default components. All examples are from testRuleLearner.py (uses titanic.tab).
learner = orange.RuleLearner()
cl = learner(data)
# print rules
for r in cl.rules:
print orngCN2.ruleToString(r)
In the first row we initialize the learner, in the second run learner on data that returns a rule classifier, and in the last two lines, the induced rules are printed on screen (uses orngCN2 module). This learner uses only default components, but we wish to change that now. The following two lines, for instance,
learner.ruleFinder = orange.RuleBeamFinder()
learner.ruleFinder.evaluator = orngCN2.mEstimate(m=50)
# print rules
cl = learner(data)
for r in cl.rules:
print orngCN2.ruleToString(r)
would change the evaluation function of learner to m-estimate of probability with m set to 50. This evaluation function is implemented in orngCN2 module (orngCN2.py). Notice that we first need to set the ruleFinder component, because the default components are not constructed when the learner is constructed, but only when we run it on data. At that time, the algorithm checks which components are necessary and sets defaults. Similarly, when the learner finishes, it destructs all default components.
Continuing with our example, assume that we wish to set a different validation function and a different bean width. This is simply written as:
learner.ruleFinder.ruleStoppingValidator = orange.RuleValidator_LRS(alpha=0.01,min_coverage=10,max_rule_complexity = 2)
learner.ruleFinder.ruleFilter = orange.RuleBeamFilter_Width(width = 50)
# print rules
cl = learner(data)
for r in cl.rules:
print orngCN2.ruleToString(r)