autowisp.diagnostics.expressions module

Class Inheritance Diagram

Named expressions over per-image diagnostics, evaluated in dependency order.

This is the whole of what an expression means: which names it references, what order a library of them has to be evaluated in, and what is wrong with one. It deliberately knows about no database of any kind. The library arrives as a {name: expression} dictionary and the data as a {diagnostic_name: array} dictionary, so the same rules apply whether the caller is the browser interface or the pipeline, both of which read the project’s library through autowisp.diagnostics.expression_library.

An exclusion rule is one more expression, taking at most one channel slot and one photometry slot, and evaluated alongside the library under rule_quantity.

Passing the values in rather than fetching them is what keeps this module free of a database and cheap to test exhaustively; it is not a facility for supplying diagnostics by hand. Building those arrays – NaN-padded onto one canonical image list so that index i is the same image in every one of them – belongs to the layer above.

class autowisp.diagnostics.expressions.QuantityLookUp(name, shared, *, definition=None, computed=None, takes_photometries=False)

Bases: object

One name an expression may read, resolved when it is asked for.

Everything a quantity needs is held here: a diagnostic owns the values fetched for it, an expression owns its text and the instantiations it has computed. Asking the top-level lookup for its channels is therefore the whole of evaluation – there is no separate pass, and no state threaded through one.

A diagnostic is the degenerate case rather than a second kind of thing: every instantiation of it is known before evaluation starts, so it arrives with its cache full and its text never consulted.

Built through library() rather than one at a time, because the lookups of one evaluation share a binding stack and an evaluator, and both are this class’s business rather than its caller’s.

__getitem__(slots)

Resolve name[slots] as the body being evaluated means it.

Python passes 1 for x[1] and (1, 2) for x[1,2], so the shapes take care of themselves. The binding read is the top of the stack, which is always the body asking: a nested evaluation finishes in here before the outer body’s next operand is touched. A quoted channel, x['G0'], is not looked up in it.

A quantity taking photometries is not resolved yet: its channels are bound, and the _ChannelBound returned resolves the second subscript, which such a quantity always has.

__init__(name, shared, *, definition=None, computed=None, takes_photometries=False)

Parameters:

• name (str) – What the expressions call it, used for error messages only – nothing resolves by it.

• shared (tuple) – The binding stack and evaluator this evaluation’s lookups share, from library().

• definition (tuple) – The expression text, and its channel and its photometry parameters as a pair, or None for a diagnostic, which has none of them because it is never evaluated.

• computed (dict) – (channels, photometries) -> array known in advance: the whole of a diagnostic, and empty for an expression.

• takes_photometries (bool) – Whether a diagnostic is read with a second subscript, binding a photometry. An expression takes photometries if it has photometry parameters.

at(channels, photometries)

Return this quantity with its parameters bound as given.

The cache is what makes an instantiation wanted twice – by two references, or by two different expressions – computed once. For a diagnostic it is also the whole of the answer, so a miss means the values were never fetched rather than that something needs computing, and is reported rather than repaired: it means get_needed_values() and whatever fetched disagree.

Nothing here guards against a reference cycle: order_expressions() refuses one statically, on names, and that is exact rather than conservative, since substitution permutes a finite parameter set and introduces no new symbols.

Parameters:

• channels (tuple) – One channel per channel parameter, in order.

• photometries (tuple) – One photometry id per photometry parameter, in order.

Returns:

The values, over the canonical image list.

Return type:

numpy.ndarray

Raises:

PipelineError – If a diagnostic was not fetched for this binding.

classmethod library(expressions, values)

Return {name: lookup} for one evaluation, ready to be asked.

The binding stack and the evaluator are created here and captured by the lookups, so neither is named outside this class. The stack is how a lookup finds the binding of the body asking it, which is nobody else’s concern; and it must belong to one evaluation rather than to the class, or two plots drawn at once would interleave their bindings on it and return wrong numbers rather than failing.

The evaluator does not come back out either: handing it over would hand over a symbol table full of lookups, and with it a way to evaluate arbitrary text with none of the parameter machinery.

Parameters:

• expressions (dict) – The library, {name: expression}.

• values (dict) – {name: {(channels, photometries): array}}, as fetched for one series, keyed exactly as get_needed_values() asked.

Returns:

A lookup per name, sharing one evaluation’s state.

Return type:

dict

class autowisp.diagnostics.expressions._ChannelBound(lookup, channels, photometry_binding)

Bases: object

A quantity taking photometries, with its channels bound.

What x[0] gives inside x[0][1]: the second subscript then binds the photometries, against the binding of the body that wrote it, and returns the values. Every read of such a quantity has that second subscript (check_expression() refuses one without), so this never reaches the arithmetic in place of values.

__getitem__(slots)

Resolve the photometries of x[...][slots], and read it.

__init__(lookup, channels, photometry_binding)

Parameters:

• lookup (QuantityLookUp) – The quantity read.

• channels (tuple) – Its channels, already bound.

• photometry_binding (dict) – The photometry binding of the body reading it, taken when its channels were bound: the same body’s, since nothing is evaluated between the subscripts.

autowisp.diagnostics.expressions._as_series(values, count)

Return values as an array of count entries.

A constant-valued expression evaluates to a scalar, which still has to plot as a series, so it is broadcast to the image count.

autowisp.diagnostics.expressions._bind_channels(slots, binding)

Return the channels slots name under binding.

A slot number is looked up; a quoted channel is already one.

autowisp.diagnostics.expressions._bind_photometries(slots, binding)

Return the photometry ids slots name under binding.

A slot number is looked up; a quoted photometry is read as the id it spells, 'ap4' as 4.

autowisp.diagnostics.expressions._body_problems(name, expression, current_library)

Return what is wrong with the text of an expression or a rule.

Everything check_expression() judges apart from the name, which is the whole of what applies to an exclusion rule as well.

Parameters:

• name (str) – What the text is evaluated under: the proposed name, or rule_quantity.

• expression (str) – The proposed text.

• current_library (dict) – As for check_expression().

Returns:

Descriptions of the problems; empty if there are none.

Return type:

list

autowisp.diagnostics.expressions._canonical_length(values)

Return the length of the image list values are padded onto.

autowisp.diagnostics.expressions._evaluation_order(targets, expressions)

Return the expressions to evaluate, each after what it references.

A depth-first walk from targets, appending a name only once the expressions it references have been appended. Only what the targets reach is visited, so asking for one expression does not drag in the whole library.

Appending on the way out is what makes this an order rather than a traversal. On the way in, two expressions reached at the same depth are indistinguishable even when one references the other: with a = b + c and c = b * 2, both b and c are reached from a together, so reversing the order of discovery can place c before the b it needs.

Raises:

PipelineError – On a reference cycle, naming the loop.

autowisp.diagnostics.expressions._evaluator_names()

Return the names a bare Evaluator already defines.

autowisp.diagnostics.expressions._get_bare_names(expression)

Return the names expression reads without a subscript.

The complement of get_indexed_names(), so between them every ast.Name is accounted for once. Mostly functions.

autowisp.diagnostics.expressions._get_read(read)

Return (name, channel_slots, photometry_slots) for one read.

read is the whole read: x[0], or x[0][1] with photometries. The photometry slots are () where there is no second subscript, and the channel slots may be empty only where there is one: name[()][0] is how a quantity taking photometries but no channel is read.

Raises:

PipelineError – As described in get_indexed_names().

autowisp.diagnostics.expressions._in_written_order(tree)

Return the nodes of tree in the order their text starts.

Rather than the breadth-first order of ast.walk, which puts y['R'] before x['B'] in 2 * x['B'] + y['R'], being shallower.

autowisp.diagnostics.expressions._literal_slots(index)

Return the slots index writes as a tuple, or None if it can’t.

Each slot is an int or a str literal; anything else, a name or a slice, writes no slot. The tuple may be empty, for the caller to judge.

autowisp.diagnostics.expressions._needed_diagnostics(targets, references, expressions)

Return the diagnostics to fetch, rejecting names that resolve to nothing.

Raises:

PipelineError – If any referenced name is neither an expression, a diagnostic, nor an evaluator builtin.

autowisp.diagnostics.expressions._nested_reads(tree)

Return the ids of the subscripts in tree that are part of another.

The x[0] of x[0][1] is a subscript node of its own, but it is part of the whole read rather than a read.

autowisp.diagnostics.expressions._slot_problems(expression, library)

Return what is wrong with how expression reads its quantities.

One rule: a quantity is read with exactly as many channel slots and photometry slots as it takes, and one taking neither is read bare. The first half is what makes a reference’s arguments match a definition’s parameters positionally, and what makes the second subscript present exactly where a _ChannelBound needs resolving; the second is what lets get_needed_values() and QuantityLookUp.library() treat a bare name as a value and still know they have seen everything. And a quoted photometry must name one.

Only names resolving to a quantity are judged, in either direction: the evaluator’s own arrays can be indexed too, and most bare names are its functions.

Parameters:

• expression (str) – The proposed text.

• library (dict) – The library it would join, {name: expression}, including the proposed expression, so that a self-reference has an arity.

Returns:

Descriptions of the problems; empty if there are none.

Return type:

list

Raises:

PipelineError – From get_indexed_names(), if an index does not name a channel slot at all.

autowisp.diagnostics.expressions._spell_channel_slots(count)

Return count channel slots, spelled for a message.

autowisp.diagnostics.expressions._spell_photometry_slots(count)

Return count photometry slots, spelled for a message.

autowisp.diagnostics.expressions._spell_read(name, channels, photometries)

Return a read of name at these slots, as an expression writes it.

The channel subscript is [()] where there are no channels but photometries follow, and the photometry one is left out where there are none.

autowisp.diagnostics.expressions._visit_needed(name, channels, photometries, expressions, needed)

Add what name bound to channels and photometries reads.

Recursively, and both kinds of reference are followed. A subscripted one is bound through channels and photometries; a bare one takes neither, being the time or an expression whose channels and photometries, if any, are all quoted – and the latter still reads diagnostics, so stopping at bare names would leave them unfetched. Other bare names are functions.

autowisp.diagnostics.expressions.check_expression(name, expression, current_library)

Return what is wrong with a proposed expression, as plain strings.

Problems are returned rather than raised so that this module stays free of any particular presentation: the browser interface turns them into a ValidationError, an importer collects them per entry.

No project is needed. What a name may mean comes from autowisp.diagnostics.diagnostic_types, which is complete: a diagnostic_type row is either seeded from the static catalogue at project creation or created by the pixel_q* branch of _save_image_diagnostics, which refuses every other name. So no project can contain a diagnostic this does not know, and an expression means the same thing everywhere – which is what lets expressions be exported from one project and imported into another.

Whether an expression is usable in a particular project is a different question, about whether rows have been recorded, and is answered by counting them rather than here.

Parameters:

• name (str) – The proposed name.

• expression (str) – The proposed text.

• current_library (dict) – The library it would join, as it stands. An existing entry of the same name is replaced by the proposed text rather than conflicting with it, so an edit can pass the library unchanged.

Returns:

Descriptions of the problems; empty if there are none.

Return type:

list

autowisp.diagnostics.expressions.check_rule(rule, library)

Return what is wrong with a proposed exclusion rule, as plain strings.

A rule is judged as an expression would be, apart from having no name of its own, and with one more constraint: it takes at most one channel slot, which is bound to the channel being decided for, and at most one photometry slot, bound to the photometry being decided for. Any other channel or photometry it reads is named by quoting it.

Whether the quoted channels and photometries exist is a question about the cameras and the configuration of a project, and is not answered here.

Parameters:

• rule (str) – The proposed rule text.

• library (dict) – The project’s library, {name: expression}.

Returns:

Descriptions of the problems; empty if there are none.

Return type:

list

autowisp.diagnostics.expressions.evaluate_quantities(wanted, expressions, values)

Evaluate the quantities of one series, each bound as asked.

Parameters:

• wanted (dict) – {quantity: set of (channels, photometries)}, as the table bound them and as get_needed_values() takes them. Asking for both axes at once is what lets an instantiation they share be computed once.

• expressions (dict) – The library, {name: expression}.

• values (dict) – {name: {(channels, photometries): array}}, every array on one canonical image list, holding what get_needed_values() asked for and keyed the same way.

Returns:

{quantity: {(channels, photometries): array}}, all of

the same length – the shape values arrives in, being the same kind of thing: a quantity read in the binding asked for.

Return type:

dict

Raises:

PipelineError – If a quantity resolves to nothing, if the expressions reference each other in a cycle, or if values lacks something they read.

autowisp.diagnostics.expressions.get_bare_aggregates(expression)

Return the NaN-propagating aggregates an expression calls.

Every array is NaN-padded to the canonical image list, so a plain median over one goes NaN as soon as a single image lacks the diagnostic – which is usual rather than exceptional. Callers use this to warn, not to refuse: a deliberate median is still a legitimate thing to write.

Parameters:

expression (str) – The expression text.

Returns:

The names called without their nan prefix.

Return type:

set

Raises:

SyntaxError – As for get_expression_names().

autowisp.diagnostics.expressions.get_channel_arity(name, expressions)

Return how many channels name has to be bound to before it can be read.

Only one of the three answers is data. A diagnostic is recorded once per channel, so it takes exactly one; time_quantity is recorded once per image and takes none; and an expression takes however many slots its body mentions, which is the only part worth deriving.

The table above a plot asks this to know how many channel dropdowns an axis needs.

Parameters:

• name (str) – A diagnostic, an expression, or time_quantity.

• expressions (dict) – The library, {name: expression}.

Returns:

The number of channels to bind.

Return type:

int

Raises:

PipelineError – If name is neither a diagnostic nor an expression nor the time.

autowisp.diagnostics.expressions.get_channel_parameters(expression)

Return the channel slots one expression takes, in order.

Derived from the body rather than declared: the parameters are the slot numbers it mentions, so there is nothing to keep in step and nothing extra to store. bg_center[0] / bg_center[1] takes (0, 1); sky_color[0,1] - sky_color[1,2] takes (0, 1, 2).

Sorted, which is what makes the order well defined when a body writes its slots out of sequence – and the order matters, since a reference’s arguments are matched onto these positionally.

A quoted channel is not a parameter: it is already bound. So x[0] - x['G0'] takes (0,), and sky_color['B', 'R'] takes none.

Parameters:

expression (str) – The expression text.

Returns:

The distinct slot numbers, ascending. Empty for an

expression that reads no channel at all, or only quoted ones.

Return type:

tuple

Raises:

SyntaxError, PipelineError – As for get_indexed_names().

autowisp.diagnostics.expressions.get_expression_dependents(name, expressions)

Return the expressions referencing name, for the delete guard.

Parameters:

• name (str) – The expression being deleted.

• expressions (dict) – The library, {name: expression}.

Returns:

Names of expressions that would break if name went away.

Return type:

set

autowisp.diagnostics.expressions.get_expression_names(expression)

Return every name one expression mentions, functions included.

nanmedian(rel) gives {"nanmedian", "rel"}: a call by bare name is an ast.Call whose func is an ast.Name, so there is nothing here to tell a function from a variable, and no attempt is made to. Which is which cannot be decided from the text anyway – it depends on the open project’s diagnostics, on what else the library holds, and on the evaluator’s symbol table – so the split is left to the caller, which is what order_expressions() and check_expression() both do.

Parameters:

expression (str) – The expression text.

Returns:

The ast.Name identifiers, whether they are used as

values or called.

Return type:

set

Raises:

SyntaxError – If the text is not a single Python expression. mode="eval" rejects statements, assignments, loops and imports, which is worth having but is not what makes evaluating an expression safe: __import__('os').listdir('.') is a perfectly valid expression. Safety comes from autowisp.evaluator.EvaluatorBase – asteval refuses imports, eval, exec, getattr and dunder traversal, and AutoWISP drops open and print on top – and from check_expression() restricting names to that symbol table plus the project’s diagnostics.

autowisp.diagnostics.expressions.get_indexed_names(expression)

Return what one expression reads, in which channel and photometry slots.

A channel slot is written as a subscript – bg_center[0], or sky_color[0,1] for a quantity taking two – and the numbers are that expression’s own formal parameters, bound to real channels only when something is plotted. So this reports the shape of what the text asks for, and says nothing about channels.

A slot may instead be a quoted channel name, bg_center['G0'], which binding passes through unchanged: it is already bound. That ties the text to a camera’s channel naming, which is the point in an exclusion rule and a convenience in a project whose cameras all name their channels alike.

A quantity taking photometries – a diagnostic fit_magnitudes produces, or an expression reading one – is read with a second subscript for them, numbered in a namespace of its own: magfit_residual[0][1] is channel slot 0 and photometry slot 1, and magfit_residual[0]['ap4'] quotes aperture 4. One taking no channel is written with an empty channel subscript, name[()][0].

Parameters:

expression (str) – The expression text.

Returns:

(name, channel_slots, photometry_slots) triples in the

order they appear, both always tuples however many were written, each slot an int or a quoted str; the photometry slots empty where there is no second subscript. Repeats are kept: sky_color[0,1] - sky_color[1,2] reads one name at two different slot pairs, which is the whole point of the parameters being formal.

Return type:

list

Raises:

• SyntaxError – As for get_expression_names().

• PipelineError – If a subscript is not a plain name indexed by integer or string literals, at most twice. Anything else – bg_center[i], bg_center[1:2], x[0][1][2] – cannot name a slot, and saying so here is clearer than letting it fail as an unresolvable name later. So is an empty channel subscript with no photometries after it.

autowisp.diagnostics.expressions.get_logical_keywords(expression)

Return the Python logical keywords an expression uses.

and, or and not ask for the truth value of their operands, which an array of more than one element refuses to have (“the truth value of an array is ambiguous”): element-wise logic is |, & and ~. On a quantity with one value, though, the keywords are legitimate, so – like get_bare_aggregates() – this is for callers to warn with, never to refuse.

Parameters:

expression (str) – The expression text.

Returns:

Those of "and", "or" and "not" that appear.

Return type:

set

Raises:

SyntaxError – As for get_expression_names().

autowisp.diagnostics.expressions.get_needed_values(wanted, expressions)

Return the diagnostics to read, and in which channels.

Runs before anything is evaluated, for two callers that both need the answer without it:

• whatever reads the values does so for a whole series in one query, so it cannot discover them as it goes;

• the table above a plot counts the images recording all of them, which is a question about rows and must never evaluate an expression – there is a table row per observing session and image type, so evaluating per row would be work proportional to the whole image collection.

It walks what evaluation walks, resolving each reference’s arguments through the binding of the body holding it, and stops at the diagnostics and the time.

Parameters:

• wanted (dict) – {quantity: set of (channels, photometries)}, each a tuple holding one channel per channel parameter of that quantity, and one photometry id per photometry parameter. A set because one quantity may be wanted at two bindings at once: bg_center in R against bg_center in B is a plot of a diagnostic between channels.

• expressions (dict) – The library, {name: expression}.

Returns:

{name: set of (channels, photometries)} – the same

shape it takes, which is what it means: what is wanted, resolved into what must be read. One entry per diagnostic, holding every binding it is read in, plus time_quantity with ((), ()) where an expression reads the time.

Return type:

dict

Raises:

PipelineError – On a reference cycle, on a name that resolves to nothing, or on a binding of the wrong length for what it binds.

autowisp.diagnostics.expressions.get_photometry_arity(name, expressions)

Return how many photometries name has to be bound to.

As get_channel_arity() counts channels: a diagnostic recorded per photometry (see photometry_diagnostic_names()) takes exactly one; any other diagnostic, and the time, take none; and an expression takes however many photometry slots its body mentions.

Parameters:

• name (str) – A diagnostic, an expression, or time_quantity.

• expressions (dict) – The library, {name: expression}.

Returns:

The number of photometries to bind.

Return type:

int

Raises:

PipelineError – If name is neither a diagnostic nor an expression nor the time.

autowisp.diagnostics.expressions.get_photometry_parameters(expression)

Return the photometry slots one expression takes, in order.

As get_channel_parameters() returns the channel slots, from the second subscripts instead: magfit_residual[0][0] - magfit_residual[0][1] takes (0, 1), and magfit_residual[0]['ap4'], whose photometry is quoted, none.

Parameters:

expression (str) – The expression text.

Returns:

The distinct slot numbers, ascending.

Return type:

tuple

Raises:

SyntaxError, PipelineError – As for get_indexed_names().

autowisp.diagnostics.expressions.get_quoted_channel_order(quantity, expressions)

Return the channels quantity quotes, in the order its text is written.

Following the expressions it references where they come, whether read bare or through a subscript: gcol = sky_color['B', 'R'] quotes B and then R, and so does anything reading gcol. Each channel is listed once, where it is first quoted.

The order is the text’s rather than any other because two things are laid out by it. A series whose channels are all quoted binds none of its own, and is shown in the first – its colour, and the frame a click on a point opens. And a quoted channel a magfit diagnostic is read in gets a column of the series table to choose its photometric reference in, and those columns follow the order the expression mentions them.

Parameters:

• quantity (str) – A diagnostic, an expression, or the time.

• expressions (dict) – The library, {name: expression}, which order_expressions() has found free of cycles.

Returns:

The channels, empty if nothing quantity reaches quotes

one – as for anything but an expression.

Return type:

list

Raises:

SyntaxError, PipelineError – As for get_indexed_names().

autowisp.diagnostics.expressions.order_expressions(targets, expressions)

Return the order to evaluate targets in, and what data that needs.

Only the dependency subtree of targets is walked, so asking for one expression does not evaluate the whole library.

What a name may mean comes from is_known_quantity() rather than from a project, because it cannot differ between projects: a diagnostic_type row is either seeded from the static catalogue or created by the quantile branch that refuses every other name. A diagnostic no image here records is therefore accepted and comes back all-NaN, which is what the padding is for.

Parameters:

• targets – The quantity names wanted. Any that are not expressions are diagnostics, and pass through to the returned set.

• expressions (dict) – The library, {name: expression}.

Returns:

list: Expression names, each after everything it references.

set: The diagnostics that have to be fetched for them.

Return type:

tuple

Raises:

PipelineError – On a reference cycle, or on a name that is neither an expression, a diagnostic, nor an evaluator builtin.

autowisp.diagnostics.expressions.rename_references(expression, old_name, new_name)

Return expression with every reference to old_name renamed.

A source-level edit, and the only rewriting anywhere in this feature. It does not contradict the rule that nothing is rewritten – that is about resolution, where an ast.unparse roundtrip would leave a stored text and a resolved text to keep straight. Here the user has asked for the change, and what comes back is what they will read and edit from then on.

So it must not reformat: the new identifier is spliced in at the exact source offsets ast reports, leaving every other byte alone. That is also what makes it safe where a textual replace is not – rel inside rel_bg, inside a string literal, or as a keyword argument’s name is left untouched, because only ast.Name nodes are moved.

Parameters:

• expression (str) – The expression text to rewrite.

• old_name (str) – The name being renamed.

• new_name (str) – What to call it instead.

Returns:

The text with those references renamed, and identical

everywhere else.

Return type:

str

Raises:

SyntaxError – As for get_expression_names().

autowisp.diagnostics.expressions.rule_quantity = '<exclusion rule>'

What an exclusion rule is evaluated under, as one more entry in the library: dict(library, **{rule_quantity: rule}). A rule taking a channel parameter (see get_channel_parameters()) is bound to each channel being decided for, (channel,), and excludes per channel; one taking none reads only quoted channels, is bound to (), and its verdict applies to every channel of the image. Photometries likewise (see get_photometry_parameters()): a rule taking a photometry parameter excludes per photometry. Not a slug, so check_expression() lets no stored expression take the name, and it reads sensibly in an error message.