Problem statement and objectives

Converting annotation files raises a difficulty that is neither technical nor linguistic, but epistemic: the user must decide, before anything happens, what to do with information that the destination cannot receive. Formats differ in what they are able to express, and those differences are not documented anywhere the user is likely to look. The decision is therefore taken blindly, and its consequences appear only afterwards — as a file that fails to be written, or worse, as a file that is written and silently amputated.

The central problem addressed by "Convert" is thus:

how to let a user convert a set of annotation files to a chosen format, while making the cost of that choice knowable in advance and the outcome verifiable afterwards, without requiring any knowledge of how formats are internally organised.

This problem is sharpened by the conditions of use. Conversion is a batch operation: it applies to a set of files whose contents are not examined at the moment the user decides. It is also a directed operation: users convert because they are moving from one tool to another, with a definite destination in mind. Convert must therefore support decisions taken once, in ignorance of individual file contents, and applied uniformly — while remaining accountable for each file individually.

The objectives follow:

• to let the user select one destination format among those that can be written, and to expose what that format is able to hold at the moment of the choice;
• to let the user state, before conversion, how annotations are to be written when the destination cannot hold them as they are, these statements being attached to the format and not to any file;
• to apply the same decisions to every file of the batch, and to report the outcome of each file separately, an individual failure never interrupting the others;
• to give the user access to the converted files, in a manner adapted to the context of use;
• to guarantee accessibility, of the interface and of the information the tool exposes;
• to remain part of the SPPAS ecosystem, reusing its reading and writing capabilities without exposing them.

"Convert" does not aim to prepare files, to edit annotations, nor to correct a corpus. It aims to make one operation — writing annotations in another format — deliberate rather than accidental.

Expressing the requirements

Choosing a destination in full knowledge

A user selects a destination format because a tool downstream requires it, not because they know what that format can hold. The choice is therefore made under a constraint the user cannot evaluate. Convert answers this by presenting, alongside each writable format, the properties it supports — several tiers or one, metadata, hierarchies, controlled vocabularies, overlapping annotations, typed labels, and so on. This information is not documentation placed beside the tool; it is part of the act of choosing, available at the moment the decision is taken. Its presentation must make a comparison possible at a glance and remain legible to someone who has never opened an annotation file in an editor: the properties of a format are a technical matter, but understanding them must not require technical expertise.

Stating export options before reading anything

When a destination cannot hold what a source expresses, something must be decided. That decision belongs to the user, and it must be expressible without knowing which files will be affected: the batch is heterogeneous and its contents are not examined beforehand. Convert therefore attaches export options to the destination format itself. The options describe how to write, not what any file contains. Chosen once, they apply to the whole batch, and their meaning must be intelligible independently of any particular file.

Delimiting the set of files to convert

The files to be converted come from the working context, not from the tool: in a working session, they are the files currently selected in the workspace; in an occasional use, they are the files the user brings to the tool. The requirement is the same in both cases — the user delimits a set, Convert operates on that set, and the set is not silently altered. The two contexts differ only in their material constraints, an occasional use being bounded by what a service can reasonably receive.

Reporting each file individually

A batch is not an operation, but a series of operations sharing the same decisions. Some will succeed, some will not, for reasons belonging to the individual file rather than to the choices made. The user must be able to see, for every file, what happened to it: written, refused, or left untouched — and why. A refusal is stated in terms the user can act upon, and it never stops the files that follow. Reporting is therefore not a side effect of the conversion; it is what makes the operation verifiable.

Reaching the converted files

A conversion that produces files the user cannot find has not converted anything. In a working session, the produced files join the workspace and remain available to the following operations. In an occasional use, they are handed back to the user. The requirement is that the outcome be reachable without the user having to know where the tool put it.

Accessibility

Everything above rests on information being exposed: supported properties, available options, statuses, causes. This information is useless if it cannot be perceived. Contrast, legibility, text alternatives to any graphical encoding, keyboard operation, consistent terminology between the interface and the concepts it manipulates: these are conditions of the tool working at all, not qualities added to it. Convert exposes a technical subject to a non-technical audience; the burden of intelligibility lies entirely on the interface.

Concepts manipulated by the application

Format

A way of writing annotations on disk, identified by an extension and associated with a piece of software.

Format property

A boolean characteristic of a format, stating what that format is able to hold. Properties belong to formats, never to files.

Destination format

The single format, among those that can be written, into which the batch is converted.

Batch

The set of files delimited by the user, heterogeneous in format, whose contents are unknown at the time decisions are taken.

Export option

A question raised by what the destination format cannot hold, together with the answers available for it. Options are derived from properties, not written per format. An option covers one property where answering alters the annotations, and several where it does not.

Answer

The user's decision for one option, applied to every file of the batch.

General option

A decision independent of the destination format: overriding an existing output, detecting the format of an input whose extension is unknown.

Remediation

A transformation applied to the annotations of one file, after reading and before writing, implementing the answer chosen for an option.

Structural loss

Information not written because the destination format cannot hold it. Determined by the destination alone, identical for the whole batch, announced once.

Contingent loss

Information not written, or content transformed, because of what one particular file contained. Determined at reading, reported for that file.

Status

The outcome of one file: written, written with remarks, refused, or skipped.

Remark

A contingent loss reported on one file.

Cause

The explicit reason a file was refused.

Deriving the options from the destination format

The correspondence between an unsupported property, the question it raises, and the answers available for it is data of the model, not code. Adding a property to the API adds an entry to this table; it does not modify Convert.

The options are of two kinds, and the line between them is whether answering alters the annotations. Some unsupported properties concern what surrounds the annotations — metadata, controlled vocabularies, media declarations. Keeping them changes nothing of what was annotated. Others concern the annotations themselves, or the shape of the result: answering transforms the content, or the number of files produced. The first kind is decided once; the second, property by property.

Preservation: one decision

Where the destination cannot hold metadata, controlled vocabularies or media declarations, the user is not asked where to put them, but whether to keep them:

Answering yes lets SPPAS write that information wherever the destination allows: as comments where the format holds them (a), in a tier otherwise (b). Answering no discards it. Where the destination allows neither, yes has the same effect as no, and the option is not offered.

The answer has a default no user has to look for: the setting of the application, interoperability, true [API-OK]. Maintaining interoperability is the normal state; accepting the loss is a choice.

That setting is a default, not the decision. It is global — it weighs on the annotations as much as on the conversions — while its effect is seen here, on a page a user reaches without ever opening the settings. Convert therefore states the effective answer where it applies, and lets it be changed for this conversion alone. Online, where no setting is kept for a given user, the value of the page is the only one there is.

Where SPPAS writes what it preserves is a mechanism, not a decision. Users divide into those who want their file and care for nothing else, and those who want nothing lost; neither arbitrates between metadata and controlled vocabularies. Offering three answers for each of three properties would ask nine questions no one has.

Structure: one decision per property

The remaining options alter the annotations or the shape of the result. They are decided individually, because answering one says nothing of the others:

alt_tag belongs here and not to preservation: discarding the alternatives of a label, however poorly scored, alters the annotation. What is written is no longer what was annotated, and that is a decision only the user can take.

(a) The property accept_comments states it, read by comments_support() [API-OK]. The API was already writing metadata as comments when the format allows it, and forgetting them otherwise; it now exposes that capability.

(b) Writing metadata, controlled vocabularies and media as a tier is done by create_unsupported_tier(), and read back by parse_unsupported_tier() [API-OK]. The tier is named DoNotEdit; see the annex for what it holds.

Properties raising no question

The remaining unsupported properties raise no question, because nothing can be fabricated and no transformation preserves the content:

• no_tiers — a file holding no tier at all cannot be written.
• empty_tier [API-OK] — a tier holding no annotation is removed where the destination refuses it, which empty_tier_support() states beforehand; see 3.3.
• interval — intervals cannot be reduced to points. A file holding intervals is refused. The converse is remediable, hence the point option above.
• hierarchy, disjoint, alt_localization, radius, gaps, overlaps — cause a structural loss, announced once for the batch.

Extensions of the API required by this section

None: the three extensions this section was stating are available [API-OK]. They are empty_tier_support(), comments_support(), and the pair create_unsupported_tier() / parse_unsupported_tier(). The annex states what they do and what they cost.

Remediation chaining

Remediations are not independent, and their order is not neutral. Removing empty tiers can leave a file with no tier, which the destination may refuse. Splitting a multi-tier file into one file per tier can produce a file whose single tier is empty. A remediation is therefore never final: it produces a new content, which is verified again against the properties of the destination.

A chain may end in a refusal. This is not a failure of the model: it means the content, once adapted as far as the destination allows, still exceeds what that destination can hold. The cause reported is the one that ended the chain, stated in terms of the user's data.

Conversion of one file

Anticipation comes first; the typed exceptions of the API act as a safety net for what anticipation did not catch.

1. Read the file, applying the general option on format detection.
2. Anticipate: compare the content read to the properties of the destination, apply the remediations chosen, chain and re-verify (3.3).
3. Write.
4. If writing fails on a typed exception identifying a remediable cause, remediate and write once more. One second pass only.
5. If it fails again, or on a non-remediable cause, refuse the file and state the cause.

The failure of one file never interrupts the batch.

The typed exceptions of the current API relevant to writing are AioMultiTiersError (1510), AioNoTiersError (1515), AioEmptyTierError (1525) and AioLocationTypeError (1530) [API-OK]. They identify refusals bound to the properties of the destination, and are usable as causes as they stand.

Requirements as specified

[010] Starting a conversion

• [011] The user can initiate a conversion of a set of annotation files.
• [012] The system initialises a conversion with no destination and no batch.

[020] Understanding the process

• [021] The user can identify what is required before a conversion can run.
• [022] The system states which decisions are missing.

[030] Controlled progression

• [031] The system allows a conversion to run only if a destination format is selected and the batch is not empty.

[100] Destination format

• [101] The user can select one destination format.
• [102] The system restricts the choice to the formats it is able to write.
• [103] The system presents, for each format, the properties it supports.
• [104] The system presents this information in a form comparable at a glance and intelligible without knowledge of file internals.
• [105] The user can select at most one destination format at a time.
• [106] The user can cancel the current selection.

[200] Batch

• [201] The user delimits the set of files to convert.
• [202] The system operates on that set and does not alter it.
• [203] The system reports when the set is empty.

[300] Export options

• [301] The system derives the options from the properties of the selected destination format.
• [302] The system presents an option for each unsupported property for which a remediation exists and answering alters the annotations, and a single option covering those for which it does not.
• [303] The user can choose one answer per option.
• [304] The system applies a default answer to any option left unanswered.
• [305] The answers apply to every file of the batch.
• [306] The options are stated before any file is read.
• [307] The system restricts the answers offered to those the destination format allows.
• [308] The system takes the answer of the preservation option from the settings of the application, states it on the page, and lets the user change it for this conversion without altering the settings.

[400] General options

• [401] The user can allow or forbid overriding an existing output file.
• [402] The user can allow or forbid detecting the format of an input file whose extension is unknown.

[500] Announcing structural loss

• [501] The system announces, once for the batch, what the destination format cannot hold.
• [502] The system does not report structural loss per file.

[600] Converting

• [601] The system reads each file of the batch.
• [602] The system applies the chosen remediations before writing.
• [603] The system re-verifies the content against the destination after each remediation.
• [604] The system writes each file in the destination format.
• [605] The system attempts a second write after remediating a typed failure.
• [606] The failure of one file never interrupts the batch.

[700] Reporting

• [701] The system reports a status for every file of the batch.
• [702] The system reports the remarks attached to a file.
• [703] The system states the cause of every refusal.
• [704] The system states the cause in terms of the user's data, not of the internals of the format.

[800] Reaching the results

• [801] The system makes the converted files reachable to the user.
• [802] The system makes them reachable without the user knowing where they were written.

[900] Restarting

• [901] The user can start a new conversion.
• [902] The system resets the decisions without resetting the batch.

[1000] UX and accessibility

• [1001] The system provides text alternatives to any graphical encoding of a property.
• [1002] The system uses consistent terminology between interface and concepts.
• [1003] The system is operable by keyboard and by screen reader.
• [1004] The system provides explicit messages at every decision point.

Data manipulated

• destination — one format, among those that can be written.
• properties — the boolean characteristics of destination, stating what it is able to hold.
• options — the questions raised by the unsupported properties for which a remediation exists, each with its available answers.
• answers — one decision per option.
• general_options — override (boolean), heuristic (boolean).
• batch — a set of files, possibly empty.
• structural_loss — what destination cannot hold.
• results — one result per file of the batch, each holding a status ∈ {written, written with remarks, refused, skipped}, the remarks attached to it, and the cause of a refusal.

Main dependencies

• destination → properties
• properties → options
• properties → structural_loss
• options → answers
• (batch ∧ destination ∧ answers ∧ general_options) → results