Options
All
  • Public
  • Public/Protected
  • All
Menu

Module core-concepts

Flux is a low code solution to create interactive applications from the description of data-flows and views. This library is the foundation of this low code solution. The flux-builder web-application enables building the applications using a user friendly interface, while flux-runner is available to run them.

Illustration of the flux-builder web application. The top panel is referenced as 'builder-panel' and the bottom panel as 'rendering-panel'

Contrasting with other low-code solutions, Flux ecosystem not only allows to define attractive UIs but also enables the description of complex data flows between processing units (called modules). These processing units can be running either on your computer or on remote severs - the latter being especially relevant for long-running computations.

A workflow is organized around modules and connections, an hypothetical workflow can be represented like this:

            physicalMdle 
                         \
                          combineMdle-->simulatorMdle-->viewerMdle
                         /
 sliderMdle-->solverMdle

This describes the data flow of the application. Modules features one or multiple inputs and/or outputs. They can react on incoming messages and emit output messages. Connections support the transmission of messages, always from upstream modules to downstream modules. Any messages emitted by a module can not be altered afterward whatsoever.

In the above workflow, two modules - sliderMdle and viewerMdle - are associated to a view, they can be arranged and styled in a layout editor in a so called rendering-panel, e.g.:

 ___________________________________
|   slider        ___viewer_____   |
|  ----|----     |              |  |
|                | ( อกโ€ขโ€ฏอœส– อกโ€ข)๐Ÿ‘Œ  |  |
|                |              |  |        
|                |______________|  |
|__________________________________|

In the above 'application' the data-flow is:

  • user move the slider, the sliderMdle emits a message containing the current value
  • the solverMdle intercept it and emit a solverModel accordingly
  • the combineMdle join and emit the created solverModel with some physicalModel produced by the physicalMdle
  • the simulatorMdle does a simulation and emit the results
  • the results are then rendered by the viewerMdle

The elementary building blocks of Flux applications are modules, they are exposed by javascript (npm) packages. In practice, one package usually gathers multiple modules related to the same domain - it is called a flux-pack.

We start in the next section the description of Flux-pack creation, and start to correlate the concepts described above with associated classes/functions. The example above is used as guideline through the discussions and examples.

What follows reference the YouWol's local environment published here. If you want to exercises in practice what is discussed you should install it. It provides a layer above your favorite stack to easily create/update/publish flux-packs and exercise them within the flux-builder web application (running locally ๐Ÿคฏ).

Creation of a flux-pack ๐Ÿ“ฆ

From YouWol's dashboard click on packages -> new package, provide a name to your package & install: that will create a skeleton project you can complete through the course of the discussions.

Let's start by implementing a flux-pack containing a (rather empty) module. Here is the declaration of the flux-pack:

//file main.ts
import { FluxPack } from '@youwol/flux-core'
// auto generated from 'package.json' - we'll go back to this latter
import { AUTO_GENERATED } from '../auto_generated'

export let pack = new FluxPack(AUTO_GENERATED}

โ• the variable name 'pack' matters: it is the normalized entry point of every flux-pack.

Below is the implementation of the module:

 // file hello-world.module.ts
 import {pack} from './main.ts'
 import {Flux, BuilderView, ModuleFlux} from '@youwol/flux-core'

 export namespace SimulatorModule{

     @Flux({
         id: 'simulator',
         pack,
         namespace: SimulatorModule
     })
     @BuilderView({
         icon: '<text> Simulator </text>',
         namespace: SimulatorModule
     })
     export class Module extends ModuleFlux {

         constructor( params ){
             super(params) 
         }
     }
 }

Those two files define a valid flux-pack, you can build & publish (in the CDN) the package and start using it in a Flux application.

Module's namespace ๐Ÿญ

All the data related to one module are encapsulated in a namespace, it includes the definition of:

  • the module itself: this is where most of the logic will go
  • the persistent data (optional - not presented yet): data of the module the user will be able to save with their applications
  • the builder view: how the user will see and interact with your module in the builder-panel of the application. This is the purpose of the BuilderView decorator.
  • the rendering view (optional - not presented yet): how the user will see and interact with your module in the rendering-panel of their applications. Not all modules have rendering view associated (e.g. a pure computational module probably won't).

๐Ÿง On top of the classes/structures explicitly written in it, decorators are generating other required data in the namespace.

The namespace acts as the Factory of the module: having it includes everything needed to create instances of the various objects related to the module (e.g. configuration, builder view, render view, etc). You can have a look to the unit tests if you are interested about coding flux applications (see also instantiateModules).

Let's move forward and include some logic in the module

Defining inputs & outputs โ‡Œ

Let's move on the simulator module and add a fake simulation triggered when a message comes in:

export class Module extends ModuleFlux {
   
   output$ : Pipe<number> 

   constructor( params ){
       super(params) 

       this.addInput({
           onTriggered: ({data} : {data:unknown}) => this.simulate( data )
       })
       this.output$ = this.addOutput()
   }

   simulate( data: unknown ){
         
       this.output$.next({data:42})
   }
}

What!? unknowns ๐Ÿ˜ฉ

The method ModuleFlux.addInput is used to add an input to the module. The key point is to provide a callback to be triggered at message reception.

The method ModuleFlux.addOutput is used to add an output to the module. It returns a Pipe: it is an handle to emit messages.

There is a lot of $ suffix in the library ๐Ÿคจ. It has been popularized by Cycle.js: 'The dollar sign $ suffixed to a name is a soft convention to indicate that the variable is a stream.' In flux-core the streams are backed by RxJs Observable (as readonly source of data) as well as Subjects (for read & write).

A word about contract

In the above snippet the type of the data is unknown. It can't be otherwise with no additional information as any modules can have been used to provide the incoming message. That being said, modules are not supposed to work with any kind of incoming data.

The developer can provide expectations in terms of accepted data-structures using the concept of contract:

// additional imports
import {compute, PhysicalModel, SolverModel } from 'somewhere'
import {contract, expectInstanceOf } from '@youwol/flux-core'

let contract = contract({
      requireds: {   
          physModel:expectSingle<PhysicalModel>({
             when: expectInstanceOf(PhysicalModel) 
          }), 
          solverModel:expectSingle<SolverModel>({
             when:  expectInstanceOf(SolverModel)
          }), 
      }
  })

type NormalizedData = {
     physModel: PhysicalModel,
     solverModel: SolverModel
}
// Decorators @Flux, @BuilderView skipped
export class Module extends ModuleFlux {
   
   output$ : Pipe<number> 

   constructor( params ){
       super(params) 

       this.addInput({
           contract,
           onTriggered: ({data} : {data:NormalizedData} ) => 
                         this.simulate( data.physModel, data.solverModel )
       })
       this.output$ = this.addOutput()
   }

   simulate( physModel: PhysicalModel, solverModel: SolverModel ){
         
       compute().then( (result) => this.output$.next({data: result}) )
   }
}

No more unknown ๐Ÿ˜‡

In this case, the contract defines two expectations on the incoming messages:

  • to retrieve exactly one variable of type PhysicalModel
  • to retrieve exactly one variable of type SolverModel

Providing a contract to an input helps in terms of pre-conditions checks, data normalization and errors-reporting. You can find out more on contract in this section

What's next ? ๐Ÿง

Adding a configuration ๐ŸŽ›

In most of the case, a module relies on some parameters with default values (e.g. a threshold property for the simulatorMdle), which should be editable by the builder of a flux application and persisted with it. This is the purpose of a module's configuration.

flux-builder auto-generates settings panel from configurations

The library provides a simple way to define configuration. The developer needs to include a class called PersistentData into the module's namespace and decorate the properties to expose, e.g.:

// additional imports
import {Schema, Property } from '@youwol/flux-core'

namespace SimulatorModule{
   
    @Schema()
    class PersistentData{

         @property({
             description: 'control the threshold of the simulation'
         })
         threshold: number

         constructor({threshold} : {threshold?: number} = {}){        
             this.threshold = threshold != undefined ? threshold : 1e-6
         }
    }
}

This is a minimalist example, an advanced use case is presented here.

The above data-structure is automatically injected in the triggered function as a named property configuration:

export class Module extends ModuleFlux {
   
   output$ : Pipe<number> 

   constructor( params ){
       super(params) 

       this.addInput({
           contract,
           onTriggered: ({ data, configuration}) => 
               this.simulate( data.physModel, data.solverModel, configuration )
       })
       this.output$ = this.addOutput()
   }

   simulate( 
       physModel: PhysicalModel, 
       solverModel: SolverModel, 
       conf: PersistentData 
       ){
         
       compute(conf.threshold).then( (result) => this.output$.next({data: result}) )
   }

This configuration argument is called the dynamic configuration as it may have been updated at run time from values included in a message; read more about configuration here.

Defining a rendering view ๐Ÿ‘

As presented in the introduction, some modules have interactive views displayed in the rendering-panel. The simplest approach to define such views is to add a new decorator RenderView to the Module class definition.

Providing execution feedbacks ๐Ÿ“ฐ

Module's execution can (and often should) provides feedbacks about their processes, it helps understanding errors, performance bottlenecks, spurious results, etc.

The library introduces the concept of journal for this purpose, they:

  • provides synthetic tree representation about the chain of functions call during the module's execution.
  • includes builtin performance metrics
  • can features registered custom views to make complex data-structure easier to apprehend (e.g. 3D views, dynamic 2D plots, etc)

Find out more about journal and its underlying companion context here.

Creating plugins ๐Ÿ”Œ

Plugins are like modules, except they can mess up with a companion ๐Ÿ˜‰. It is a feature used quite often in various use case, e.g.:

  • for 3d viewer, plugins are used to add various features: object picker, controls in the viewer, compass
  • they can serve to register multiple modules to an orchestrator
  • they can intercept input messages and decorate them
  • etc

Some documentation about plugins can be found here

Optimizing performances ๐Ÿš„

The functional nature of Flux (close to nothing is allowed to mutate) allows to provide a simple, yet powerful, caching mechanism that enabled important optimization in common scenarios in scientific computing. Find out more here.

Going beyond simple builder view

It is possible to go beyond a simple representation of the modules in the builder panel. It is possible for instance:

  • use any kind of representation as long as it can be expressed in a SVG element
  • enable communication between module processes and the view, e.g. to provide visual hints about module execution

The ModuleRendererBuild documentation is a good place to start if you are interested.

Index

Type aliases

ConfigurationAttributes

ConfigurationAttributes: {}

Type declaration

  • [key: string]: AnyJson

Factory

Factory: { BuilderView: any; Configuration: any; Module: any; PersistentData: any; RenderView: any; consumersData: {}; displayName: string; id: string; isPlugIn: boolean; packId: string; resources: {}; uid: string }

A module's Factory is actually the namespace englobing it, the properties are initialized when using the decorators Flux, BuilderView, RenderView.

The factory can hold any other data as it actually is the namespace of the module and any variables can be defined in it. For instance it can be useful to define some static variables of your module you need in some places where the Factory is provided.

For consuming applications, data can be stored/retrieved using the consumersData attribute.

๐Ÿค“ In regular scenarios, most of the properties belonging to Factory are generated using the decorators mentioned above. You can still turn a namespace into module's factory by providing all of them explicitly, without using decorators.

Type declaration

  • BuilderView: any

    Builder view constructor, inheriting ModuleRendererBuild

    Issued from the decorator BuilderView

    ๐Ÿ’ฉ Should not be declared as 'any'

  • Configuration: any

    Configuration (ModuleConfiguration)

    Issued from the decorator Flux

  • Module: any

    Module constructor, inheriting ModuleFlux, explicitly written.

    ๐Ÿ’ฉ Should not be declared as 'any'

  • PersistentData: any

    Persistent data constructor, explicitly written.

    To get automatic schema generation in Flux, this class should be decorated with Schema.

    ๐Ÿ’ฉ Should not be declared as 'any'

  • RenderView: any

    Render view constructor, inheriting ModuleRendererRun

    Issued from the decorators RenderView

    ๐Ÿ’ฉ Should not be declared as 'any'

  • consumersData: {}

    Dynamic data the consumer (host application) may want to associate to the factory.

    • [key: string]: any
  • displayName: string

    display name.

    Issued from the decorator Flux

  • id: string

    id of the module's factory.

    Issued from decorator Flux

  • isPlugIn: boolean

    Whether or not the factory generate a module of type PluginFlux.

    Issued from the decorator [[@Flux]]

  • packId: string

    A unique id that references your package.

    Issued from decorator Flux.

  • resources: {}

    mapping <name, url> of resources

    Issued from the decorators Flux

    • [key: string]: string
  • uid: string

    combination of [[id]] & packId: uniquely identifies your module in the ecosystem Flux.

Message

Message<T>: { configuration?: ConfigurationAttributes; context?: UserContext; data: T }

A message is what is transmitted from modules to modules through a connection.

Messages have three components: data, configuration and context.

data

The data part of the message vehicles the core of the information. This is what is emitted by the modules in their output slots.

configuration

The configuration part of the message includes some properties meant to override the default configuration of the destination module.

Setting the configuration component of the message is most of the time achieved using an adaptor.

context

The context part of the message is a mapping {[key:string]: any} that the builder of a Flux application provide (here again, usually using an adaptor).

It is an append only data-structure transferred through the execution paths of flux applications that gets completed along the flow.

It serves at transmitting information from a starting point to a destination point away from multiple connections. The modules are not supposed to use it, they are just in charge to forward it in the most meaningful way (most of the times it is transparent for the modules' developers).

Type parameters

  • T = unknown

    type of the data part in the message

Type declaration

Pipe

Pipe<T>: Subject<Message<T>>

A pipe is the data-structure used by the modules to emit data:

  • in their constructor, the modules declare one or multiple pipes using ModuleFlux.addOutput and store the reference.
  • the reference is used anywhere in the module to emit a message in the output connection.

A typical usage:

 export class Module extends ModuleFlux {

     output$ : Pipe<number>
     
     constructor( params ){
         super(params) 
         this.output$ = this.addOutput({id:'output'})
         //...
     }
     
     someFunction(value: number, context: Context){
         this.output$.next({data:5, context})    
     }
  }

! ๐Ÿค“ a Pipe is actually a RxJs Subject templated by the Message type

Type parameters

  • T

UserContext

UserContext: {}

Type declaration

  • [key: string]: any

Functions

instanceOfSideEffects

  • instanceOfSideEffects(object: any): object is SideEffects

uuidv4

  • uuidv4(): string

Generated using TypeDoc