Using Joule

Joule decentralizes signal processing into discrete modules. These modules are connected by streams as shown in the figure below. The interconnection of modules and streams form a data pipeline. A pipeline may execute as a single proces, a collection of processes, or even be distributed across multiple nodes in a network without adjusting any module code.

_images/data_pipeline.png

Joule pipelines are composed of modules and streams

Joule constructs the pipeline based on module and stream configuration files. Files must end with the .conf suffix and by default should be placed in /etc/joule/module_configs and /etc/joule/stream_configs respectively. The pipeline must form a directed acyclic graph (DAG). Circular paths are not allowed.

Modules

Modules are executable programs. Joule runs each module as a separate process. This enfores isolation and improves resiliency. Malfunctioning modules do not affect other parts of the pipeline and can be restarted without interrupting the data flow. The client API provides three basic module types: Reader Modules, Filter Modules, and Composite Modules.

The module configuration format is shown below:

Module Configuration File
[Main] #required settings (examples) exec_cmd = /path/to/module.py --args name = Processing Module #optional settings (defaults) description = [Inputs] in1 = /stream/path/input1 in2 = /stream/path/input2 #additional inputs ... [Outputs] out1 = /stream/path/output1 out2 = /stream/path/output2 #additional outputs ...

See the list below for information on each setting.

[Main]
  • exec_cmd – path to module executable. This can include command line arguments
  • name – identifies the module in Joule CLI output
  • description – optional description used in Joule CLI output
[Inputs]
  • name = /stream/path – association between module input and data stream
[Outputs]
  • name = /stream/path – association between module output and data stream

The input and output names should be provided by the module author. Inputs may be reused between modules, but outputs must be unique. All input and output streams must have corresponding configuration files.

Streams

Streams are timestamped data flows that connect modules. Modules read and write to streams through Joule Pipes. Streams can be visualized as a tabular data structure. Timestamps are in Unix microseconds (elapsed time since January 1, 1970).

Timestamp Element1 Element2 ElementN
1003421 0.0 10.5 2.3
1003423 1.0 -8.0 2.3
1003429 8.0 12.5 2.3
1003485 4.0 83.5 2.3

The stream configuration format is shown below:

Stream Configuration File
[Main] #required settings (examples) name = stream name path = /stream/path datatype = float32 keep = 1w #optional settings (defaults) decimate = yes [Element1] #required settings (examples) name = stream name #optional settings (defaults) plottable = yes discrete = no offset = 0.0 scale_factor = 1.0 default_max = default_min = #additional elements...

See the list below for information on each setting.

[Main]
  • name – stream identifier, white space is permitted

  • path – unique identifier which follows the Unix file naming convention. The web UI visualizes the path as a folder hierarchy.

  • datatype – element datatype, must be one of the following values:

    float32 int8 uint8
    float64 int16 uint16
      int32 uint32
      int64 uint64
  • keep – how long to store stream data. Format is a value and unit.

    Units are h: hours, d: days, w: weeks, m: months, y: years. For example 6d will keep the last six days of data. A value of false means no data will be stored for this stream.

  • decimate – whether decimated data will be stored for this stream. Decimation roughly doubles the required storage but enables web UI visualization.

[Element#]
  • name – element identifier, may contain whitespace
    NOTE: the following settings apply to visualizations in the web UI
  • plottable[yes|no] whether the element can be plotted
  • type[continuous|discrete|event] controls the plot type
  • offset– apply linear scaling to data visualization y=scale_factor*(x-offset)
  • scale_factor– apply linear scaling to data visualization y=scale_factor*(x-offset)
  • default_max– control axis scaling, leave blank to auto scale
  • default_min– control axis scaling, leave blank to auto scale

Command Line Interface

jouled – controls pipeline execution, runs as a system daemon

Command Line:
# use service to control jouled: # NOTE: restart the service to apply configuration file changes $>sudo service jouled [start|stop|restart|status] # by default jouled starts at boot, this can be enabled or disabled: $>sudo systemctl [enable|disable] jouled.service # jouled may be run in the foreground if the service is stopped $> sudo jouled # exit with Ctrl-C

joule modules – view currently executing modules

Command Line:
$>joule modules +-------------+--------------+----------------+---------+-----+ | Module | Inputs | Outputs | Status | CPU | +-------------+--------------+----------------+---------+-----+ | Demo Reader | | /demo/random | running | 0% | | Demo Filter | /demo/random | /demo/smoothed | running | 0% | +-------------+--------------+----------------+---------+-----+

joule logs – view stdout and stderr from a module

Joule keeps a rolling log of module output. By default the last 100 lines are stored, see System Configuration to customize this value.

Command Line:
$>joule logs "Demo Filter" [27 Jan 2017 18:22:48] ---starting module--- [27 Jan 2017 18:22:48] Starting moving average filter with window size 9 #... additional output

joule docs – manage module documentation

Manage the contents of the Module Documentation. See sec-module-documentation for details on writing module documentation.

Command Line:
$>joule docs add joule-random-reader added documentation for [Random Reader] $>joule docs update joule-random-reader updated documentation for [Random Reader] $>joule docs list Documented modules: Random Reader ...other modules $>joule docs remove "Random Reader" removed documentation for [Random Reader]

System Configuration

Joule uses a set of default configurations that should work for most cases. These defaults can be customized by editing /etc/joule/main.conf. Start joule with the –config flag to use a configuration file at an alternate location. The example main.conf below shows the full set of options and their default settings:

/etc/joule/main.conf
#default settings shown [NilmDB] url = http://localhost/nilmdb InsertionPeriod = 5 CleanupPeriod = 600 [ProcDB] DbPath = /tmp/joule-proc-db.sqlite MaxLogLines = 100 [Jouled] ModuleDirectory = /etc/joule/module_configs StreamDirectory = /etc/joule/stream_configs ModuleDocs = /etc/joule/module_docs.json IPAddress = 127.0.0.1 Port = 1234

See the list below for information on each setting.

NilmDB
  • url – address of NilmDB server
  • InsertionPeriod – how often to send stream data to NilmDB (in seconds)
  • CleanupPeriod – how often to remove old data as specified by stream keep parameters
ProcDB
  • DbPath – path to sqlite database used internally by joule
  • MaxLogLines – max number of lines to keep in a module log file (automatically rolls)
Jouled
  • ModuleDirectory – folder with module configuration files (absolute path)
  • StreamDirectory – folder with stream configuration files (absolute path)
  • ModulesDocs – JSON data file for module documentation
  • IPAddress – address to listen for standalone modules
  • Port – port to listen for standalone modules