Joule: Decentralized Data Processing¶
Joule is a framework for decentralized data processing. Joule distributes computation into independent executable Modules that are connected by Pipes which carry timestamped data flows called Streams.
A typical deployment is shown below. Embedded sensors collect high bandwidth data (module 1) and perform feature extraction locally (module 2). This lower bandwidth feature data is transmitted to local nodes that convert this into actionable information by applying machine learning (ML) models (module 3). Aggregation nodes at the data center collect streams from a variety of local nodes and perform more computationally intensive tasks like training new ML models (module 4).
Contributing & Running Tests¶
Contribution is always welcome. Please include tests with your pull request. Unittests can be run using nose2, see joule/htmlcov for code coverage.
$> cd joule $> nose2 # run all unittests
- Quick Start
- Using Joule
- Command Line Interface
- API Reference