When we model and optimize energy consumption of ships we typically deal with large geospatial and temporal datasets. For instance, most commonly we need to find the values of various meteorological forecasts along the whole of a vessel’s route. Such a route may span across oceans and can take many days for a ship to complete. Also, we would like the data to be delivered on-board as fast as possible. Suffice it to say, our production environment needs at least robust and reliable storage of the data and effective access to it.
Earth is not flat, although, on many occasions, I wished it was. Geometric calculations such as finding the distance between two points or the closest distance between a line and a point are simple on flat surfaces, but things become much more complex on a curved surface. Instead of geometric, these operations are called geodetic calculations.
In the previous part of this series, we introduced Lusmu, our minimal reactive/dataflow programming framework in Python. We clarified dataflow terminology and some of our design choices using simple usage examples.
In this part, we create a stripped-down implementation of the framework to highlight some characteristics of the framework:
- Synchronous Activation
- Pull Data (invalidate / lazy-revalidate)
- Doubly Linked Adjacency List as the graph data structure
At Eniram, we help shipping companies reduce fuel consumption using a data-driven approach. Our servers crunch measurement data for hundreds of variables from hundreds of ships sailing around the world.
We validate incoming data, and perform analysis and statistical modeling based on the data. In some cases, processing takes place on board ships without connectivity or on workstations of data analysts.
It’s desirable to have one unified toolchain for data processing. The tools should enable non-programmers to specify rules for data validation and analysis.