Provide a proper databse backend for the dataserver
The NephelaeDataServer class implementation manages data samples on an ad-hoc basis.
- The current data model is 4 ordered lists with duplicated data
- Reading requires a bisection search O(log(n))
- Adding new samples requires a list insertion operation
- The search and insertion process is coded in python
- No long-term data storage. Currently data is stored as a pickle memory save.
CAMS should use an existing data model and save format to ensure long-term data availability and interoperability with other geographic information systems.
Use the standard OGC backed GeoPackage format for storage and an existing library to manipulate this file format:
- GDAL/OGR (lowlevel C library with python bindings)
- fiona (pythonic program interface to OGR)
- GeoPandas (Pandas-like interface to fiona)
Only GDAL provides the most fine-grained and most feature-full interface to GeoPackage at the expense of a rough API. Fiona is an interesting alternative but the current stable version lacks support for "select where" filters required to mimic the incumbent NephelaeDataServer search functionnality (the "2.0dev" version can do this nevertheless). Geopandas is better suited for post-mission analysis.
Updated by Rafael Bailon-Ruiz almost 3 years ago
- Status changed from New to In Progress
- % Done changed from 0 to 80
A new database format —based on GeoPackage and for vector features— is implemented in the
new_database branch. The new functionality is coded in the
Data sample insertion and retrieval are implemented as well as the integration with the dataviews and the GUI.
The scenario replay script must be rewritten from scratch as it depends on the .txt representation of a
NephelaeDatabase to work.