Working with the database¶

The feature database is CAMS managing the storage and access of vector data, pieces of information like sensor measurements that can be described in space with a geometric figure such as a points, lines or polygons. See http://wiki.gis.com/wiki/index.php/Vector_data_model to read more about vector data models.

The basic piece of information of this data model is the feature, defined by a geometry that indicates unequivocally its position and shape in the world, and a set of attributes which are the characteristics associated to that location. Related features sharing a common geometry definition and attribute set are usually grouped together.

• Table of contents
• Working with the database
  • Data model
    • Dataset
    • Collection
    • Feature
  • Code architecture
    • Class diagram
  • Code examples
  • Post-mission analysis

Data model¶

The CAMS database model is takes inspiration from the OGR data model and the OGC OpenPackage specification.

Dataset¶

A dataset is encompasses a set of feature collections stored in the same database or file.

Collection¶

A collection describes the characteristics of features of the same kind or category. I.e.: "Wind", "UAV state", "Liquid water content", etc. It corresponds roughly to a table in a relational database or a layer in many geographic information models.

A collection is defined by the following parameters:

1. A computer id (name_id),
2. A human-readable name,
3. An coordinate reference system as an EPSG code,
4. A geometry type (As of 12/2020 only the "point" geometry is supported),
5. A ordered set of attributes and corresponding types (Attributes can be of type int, str, float, or datetime),
6. And, optionally, a long description.

Feature¶

The collection field is used to identify the collection to with a particular field belongs; thus determining the geometry and attribute set.

General attributes, t (time) and producer , are mandatory for features generated repeatedly by UAV sensors. The time attribute is represented by a date and time (datetime.datetime in python) in Coordinated Universal Time (UTC). The producer attribute is a string.

Specific attributes are unique to a particular collection. All features of the same collection must have the same attributes, but features of different collections do not need to share specific attributes unlike general ones. For instance, a "wind" collection can have the "east" and "west" attributes to describe the wind vector components.

Code architecture¶

The GeoPacakgeDatabase and MemoryDatabase provide two alternative feature storage strategies for the FeatureDatabase. The first uses the GDAL/OGR library to write and read GeoPackage files and the second implements a memory-backed database tailor-made to provide fast access to common simple queries. Depending on the request complexity, the FeatureDatabase query method chooses one of the storage backends will use, the MemoryDatabase when possible or the GeoPackageDatabase otherwise.

The GeoPackageDatabase class use Sqlite transactions that can be slow for writing or reading small pieces of data. When writing features it is advised to use the register_features method to delay disk I/O operations and reduce the number of transactions. Anyway, fetching information from the database triggers a write transaction beforehand to ensure data integrity.

The DataServer class receives AircraftStatus and SensorSample objects from the add_sample and add_status events and converts them to database features.

Class diagram
¶

Code examples¶

nephelae_base/unittests/test_feature_database.py provides many examples on using the CAMS database.

Important: You should be aware that neither CAMS or GDAL/OGR sanitize SQL statements. Your program may be the target of SQL injection attacks by malicious users resulting on important data loss and/or serious denial-of-service.

fdb = FeatureDatabase("database.gpkg")  # Create a FeatureDatabase with memory and geopackage storage backends

lwc_attrs = (("t", "datetime"), ("producer", "str"), ("humidity", "float"))
lwc_collection = CollectionSchema(
    "lwc", "Liquid Water Content", 32631, "point", lwc_attrs,
    description="The liquid water content measurements")  # epsg:32631 corresponds to WGS84/UTM31N

fdb.add_collection(lwc_collection)

# Define some features from a liquid water content sensor on UAV "200" 
lwc_feature = Feature('lwc', (360347.0, 4813681.0, 300.0), 
    datetime.datetime(2020, 3, 5, 14, 35, 20, int(123.0 * 1000)),
    200,
    {"humidity": 0.0125})
lwc_feature2 = Feature('lwc', (360347.0, 4813681.0, 300.0),
    datetime.datetime(2020, 3, 5, 14, 35, 20, int(123.0 * 1000)),
    "202",
    {"humidity": 0.0125})
lwc_feature3 = Feature('lwc', (361347.0, 4814681.0, 300.0),
    datetime.datetime(2020, 3, 5, 14, 35, 22, int(123.0 * 1000)),
    "200",
    {"humidity": 0.0125})

# Add them to the database
fdb.insert(lwc_feature, lwc_feature2, lwc_feature3)

# Get all featres from the "lwc" collection
result_iter = fdb.query("lwc") 

# The result is an iterator (the actual reading operation is performed 
# lazily and makes it easier to combine with further filtering code
# without extra memory usage.
list_of_lwc = list(result_iter)  # But you can have a list if needed

# complex_r is a complex request that requires an sql engine to be processed
complex_r = list(empty_feature_db.query(
            "lwc", where="\"producer\" == \"200\"", 
            order_by="t", direction="asc"))

# (minx, miny, minz, maxx, maxy, maxz)
bbox = (lwc_feature.geometry[0] - 0.1, lwc_feature.geometry[1] - 0.1,
        -math.inf,
        lwc_feature.geometry[0] + 0.1, lwc_feature.geometry[1] + 0.1,
        math.inf)
# Simple bounding box request. Fast result from the memory database
bbox_r = list(empty_feature_db.query("lwc", bounding_box=bbox))

Post-mission analysis¶

While GeoPackage .gpkg files generated by CAMS during a mission can be read using this software, it is better to use general purpose geographic information systems or more mature GIS libraries to process the information.

Popular python libraries are fiona —a pythonic style interface to the popular GDAL/OGR library— and geopandas, extending the python pandas library model to geographic data. QGIS is an easy option for non-developpers to visualize geospatial data and visually combine the information with other sources.