Why LiDAR Data Organization is Critical to Automation

In a previous post we talked about the Stages of a LiDAR Survey, in each of the stage it’s important to remember that errors in a previous stage will compound as they move through the workflow creating additional (and larger!) errors downstream.

Considering this and the cost associated with acquisition (particularly with airborne systems) one can see how much business risk an organization is exposed to if they do not properly manage & organize their data.

To reduce this risk, GNO-SYS recommends that clients invest in implementing a standard data organization scheme that begins as soon as data is extracted from the acquisition platform and is carried all the way through into the production process.  Having a consistent organization scheme will minimize risk of data loss, while also allowing clients to easily scale up their acquisition to multiple platforms.

Raw Data Payload

A raw ‘raw data payload’ is all of the data from an acquisition session (flight, drive, etc) bundled nicely into a folder structure. The critical thing is to maintain a consistent folder structure from one flight to another as this enables programmatic traversal of contained data and downstream automated processing.

Generally, each payload needs to be assigned a name that is unique across the survey.  This can usually be achieved through a combination of acquisition platform name and time of survey session.  Ensuring each payload has a unique name enables acquisition teams to combine multiple payloads onto a single drive when transmitting data offsite.  Here’s a simple example payload for a platform called ‘PLATYPUS-A’ that has one LiDAR sensor, two cameras, and IMU/GNSS:

Extension and Automation

Implementing a folder solution like the one above means we can easily extend the format to include as many on-board sensors as necessary and that each sensor can maintain its own organization scheme.  Further, an automated quality-control solution can be run on a series of raw data payloads to programmatically check the quality of each data product, relieving the field team of having to do so manually.

Generally speaking, a ‘ready data payload’ would mirror the format of the ‘raw data payload’ but with more data available within it (such as LAS/LAZ files or processed JPEG imagery).

Once the quality of the raw data has been validated then any downstream preparation processes can we performed in a semi- or fully-automated way. Since the folder structures are consistent these data preparation processes only need to know which payload it’s working on.  This reduces the overhead required to prepare and ingest data while also enabling automated, cloud-based solutions to handle any incoming payload.

With a consistent raw data payload format we can lever the organization scheme to automate post-processing and validation, as well as automatically generating a manifest of all contained data products at time-of-extraction.  This allows us to then ensure every single file is present and un-changed when data payloads are received offsite.

Thinking about the data structures involved in LiDAR data collection may seem basic, but having a consistent and reliable system for organizing and managing your data is the first step in automating the processing.