The problem of detecting / segmenting features from Earth Observation (EO) imagery without human involvement and at scale is addressed in this report.
NJTPA wants to be at the forefront of conducting research for advanced feature extraction methods based on Machine Learning (ML) and Artificial Intelligence (AI). At the same time NJTPA understands that automation is required for the organization to be able to roll out ML models and AI methods in production.
In this report, we address a use case that has attracted recently the attention of urban and suburban planners, robotic delivery companies and other commercial entities: sidewalk mapping in the NJTPA region. There is no comprehensive database of sidewalks in the NJTPA region while feature mapping methods require the involvement of GIS specialists and consultants.
Premise of the project
The sidewalk mapping project delivered in June 2020 a Proof of Concept (PoC) that was able to map sidewalks using a deep learning based approach in one NJ county. The second phase of the sidewalk mapping project started in Jan 2021 and ended in May 2021, inheriting the PoC implementation. Upon closer inspection of the PoC, it became clear by Feb 2021 that we need to re-engineer the solution to:
Allow NJTPA to create GIS data catalogues at scale. A resilient data storage facility was needed to be able to store more than 2.5 TB of imagery and metadata information that was required just for this project. It is noteworthy that ML/AI requires batch processing of such data in data pipelines and that all training data must be accessed in a so called model training job such that a machine can learn how to map features. This is in contrast to serving GIS data for pure visualization purposes in GIS desktop tools such as ArcGIS Pro. Therefore the data pipelines are very complex and we need horizontal scaling for their implementation.
Allow NJTPA to use very high resolution imagery (eg. 1ft) for complex feature extraction. High resolution requires high end accelerators Graphical Processing Units (GPUs) with significant memory footprints for efficient training and points to the requirement for vertical scaling.
Allow NJTPA to have an economically viable solution for its production deployment. As the previous point made it clear, employing high end accelerators 24/7 is not acceptable both from the cost and the application resiliency perspective. This means that for production (inference) the solution must be able to run in a serverless infrastructure.
Allow NJTPA to operate the system in a loop. Data is never a static asset. New imagery is constantly made available and the system must be engineered to be able to react to these events - these systems are called reactive for this reason. We need to be able to feed new imagery into the system and obtain updated maps without any human involvement.
Delivered Outcomes
We have adopted AWS as our development, QA and production environment. NJOGIS has partnered with AWS and hosts some of the NJ imagery in AWS S3 (simple storage service) as described in the Datasets section of this report and AWS was therefore a natural choice given the experience of the development team with it. We uploaded into S3, regarded as the first layer of a data lake (see below), all PoC data that were stored in a local workstation at NJIT.
We created a data lake based on AWS Lakeformation to be able to experiment with data cataloging. We selected to not deploy the data lake as there is a strong linkage between application APIs and data. For example, NJTPA may want to create a temporal catalog to satisfy differential queries such as those highlighting the feature differences between two points of time (eg before and after an event). Last but not least, a data lake design is not only a sidewalk-only discussion but it depends on stakeholders within the NJTPA organization.
We have rewritten from scratch the PoC code that extracts sidewalks from the 1-ft imagery and replaced the model with
Yolact++, a model that offers improved performance relative to the earlier PoC model. We rolled out a plugin architecture where the encoder and other neural components can be swapped out easily and tried out. Its not only data that is constantly changing - ML / AI is making advances almost daily and having an environment where a higher level interface can call model-specific containers is needed.We revamped the documentation that is now in an industry standard format (markdown) and can be synthesized. In addition, every API implemented is automatically documented and always up to date.
We authored a Command Line Interface (CLI) that can be used to trigger the training and inference pipelines from a terminal inside a docker container. The investment on the CLI will prove worthy when the system moves into production.
Closing, we want to highlight automatic feature extraction can bring to NJTPA. With the proper cloud-driven engineering and mindset it is feasible to create an architecture where ML models are continuously updated with newly arriving data while AI uses the models to make better more accurate world models (digital twins) of any geographical region and simulate physical phenomena on such models (such as impact of weather).
Attributions
Several people have contributed towards this work. Ning Huan and Xinyue Ye, Ph.D developed the original implementation of the sidewalk mapping in 2020 based on a desktop environment. In the spring of 2021 Pantelis Monogioudis and Richanshu Jha re-coded the original implementation and engineered the ML pipelines to make them operational in the cloud (AWS EC2/S3). In the spring of 2022 the codebase was further enhanced with the adoption of the Auraison platform as the hosting environment - based on a Hanshicorp Nomad cluster. This allowed the project to infer sidewalks for the whole state of NJ that involved processing several TBs of cloud optimized GeoTIFFs. We received strong technical and financial support by Rick Cippoletti (NJTPA) and project management support by Munir Cochinwala. DVRPC contributed data from their extensive annotation exercise in the DVRPC region - we are thankful for their contribution to this project.