Deep learning solutions in CCTV pipe inspections

Recently, I wrote a piece describing some of the machine learning challenges which I’d encountered during my time working with stormwater and sewer pipe inspection footage at VAPAR. Like any other industry, pipe infrastructure brings its fair share of issues that need to be resolved if valid and accurate results are to be obtained from AI models and provided to clients.

In this piece, I’d promised to also outline the deep-learning (which I’ll do here) and computer vision challenges (which I’ll be doing soon) that I’ve encountered during my VAPAR tenure so far.

So, with no further delay, let’s get into the deep learning struggles!

Correct Defect Identification (and Mixtures of Defects)

In total, we identify and categorise 80+ kinds of defects in pipe infrastructure – these are guided by the regional standard defect codes, such as those from Australia, NZ and UK. Since many of these defects are extremely similar, or define levels of severity within a defect type, distinguishing the difference between them is not an easy task. Even an experienced, specialised asset engineer won’t enjoy a high rate of success (and anecdotally the average accuracy industry-wide is 50%). With this in mind, it’s clear that performing this function accurately is also going to be challenging for an AI-based system.

Detecting the Proper Scale of a Defect

When a contractor records a defect, they will sometimes zoom in to inspect the defect more closely. Since our AI models identify and classify defects based directly from this footage, instances of zooming can lead these models to incorrectly classify defects as being larger than they actually are as a result of zooming. In turn, this will affect condition scores and repair recommendations which we would provide to a client.

Localization of defects

Another deep learning challenge which we encountered related to the localisation of identified defects. Localisation refers to the precise physical location of an identified defect within a pipe. Since the footage provided to us by clients does not contain any data relating to localisation or telemetry, we are immediately presented with a challenge – providing information relating to defect location through an AI model.

So, how did we overcome these problems?

Firstly, let’s discuss overcoming the challenges associated with correct defect identification, as well as instances of multiple defects within a single frame.

Solutions – Correct Defect Identification (and Mixtures of Defects)

For pipe inspections, VAPAR uses various defect classifications, initially identified by a pre-trained deep learning mode. However, based on my previous experience, I knew that only utilizing transfer learning and fine-tuning techniques, we could not achieve optimal results. 

Features in the final layer of our pre-trained model (before the classification layer) usually have small dimensions, which is not suitable for our application. Since we deal with a large range of defect types, the core problem to solve was establishing how to alter the layers in our pre-trained convolutional neural network to achieve optimum results. We eventually managed to achieve this by combining our domain and application-specific knowledge with our nuanced understanding about the deep learning and convolutional neural network.

Imbalances in data was another issue when aiming to optimize our defect identification. The graph above illustrates the distribution of training data which our deep learning model utilised. In this graph, it’s clearly evident that certain classifications are far more prevalent within the data set, which influences the results of the model toward those categories most commonly represented in the data set – potentially skewing classification when performing inspections for clients.

To resolve this imbalance, researchers and experts typically use two techniques – balance sampling and weighted loss function. In this instance, we utilized the combination of these two techniques to help us take the most out of our model and improve performance by around 25 percent.

To combat the issue of having multiple defects in one frame (see the image below for an example), we combined the results of our deep learning model with our machine learning model, and developed an AI-based algorithm to effectively account for instances of multiple defects.

Solutions – Detecting the Proper Scale of a Defect

To correctly determine the proper scale of identified defects, we defined and developed a new AI model, taking the benefits of the three most important AI techniques (computer vision, machine learning and deep learning). First we developed a deep learning-based solution for measuring the scale of the defect in each relevant frame. Then we utilized a machine learning solution to find similarities between different frames. Within the machine learning model, we utilized computer vision techniques to provide the data required for the model. Successfully executing this solution allowed us to deliver strong performance when dealing in accounting for camera zoom to correctly capture the scale of defects.

Localization of defects

Personally, I found solving the issues around localisation to be the most satisfying resolution of all those which I’ve outlined so far.

With available deep learning and image segmentation techniques along with the right dataset, localization is an achievable task to undertake. However, most industrial projects (like ours) carry huge time and cost requirements if this data is to be provided, 

This left us at a crossroads – do we abandon this functionality for our clients? Perhaps, an ordinary team might, but I’m proud to say our innovative and driven team managed to come up with a fantastic solution, using the latest state-of-the-art innovation in the deep learning discipline.

The images below illustrates some of the results obtained with our solution. 

Original Image	Localisation data

For me, the coolest part of our innovative solution is that we can classify and localize the defect at same time without any memory or time cost.

=================

Interested in learning about the computer vision challenges in CCTV pipe inspections (and how we’ve overcome them)? Stay tuned for future blogs!

Alternatively, check out the piece we already completed relating to machine learning challenges.

=================

Saeed Amirgholipour PhD. is an AI Architect, Full Stack Data Scientist, and Data Science/ AI Lead Trainer with over 10 years of industry experience, including CSIRO’s Data61, Australia’s leading data innovation group. His experience spans end-to-end large-scale innovative AI, Data Science, and analytics solutions. Saeed has a passion for solving complex business problems utilizing Machine Learning (ML) and Deep Learning (DL) models.