Optimizing Autonomous Systems with Dataset Quality Analytics

Safe AD or ADAS vehicles require an excellent perception system to be able to perceive the surroundings, interpret them and move accordingly in a safe way. The foundation of such systems often relies on deep neural networks which are trained on massive amounts of data. As autonomy becomes increasingly viable as a business, with companies like Waymo scaling to over 250,000 paid autonomous rides weekly, the quality of training data becomes even more critical.

Our customers are at the forefront of this revolution, building systems that are reshaping how everything that moves becomes autonomous. When issues in their perception systems arise, they excel at debugging their data pipelines and experimenting with different architectures. However, we've observed that perception system developers frequently struggle with two critical questions when debugging their models:

• Do I have sufficient volume of data to train this model on this scenario or class?

• Is the label quality sufficiently good to train this model on this scenario or class?

In this blogpost we'll focus on the second question, where our newest data quality product plays a crucial role in helping teams determine if they have the right data. And don't worry - we'll address the first question in an upcoming blog post about our Data Coverage product release 👀. Now, let's dive into today's topic. 👇

Is the label quality sufficiently good to train this model on this scenario or class?

The foundation of any autonomous system is trustworthy data that can be used for training and validation. Understanding if annotations are good enough is a challenge that many companies face, and there's often no clear approach to solving it.

Most teams spend considerable time browsing through data trying to assess overall structure quality. At Kognic, we've observed this work is typically semi-structured, identifying mislabeled data points that need correction. While annotations inevitably contain some errors due to their manual nature, there are usually more data irregularities resulting from different interpretations and labeling guidelines - issues that are difficult to spot by simply browsing data.

As autonomy datasets grow exponentially (with fully-featured autonomous stacks producing up to 4TB of raw data per vehicle per day), you need objective dataset quality metrics to determine if your dataset is good enough. At Kognic, we've developed methods to measure dataset quality robustly, enabling us to identify structured errors and help clients understand if their data meets their needs. We're proud to announce that our newest product: Dataset Quality Analytics is now available to help you tackle these challenges. ⚡

Dataset Quality Analytics. What's in it for you?

Licensing our Dataset Quality Analytics product offers significant advantages for teams developing perception systems.

First, DQA eliminates the need to browse massive amounts of data for quality assessment. Instead, you can easily navigate through quality metrics and investigate suspicious ones to understand underlying issues. When metrics indicate room for improvement, the most effective approach is often clarifying guidelines to enhance annotator understanding.

Beyond confirming data quality and identifying issues, DQA helps you discover aspects of your dataset that might limit your training and validation capabilities, and understand why. When analyzing model performance or improvement potential, DQA allows you to leverage quality insights - all through measuring your dataset quality!

Let’s get practical

Consider working on a regression problem to predict object distance or length, using mean squared error (MSE) for model validation. Even with a perfect model, noisy data limits your ability to assess improvements from model tweaks. The example below illustrates this: low-noise data yields an MSE of 0.0090, while high-noise data produces 0.160.

With our dataset quality metrics, you can infer your actual model performance by statistically modeling noise and measuring data variability. In this example, assuming normally distributed noise, calculating actual performance is straightforward: Variance(measured) = Variance(model) + Variance(data).

Here's another example: if you validate an object detection system with an 85% detection rate against test data, this might seem adequate. However, if your test dataset contains only 90% of all objects, your true system performance drops to 0.85 x 0.9 = 0.765 - significantly lower than measured. To accurately assess your model, you must incorporate dataset quality.

Real-world scenarios require sophisticated statistical models to draw similar conclusions, but the principle remains: acknowledging data variability is essential for understanding true performance and identifying quality limitations.

The efficient way to correctly interpret your model performance and draw informed conclusions

Dataset Quality Analytics helps you identify your true model performance and the upper limit on what you can expect to achieve. This prevents wasted time trying to improve models where measurable improvement isn't possible. Hello, efficiency 👋!

Our dataset quality metrics help you understand if dataset quality constrains your system development capabilities. By accessing our statistical quality models that support common perception modalities, you can ensure your models undergo exhaustive validation against real-world applications, increasing engineering velocity and aligning with our mission to be the leader in autonomy data annotation.

So we ask: are you already empowering your team with this knowledge? We understand building safe perception systems takes time and requires the right tools. Once you equip your engineers with powerful tools like Dataset Quality Analytics, true progress becomes possible. We're here to support your journey toward creating safer, more reliable autonomous systems.