The quality of your machine learning outcomes depends directly on the quality of your data. At Kognic, we've established ourselves as the price leader in autonomous driving annotation by mastering three essential elements: Platform, Processes, and People. Our focus on these three pillars allows us to deliver what matters most - ensuring customers get the most annotated data for their budget while maintaining exceptional quality standards.
When setting up your annotation project, the first crucial question to address is whether you're annotating sensor data or the underlying reality it represents. Many assume annotations automatically provide "ground truth," but this isn't always the case. What human annotators see is merely a snapshot of reality captured through sensors (cameras, LiDAR, etc.) - each with their own limitations and distortions.
Sensor data inherently contains various distortions that differ between camera and LiDAR systems, and even vary across different models and manufacturers. While some distortions are easily identified and corrected, others present more complex challenges. Let's examine three common phenomena that impact annotation accuracy in autonomous driving applications:
Camera images often suffer from the rolling shutter effect, beautifully demonstrated in this video from the 'Smarter Every Day' YouTube channel. This effect causes moving vehicles to appear wider or narrower depending on both camera shutter speed and vehicle velocity. Even the most meticulous annotation cannot provide a true representation of reality without accounting for this distortion.
LiDAR sensors face similar challenges since they scan environments sequentially using laser beams. Depending on the scan pattern, moving vehicles can appear strangely deformed - making it challenging to accurately estimate their true dimensions.
The images below compare camera view with Luminar LiDAR point cloud data from the Cirrus dataset. Note how the approaching van appears elongated in the point cloud with diagonal outlines at both ends - a perfect illustration of how LiDAR data doesn't perfectly mirror real-world objects.
When your vehicle has multiple LiDAR sensors and is scanning moving objects, you'll often see the same object captured at different positions within your point cloud. This occurs because different sensors detect the vehicle at slightly different times and locations. This multiplicity further complicates accurate vehicle length estimation.
This phenomenon is clearly visible in Audi's A2D2 dataset, which utilizes five LiDARs. This video demonstrates the effect dramatically when focusing on the bright points representing a vehicle's license plate.
LiDARs measure distance by calculating how long light beams take to return after emission. One significant challenge is that highly reflective surfaces often appear larger than their actual size - known as the blooming effect. While LiDARs typically maintain precision within 2 centimeters, this effect can lead to overestimation of reflective object dimensions.
The image below illustrates this phenomenon, where points to the right of the truck represent LiDAR noise rather than the actual vehicle. Untrained annotators might incorporate these points into their bounding boxes, resulting in width overestimation.
Blooming effect - Kognic Platform
This brings us to our core question: what exactly are you annotating - sensor data or reality? The approach fundamentally changes how annotations are created:
Annotating sensor data: Drawing bounding boxes around all pixels or points belonging to an object, regardless of distortion.
Annotating reality: Drawing bounding boxes where objects actually exist, potentially excluding some sensor points identified as noise.
At Kognic, we invest heavily in our Annotator Academy to train professionals to distinguish between actual object data and sensor artifacts. By specializing exclusively in automotive data, our annotators deliver higher quality results that more accurately reflect reality.
This distinction leads to another critical consideration: what level of annotation precision can you realistically expect?
Understanding the difference between annotating sensor data versus reality prompted us to investigate annotation precision limits. Since comparing annotations to actual geo-referenced trajectories isn't always feasible, we developed an alternative metric: inter-annotator agreement - measuring how consistently different annotators label the same data with identical information access.
We conducted an experiment where trained annotators independently labeled the same highway LiDAR sequence twice, allowing us to calculate agreement levels for various vehicle dimensions.
The technical challenge stems from LiDAR's inherent sparsity. For instance, a Luminar Hydra with 0.07-degree horizontal resolution provides points spaced 12.2 centimeters apart at 100 meters distance. When annotating vehicles traveling toward you at highway speeds, this creates fundamental precision limitations.
Our findings confirmed that width and height measurements showed strong inter-annotator agreement, with discrepancies rarely exceeding 30 centimeters. However, length estimation (more challenging from a rear view) occasionally showed disagreements up to 80 centimeters - revealing important insights into annotation reliability thresholds.
Some annotation tools attempt to bridge the reality gap using more sophisticated methods. The "nyc3dcars-labeler" developed by Kevin James Matzen for the NYC3DCars dataset aligns images to real-world coordinates and places 3D CAD models on the ground plane to estimate object positions.
While this approach can increase confidence in annotations, it also introduces risks by potentially creating false precision. The paper "3D Object Detection and Viewpoint Estimation with a Deformable 3D Cuboid Model" demonstrates this hazard - initial results appear impressive, but closer examination reveals incorrect vehicle orientations and classifications that aren't immediately apparent.
3D Object Detection and Viewpoint Estimation with a Deformable 3D Cuboid Model - results
At Kognic, we've developed annotation tools supporting both sensor data and reality-based approaches. Our platform combines three essential elements:
We focus exclusively on the automotive sector, continuously improving our capabilities to deliver both sensor-accurate and reality-representative annotations. Whether you should pursue one approach or the other depends entirely on your specific use case.
Despite advancing automation capabilities, we firmly believe human oversight remains essential for maintaining annotation quality and road safety. The world's complexity and ever-emerging edge cases mean fully automated annotation cannot deliver the trustworthiness required for safety-critical applications.
Our fundamental principle is simple: machines learn faster with human feedback. We establish trust through human verification of every annotation, building the control plane for human feedback that routes attention where it matters most while integrating automation wherever possible.
Ready to enhance your autonomous driving development with industry-leading annotation quality and efficiency? Contact us today to discuss how Kognic can accelerate your path to production!