Large Scale Annotation for the Future of Autonomous Driving. A Testimonial.

7 minutes read

With more AD/ ADAS platforms, sensors and self-driving algorithms trying to make it onto the market, the future of autonomous driving is being decided in large AI training and validation projects. To develop and test a self-driving AI requires more data, more engineering and more demand for data labeling services than the automotive OEMs can handle alone. That’s why one of the world’s largest Engineering and R&D companies L&T Technology Services (LTTS) teamed up with understand.ai (UAI) to give the automotive customer the much needed edge to lead the autonomous driving race. LTTS’ world class services propelled by the UAI annotation automation empower the largest car makers in the autonomous driving market today.

L&T Technology Services Autonomous Driving & ADAS

Highly Optimized Imaging Algorithms for Autonomous Driving

LTTS tapped into the ADAS perception market early, exploring their own autonomous driving functions, experimenting with object detection, navigation and road simulation. So when the first requests came to label data for self-driving AI, LTTS had already built up the project expertise necessary to support their automotive customers. LTTS knew there were 3 critical points that determine the success of a large scale AD/ADAS annotation project.

1. Tooling ready to scale

Not every labeling tool is equipped to handle complex annotations in large teams. What are the must-haves? High throughput of annotated data, split tasks, smart workflows organizing the work of hundreds of labelers and a detailed overview over performance and cost targets.

2. Risk diversification

Once the stakes (and the data volumes) are high, trusting a single tooling supplier can jeopardize the entire project and, eventually, cost you the trust of your end customer. Any annotation tooling bottleneck or issue is impacting LTTS throughput and the delivery deadline. It’s smart to have a back up.

3. Flexibility and agile customer service

Autonomous driving is a volatile and highly competitive market with an increasing rate of project complexity. Customer demands and specifications change frequently. If the supplier is not able to keep up and adapt, the customer dissatisfaction can cost you the whole business.

Picking the Right Annotation Tooling

Let’s focus on a particular AI training project of a major car manufacturer. The annotation scope covered 7 use cases:

1. 2D bounding boxes

2. 3D sensor fusion annotation (LiDAR & Camera)

3. Polygones

4. Polylines

5. Lane change

6. Semantic segmentation

7. Speed change/ speed limit annotation

The annotated data quality target was set at 99% with throughput aimed at 3600 sequential frames per labeler per month. Since LTTS didn’t have an internal annotation tooling technology, they decided to partner up with a tool supplier to fill the gap. The LTTS domain architect crafted the requirements and measurement criteria for the tooling, some of which were:

  • Throughput - how much data can the tool handle per year
  • Quality - the tool has to support for the final 99% quality target
  • Sequential data tracking
  • Possibility to break down the annotation process to smaller tasks be handled by multiple labelers
  • Workflow management capability to support 500-600 people working on the same labeling project
  • Advanced dashboards to keep a detailed overview of the project

Despite not participating in the tooling selection round, understand.ai’s UAI Annotator was added as the second annotation tooling for 3D sensor fusion later. LTTS discovered UAI and the tool’s potential in a smaller project with a different customer.

UAI-Annotator-workflow-validation-project
UAI Annotator workflow management

What were the reasons to take a second tooling on board at all?

  1. In order to diversify the risk, LTTS decided not to depend on one single tool for such a large piece of business. Any stop in the data production line would bring the entire project to a standstill.
  2. The original 3D tooling was facing recurring throughput issues. A lot of effort was invested into the project without tangible results such as realized annotated volume. Engineers were sitting idle because of the lack of incoming data. And since the customer pushed for outcome and deadlines, an alternative had to be found.

When it works

A new benchmark was soon established. Impressive throughput outcomes were accompanied by the ability to understand requirements specific to autonomous driving and smooth adjustments to the tooling. UAI’s annotated volume grew first to 50% and within the next month, UAI Annotator became the sole tooling used to label 3D bounding boxes for the multi-year AD/ADAS training project. Why?

How annotation automation changed the game

It’s all about the business and annotation is not much different. You try to maximize your output at the lowest cost possible. Despite the latest innovations, labeling still requires manual work. It’s the biggest item of the cost structure. And while increasing the efficiency of the labeling process is vital, the only way to make a decisive leap is automation.

UAI tools are built to take advantage of artificial intelligence and constantly increase the automation rates. During benchmark tests at stable conditions, the throughput of UAI Annotator increased by 380% compared to the tooling that won the initial tender, all due to algorithmic annotations and tool features.

Throughput increase with UAI Annotator - annotation automation
3D bounding box per labeler per day

The ADAS business has become more competitive and the customers are asking for increasing levels of automation so they can reduce their costs. Even though the focus is on manual labeling services, LTTS is constantly on the lookout on how to reduce resources. UAI helped to raise LTTS’ overall annotation efficiency by 50% despite changing inputs and conditions. The potential of automation is great, once the labelers are aligned to a steady process and the sensor set-up, the automation rates go up.

When There’s No Standard Project - Annotation Tooling with Expertise

Autonomous driving technology is still quite new and constantly evolving, making the training and validation projects prone to frequent changes and customizations. However mature the annotation tooling, customers will still require expertise - people with sound understanding of their requirements and the ability to make the right adjustments. Collaboration with UAI was always based on delivering the best-in-class expertise and on the willingness to come up with a solution.

UAI Annotator Performance Dashboards - annotation automation
UAI Annotator performance dashboards

And there’s more. Good technical support and customer service is critical. Fixed stuff that doesn’t break again is also critical. Tool features, its scalability and flexibility is of paramount importance. All reasons why LTTS decided to expand UAI tooling coverage to 2D bounding boxes, 2D polygons, 2D polylines, lane change, 2D semantic segmentation, speed change/ speed limit annotation going forward.

Future of Autonomous Driving is in Automated Data Pre-labeling

LTTS predicts huge potential for autonomous driving and AI based data annotation. There’s a long way until the L5 cars are on the road, but the road to full autonomy will be shaped by more intelligent, more automated tools, capable of automated pre-labeling. LTTS sees UAI as a strategic partner capable of such paradigm shift in annotation technology. Understand.ai is a partner who is ready and excited to get things done to win in this business.

Indrajit Sen, Vice President and Regional Head of DACH & EE at LTTS

About LTTS

L&T Technology Services Limited is a global leader in Engineering and R&D (ER&D) services. With 550 patents filed for 53 of the Global Top 100 ER&D spenders, LTTS lives and breathes engineering. LTTS’ expertise in engineering design, product development, smart manufacturing, and digitalization touches every area of human lives. L&T Technology Services helps automotive industries to enhance the safety and driving experience with vision-based ADAS systems.