Case Study

A leading autonomous driving company (e.g. Tesla) requires a large image labeled dataset to improve its vehicle recognition algorithms. The task involves labeling various elements in driving scenes, such as cars, pedestrians, traffic signs, and road conditions.

Step-by-Step Workflow:

Step 1: Task Creation

Requester (e.g. Tesla) submits a task on the DataCrowd platform, with specific requirements including:

• Data Type: Images from dashboard cameras

• Labeling Type: Bounding box labeling of cars, pedestrians, and traffic signs

• Quantity: 100,000 images

• Quality Requirement: Labeling accuracy must reach 90%

Step 2: Task Assignment Algorithm

DataCrowd uses a task assignment algorithm to distribute the task to contributors based on their skills, past performance, and engagement level.

Sample Algorithms:

• Skill Matching Algorithm: The algorithm evaluates contributors past performance on similar tasks and matches them to the current task based on their skill level.

• Dynamic Task Assignment: Contributors are ranked based on previous task completion, quality of past labeling tasks (accuracy score) and engagement level (frequency of contributions)

Task Assignment Example: Identifying Contributors A, B, and C as the best candidates:

• Contributor A: 95% accuracy, high engagement

• Contributor B: 85% accuracy, moderate engagement

• Contributor C: 90% accuracy, low engagement

Result: The algorithm assigns 60% of the task to Contributor A, 30% to Contributor B, and 10% to Contributor C.

Step 3: Data Labeling

Contributors use a user-friendly interface to label images, allowing them to:

• Draw bounding boxes around identified objects

• Label each box with the relevant category (e.g., "car," "pedestrian," "traffic sign")

• Provide additional metadata if necessary (e.g., weather conditions)

Step 4: Validation Algorithm

Once contributors submit their labels, the platform evaluates the quality of the work using a validation algorithm.

Validation Process:

• Automatic Quality Check: The system uses algorithms to analyze the consistency and accuracy of the labeling samples. For example, comparing the labels with a pre-labeled data-set, or using object detection models to validate the position of the bounding boxes.

• Peer Review System: A portion of tasks is sent to other contributors for peer review, ensuring an additional layer of quality control

Step 5: Reward Distribution

After the validation process, contributors receive rewards based on their performance metrics through the evaluation and reward system.