Inside Tesla’s autopilot labeling units

Business Insider spoke with 17 current and former employees of Tesla’s data annotation team to understand what it’s like to operate the company’s Autopilot and Full Self-Driving software. The team is split across three different Tesla facilities in Buffalo, New York, Palo Alto, California, and Draper, Utah.

Team projects can last anywhere from months to days and range from workflows that require staff to tag short clips to track still images or overlay satellite data.

“Sometimes it can get monotonous,” said one former worker. “You could spend eight hours a day, for months, just tagging lane lines and curbs on thousands of videos.”

An “intimate look into someone’s life”

The clips can provide a unique window into the daily lives of Tesla drivers. At one point, there was a project that required workers to tag data taken from owners’ garages through Tesla’s Sentry Mode feature, five workers said.

Another project, called “Selfie,” required some annotators to label data taken from Tesla’s in-cab cameras, according to two workers who saw employees working on the project. Four additional workers said they were aware of the program. The Selfie program was designed to teach Tesla’s system how to identify when a driver wasn’t paying attention to the road while Autopilot was in use, they said.

Tesla said the vehicles’ dash cam “shares short dash cam videos with Tesla to help us develop future safety improvements and continually improve the intelligence of features that rely on the dash cam,” according to its owner’s manuals . Tesla owners must first opt ​​in to share their data so Tesla taggers can access the videos, the company says.

In other cases, workers found themselves tagging routes associated with YouTubers and even Elon Musk himself, BI previously reported.

“There’s something very strange about having this very intimate view of someone’s life,” said one current employee. “It seems strange to see someone’s daily commute, but it’s also an important part of correcting and refining the program.”

The videos are taken from all over the US, as well as some regions in Europe and South America, 15 workers said. Two workers recalled tagging videos they believed were taken from customers’ cars in Ukraine during the Russian invasion.

Business Insider reached out to Tesla, Musk and his legal team for comment, but did not hear back before publication.

Workers can find data from any number of countries in a single workflow, meaning they must constantly be aware of the different rules of the road for each region. At times, Tesla appeared to take a more relaxed stance on those rules, seven former and current workers said. For example, some workers said they were told to ignore “No Turn on Red” or “No U-Turn” signs, meaning they would not instruct the system to adhere to those signs.

“It’s a driver-first mentality,” said one former worker. “I think the point is that we want to train him to drive like a human would, not a robot that just follows the rules.”

Sometimes the role requires workers to tag videos of accidents and near misses. Seven workers recalled tagging videos that included Tesla crashes or those involving nearby vehicles. At one point, a worker even shared a video among employees of an incident in which a boy on a bicycle was hit by a Tesla, four workers said. It was one of many videos and memes the workers were swapping, they said.

Last year, Reuters was the first to report on the bike clip and potential privacy issues on annotation sites. Shortly after the article was published, Tesla began restricting access to clips outside of worker-designated projects and added watermarks to some of the videos and images so it could easily track which employees were sharing images, 9 workers told BI .

Tesla employee monitoring systems

Tesla has pretty strict employee monitoring systems at its headquarters in Buffalo. The location has a number of surveillance cameras at the site that overlook the workspace, 11 workers told Business Insider.

Employees are also closely monitored using two different software systems.

One system, called HuMans, estimates how long it should spend on each clip, four workers said. Annotators who consistently take longer than their allotted time are likely to receive poor performance reviews or be subject to a performance improvement plan, or PIP, they said. The software was originally designed to assist U.S. Air Force pilots and also has the ability to track employee eye movement and make audio recordings, according to its website. But it’s unclear whether Tesla uses the software to track staff eye movement.

The company also uses a metric called “Flide Time” to track annotators’ active time on the tagging software, 17 workers said. It can track keystrokes and how much time workers spend with the tagging software open, but it won’t track the time workers spend using other tools on their computer, they said. Depending on their level, workers can be expected to log anywhere from five to seven and a half hours of Flide Time, meaning they must be active on the software for at least that amount of time.

If workers have less than five minutes to reach their designated flight time, they could face disciplinary action, six workers said. If they miss Flide Time three times within six months, they could be fired, the workers added.

Some Tesla workers have tried to push back on the company’s staff values ​​with little success.

In February 2023, some workers at Tesla’s Buffalo facility attempted to unionize. Union organizers at the Buffalo facility told Bloomberg the company is tracking their keys and said some workers are tired of “being treated like robots.”

That same month, Tesla laid off dozens of workers at the Buffalo plant. At the time, the National Labor Relations Board (NLRB) filed a complaint alleging that Tesla illegally fired some of its staff “in retaliation for union activity and to discourage union activity.” But Tesla has denied the allegation, saying the employees were fired due to poor performance. The NLRB did not respond to a request for the current status of the complaint.

When Tesla began building its driver assistance program in 2016, the company outsourced data tagging to a California company that had offices in Kenya, but Tesla brought the program in-house in 2019, Reuters reported.

More recently, Tesla’s Autopilot team was hit with company-wide layoffs in April. Tesla has lost nearly 300 jobs in Buffalo, according to a WARN notice.

Tesla has said that its neural network will be able to train itself one day, but for now it relies on human labor.

The work is vital to Musk’s vision for the car company.

Over the years, Tesla’s CEO has repeatedly emphasized the importance of Tesla’s efforts to achieve autonomous driving. In 2022, Musk said that Tesla’s self-driving technology is “the difference between Tesla being worth a lot of money or being worth basically nothing.”

Tesla plans to unveil its self-driving Robotaxi service later this year, which is expected to be built on the same self-driving software — and, of course, the tedious, clip-by-clip analysis of its taggers.

Do you work for Tesla or have a tip? Contact the reporter via email and a non-duty device at [email protected] or through the secure messaging app Signal at 248-894-6012.