The work being done inside content-moderation facilities at the moment has a subtle unsettling quality. It’s not because the individuals there are acting improperly. They are carrying out the exact tasks for which they were hired. What makes you pause is what that work is turning into.
Tens of thousands of people sit at screens in contracted facilities in the Philippines, Kenya, India, and parts of Eastern Europe, evaluating text, images, and videos for platforms that the majority of us use mindlessly. They determine what remains upright and what descends. Even in the absence of the psychological impact of what they are viewing, they label content—graphic, non-graphic, political, benign, and violent—at a rate that would be difficult to maintain. Increasingly, each label they add contributes to a dataset that will eventually be absorbed by a machine learning model.
The harsh truth is that a large number of these employees are training the systems that will eventually replace them. There is no conspiracy. It is the business’s logic.
Some labor advocates and researchers believe that the industry hasn’t been completely open about this cycle. Usually, the pitch to moderators focuses on user protection and safety. which, insofar as it goes, is accurate. However, the long-term goal of creating labeled datasets that eliminate the need for human reviewers is at odds with that framing. As this has developed over the past few years, it’s difficult to ignore the fact that the discussion of AI safety hardly ever returns to the individuals laying the foundation.
Facilities for content moderation are not glamorous. Low natural light, rows of workstations, headphones everywhere, and little break time are common characteristics of the ones that journalists and researchers have documented. In certain places, employees have reported going over hundreds of clips every shift. Some facilities provide counseling services, but employees don’t always feel comfortable using them, and access and quality vary greatly. The content itself, such as images of child exploitation, beheadings, and self-harm, is the type that builds up subtly and doesn’t always become apparent right away.
The explicit incorporation of annotation work into AI pipelines is what has recently changed. Previously, the main goal of content moderation was to prevent harmful content from appearing on the platform. Data labeling, or teaching models to identify categories, patterns, and edge cases, now accounts for an increasing portion of that work. Because the result is more than just a safer platform, the distinction is important. It’s a set for training.
Whether the major platforms have fully considered the implications for their contractor workforces is still up for debate. As their models have improved, some businesses have reduced the amount of human review in specific content categories. It is presented as an increase in efficiency. The framing feels a little different to those whose jobs become more limited as a result.
All of this does not imply that AI-assisted moderation is a bad idea. Automated systems are able to capture volumes that are practically impossible for a human team. The technology is not the issue. It’s whether the transition is being managed with any real recognition of the people who laid the groundwork for it.
A generation of employees reviewed content for years that most people would never see, and their choices influenced how algorithms learned to consider harm. Product announcements typically don’t mention that contribution. It appears subtly in the models themselves.
