Generating a convincing voice clone now takes seconds, and its impact has the potential to influence civic conversations and change election outcomes. That puts more pressure on content moderation teams. Reacting too fast risks silencing honest discourse, but a delay could allow fake posts to mislead the public.

“The operational burden of defending this ecosystem ultimately rests on “human-in-the-loop” (HITL) moderation teams within business process outsourcing (BPO) partners,” says Gabriel Camargo, Senior Researcher at TaskUs, in a white paper. “BPO frontline teams often help operationalize client policies through contracted review and escalation processes.”

Stopping deception while protecting legitimate voices

Guarding against synthetic manipulation while protecting civic conversations is an intricate balance. On one hand, generative AI’s (gen AI) speed enhances communication. On the other, its ability to produce highly realistic media makes identifying deception even harder.

Content spans a wide spectrum of intent. Campaigns use AI to translate outreach materials for non-English speaking constituents and bad actors deploy voice clones to misinform citizens. The gray area in between (e.g., synthetic satire, hyper-personalized messaging and uncaptioned candidate avatars) needs nuanced evaluation.

“Ultimately, content moderation should be designed and implemented to reduce the risks of over-enforcement, perceived bias or the unwarranted suppression of legitimate civic discourse, while recognizing that moderation at scale involves judgment and cannot eliminate all errors,” says Gabriel.

Why automated software cannot catch everything

Automated detection tools cannot fully catch synthetic media on their own. Gabriel points to three reasons.

First, automated scanners search for digital artifacts in images and sound clips, but gen AI models evolve so quickly that detection models struggle to keep pace. This degrades precision and leaves genuine news coverage or authentic personal opinions vulnerable to wrongful censorship.

Second, digital tracking tags (like C2PA) can confirm where a file came from, but actions like taking a screenshot, compressing a file or re-sharing a video often strip or corrupt this metadata.

Finally, bad actors exploit public skepticism around deepfakes by organizing mass-flagging campaigns against authentic, unflattering news clips, tricking automated systems into taking them down.

“Treating detector output as conclusive evidence can lead to incorrect action against legitimate political discourse,” Gabriel explains. “Detector output should therefore be treated as one signal in a documented review process rather than as the sole basis for removal.”

Fragmented and shifting regulations further complicate the process. Federal regulators take divergent tracks. Agencies like the FCC penalize AI-generated robocalls that confuse voters, while campaign finance authorities debate oversight for online ads. Meanwhile, individual states continue to enact a conflicting patchwork of disclosure mandates and pre-election restrictions.

“The number and status of these laws change frequently, and requirements differ by jurisdiction, covered medium, speaker, intent, timing, exceptions, remedies and constitutional posture,” notes Gabriel. “This state-level patchwork creates complex compliance dynamics for platforms and BPO teams.”

Empirical research also reveals a “disclaimer effect:” applying mandatory AI warning labels on political material can actually lower voter trust and candidate appeal, even when the underlying message is authentic.

Four practical steps for review teams

Gabriel recommends the following steps to platforms and their moderation partners:

Focus on verifiable facts over intent: Workflows should guide reviewers to evaluate clear factual details — voting dates, times, polling sites — rather than trying to guess a creator’s intent. Where claims are clearly inaccurate, platforms can apply proportionate measures like context notes, reduced reach or removal, while keeping space open for satire, opinion and standard outreach.

“Disclosure can be considered where required or appropriate, but labeling should not be treated as a universal legal safe harbor,” he notes.

Set time-limited emergency reviews: During peak periods, like the days right before an election, fast escalation paths help teams handle urgent, high-risk cases quickly. Any temporary pause on a post should include a set expiration time so routine discourse isn’t left in limbo while voting is underway.

Queue telemetry as real-time intelligence: Integrate frontline queue metrics into overall risk monitoring. Sudden spikes in flagged audio clips or clusters of localized content can be early warnings for coordinated campaigns, enabling teams to step in before posts go viral.

Review workflows should also prompt frontline staff to record standardized metadata during their checks. Gabriel explains: “Collection should be limited to information necessary for the defined purpose and governed by applicable privacy, security, access, retention, localization and client-contract requirements.”

Safety-by-design platform controls: A strong defense begins at the architectural level. Emerging research suggests four key steps: treating posts as claims rather than facts, weighing alternative explanations, avoiding binary (real or fake) decisions and redesigning feeds so algorithms stop rewarding outrage with reach.

“This approach is intended to reduce, but cannot prevent, the ingestion or propagation of inaccurate information,” says Gabriel.

The future of human-in-the-loop AI moderation

With technology only getting more sophisticated, Gabriel suggests that platforms have a civic duty to respond. “The integration of generative AI into the electoral process calls for consideration of proactive governance, operational, and technical controls,” he says. “Organizations should adopt controls proportionate to their role, technology, jurisdiction, and risk profile, and should test whether those controls operate as intended.”

For more on building resilient systems, enforcing objective policies and equipping skilled teams with actionable threat intelligence, read Gabriel’s full whitepaper: Safeguarding Civic Communications: Balancing Automated Moderation and User Expression in Generative AI Ecosystems.