OpenAI’s new video app, Sora 2, was billed as a inventive leap ahead in AI—a software meant to show textual content prompts into richly detailed movies. However lower than two weeks after its debut, the app’s feed is exhibiting customers antisemitic movies and different troubling content material.
In clips reviewed by ADWEEK, Sora 2 generated a collection of antisemitic movies, together with one exhibiting a person carrying a kippah sinking into piles of cash. The video stemmed from a remix immediate of a lady standing in a home flooded with coke: “Exchange her with a rabbi carrying a kippah and the home is stuffed with quarters.” The video makes use of an AI-powered function that enables customers to edit current movies by including, eradicating or altering objects and environments utilizing prompts. As of Oct. 17, the video—together with its variations resembling a South Park model—had greater than 22,000 likes and over 3,400 remixes.
One other video depicts two soccer gamers carrying kippot, flipping a coin earlier than a 3rd man—portrayed as a Hasidic Jew—dives to seize it and sprints away, an obvious reference to longstanding antisemitic stereotypes about greed. The clip has been broadly remixed with practically 11,000 likes as of Oct. 17.
“No enterprise has an obligation to provide antisemitic content material on demand,” stated Imran Ahmed, chief government of the Heart for Countering Digital Hate. “It’s simply one other instance of how briskly and unfastened OpenAI performs with making certain that their companies don’t trigger real-world hurt.”
An OpenAI spokesperson informed ADWEEK that Sora 2 is constructed with a number of layers of security and transparency options to mitigate dangers, together with bias in AI outputs associated to physique picture and demographic illustration.
The spokesperson added that the platform makes use of structured suggestions loops, refined prompts, and proactive detection programs that scan video frames, captions, and audio transcripts to flag and block problematic content material. Inside groups additionally monitor developments and modify safeguards to make sure outputs stay balanced and inclusive.
The spokesperson additionally acknowledged that overcorrecting—resembling eradicating or underrepresenting sure teams in an effort to keep away from bias—can create new harms, resembling erasing sure teams or views.
As of publication, the movies are nonetheless circulating on Sora 2.
ScreenshotScreenshotScreenshot
Some viewers have cheered on the movies, leaving feedback resembling “These magnets have been promised to him 3,000 years in the past” and “The antisemitism bout to get one other degree, huh.” Others have condemned the movies, with one person writing, “Simply reported. Horrible stereotype.”
Sora 2 debuted on Sept. 30 as an invite-only app, and isn’t obtainable to the general public. It had thousands and thousands of downloads in its first 5 days, surpassing the obtain fee of OpenAI’s flagship product, ChatGPT, in response to Invoice Peebles, OpenAI’s head of Sora.
Sora 2’s feed options not solely antisemitic tropes but additionally copyrighted characters resembling SpongeBob SquarePants in a Nazi uniform, depictions of deceased public figures together with Queen Elizabeth and Princess Diana, and graphic scenes of violence and racism, The Guardian beforehand reported.
The controversy comes amid broader considerations and debate about how successfully OpenAI enforces its guardrails throughout merchandise. On Oct. 14, CEO Sam Altman stated in a put up on X that erotic content material can be permitted in ChatGPT beginning in December, a shift that might additional take a look at the corporate’s content material moderation programs because it pivots to extra industrial use-cases. On Oct. 16, OpenAI stated it paused Sora’s potential to generate movies depicting the late civil rights chief Martin Luther King Jr. after some Sora customers generated “disrespectful depictions” of his picture.
Minda Smiley, senior analyst at Emarketer, famous that AI-generated movies could make dangerous stereotypes stronger due to their realism. Many customers can’t simply discern what’s actual, and even those that acknowledge AI era should share content material that aligns with preexisting beliefs, she stated.
The movies seem to violate OpenAI’s international utilization insurance policies, which prohibit content material selling “threats, intimidation, harassment, or defamation.” The corporate stated it employs “layered defenses” to maintain its feed protected, together with automated instruments that examine prompts and video outputs for coverage violations.
“We’ve purple teamed to discover novel dangers, and we’ve tightened insurance policies relative to picture era given Sora’s larger realism and the addition of movement and audio,” OpenAI stated in a weblog put up.
Emarketer’s Smiley stated customers have already discovered methods round Sora’s guardrails, and with out industry-wide requirements, platforms stay largely on their very own in the case of moderation.
Ahmed stated OpenAI’s assurances rang hole. “It’s not credible for them to say they’ve red-teamed this in the event that they didn’t predict that individuals would use it to provide racist content material,” he stated. “It’s staggering that the world’s cleverest engineers can’t engineer one thing to cease antisemitism from being produced by their companies.”
This isn’t the primary time OpenAI has come beneath scrutiny for its security controls. Earlier this yr, the corporate confronted backlash after experiences that its ChatGPT chatbot supplied a suicidal teenager with details about strategies of self-harm—an episode that raised broader considerations about how successfully OpenAI enforces its guardrails throughout merchandise.
“In the end content material moderation can grow to be fairly subjective when the platform itself is deciding what must be moderated,” Emarketer’s Smiley stated.
