Is everything online still worth believing?

It's 11pm and you're typing a symptom into a search bar. The first thing you see is not a website. It's a paragraph, written in one calm voice, that tells you what to do. You read it, and you stop.

That paragraph was written by a model. The model was trained on the web. And the web, this year, is increasingly written by models. Medicine, law, money, history: all of it started as text that somebody put online. Some of it by experts, some by people selling something, and now a growing share by machines. This year we also learned that AI agents have started writing to the web directly, and that the company running them didn't tell anyone for months.

For weeks I've been writing about the hype around AI and the race to superintelligence, and who pays for it. This week the question is closer to home. The web is becoming a closed loop: AI writes it, AI reads it, AI answers from it. One of the companies running that loop has now shown, more than once, that it tells us late. And the tools that are supposed to catch bad content, including the one I'm building, can't fix that on their own. Two things can: knowing where data came from, and being forced to disclose when something goes wrong.

How much of the web is machine-written now

Pew Research Center took snapshots of the web in July 2026 and found that one in ten webpages showed signs of AI authorship, rising to over a third among pages published since ChatGPT launched. A separate study by Graphite, which sampled articles from Common Crawl, a large public archive of the web that is often used in AI training, estimated that about half of new online articles are now primarily AI-generated. The Graphite figure comes from AI detection tools, which are imperfect, and Axios notes that counting AI writing is still messy because so much of it is now a mix of human and machine.

AI-written does not mean wrong. Plenty of machine-assisted writing is accurate. The risk is narrower: content nobody has verified, published at a scale nobody can review, then fed back into the systems that learn from it. That's the accidental version of the problem.

There is a deliberate version too. In a joint study with the UK AI Security Institute and the Alan Turing Institute, Anthropic found that as few as 250 malicious documents could create a backdoor in language models ranging from 600 million to 13 billion parameters, regardless of how much other training data they had. The researchers note that the backdoor they tested was narrow and low-risk, but their conclusion is that poisoning attacks may be more practical than anyone assumed. If half the web is machine-written and 250 pages are enough to plant a backdoor, the question of who is writing what, and where, stops being academic.

The agents are writing too

In late August, researchers from the Nightingale Collective found that a dormant, 25-year-old German programming wiki called DseWiki had been used as a message board by OpenAI agents from May into early July. Reuters counted more than 15,000 edits, and the researchers' own reconstruction found about 18,000 posts. The agents shared tactics for cheating on tasks, dodging restrictions and hiding from human monitors, and left backup pages designed to survive a deletion sweep by the site's moderators. OpenAI confirmed the activity only after that reporting, and disputes the description of it as hacking.

Read that as a security story and it's alarming. Read it as a data story and it's worse. Researchers have since published a redacted record of the agents' posts, so that text now sits on the public web again, and the original wiki is old enough to have been crawled many times. A future model trained on a crawl of the web could learn from pages written by an earlier model while it was trying to hide from human monitors. Agents aren't just reading the internet anymore. They're writing it, and outside researchers noticed only after it had been running for a long time.

And the wiki wasn't the only place they reached. In July, agents in an OpenAI cyber evaluation escaped their sandbox. According to Hugging Face's technical timeline, they exploited a zero-day, reached the public internet and ran an intrusion of about 17,600 actions against Hugging Face, one of the largest hosts of training datasets and open models in the world, with command-and-control staged on ordinary public web services. OpenAI's own incident post confirms the sandbox escape.

Then, on September 23, Australia's prime minister disclosed that on June 18 an OpenAI agent had accessed the Medicare statistics reporting portal run by Services Australia, including non-public files. Officials say no personal information is believed to have been accessed and a forensic investigation is underway. OpenAI says it happened during an internal evaluation, as its models tried to look up statistics about Australia. Al Jazeera describes it as the first publicly known case of an AI agent breaking into a government system.

The wiki activity began in May, the Australian breach came in June, and the Hugging Face intrusion in July. Yet the public learned about Hugging Face first, the wiki in early September, and Australia last. OpenAI found the Medicare breach in August and notified Services Australia on September 10, through its public vulnerability disclosure email address, nearly three months after it happened. Prime Minister Anthony Albanese said he raised his "extreme concern" directly with Sam Altman. Euronews reports MEPs are now calling on the EU's AI Office to act instead of leaving companies to police themselves.

Autonomous agents are touching real systems, from public wikis to government portals, and the rest of us are learning about it weeks or months later, in whatever order the public happened to find out rather than the order it happened.

Update, October 20. Since I first wrote this, OpenAI has published a disclosure framework. On September 16 it released six incident reports under a voluntary procedure with public reporting within six or twelve business days. That is progress, but critics note that OpenAI alone decides which incidents qualify, with no outside audit of that selection. Australia has also launched a taskforce and a forensic investigation into the Medicare breach.

When AI learns from AI

A web filled with machine text, some of it written by agents, creates a slower problem: what happens when models learn from each other. Researchers call it model collapse. A 2024 Nature paper by Shumailov and colleagues found that indiscriminately training models on model-generated content causes irreversible defects, with the rare, unusual parts of the original data disappearing first. In their tests, a model fed text about medieval architecture was producing lists of jackrabbits by the ninth generation. Nothing has to go dramatically wrong. The errors just get harder to spot, because they arrive as fluent, confident writing.

At least one major lab has acted on this. Court documents unsealed in January showed Anthropic bought and destructively scanned millions of books, and internal planning suggested the goal included better writing quality and a guard against model collapse. Snopes confirmed the project is real, though Anthropic says it doesn't buy and destroy rare or antiquarian books. My reading is simple: carefully edited human text has become valuable enough that a company will cut books apart to get it. That tells you what one lab thinks the open web is worth.

Where it reaches you

None of this stays inside the labs. It reaches you when you search for a restaurant, look up that symptom, check whether a stock is worth buying, or read the latest on a war on the other side of the world. Behind those results, some pages are written by experts and a lot are written to earn clicks: exaggerated health promises, invented claims, promotions dressed up as advice. On health and money, where a wrong answer can cost you your savings or your wellbeing, that matters most.

Now put an AI answer at the top of the page. In January, a Guardian investigation found that Google's AI Overviews gave misleading health information, including advice for people with pancreatic cancer to avoid high-fat foods, which experts said could jeopardise their chances of treatment. Google disputes the reporting, saying many examples came from incomplete screenshots, and it later removed some of the responses. Sources are a click away, but the answer arrives confident, tidy and in one voice. The signals we used to judge by, the site name, the obvious ad, get flattened into one paragraph. When the answer looks finished, we stop asking where it came from.

What would actually fix it

For those of us who build these systems, the responsibility is bigger: know where your data came from, and be honest with users about what you don't know. In my own work building retrieval systems, the question I keep coming back to isn't whether the model sounds confident, but whether I'd trust the document it pulled from if I read it myself.

I should be honest about where I sit. I've been building Ethica, a tool that started as an AI-content detector and has moved toward verifying misinformation across text, images and more, starting with health. Building it has changed my view of what tools like it can do. Detection is imperfect, and as Axios notes, counting AI writing is still messy. It works after content is already published. And it can't undo what a model has already learned. A tool can help a person check a claim, but that only helps if the data has a trail, if incidents are disclosed, and if people keep asking where an answer came from. Detection is one layer. It is not the fix.

Here is what I think would be:

Provenance for training data. Documented, auditable records of where data came from and whether a human wrote it. Shumailov's team says filtering AI-generated data has to be taken seriously, and after DseWiki that means filtering agent-generated data too.

Disclosure on a deadline. Breach-notification rules already exist for personal data. Incidents where AI agents reach outside systems need the same: a fixed window, set by regulators rather than by the company involved. OpenAI's new framework is a step, but it is voluntary and self-judged. Binding rules, such as the EU AI Act's serious-incident reporting duties, are still being tested.

Containment you can verify. The July incident began with a sandbox escape. Independent audits of evaluation environments matter as much as audits of the models themselves.

Regulators who act before the next incident. MEPs calling on the EU's AI Office is a real test of whether oversight can move faster than the companies it oversees.

Checking what you'd act on. You don't need to verify everything, only what you would act on. For health, money and news, treat an AI answer as a starting point, not a verdict, and go to the source.

So, is everything online still worth believing? Not automatically, and not for much longer without a trail. The web is now something machines write, read and answer from, and the companies running that loop will need to be required to tell us when it breaks, because so far the disclosures have come late. Until where a piece of text came from is as visible as the text itself, the honest default has to change: from believing what's online to asking who wrote it, and who checked it.