The AI dooooooom thread

[Zoupa]

Indeed.

But replicating every relevant incremental experiment since 2025 in your field is going to become an overwhelming burden on individual researchers very quickly.

Replicating every relevant incremental experiment since the dawn of time up to 2025 would have been an overwhelming burden as well.

There was little need to replicate an experiment previously. That's the whole point of scientific journal publishing.

I guess you haven't heard of replication crisis, which was a big thing years before AI.  Good thing there was little need to replicate experiments, because it turned out most of the time you couldn't.  Humans were very much capable of bullshitting and willful lack of skepticism on their own, especially when academic career was on the line.

I was unphased. Crap studies and crap papers have always been around. The explosion we're seeing now is orders of magnitude. I'm referring to medical journals, as I'm not familiar with other fields of research.

[crazy canuck]

Peer-review doesn't involve checking if an experiment was actually performed or what the results actually were. The basic quality assurance method of the scientific system is replication.

That is accurate. One of the important functions of peer review of a scientific paper to both ensure that the procedures used by the scientists are adequately, explained so that the experiment can be replicated by somebody reading the paper and also considering the data that was derived from the experiments to ensure that the conclusion drawn from the experiments and reported in the manuscript are valid.

Indeed.

But replicating every relevant incremental experiment since 2025 in your field is going to become an overwhelming burden on individual researchers very quickly.

Replicating every relevant incremental experiment since the dawn of time up to 2025 would have been an overwhelming burden as well.

There was little need to replicate an experiment previously. That's the whole point of scientific journal publishing.

I guess you haven't heard of replication crisis, which was a big thing years before AI.  Good thing there was little need to replicate experiments, because it turned out most of the time you couldn't.  Humans were very much capable of bullshitting and willful lack of skepticism on their own, especially when academic career was on the line.

Yi and DG have missed the point. 

AI exacerbates the problem. Nobody is claiming that a problem didn't exist prior to AI.

[Baron von Schtinkenbutt]

But replicating every relevant incremental experiment since 2025 in your field is going to become an overwhelming burden on individual researchers very quickly.

Going to be?  In many fields, it has been for decades.  The "replication crisis" well predates generative ML models.

[Baron von Schtinkenbutt]

AI exacerbates the problem. Nobody is claiming that a problem didn't exist prior to AI.

You don't interpret "[t]here was little need to replicate an experiment previously" as a denial that there was a problem previously?  That's how I do.

In many fields, like machine learning, human slop has been deluging reviewers for a couple decades.  I worked for a company owned by a Computer Science professor, and she bemoaned the slop reviewers were being subject to back in the mid 2010s.  LLMs have definitely made the problem worse, as they significantly improve the "productivity" of generating human research slop, but it's not "orders of magnitude" as Zoupa says unless you're in an area that was not subject to high volumes of slop coming from questionable researchers in China and India.

[Jacob]

Okay that was wrong. Mea culpa.

[crazy canuck]

AI exacerbates the problem. Nobody is claiming that a problem didn't exist prior to AI.

You don't interpret "[t]here was little need to replicate an experiment previously" as a denial that there was a problem previously?  That's how I do.

In many fields, like machine learning, human slop has been deluging reviewers for a couple decades.  I worked for a company owned by a Computer Science professor, and she bemoaned the slop reviewers were being subject to back in the mid 2010s.  LLMs have definitely made the problem worse, as they significantly improve the "productivity" of generating human research slop, but it's not "orders of magnitude" as Zoupa says unless you're in an area that was not subject to high volumes of slop coming from questionable researchers in China and India.

Where I say they are going wrong is the suggestion of that the solution to the problem is the same as it has ever been, replicating the experiment.

We are putting our heads in the sand if we pretend that the replication problem is the same now as it has ever been.  That is clearly ridiculous given the volume of fraudulent papers that are being submitted to the journals.

[Sheilbh]

But replicating every relevant incremental experiment since 2025 in your field is going to become an overwhelming burden on individual researchers very quickly.

Going to be?  In many fields, it has been for decades.  The "replication crisis" well predates generative ML models.

Yeah I think this is the interesting thing because in many ways I think that replication crisis was generated by our existing artificial intelligence: boards of administrators and funding criteria.

On the input side for some time we have been collecting vastly more data than we are able to meaningfully or reliably extract information from, or it is simply too complex or open to "solve" (the protein fold problem). And on the output side we have turned academia itself into something that is generating datapoints. The whole "publish or perish" approach is basically about ensuring you hit metrics in order to be reviewed by (non-expert, non-peer) funding bodies and administrators. That has I think directly fed into the replication crisis and also other trends within academia such as ever deeper and narrower specialisation (which I think is less likely to produce breakthroughs). We are already in a world of academic slop, it's just artisanal slop.

As I say I think AI is actually going to dissolve some of the problems from the input, as it may be better and able to produce meaningful output from the vast datasets we've been collecting. At the same time - especially if the metrics that we monitor and care about from an administrative and funding body perspective do not change - it will almost certainly exacerbate the academic slop problem.

But I think it is impossible to talk about the impact of AI and academia without having AlphaFold in that conversation. The team behind that literally won the Nobel Prize in Chemistry a year or two ago - from what I read there was some grumbling that it was too soon (because Nobel Prizes are often more lifetime achievement), but that the breakthrough was deserving was not doubted. Again I know nothing about this but my understanding is that protein folding was seen as an intractable problem in the field because the number of possible bonds was basically infinite. Within that area people did not expect their to be a solution to this in their lifetime, if ever (I read an article that noted that just doing it sequentially which is what we've done so far, for all the proteins we have would take longer than the age of the universe).

The first iteration of AlphaFold came out in 2018 and it was already a revolutionary breakthrough from our current understanding in its ability to predict the protein folds. But there were still accuracy issues, particularly the more complex the protein but it has continued to improve and the latest version is significantly better. There are still limitations and issues - which means you keep working and re-iterating and building new versions - but from what I've read it is a seismic shift in that area of research. We'll see the impact play out in the comming years when the other research or discovery that is based on our requires structural biology now have this new foundational tool to build on.

But I think this is what I mean by AI being able to help on one side while also accelerating on the other and, you know, you kind of hope that humans and academia will be able to (and increasingly better able to) dicriminate between the two.

[Baron von Schtinkenbutt]

Where I say they are going wrong is the suggestion of that the solution to the problem is the same as it has ever been, replicating the experiment.

I agree there.  My point was just that some fields reached the point where they were so overwhelmed with fraud that "just replicate" was infeasible well before LLMs.

We are putting our heads in the sand if we pretend that the replication problem is the same now as it has ever been.  That is clearly ridiculous given the volume of fraudulent papers that are being submitted to the journals.

This is where I partially disagree.  The nature of the replication problem is the same as it has been (unethical researchers flooding journals with fraudulent papers).  The magnitude has increased significantly, though.  For some fields, this shifts the dynamic from "annoyed by frauds" to "overwhelmed by frauds", which is significant.  For others, it shifts the dynamic from "overwhelmed by frauds" to "very overwhelmed by frauds", which (in my opinion) isn't significant.

A pernicious problem that I think LLMs will greatly exacerbate is instances of high-quality, targeted fraud.  The ability of LLMs to craft well-worded bullshit makes it easier to craft papers with an agenda, where it will be harder to detect the fraud.  It's an extension of using LLMs to craft misinformation.  The peer review system has long had a problem here, since it really isn't set up to assume submitters are high-effort liars.  Making this significantly easier is something I think could change the nature of the replication problem.

[garbon]

Yeah I think this is the interesting thing because in many ways I think that replication crisis was generated by our existing artificial intelligence: boards of administrators and funding criteria.

On the input side for some time we have been collecting vastly more data than we are able to meaningfully or reliably extract information from, or it is simply too complex or open to "solve" (the protein fold problem). And on the output side we have turned academia itself into something that is generating datapoints. The whole "publish or perish" approach is basically about ensuring you hit metrics in order to be reviewed by (non-expert, non-peer) funding bodies and administrators. That has I think directly fed into the replication crisis and also other trends within academia such as ever deeper and narrower specialisation (which I think is less likely to produce breakthroughs). We are already in a world of academic slop, it's just artisanal slop.

As I say I think AI is actually going to dissolve some of the problems from the input, as it may be better and able to produce meaningful output from the vast datasets we've been collecting. At the same time - especially if the metrics that we monitor and care about from an administrative and funding body perspective do not change - it will almost certainly exacerbate the academic slop problem.

But I think it is impossible to talk about the impact of AI and academia without having AlphaFold in that conversation. The team behind that literally won the Nobel Prize in Chemistry a year or two ago - from what I read there was some grumbling that it was too soon (because Nobel Prizes are often more lifetime achievement), but that the breakthrough was deserving was not doubted. Again I know nothing about this but my understanding is that protein folding was seen as an intractable problem in the field because the number of possible bonds was basically infinite. Within that area people did not expect their to be a solution to this in their lifetime, if ever (I read an article that noted that just doing it sequentially which is what we've done so far, for all the proteins we have would take longer than the age of the universe).

The first iteration of AlphaFold came out in 2018 and it was already a revolutionary breakthrough from our current understanding in its ability to predict the protein folds. But there were still accuracy issues, particularly the more complex the protein but it has continued to improve and the latest version is significantly better. There are still limitations and issues - which means you keep working and re-iterating and building new versions - but from what I've read it is a seismic shift in that area of research. We'll see the impact play out in the comming years when the other research or discovery that is based on our requires structural biology now have this new foundational tool to build on.

But I think this is what I mean by AI being able to help on one side while also accelerating on the other and, you know, you kind of hope that humans and academia will be able to (and increasingly better able to) dicriminate between the two.

I don't really understand what you are talking about when you are distinguishing between quant and theory and also input and output. Using AI to help examine the quant data points (as an input) won't likely diminish quant outputs if defined as 'The whole "publish or perish" approach is basically about ensuring you hit metrics in order to be reviewed by (non-expert, non-peer) funding bodies and administrators'. I don't see why that would disappear.  It just might help increase the robustness of papers as we can now efficiently see connections between the inputs that we could never have managed by human brainpower alone.  Nothing there feels less quanty.

AI on the outputs (aka writing papers) then seems like it will only increase slop life...so while there may be more robust papers coming out, they also may get lose in some of the noise that humans will need sort through. Other than using AI to detect AI, I don't see why humans would suddenly get better at discriminating between the robust and the AI slop given the sheer volume of papers that will/do exist.

[Jacob]

This is where I partially disagree.  The nature of the replication problem is the same as it has been (unethical researchers flooding journals with fraudulent papers).  The magnitude has increased significantly, though.  For some fields, this shifts the dynamic from "annoyed by frauds" to "overwhelmed by frauds", which is significant.  For others, it shifts the dynamic from "overwhelmed by frauds" to "very overwhelmed by frauds", which (in my opinion) isn't significant.

A pernicious problem that I think LLMs will greatly exacerbate is instances of high-quality, targeted fraud.  The ability of LLMs to craft well-worded bullshit makes it easier to craft papers with an agenda, where it will be harder to detect the fraud.  It's an extension of using LLMs to craft misinformation.  The peer review system has long had a problem here, since it really isn't set up to assume submitters are high-effort liars.  Making this significantly easier is something I think could change the nature of the replication problem.

This seems a reasonable take to me.

[crazy canuck]

Where I say they are going wrong is the suggestion of that the solution to the problem is the same as it has ever been, replicating the experiment.

I agree there.  My point was just that some fields reached the point where they were so overwhelmed with fraud that "just replicate" was infeasible well before LLMs.

That's fair

We are putting our heads in the sand if we pretend that the replication problem is the same now as it has ever been.  That is clearly ridiculous given the volume of fraudulent papers that are being submitted to the journals.

This is where I partially disagree.  The nature of the replication problem is the same as it has been (unethical researchers flooding journals with fraudulent papers).  The magnitude has increased significantly, though.  For some fields, this shifts the dynamic from "annoyed by frauds" to "overwhelmed by frauds", which is significant.  For others, it shifts the dynamic from "overwhelmed by frauds" to "very overwhelmed by frauds", which (in my opinion) isn't significant.

A pernicious problem that I think LLMs will greatly exacerbate is instances of high-quality, targeted fraud.  The ability of LLMs to craft well-worded bullshit makes it easier to craft papers with an agenda, where it will be harder to detect the fraud.  It's an extension of using LLMs to craft misinformation.  The peer review system has long had a problem here, since it really isn't set up to assume submitters are high-effort liars.  Making this significantly easier is something I think could change the nature of the replication problem.

Yes, I think this is where we depart.  The manipulation of data is being caught, largely and ironically by the use of other AI tools (as mentioned before, AI tools are very good at pattern recognition) but the systems within Academia for dealing with research integrity issues are becoming overwhelmed by the sheer quantity of manuscripts that have been identified.  Just the number of implicated manuscripts using Western blots has significantly increased. These are not high quality frauds, they are largely easily detectably image manipulation/duplication type issues.  But the sheer number of instances now being detected has become unmanageable.

Also, the use of LLM's to draft manuscripts is very easy to spot.  Stupid mistakes that no human author would made abound.  But again, it is the sheer number of such manuscripts that overwhelms.

[crazy canuck]

Edit, I should inject one point of hope - Journals are creating editorial policies which now require that all data and original uncropped images be submitted along with the manuscript so that there can be an initial scan to detect potential data or image manipulation before the manuscript is sent for peer review.

[Tamas]

https://www.theguardian.com/politics/2026/jan/25/ai-generated-british-schoolgirl-becomes-far-right-social-media-meme

[Sheilbh]

I don't really understand what you are talking about when you are distinguishing between quant and theory and also input and output. Using AI to help examine the quant data points (as an input) won't likely diminish quant outputs if defined as 'The whole "publish or perish" approach is basically about ensuring you hit metrics in order to be reviewed by (non-expert, non-peer) funding bodies and administrators'. I don't see why that would disappear.  It just might help increase the robustness of papers as we can now efficiently see connections between the inputs that we could never have managed by human brainpower alone.  Nothing there feels less quanty.

AI on the outputs (aka writing papers) then seems like it will only increase slop life...so while there may be more robust papers coming out, they also may get lose in some of the noise that humans will need sort through. Other than using AI to detect AI, I don't see why humans would suddenly get better at discriminating between the robust and the AI slop given the sheer volume of papers that will/do exist.

Fair and I think this is possibly me being a bit sloppy in my language.

By output I menat that AI being used as the research tool in itself - as in AlphaFold (or their newly announced project - which I'll dig out the article about). The research itself, the new knowledge that is generated is from the use of AI. In that case the ability to predict protein folds which, from my understanding, gives a tool for structural biologists that they've never had - we don't know where that'll lead but it's a breakthrough in that field. But it's a breakthrough because of the way AI has been applied to generate the research (I'm not sure which is the better analogy here - a new machine or a new technique?) rather than because of the paper itself being the output generated by AI. My understanding is it could have a big impact on RNA research, for example - (not sure if this would be directly affected) but, inevitably, I have to note that Moderna' CEO has said the company doesn't pla to invest in new late-stag vaccine trials because of growing opposition to immunisation from American officials.

On the output as journal article side of things - I agree. What I'd note is that the "publish or perish" culture and metrics are human incentives that we have created. It is in our gift to change that. As the piece I posted said it is going to increase the volume in that academic culture. He described it as "slop-plus" because they were fine to a point:

The first thing is probably to stop calling it "slop". As Max Read notes, slop "suggests a set of qualities—forgettability, predictability, unoriginality, lifelessness—rather than a particular origin." The coming AI-generated papers may be unoriginal but they aren't lifeless in that way. They're technically proficient. They follow the form. They're adequate. They're easy to do and require little creativity, but also constitute the kind of legitimate incremental work that Thomas Kuhn called "normal science".

Call it Slop-Plus? Premium Slop? Maybe that's too harsh. The German term for Kuhn's normal science is Normalwissenschaft, so maybe Automatenwissenschaft?

So that's what I meant by it help solve the input problem of huge datasets, while also accelerating the output problem of vast numbers of papers of relatively little value in a "publish and perish" world.

That may then lead to a split between the regular journals and elite ones because the skill will be in discerning the valuable paper from the slop-plus. In the same way as recognising the human in art (for now) - what will get into the elite side of things is the original, striking, innovative theory or (in social "science"/humanities areas) elegant stylists. If the academic publishing culture of the last few decades has made everyone write like an AI then the way to break out of the emergence of AI, is to not write like AI.

On the Deep Mind thing I mentioned - from the paper (I imagine written by humans) in Nature today:

Google DeepMind launches AI tool to help identify genetic drivers of disease
AlphaGenome can analyse up to 1m letters of DNA code at once and could pave way for new treatments
Ian Sample Science editor
Wed 28 Jan 2026 16.15 GMT

Researchers at Google DeepMind have unveiled their latest artificial intelligence tool and claimed it will help scientists identify the genetic drivers of disease and ultimately pave the way for new treatments.

AlphaGenome predicts how mutations interfere with the way genes are controlled, changing when they are switched on, in which cells of the body, and whether their biological volume controls are set to high or low.

Most common diseases that run in families, including heart disease and autoimmune disorders, as well as mental health problems, have been linked to mutations that affect gene regulation, as have many cancers, but identifying which genetic glitches are to blame is far from straightforward.

"We see AlphaGenome as a tool for understanding what the functional elements in the genome do, which we hope will accelerate our fundamental understanding of the code of life," Natasha Latysheva, a DeepMind researcher, told a press briefing on the work.

The human genome runs to 3bn pairs of letters – the Gs, Ts, Cs and As that comprise the DNA code. About 2% of the genome tells cells how to make proteins, the building blocks of life. The rest orchestrates gene activity, carrying the crucial instructions that dictate where, when and how much individual genes are switched on.

The researchers trained AlphaGenome on public databases of human and mouse genetics, enabling it to learn connections between mutations in specific tissues and their impact on gene regulation. The AI can analyse up to 1m letters of DNA code at once and predict how mutations will affect different biological processes.

The DeepMind team believes the tool will help scientists map out which strands of genetic code are most essential for the development of particular tissues, such as nerve and liver cells, and pinpoint the most important mutations for driving cancer and other diseases. It could also underpin new gene therapies by allowing researchers to design entirely new DNA sequences – for example, to switch on a certain gene in nerve cells but not in muscle cells.

Carl de Boer, a researcher at the University of British Columbia in Canada, who was not involved in the work, said: "AlphaGenome can identify whether mutations affect genome regulation, which genes are impacted and how, and in what cell types. A drug could then be developed to counteract this effect.

"Ultimately, our goal is to have models that are so good we don't have to do an experiment to confirm their predictions. While AlphaGenome represents a significant innovation, achieving this goal will require continued work from the scientific community."

Some scientists have already begun using AlphaGenome. Marc Mansour, a clinical professor of paediatric haemato-oncology at UCL, said it marked a "step change" in his work to find genetic drivers for cancer.

Gareth Hawkes, a statistical geneticist at the University of Exeter, said: "The non-coding genome is 98% of our 3bn base pair genome. We understand the 2% fairly well, but the fact that we've got AlphaGenome that can make predictions of what this other 2.94bn base pair region is doing is a big step forward for us."

Edit: And worth noting - as with AlphaFold, AlphaGenome is open-source for non-commercial use. And I think Google DeepMind have said there's over a million API calls a day with over 3,000 users across 160 countries at this point.

[crazy canuck]

Yes, there are definitely uses for certain AI tools.

The problem is people generalizing that utility across all AI tools and especially generative AI tools.