The AI dooooooom thread

[Zoupa]

Indeed.

But replicating every relevant incremental experiment since 2025 in your field is going to become an overwhelming burden on individual researchers very quickly.

Replicating every relevant incremental experiment since the dawn of time up to 2025 would have been an overwhelming burden as well.

There was little need to replicate an experiment previously. That's the whole point of scientific journal publishing.

I guess you haven't heard of replication crisis, which was a big thing years before AI.  Good thing there was little need to replicate experiments, because it turned out most of the time you couldn't.  Humans were very much capable of bullshitting and willful lack of skepticism on their own, especially when academic career was on the line.

I was unphased. Crap studies and crap papers have always been around. The explosion we're seeing now is orders of magnitude. I'm referring to medical journals, as I'm not familiar with other fields of research.

[crazy canuck]

Peer-review doesn't involve checking if an experiment was actually performed or what the results actually were. The basic quality assurance method of the scientific system is replication.

That is accurate. One of the important functions of peer review of a scientific paper to both ensure that the procedures used by the scientists are adequately, explained so that the experiment can be replicated by somebody reading the paper and also considering the data that was derived from the experiments to ensure that the conclusion drawn from the experiments and reported in the manuscript are valid.

Indeed.

But replicating every relevant incremental experiment since 2025 in your field is going to become an overwhelming burden on individual researchers very quickly.

Replicating every relevant incremental experiment since the dawn of time up to 2025 would have been an overwhelming burden as well.

There was little need to replicate an experiment previously. That's the whole point of scientific journal publishing.

I guess you haven't heard of replication crisis, which was a big thing years before AI.  Good thing there was little need to replicate experiments, because it turned out most of the time you couldn't.  Humans were very much capable of bullshitting and willful lack of skepticism on their own, especially when academic career was on the line.

Yi and DG have missed the point. 

AI exacerbates the problem. Nobody is claiming that a problem didn't exist prior to AI.

[Baron von Schtinkenbutt]

But replicating every relevant incremental experiment since 2025 in your field is going to become an overwhelming burden on individual researchers very quickly.

Going to be?  In many fields, it has been for decades.  The "replication crisis" well predates generative ML models.

[Baron von Schtinkenbutt]

AI exacerbates the problem. Nobody is claiming that a problem didn't exist prior to AI.

You don't interpret "[t]here was little need to replicate an experiment previously" as a denial that there was a problem previously?  That's how I do.

In many fields, like machine learning, human slop has been deluging reviewers for a couple decades.  I worked for a company owned by a Computer Science professor, and she bemoaned the slop reviewers were being subject to back in the mid 2010s.  LLMs have definitely made the problem worse, as they significantly improve the "productivity" of generating human research slop, but it's not "orders of magnitude" as Zoupa says unless you're in an area that was not subject to high volumes of slop coming from questionable researchers in China and India.

[Jacob]

Okay that was wrong. Mea culpa.

[crazy canuck]

AI exacerbates the problem. Nobody is claiming that a problem didn't exist prior to AI.

You don't interpret "[t]here was little need to replicate an experiment previously" as a denial that there was a problem previously?  That's how I do.

In many fields, like machine learning, human slop has been deluging reviewers for a couple decades.  I worked for a company owned by a Computer Science professor, and she bemoaned the slop reviewers were being subject to back in the mid 2010s.  LLMs have definitely made the problem worse, as they significantly improve the "productivity" of generating human research slop, but it's not "orders of magnitude" as Zoupa says unless you're in an area that was not subject to high volumes of slop coming from questionable researchers in China and India.

Where I say they are going wrong is the suggestion of that the solution to the problem is the same as it has ever been, replicating the experiment.

We are putting our heads in the sand if we pretend that the replication problem is the same now as it has ever been.  That is clearly ridiculous given the volume of fraudulent papers that are being submitted to the journals.

[Sheilbh]

But replicating every relevant incremental experiment since 2025 in your field is going to become an overwhelming burden on individual researchers very quickly.

Going to be?  In many fields, it has been for decades.  The "replication crisis" well predates generative ML models.

Yeah I think this is the interesting thing because in many ways I think that replication crisis was generated by our existing artificial intelligence: boards of administrators and funding criteria.

On the input side for some time we have been collecting vastly more data than we are able to meaningfully or reliably extract information from, or it is simply too complex or open to "solve" (the protein fold problem). And on the output side we have turned academia itself into something that is generating datapoints. The whole "publish or perish" approach is basically about ensuring you hit metrics in order to be reviewed by (non-expert, non-peer) funding bodies and administrators. That has I think directly fed into the replication crisis and also other trends within academia such as ever deeper and narrower specialisation (which I think is less likely to produce breakthroughs). We are already in a world of academic slop, it's just artisanal slop.

As I say I think AI is actually going to dissolve some of the problems from the input, as it may be better and able to produce meaningful output from the vast datasets we've been collecting. At the same time - especially if the metrics that we monitor and care about from an administrative and funding body perspective do not change - it will almost certainly exacerbate the academic slop problem.

But I think it is impossible to talk about the impact of AI and academia without having AlphaFold in that conversation. The team behind that literally won the Nobel Prize in Chemistry a year or two ago - from what I read there was some grumbling that it was too soon (because Nobel Prizes are often more lifetime achievement), but that the breakthrough was deserving was not doubted. Again I know nothing about this but my understanding is that protein folding was seen as an intractable problem in the field because the number of possible bonds was basically infinite. Within that area people did not expect their to be a solution to this in their lifetime, if ever (I read an article that noted that just doing it sequentially which is what we've done so far, for all the proteins we have would take longer than the age of the universe).

The first iteration of AlphaFold came out in 2018 and it was already a revolutionary breakthrough from our current understanding in its ability to predict the protein folds. But there were still accuracy issues, particularly the more complex the protein but it has continued to improve and the latest version is significantly better. There are still limitations and issues - which means you keep working and re-iterating and building new versions - but from what I've read it is a seismic shift in that area of research. We'll see the impact play out in the comming years when the other research or discovery that is based on our requires structural biology now have this new foundational tool to build on.

But I think this is what I mean by AI being able to help on one side while also accelerating on the other and, you know, you kind of hope that humans and academia will be able to (and increasingly better able to) dicriminate between the two.

[Baron von Schtinkenbutt]

Where I say they are going wrong is the suggestion of that the solution to the problem is the same as it has ever been, replicating the experiment.

I agree there.  My point was just that some fields reached the point where they were so overwhelmed with fraud that "just replicate" was infeasible well before LLMs.

We are putting our heads in the sand if we pretend that the replication problem is the same now as it has ever been.  That is clearly ridiculous given the volume of fraudulent papers that are being submitted to the journals.

This is where I partially disagree.  The nature of the replication problem is the same as it has been (unethical researchers flooding journals with fraudulent papers).  The magnitude has increased significantly, though.  For some fields, this shifts the dynamic from "annoyed by frauds" to "overwhelmed by frauds", which is significant.  For others, it shifts the dynamic from "overwhelmed by frauds" to "very overwhelmed by frauds", which (in my opinion) isn't significant.

A pernicious problem that I think LLMs will greatly exacerbate is instances of high-quality, targeted fraud.  The ability of LLMs to craft well-worded bullshit makes it easier to craft papers with an agenda, where it will be harder to detect the fraud.  It's an extension of using LLMs to craft misinformation.  The peer review system has long had a problem here, since it really isn't set up to assume submitters are high-effort liars.  Making this significantly easier is something I think could change the nature of the replication problem.

[garbon]

Yeah I think this is the interesting thing because in many ways I think that replication crisis was generated by our existing artificial intelligence: boards of administrators and funding criteria.

On the input side for some time we have been collecting vastly more data than we are able to meaningfully or reliably extract information from, or it is simply too complex or open to "solve" (the protein fold problem). And on the output side we have turned academia itself into something that is generating datapoints. The whole "publish or perish" approach is basically about ensuring you hit metrics in order to be reviewed by (non-expert, non-peer) funding bodies and administrators. That has I think directly fed into the replication crisis and also other trends within academia such as ever deeper and narrower specialisation (which I think is less likely to produce breakthroughs). We are already in a world of academic slop, it's just artisanal slop.

As I say I think AI is actually going to dissolve some of the problems from the input, as it may be better and able to produce meaningful output from the vast datasets we've been collecting. At the same time - especially if the metrics that we monitor and care about from an administrative and funding body perspective do not change - it will almost certainly exacerbate the academic slop problem.

But I think it is impossible to talk about the impact of AI and academia without having AlphaFold in that conversation. The team behind that literally won the Nobel Prize in Chemistry a year or two ago - from what I read there was some grumbling that it was too soon (because Nobel Prizes are often more lifetime achievement), but that the breakthrough was deserving was not doubted. Again I know nothing about this but my understanding is that protein folding was seen as an intractable problem in the field because the number of possible bonds was basically infinite. Within that area people did not expect their to be a solution to this in their lifetime, if ever (I read an article that noted that just doing it sequentially which is what we've done so far, for all the proteins we have would take longer than the age of the universe).

The first iteration of AlphaFold came out in 2018 and it was already a revolutionary breakthrough from our current understanding in its ability to predict the protein folds. But there were still accuracy issues, particularly the more complex the protein but it has continued to improve and the latest version is significantly better. There are still limitations and issues - which means you keep working and re-iterating and building new versions - but from what I've read it is a seismic shift in that area of research. We'll see the impact play out in the comming years when the other research or discovery that is based on our requires structural biology now have this new foundational tool to build on.

But I think this is what I mean by AI being able to help on one side while also accelerating on the other and, you know, you kind of hope that humans and academia will be able to (and increasingly better able to) dicriminate between the two.

I don't really understand what you are talking about when you are distinguishing between quant and theory and also input and output. Using AI to help examine the quant data points (as an input) won't likely diminish quant outputs if defined as 'The whole "publish or perish" approach is basically about ensuring you hit metrics in order to be reviewed by (non-expert, non-peer) funding bodies and administrators'. I don't see why that would disappear.  It just might help increase the robustness of papers as we can now efficiently see connections between the inputs that we could never have managed by human brainpower alone.  Nothing there feels less quanty.

AI on the outputs (aka writing papers) then seems like it will only increase slop life...so while there may be more robust papers coming out, they also may get lose in some of the noise that humans will need sort through. Other than using AI to detect AI, I don't see why humans would suddenly get better at discriminating between the robust and the AI slop given the sheer volume of papers that will/do exist.

[Jacob]

This is where I partially disagree.  The nature of the replication problem is the same as it has been (unethical researchers flooding journals with fraudulent papers).  The magnitude has increased significantly, though.  For some fields, this shifts the dynamic from "annoyed by frauds" to "overwhelmed by frauds", which is significant.  For others, it shifts the dynamic from "overwhelmed by frauds" to "very overwhelmed by frauds", which (in my opinion) isn't significant.

A pernicious problem that I think LLMs will greatly exacerbate is instances of high-quality, targeted fraud.  The ability of LLMs to craft well-worded bullshit makes it easier to craft papers with an agenda, where it will be harder to detect the fraud.  It's an extension of using LLMs to craft misinformation.  The peer review system has long had a problem here, since it really isn't set up to assume submitters are high-effort liars.  Making this significantly easier is something I think could change the nature of the replication problem.

This seems a reasonable take to me.

[crazy canuck]

Where I say they are going wrong is the suggestion of that the solution to the problem is the same as it has ever been, replicating the experiment.

I agree there.  My point was just that some fields reached the point where they were so overwhelmed with fraud that "just replicate" was infeasible well before LLMs.

That's fair

We are putting our heads in the sand if we pretend that the replication problem is the same now as it has ever been.  That is clearly ridiculous given the volume of fraudulent papers that are being submitted to the journals.

This is where I partially disagree.  The nature of the replication problem is the same as it has been (unethical researchers flooding journals with fraudulent papers).  The magnitude has increased significantly, though.  For some fields, this shifts the dynamic from "annoyed by frauds" to "overwhelmed by frauds", which is significant.  For others, it shifts the dynamic from "overwhelmed by frauds" to "very overwhelmed by frauds", which (in my opinion) isn't significant.

A pernicious problem that I think LLMs will greatly exacerbate is instances of high-quality, targeted fraud.  The ability of LLMs to craft well-worded bullshit makes it easier to craft papers with an agenda, where it will be harder to detect the fraud.  It's an extension of using LLMs to craft misinformation.  The peer review system has long had a problem here, since it really isn't set up to assume submitters are high-effort liars.  Making this significantly easier is something I think could change the nature of the replication problem.

Yes, I think this is where we depart.  The manipulation of data is being caught, largely and ironically by the use of other AI tools (as mentioned before, AI tools are very good at pattern recognition) but the systems within Academia for dealing with research integrity issues are becoming overwhelmed by the sheer quantity of manuscripts that have been identified.  Just the number of implicated manuscripts using Western blots has significantly increased. These are not high quality frauds, they are largely easily detectably image manipulation/duplication type issues.  But the sheer number of instances now being detected has become unmanageable.

Also, the use of LLM's to draft manuscripts is very easy to spot.  Stupid mistakes that no human author would made abound.  But again, it is the sheer number of such manuscripts that overwhelms.

[crazy canuck]

Edit, I should inject one point of hope - Journals are creating editorial policies which now require that all data and original uncropped images be submitted along with the manuscript so that there can be an initial scan to detect potential data or image manipulation before the manuscript is sent for peer review.