The AI dooooooom thread

[DGuller]

I mean, that's not really surprising.  The nature of 'AI' is such that it can never truly be accurate.  All it does is throw in the most probable words one after another, and if pushed back on tells the user exactly what they want to hear. 

That's like saying that all a nuclear bomb does is trigger a runaway chain reaction.  It's a technically accurate thing to say, but it trivializes a very complex thing behind the "all it does".  Getting the conditional probabilities of the next token right depending on the context is kind of where the synthesis of concepts happens in an LLM.

[Sheilbh]

I think this is a fantastic summary by a very smart engineer in the Guardian's newsroom AI projects on what LLMs are (and aren't):
https://medium.com/@joelochlannsmith/mythbusting-large-language-models-2f4ef31a7ce1

I agree with his summary at the top:

Large Language Models (LLMs) are remarkable tools with major limitations. In the two years since ChatGPT made them famous, their use has spread to search engines, coding assistants, document editors, even WhatsApp.

Unfortunately, as their use has spread, clarity about how they work has not. Over the past two years as tech lead of the Guardian's fledgling Newsroom AI team, I've learned a lot about LLMs. But I've also noticed that the applications built on top of them conceal what's really going on, leading to widespread misunderstanding.

Then runs through some mythbusting which I think often boil down to some degree of anthromophisation which isn't appropriate. And I think if you try to understand what they are you can probably work out what they could be used for and what they shouldn't. I think a lot of the horror stories are people using it for things that it just shouldn't be used for - and that is not helped by the anthropomorphised layer on top.

(I'd add despite that paragraph and my awareness of carbon impact of each prompt - 9 seconds of TV - and the pleas of folks like Sam Altman, I still say please and thank you whenever I use one of these )

[DGuller]

I think this is a fantastic summary by a very smart engineer in the Guardian's newsroom AI projects on what LLMs are (and aren't):
https://medium.com/@joelochlannsmith/mythbusting-large-language-models-2f4ef31a7ce1

I agree with his summary at the top:

Large Language Models (LLMs) are remarkable tools with major limitations. In the two years since ChatGPT made them famous, their use has spread to search engines, coding assistants, document editors, even WhatsApp.

Unfortunately, as their use has spread, clarity about how they work has not. Over the past two years as tech lead of the Guardian's fledgling Newsroom AI team, I've learned a lot about LLMs. But I've also noticed that the applications built on top of them conceal what's really going on, leading to widespread misunderstanding.

Then runs through some mythbusting which I think often boil down to some degree of anthromophisation which isn't appropriate. And I think if you try to understand what they are you can probably work out what they could be used for and what they shouldn't. I think a lot of the horror stories are people using it for things that it just shouldn't be used for - and that is not helped by the anthropomorphised layer on top.

(I'd add despite that paragraph and my awareness of carbon impact of each prompt - 9 seconds of TV - and the pleas of folks like Sam Altman, I still say please and thank you whenever I use one of these )

I read the full article.  To be frank, while everything that is written there is true, IMO almost none of it is relevant or useful (which in my experience is as good at it gets on Medium when it comes to data science articles).  What useful bit of understanding is imparted by "busting" the myth that LLMs take text (rather than tokens)?  It sounds like the kind of mythbusting a high school physics students would do to show off his newfound knowledge:  "actually, there is no such thing as centrifigal force, there is only centripetal force".

I think most people who looked into this for a while understand that LLMs are predicting the next word (yeah, yeah, token, I know).  I think the relevant question is when does that mechanism of predicting the next word results in something that isn't too functionally different from real intelligence.  I don't think this article does much to even get you started answering that question.

[Sheilbh]

Well I suppose the audience matters there because to my view - it's not a data science article or about the cutting edge of LLMs - but an engineer working in that area in a newsroom. So surrounded by non-technical people who will not stop asking "why" and "how".

I saw it via a friend there who described it as for non-technical people (me - but also I suspect 90% of the people he is working with) looking for a deeper understanding of how they actually work and what they're doing - and I think at that it is very good. In that context for example, I think in particular emphasising the very simple point that the user-friendly, helpful (anthropomorphisable) interface is not the LLM - I also feel the token/word point is maybe not hugely important but makes me mindful given that I'm in a wordy profession (and he's working surrounded by people in wordy professions).

[DGuller]

Well I suppose the audience matters there because to my view - it's not a data science article or about the cutting edge of LLMs - but an engineer working in that area in a newsroom. So surrounded by non-technical people who will not stop asking "why" and "how".

I saw it via a friend there who described it as for non-technical people (me - but also I suspect 90% of the people he is working with) looking for a deeper understanding of how they actually work and what they're doing - and I think at that it is very good. In that context for example, I think in particular emphasising the very simple point that the user-friendly, helpful (anthropomorphisable) interface is not the LLM - I also feel the token/word point is maybe not hugely important but makes me mindful given that I'm in a wordy profession (and he's working surrounded by people in wordy professions).

I get that I'm not the audience, since for one I already knew all that.  That said, my evaluation of that article was on the merits of its usefulness to the non-technical audience, not to someone like me.  Whether you're technical audience or not, the question that I posed at the end of my last reply IS by far the most important question.  I think this article fails both at usefully educating those who know very little, and also those with little knowledge that we all know is dangerous.

[Syt]

Tbf, I think this is the kind of article that (if summarized to a power point ) would be useful at work - some in our organization seem to think AI is more than it is (or misunderstanding entirely).

[Josquius]

Yes, definitely.
I've heard some senior people suggesting uses which are...seriously worrying.
But in all this the actual practical uses in say transcribing meetings are getting quite overshadowed.

I mean, that's not really surprising.  The nature of 'AI' is such that it can never truly be accurate.  All it does is throw in the most probable words one after another, and if pushed back on tells the user exactly what they want to hear.

This differs to a flesh and blood marketing middle manager how? :p

[garbon]

Whether you're technical audience or not, the question that I posed at the end of my last reply IS by far the most important question.  I think this article fails both at usefully educating those who know very little, and also those with little knowledge that we all know is dangerous.

Do you care to elaborate? At first thought, I would think that your question would be dangerous for those who know little.

[Syt]

transcribing meetings are getting quite overshadowed.

Such a huge quality of life benefit! No longer someone having to devote their brain power to chronicling the meeting. Much more efficient to have AI do it and then review/edit before sending to participants.

[Sheilbh]

Such a huge quality of life benefit! No longer someone having to devote their brain power to chronicling the meeting. Much more efficient to have AI do it and then review/edit before sending to participants.

I don't know if I shared it but enjoyed the Popbitch story on this

>> AI Goes Popbitch <<
ITV staff - your mic is on!

Following the company-wide adoption of Google's AI product Gemini at ITV, all meetings now have the option of transcriptions and summaries. Handy, right!

Maybe, but it has also led to some awkward office casualties linked to the shiny new feature.

In one online session – which was being transcribed and summarised - a couple of employees stayed on after the meeting had ended and had a good old bitch.

A pithy summary of their bitching session was duly appended to the meeting notes, with a word-for-word transcription also available for anyone who wanted a deep-dive.

Also a class action in the US on this area - which is the way of US law (and, honestly, increasingly I'm not so sure it's less effective than European regulation plus intermittent underpowered enforcement) - because they have quite strict laws on recording without consent. So some tools, especially Otter which is super-popular, don't have consent or the "this meeting is being recorded" by default.

Tbf, I think this is the kind of article that (if summarized to a power point ) would be useful at work - some in our organization seem to think AI is more than it is (or misunderstanding entirely).

Yes, definitely.
I've heard some senior people suggesting uses which are...seriously worrying.

So with both of these it has been a bit of eureka moment for me using it on my small publishing project because it's been really helpful.

I wouldn't use it to do any of the actual work but at the level of what are the steps I need to take, what's the rough timeline, directories for x type of person, high level (and no doubt out of date) market research and positioning etc - it is very helpful.

So what I'm suddenly understanding is why strategy teams and senior leader types are so impressed and excited - but because of the sort of thing they're experimenting with it for less aware of how it technically works or how it can/can't be used effectively. Because my experience is that, with the exception of a few specific areas (like engineers and data scientists), there's a huge gap between senior excitement and expectation and then actual results/use (this is where I think that MIT point around identifying specific pain points that can be automated is really key - at this stage).

[garbon]

If I am in a meeting where it is noted the meeting is being transcribed and will have AI summaries, I clam up. Whole point of a meeting was to minimize verbatim written record!

[HVC]

let the underlings take the fall

[Sheilbh]

Yes. Or if it's that type of meeting at least get the lawyers to take notes so you can try to run a privilege argument

But then I am regularly astonished at the things often very senior smart people write in chats or WhatsApp.

[Neil]

I mean, that's not really surprising.  The nature of 'AI' is such that it can never truly be accurate.  All it does is throw in the most probable words one after another, and if pushed back on tells the user exactly what they want to hear. 

That's like saying that all a nuclear bomb does is trigger a runaway chain reaction.  It's a technically accurate thing to say, but it trivializes a very complex thing behind the "all it does".  Getting the conditional probabilities of the next token right depending on the context is kind of where the synthesis of concepts happens in an LLM.

Sure, but the point is that everything a LLM does is worthless, since you can never be sure if it's lying to you or not, and it can never be dependable. 

[garbon]

I mean, that's not really surprising.  The nature of 'AI' is such that it can never truly be accurate.  All it does is throw in the most probable words one after another, and if pushed back on tells the user exactly what they want to hear. 

That's like saying that all a nuclear bomb does is trigger a runaway chain reaction.  It's a technically accurate thing to say, but it trivializes a very complex thing behind the "all it does".  Getting the conditional probabilities of the next token right depending on the context is kind of where the synthesis of concepts happens in an LLM.

Sure, but the point is that everything a LLM does is worthless, since you can never be sure if it's lying to you or not, and it can never be dependable. 

Lying feels like the wrong word as that feels like attributing human motivations to it.

It can provide inaccurate/false information. Of course, it generally is the case that predictions aren't 100% accurate. After all, when companies buy market research, it isn't because they are going to get a 100% accurate picture of what is currently happening and predictions for the future but that it is better than the knowledge they have if they only relied on their fieldforce, annecdotal customer feedback and sales data.