The AI dooooooom thread

[crazy canuck]

Ye olde R-R Excel error thread: https://languish.org/forums/index.php/topic,9694.msg565989.html#msg565989

Reihart-Rogoff!  Yeah that was a memorable one.  That paper already had serious issues before the Excel error came out, and it was compounded by the fact that even taking it on its face, it was cited for broader propositions than it ever really stood for.

And to bring us back on topic, is it any wonder AI tools are so unreliable when they were trained on this sort of material?

[mongers]

Leaving aside my feigned ignorance of what the Pope said, I seems notable that he chose to make it in concert with the Anthropic CEO chap, who also had some interesting things to say.

[Norgy]

I have been using Fitbit devices for about a decade plus to track physical activity. It was never going to be good news when Google bought them, and now the app is chock full of an AI coach. The familiar interface, all historical data and some functionality has been wiped, and replaced by... junk.

Fuck you all, tech bros. I hope your robots eat you.

[PJL]

I have been using Fitbit devices for about a decade plus to track physical activity. It was never going to be good news when Google bought them, and now the app is chock full of an AI coach. The familiar interface, all historical data and some functionality has been wiped, and replaced by... junk.

Fuck you all, tech bros. I hope your robots eat you.

Like I've said before, I'm increasingly having the same opinion of the tech bros as I have of the marketing department of the Sirius Cybernetics Corporation.

[Syt]

Like I've said before, I'm increasingly having the same opinion of the tech bros as I have of the marketing department of the Sirius Cybernetics Corporation.

They're a bunch of mindless jerks who'll be the first against the wall when the revolution comes.

[crazy canuck]

Microsoft dropped the anthropic licenses. It was using internally.  Why?  Because it's too expensive for the benefit it produces.

If one of the most wealthy companies in the world cannot use AI tools economically, who can?

[Syt]

Meanwhile, Claude has become the Holy Grail in our company. "Ugh, ChatGPT ... can I get Claude access? But please without token limits."

[crazy canuck]

My kind of AI

https://www.instagram.com/reel/DY9rEssxgA3/?igsh=d256ZGM2ejg3OHNy

[Sheilbh]

Fascinating article on impact on software engineering with lots of interviews from people in the industry and universities - interesting because from everything I've read (but also just speaking to engineers) this is the cutting edge:
https://www.politicshome.com/news/article/software-engineering-teach-us-ai-future-work

[crazy canuck]

I think the article accurately describes what is happening now, but I disagree with your inference that it is the way of the future.

What is happening now is the competition for experienced coders has increased dramatically because they are the only ones who can spot when the AI is making mistakes. Salaries for those workers are increasing fairly dramatically.  And I would say unsustainably because of the savings in using the AI is negligible or nonexistent and so really what is happening is the overall cost of the company has increased due to the cost of the AI tool licensing and the increased salaries of the people needed to be hired to make sure mistakes aren't being made.

As a sidenote, productivity is also going down because so much time is spent tracking down the errors and trying to fix them.

[Josquius]

I've seen a lot of claims the tech firing they're blaming  on AI is mostly just the standard cycle after over hiring during the pandemic and the ending of Biden era subsidies.
Blaming AI helps them with HR and let's them promote a thing they have a hand in selling.

[crazy canuck]

I've seen a lot of claims the tech firing they're blaming  on AI is mostly just the standard cycle after over hiring during the pandemic and the ending of Biden era subsidies.
Blaming AI helps them with HR and let's them promote a thing they have a hand in selling.

E-Suite belief in the AI hype is real.   

[Jacob]

In my little corner of the games industry, I have not seen much in the way of AI impact.

We have a handful of people who are strong AI boosters, and many many more who are in the "I'm not against AI at all and want to figure out how to leverage it... but so far it hasn't made business sense" camp. In my personal life I'm strongly anti-AI, but professionally I'm actively looking for places where there are a strong business cases. Nothing has really turned up yet.

We do have some engineers who work with partners that have some kind of "use AI" mandate for their own staff, that we are also conforming to on those projects. What I've heard anecdotally (and from friends working at places with similar mandates) is pretty middling. More like "sure, it helps sometimes", but certainly no "it's revolutionizing workflows and productivity."

In the last little while we haven't been hiring juniors, that's true, but that has more to do with general headwinds in the business environment and not because AI has replaced their roles. Things have improved for us recently and if they continue to be decent, I expect we'll be hiring (strong) juniors at the same pace we did in the past. IMO, interest rates (i.e. the availability of money), risk appetite, and the project pipeline is a much greater factor in game industry hiring than AI.

For us specifically, when we're hiring engineers we are still looking for strong fundamentals (and have adjusted our hiring process to minimize the chance of a candidate using AI to fake having strong fundamentals). If those strong fundamentals can be further enhanced with AI that's great, but that's not a primary concern.

As a general observation it's also my experience that "speed of writing code" is not a primary concern in game dev, and "volume of code per unit of time" even less so. Maintainability, architecture, accountability, and managing the accumulation of tech debt is much more important; and so far it seems to me that going all in on AI code is basically a tech debt generating machine. I rather imagine it's much the same for most software development that's not Silicon Valley "ship it and modify it on the fly based on KPIs" software or the simplest of stand-alone templates software (like building simple websites etc).

Also - and this is game dev specific, but I think it may echo elsewhere as well - people investing tens if not hundreds of millions into game projects tend to look for ways to maximize returns while minimizing risk. At large scale development, a non-trivial part of that comes down to predictability. And no-one at this time (I think) has figured out how the project trajectory is affected by large-scale adoption of AI in dev. Do AI-heavy dev teams reach an Alpha-build faster (or cheaper) than non AI-heavy teams? And if they do hit Alpha faster, do they have a longer more painful (and therefore expensive) slogs from Alpha to ship due to accumulated tech debt? And how does it impact the costs of  post-launch maintenance and support? How big are the potentially savings really, and what does it do to risk?

My sense is that executives greenlighting projects are not necessarily super keen to actively take on that risk yet. This is further compounded by the fact that currently "this AAA game was built with an AI-first approach" is not necessarily a positive PR story at the moment, which in turn adds further to the risk. Now, if someone shows up with a "here's a game that's practically the same scope as GTA, with the same cultural resonance as that game; but you can buy it for $30 because we made it that much cheaper with AI", and it has enough sales, that's going to rock the industry to its foundation. But I see basically no signs of that happening yet.

Some of these things are specific to the game industry, but the "what does this do to the cost structure and risk profile of my project over its entire lifecycle" issue is fundamental to software dev in general, if not business as a whole. Obviously that's going to depend on specifics from industry to industry, and project to project; but at this stage I'm less convinced that AI replacing juniors is going to happen than I was a year or two ago.

Maybe I'm being myopic. We'll see, I suppose.

[crazy canuck]

Thanks, that is a really interesting perspective.

I am coming to the view that some limited AI agents, when used properly, can be helpful.

But those are big caveats.  An example is an AI tool being developed at the UBC law school to assist with legal research. It is not designed to give answers, but it is designed to get the researcher to think about the questions they ask themselves and the topics they are going to research.  That sort of collaboration between a researcher and an AI tool could prove very beneficial to ensure that an area that should've been researched isn't missed.

But what I'm seeing in practice is people relying on generative AI tools to spit out research answers, and that's where we see fabrications of cases and legal principles that simply do not exist.

[Jacob]

I used ChatGPT the other day to look at my crash log from (my overly modded) Skyrim and tell me what went wrong. I haven't iterated on the process to know if it'll get me out of the hole (i.e. I made changes based on the feedback, but haven't really played since so I don't know if it works). If it does I'll happily concede that that's a good use case for AI.