The AI dooooooom thread

[Jacob]

He also said that "prompt engineering" is not trivial to get "what you want". That current workflows involves working with one chat (agent?) to figure out the prompts you want to feed the chat that's doing the actual coding, and that this is complicated by Claude telling you that you're super smart and totally right when you're wrong.

[Jacob]

I read that as a worker checks the ais work. If thats so it raises a question if that means checked because it needed to be corrected or juar checked as a quality control thing.

No that's not what I meant. I meant that of all the work done by people using AI, around 15% of that output is actually used.

[HVC]

Ah, nvm then

[Valmy]

I read that as a worker checks the ais work. If thats so it raises a question if that means checked because it needed to be corrected or juar checked as a quality control thing.

No that's not what I meant. I meant that of all the work done by people using AI, around 15% of that output is actually used.

I wonder how much Amazon is paying per line of usable code. A few million dollars?

[Jacob]

I wonder how much Amazon is paying per line of usable code. A few million dollars?

Good question.

And to be fair, that number would mean more if there was a comparable statistic for non-AI code. Like, if non-AI-assisted code is usable at 8% (seems a stretch, but for arguments sake) then AI is really good. If it's closer to 90% then AI coding is very inefficient... assuming, of course, you've corrected for time to get those statistics (i.e. if AI code is usable at 15% compared to 45% for non-AI-assisted coding, but AI generates code at 15x over time then it's still potentially efficient).

[HVC]

But think of the labour savings! They must be saving at least 100k in labour for every million spent on ai and on top of that some manager is making bank on labour cost cutting kpi's

[DGuller]

Anecdotal note - I just had a conversation with someone who recently left Amazon. He said that they have measurements of how much AI generated work gets checked into the code base, and it hovers around 15%.

I think that underestimates the impact of AI.  I personally do not let AI write code, unless it's a trivial but tedious piece of code that is easy to describe but takes some time to do right with due diligence. 

However, in the past, I often asked AI to give me an example, or to tell me how to code something, or to create documentation based on my code.  None of that is AI-generated code checked into anything, but it's still QOL improvement, or it saves me a lot of research time.  I do that much less now, but only because in my new job I have Copilot instead of something that functions.

[DGuller]

He also said that "prompt engineering" is not trivial to get "what you want". That current workflows involves working with one chat (agent?) to figure out the prompts you want to feed the chat that's doing the actual coding, and that this is complicated by Claude telling you that you're super smart and totally right when you're wrong.

I never used Claude, but at least with ChatGPT I found it very effective to tell it in the general instructions that you're looking for Dutch level bluntness, not bootlicking.  I also tell in my general instructions that I would be regularly testing it by saying something deliberately wrong and seeing whether I get any pushback.  Sometimes it works too well, and it lectures me that Kyle Busch is still alive, assuming that I'm testing it, but for the most part it calibrates well.

[Jacob]

I think that underestimates the impact of AI.  I personally do not let AI write code, unless it's a trivial but tedious piece of code that is easy to describe but takes some time to do right with due diligence.

Well it's a metric specific to how Amazon uses AI in its coding workflows. It's probably nothing more than a single data point if you're assessing the impact of AI overall, and you'd probably need additional context on how Amazon operates to make that data point useful.

However, in the past, I often asked AI to give me an example, or to tell me how to code something, or to create documentation based on my code.  None of that is AI-generated code checked into anything, but it's still QOL improvement, or it saves me a lot of research time.  I do that much less now, but only because in my new job I have Copilot instead of something that functions.

Yeah that's sensible enough. I think that data-point is more to the "can we replace devs with AI" conversation, and less to the "how much of an efficiency multiplier is AI for individual devs" one.

[Jacob]

I never used Claude, but at least with ChatGPT I found it very effective to tell it in the general instructions that you're looking for Dutch level bluntness, not bootlicking.  I also tell in my general instructions that I would be regularly testing it by saying something deliberately wrong and seeing whether I get any pushback.  Sometimes it works too well, and it lectures me that Kyle Busch is still alive, assuming that I'm testing it, but for the most part it calibrates well.

I'm glad you've find ways to use it that works for you.

If you were to estimate how much faster / better you've gotten, could you put a number on it?

Like, has your positive outcomes output increased by 10% compared to pre-AI? By 100%?

[Sheilbh]

Good question.

And to be fair, that number would mean more if there was a comparable statistic for non-AI code. Like, if non-AI-assisted code is usable at 8% (seems a stretch, but for arguments sake) then AI is really good. If it's closer to 90% then AI coding is very inefficient... assuming, of course, you've corrected for time to get those statistics (i.e. if AI code is usable at 15% compared to 45% for non-AI-assisted coding, but AI generates code at 15x over time then it's still potentially efficient).

Yeah - I mean I think it's also sort of fundamentally slightly different things. So first of all I slightly wonder about volume of code as a metric (see Elon Musk taking over X and judging engineers by how much code they write). In a similar way, on this I have a friend who's an academic research scientist rather than an engineer but obviously being able to do statistical analysis is a huge part of his job. He was the earliest and heaviest adopter of AI I know and it wasn't necessarily because it could do what he needed - but because he could explain what he needed and then see what it produced (often wrong, but explicably wrong) which he could then refine. It might not reach the answer but was a useful way in getting him to the answer.

On the flipside I've known engineering teams who have outsourced explicitly on the instruction not to use AI and then got code that is clearly AI-generated and is unusable - but part of that is because what they need is a human looking at their (very complex and specific/build culture) codebase. In that case I actually slightly suspect AI might be a better solution than outsourcing but I'm not 100%

But also the bit I find particularly interesting is that my experience is that there'll be a four eyes review before code is pulled, but I'm not necessarily sure that checking and assessing engineers via % of code produced is necessarily how engineering teams work in my experience. So I wonder if the possibly better and slightly more alarming metric is the amount of time it takes check that AI-generated code v the time to train, mentor and improve an entry-level engineer to reach that point (and then how you produce the next generation of checkers - or if that becomes the job?).

Edit: And as someone from a similar-ish job (lawyers and engineers are very, very similar ) I don't know how that works because I have learned by doing. At least in the UK the fundamental way of training lawyers is basically like an apprenticeship. And I don't know how you skip to training people capable of checking work - that feels to me like a fundamentally different skillset.

[DGuller]

I never used Claude, but at least with ChatGPT I found it very effective to tell it in the general instructions that you're looking for Dutch level bluntness, not bootlicking.  I also tell in my general instructions that I would be regularly testing it by saying something deliberately wrong and seeing whether I get any pushback.  Sometimes it works too well, and it lectures me that Kyle Busch is still alive, assuming that I'm testing it, but for the most part it calibrates well.

I'm glad you've find ways to use it that works for you.

If you were to estimate how much faster / better you've gotten, could you put a number on it?

Like, has your positive outcomes output increased by 10% compared to pre-AI? By 100%?

I really hate putting numbers on things that are not quantifiable, and in my case the contribution of AI is not quantifiable.  I'm not a software engineer, I'm a data scientist.  I'm not limited by the number of lines of code my fingers can type out.  My value add comes from coming up with good insights, or finding the best modeling solution.

In my previous job where I did have functional AI, the main benefit was research assistance and documentation.  The benefit of research assistance was enabling me to invent an algorithm that I'm not sure I would be able to invent the old-fashioned way.  How much did it improve my productivity?  No idea, but if I didn't invent the algorithm, the model would've probably been taken down to avoid regulatory risk.  How much did documentation add to my productivity?  I have no idea, but it definitely added to my actuarial life expectancy, because documenting code takes a toll on my mental health.

If a company were to decide whether they would be better off with two DGullers, or one DGuller with good AI, my guess is they'd be better off with two DGullers, especially since now DGuller would have someone smart to talk to.  So I guess the productivity enhancement is less than 100%, but beyond that, I have no idea how I would even begin to quantify it.

[crazy canuck]

But the business case for AI its productivity impact is quantifiable.

[garbon]

Ah, nvm then

Hey same

[Jacob]

I really hate putting numbers on things that are not quantifiable, and in my case the contribution of AI is not quantifiable.  I'm not a software engineer, I'm a data scientist.  I'm not limited by the number of lines of code my fingers can type out.  My value add comes from coming up with good insights, or finding the best modeling solution.

In my previous job where I did have functional AI, the main benefit was research assistance and documentation.  The benefit of research assistance was enabling me to invent an algorithm that I'm not sure I would be able to invent the old-fashioned way.  How much did it improve my productivity?  No idea, but if I didn't invent the algorithm, the model would've probably been taken down to avoid regulatory risk.  How much did documentation add to my productivity?  I have no idea, but it definitely added to my actuarial life expectancy, because documenting code takes a toll on my mental health.

If a company were to decide whether they would be better off with two DGullers, or one DGuller with good AI, my guess is they'd be better off with two DGullers, especially since now DGuller would have someone smart to talk to.  So I guess the productivity enhancement is less than 100%, but beyond that, I have no idea how I would even begin to quantify it.

That's all very resonable.

From one management perspective, if you feel the tool helps you, you like using it, and you're confident it makes you more productive then that's enough for me... as long as the costs are reasonable.

As the costs become less reasonable (and what that is will vary across businesses), the pressure for showing measurable improved outcomes will increase.

At an executive and strategic level, they have to evaluate whether completely upending workflows across the organization provides an advantage of some sort... or whether it doesn't. Typically, at organizations the scale of Amazon, it has to be quantifiable and measurable.

I think that 15% number is an attempt to start coming to grips with that question, even if its far from adequate on its own. But as an organization, it needs to figure it out at some point.

I too am curious about the actual impact beyond "yeah, I find it helpful", though I'm more ready to be satisfied by individual anecdotes compared to Amazon execs.