Someone I know works for a municipality in digital transformation. They have a public facing website where local residents can report things to the municipality, such as potholes, water system issues, noise complaints, etc.
The UI has this huge taxonomy with like 200 categories with three levels of nesting that route to different departments. There are multiple "other" categories.
When a resident chooses other, it becomes an employee's job to choose a different category. But 30% of the categories employees choose are wrong (about the same for residents).
It's a UX problem, a taxonomy problem, a training problem, a change management problem, a work routing problem, and all these contribute to longer resolution times.
My friend started tinkering with a small quantized local LLM, testing whether it could classify the reported issues more accurately than the public/staff. Of course it could. They're preparing to integrate it into production. Installing it will dramatically improve the UX for the residents, save staff time (at least hundreds of hours a year), improve resolution time, etc.
My friends boss mentioned it at a conference and apparently "no one else is doing this".
So yeah, we are just barely scratching the surface of the productivity opportunities LLMs offer. It's early, not because the technology isn't developed enough to help (it is), but because most people are still figuring out how it can help.
I think a lot of the reasons for this is because AI helps provide a productivity boost to non-profitable sectors of most businesses, i.e. software development, finance, HR, etc. Since these departments do not directly drive profits, there's no visible bottom line to make meaningful observations on.
I do software for a retail company now and we've been having a similar debate: AI helps me and other departments do work more efficiently, but me getting a feature out the door faster is better for the business but doesn't get more products off the shelves. So, to the shareholders and the C-suite, is AI doing anything for the company?
> Since these departments do not directly drive profits, there's no visible bottom line to make meaningful observations on.
“Bottom line” is a reference to costs, it doesn’t matter whether a department is a profit center. If AI is making these departments more efficient, it should show up in the bottom line.
AI does show up in the bottom line, but it’s not attributable to AI. How can you tell if the last month’s 20 new features vs 10 new features the same period one year ago are due to AI or simply to your developers being smarter/working harder/returning to office, etc?
> > I think a lot of the reasons for this is because AI helps provide a productivity boost to non-profitable sectors of most businesses, i.e. software development, finance, HR, etc. Since these departments do not directly drive profits, there's no visible bottom line to make meaningful observations on.
So the right place to look should be looking at free time of employees or cognitive load/stress they are under.
How is it possible to measure that?
Anyways Goldman might not be the right firm to measure it because they are not interested in anything that isn't money.
I think finding out whether or not AI is actually boosting productivity is a problem of measuring productivity period, which at the very least my current company is pretty bad at. For a developer, is their productivity their lines of code produced, hours worked, project tasks completed per unit of time, agile points completed per unit of time, PRs reviewed, PRs submitted? In more human metrics, is it what their coworkers say about them, what leadership says about them, what customers say about them, testers, QA? The amount of bugs they fix, the amount of bugs they don't ship?
Sorry for the ramble, but apply this productivity measurement conundrum to entire corporations and it's no wonder that no productivity boost is being recorded. I'd be surprised if semi-accurate productivity measurements were taken in the first place.
Wrapping business processes around these LLMs is the same kind of hard organizational problem plaguing most internal IT projects. People are still the bottleneck.
You also run into the issue of accuracy compounding. Running multi step flows with AI compounds the success rate and dramatically increases the chances of a full-job failure. E.g. even at 99% success rate for any single step, a 30-step process is only likely to succeed 75% of the time without errors. If you go down to 95% success for each, you only have a 75% likelihood of flawless execution at about 6 steps.
So it’s also about getting those per step success rates way up.
Something I'm noticing is that people in my network have lost some kind of concept of time passing
I had a conversation with an acquaintance a few weeks ago who was adamant that ChatGPT only really showed up less than a year ago. They were absolutely mind blown when I pointed out that ChatGPT got crazy popular literally years ago, late 2022, early 2023. They were convinced this was still very new stuff, like 8 months ago or so
I don't really blame people either. Personally I feel like the years since COVID have been a weird blur. People don't realize that lockdowns were half a decade ago already
> People don't realize that lockdowns were half a decade ago already
Myself included. My perception of time has been off ever since, and 2020 quite literally feels like it was just last year.
Same for my peers. I'd really be curious to see some study done on this, and why this distorted perception of time exists now. I'm not sure I can attribute it to lock-down necessarily. I was already fully remote pre-COVID, and my habits nor work schedule changed during lock downs other than wearing a mask whenever I left the house.
If it isnt showing up the they are probably depreciating the massive hardware spending over a long time, assuming that the announced billions are actually occuring.
Lots of companies are using AI already, which means there is no competitive edges against another company.
It mainly helps with mundane tasks. I think mostly employees have better life within the company do those stupid task or another email or meeting notes.
I’m not trying to hurt your pride; I’m trying to communicate a minimum comment quality.
A not-uncommon reply at this point might be: “Why don’t you tell me my mistakes?” That would be a rather shallow criticism. If you don’t see them, run your comment through a modern LLM.
You’ll learn more anyway by finding mistakes yourself than if I identify them for you.
If a comment doesn’t survive a basic sanity check with a large language model, I suggest that it’s a waste of our time here.
Keep in mind that a comment gets read more times that it is written. So, if you respect your audience, fix your logical errors.
Finally, passive aggression is common in places but hardly something to strive for. You may not appreciate this now, but one can appreciate and benefit from sincere feedback of all kinds.
My guess is they don't really know how to price it in yet - but also they seem to massively reduce headcount in areas where you can loftily make assumptions that "ai is boosting productivity"
Seems like hand waving and layoffs will have to stop before we get real data
The Altair 8800 came out in 1974 with no video output. The Commodore PET and Apple II were released in 1977 as non-kit computers with video output. VisiCalc came out in late 1979. We are approximately at the VisiCalc release date now with LLMs.
Anecdote:
Someone I know works for a municipality in digital transformation. They have a public facing website where local residents can report things to the municipality, such as potholes, water system issues, noise complaints, etc.
The UI has this huge taxonomy with like 200 categories with three levels of nesting that route to different departments. There are multiple "other" categories.
When a resident chooses other, it becomes an employee's job to choose a different category. But 30% of the categories employees choose are wrong (about the same for residents).
It's a UX problem, a taxonomy problem, a training problem, a change management problem, a work routing problem, and all these contribute to longer resolution times.
My friend started tinkering with a small quantized local LLM, testing whether it could classify the reported issues more accurately than the public/staff. Of course it could. They're preparing to integrate it into production. Installing it will dramatically improve the UX for the residents, save staff time (at least hundreds of hours a year), improve resolution time, etc.
My friends boss mentioned it at a conference and apparently "no one else is doing this".
So yeah, we are just barely scratching the surface of the productivity opportunities LLMs offer. It's early, not because the technology isn't developed enough to help (it is), but because most people are still figuring out how it can help.
Wait for the next admin to unplug the decision tool so they can re-add the jobs and backfill with their private staffing service.
Persistent negative thinking. I'm pretty sure someone posted a study just yesterday about how long term PNT seems to correlate strongly with dementia.
You know you can choose to not always assume the worst about everything, right?
I think a lot of the reasons for this is because AI helps provide a productivity boost to non-profitable sectors of most businesses, i.e. software development, finance, HR, etc. Since these departments do not directly drive profits, there's no visible bottom line to make meaningful observations on.
I do software for a retail company now and we've been having a similar debate: AI helps me and other departments do work more efficiently, but me getting a feature out the door faster is better for the business but doesn't get more products off the shelves. So, to the shareholders and the C-suite, is AI doing anything for the company?
> Since these departments do not directly drive profits, there's no visible bottom line to make meaningful observations on.
“Bottom line” is a reference to costs, it doesn’t matter whether a department is a profit center. If AI is making these departments more efficient, it should show up in the bottom line.
AI does show up in the bottom line, but it’s not attributable to AI. How can you tell if the last month’s 20 new features vs 10 new features the same period one year ago are due to AI or simply to your developers being smarter/working harder/returning to office, etc?
I would pose that question back to you, if it isn’t measurable, how are you certain that it is affecting the bottom line?
> > I think a lot of the reasons for this is because AI helps provide a productivity boost to non-profitable sectors of most businesses, i.e. software development, finance, HR, etc. Since these departments do not directly drive profits, there's no visible bottom line to make meaningful observations on.
So the right place to look should be looking at free time of employees or cognitive load/stress they are under.
How is it possible to measure that?
Anyways Goldman might not be the right firm to measure it because they are not interested in anything that isn't money.
Very much agree on your last point.
I think finding out whether or not AI is actually boosting productivity is a problem of measuring productivity period, which at the very least my current company is pretty bad at. For a developer, is their productivity their lines of code produced, hours worked, project tasks completed per unit of time, agile points completed per unit of time, PRs reviewed, PRs submitted? In more human metrics, is it what their coworkers say about them, what leadership says about them, what customers say about them, testers, QA? The amount of bugs they fix, the amount of bugs they don't ship?
Sorry for the ramble, but apply this productivity measurement conundrum to entire corporations and it's no wonder that no productivity boost is being recorded. I'd be surprised if semi-accurate productivity measurements were taken in the first place.
It’s still too early to see real benefits.
Wrapping business processes around these LLMs is the same kind of hard organizational problem plaguing most internal IT projects. People are still the bottleneck.
You also run into the issue of accuracy compounding. Running multi step flows with AI compounds the success rate and dramatically increases the chances of a full-job failure. E.g. even at 99% success rate for any single step, a 30-step process is only likely to succeed 75% of the time without errors. If you go down to 95% success for each, you only have a 75% likelihood of flawless execution at about 6 steps.
So it’s also about getting those per step success rates way up.
It has been years, when will it not be too early?
Something I'm noticing is that people in my network have lost some kind of concept of time passing
I had a conversation with an acquaintance a few weeks ago who was adamant that ChatGPT only really showed up less than a year ago. They were absolutely mind blown when I pointed out that ChatGPT got crazy popular literally years ago, late 2022, early 2023. They were convinced this was still very new stuff, like 8 months ago or so
I don't really blame people either. Personally I feel like the years since COVID have been a weird blur. People don't realize that lockdowns were half a decade ago already
> People don't realize that lockdowns were half a decade ago already
Myself included. My perception of time has been off ever since, and 2020 quite literally feels like it was just last year.
Same for my peers. I'd really be curious to see some study done on this, and why this distorted perception of time exists now. I'm not sure I can attribute it to lock-down necessarily. I was already fully remote pre-COVID, and my habits nor work schedule changed during lock downs other than wearing a mask whenever I left the house.
If it isnt showing up the they are probably depreciating the massive hardware spending over a long time, assuming that the announced billions are actually occuring.
Lots of companies are using AI already, which means there is no competitive edges against another company.
It mainly helps with mundane tasks. I think mostly employees have better life within the company do those stupid task or another email or meeting notes.
Bad logic on at least two counts. Try again.
Thank You for your input. May be next time dont reply.
I’m not trying to hurt your pride; I’m trying to communicate a minimum comment quality.
A not-uncommon reply at this point might be: “Why don’t you tell me my mistakes?” That would be a rather shallow criticism. If you don’t see them, run your comment through a modern LLM.
You’ll learn more anyway by finding mistakes yourself than if I identify them for you.
If a comment doesn’t survive a basic sanity check with a large language model, I suggest that it’s a waste of our time here.
Keep in mind that a comment gets read more times that it is written. So, if you respect your audience, fix your logical errors.
Finally, passive aggression is common in places but hardly something to strive for. You may not appreciate this now, but one can appreciate and benefit from sincere feedback of all kinds.
My guess is they don't really know how to price it in yet - but also they seem to massively reduce headcount in areas where you can loftily make assumptions that "ai is boosting productivity"
Seems like hand waving and layoffs will have to stop before we get real data
https://archive.ph/viAgg
I remember identical assertions about the pc. If you want to understand business trends, ask an accountant. In 7-10 years.
The Altair 8800 came out in 1974 with no video output. The Commodore PET and Apple II were released in 1977 as non-kit computers with video output. VisiCalc came out in late 1979. We are approximately at the VisiCalc release date now with LLMs.