Kinney Drugs pulls back AI phone assistant after hundreds of customer complaints

FWIW I know of a pharmacy network that's implementing AI. Not only does the AI make egregious errors like speaking in Spanish when the conversation is in English. It also has a limited context window for ground rules of 5000 tokens or less. So correcting all the various errors goes over the instructions limit. As a result if the implementation goes through it will be of poor quality. All I can think is that people with no technical expertise are making these decisions, it is easy for any techie to predict that AI of this level won't be capable.

So - I run a company that does AI agents for pharmacy. We are at Series B now and growing well - series C in 9-12 months given growth. We just acquired a competitor of that vendor so very much an insider.

First, The technology works, and it scales, but the whole bottleneck is domain expertise and implementation. These are expensive, and hard to scale. We hire pharmacists as project managers, that's how important domain expertise and implementations are.

Second, the amount of noise of "Voice AI for <industry>" is incredible. Most of them are completely clueless about the industry & basically "YC-striver" type who can only sell to other yc companies. They fail hard the moment they touch critical functions of the real world.

We have hundreds of pharmacies with us now and they mostly successful and happy. Our biggest friction is customers being burned by vendors like this and writing off AI altogether as "not ready".

Cases like this raise a question. As an engineer, the errors that coding AIs make are an annoyance, and reassuring in terms of my job prospect.

As a consumer, the errors that AIs make are more than an annoyance. I have yet to meet an AI phone assistant that can do anything more than an explicitly programmed phone tree, and several, including the one for the pharmacy I use, that do far less. They are undoubtedly much more expensive to create and test.

Clearly there's a bubble going on, and unprincipled people chasing funding and promotions as usual, but why are companies like CVS paying more to get less?

I think it is literally to waste our time. It's one more layer of defense to stop you from talking to a person.

There are two big wins in health care. One is to increase the scope of health care that gets paid for. More and more expensive treatments, to bring the money in. The second big win is to stop people from getting treatment. Deny coverage, confuse them and wear them out with bureaucracy, and in this case stop them from talking to an expensively educated human pharmacist. That's what an AI does better than a phone tree, because it more effectively creates a sense of helplessness and valuelessness in the customer.

Expanding the scope of care you're entitled to and then fighting against your ability to get it isn't a contradiction any more than it's a contradiction when a person breathes in and breathes out. It's two synergistic parts of a system whose goal ultimately has nothing to do with health care.

Companies will pawn off thier customers to AI at their peril.

In many ways this is a repeat of the India call center train wrecks of the 00s. On paper, letting someone in Bangalore vs onshore handle incoming customer service calls looked like a path to amazing savings. In practice the customer experience was horrendous and companies CTRL-Zed these decisions and rapidly brought customer service back onshore again. AI is just that story of shortsighted decisions by weak leadership playing out all over again.

Pharmacy phone lines are a terrible first use case for voice AI. Older callers, drug names, insurance mess, zero patience. Automating "store hours" is fine. Automating refill exceptions is how you get this outcome.

This is my favorite AI customer service story:

https://arstechnica.com/tech-policy/2024/02/air-canada-must-...

> Air Canada essentially argued that “the chatbot is a separate legal entity that is responsible for its own actions”

Actual personal service from a human being is the main reason that I'd choose a small regional pharmacy over a local CVS or mail-order. It seems to me like personal service is the ONLY advantage a regional chain like this has. If I were them, I'd lean into it rather than burning customer's goodwill with AI.

my experience with AI assistants has not been that great so far, they ask for customer id numbers which are long as ... with several digits and if you group them they don't understand, for example if the number begins with two 1s and you say eleven instead of one one they don't understand you, then they are usually not trained for the local pronunciation or dialect which makes it very hard communicating, my blood pressure rises into unhealthy ranges, but if I use expletives enough I get a human on the line

Not Pharmacy, but I literally had Shopify checkout button clickjack me and then AI agent and human agents (chat, email, and phone) refuse to cancel the resulting order today. Despite contacting within 5 minutes. Given how hostile so many companies are, I question why they even bother appearing to have support.

But this doesn't seem likely to translate well to pharmacy, where there will be consequences for treating customers like vermin, as opposed to actual incentives to do it.

I wouldn't be shocked if the tech needed in places like pharmacy or other sectors where there's some level forced responsibility leads to the version that's actually palatable elsewhere.

I guess everything is ok these days as long as the President says "We screwed up [our most basic responsibility to customers], and we own that."

My partner got trapped in an email loop with a support agent and my goodness was she pissed.

Still, I got it to send a chocolate chip cookies recipe to us.

These things are horrible, but then coding agents were dumb as dogshite not too long ago, so maybe in a few years AI support agents will become useful.

if you've tried to legitimately productionize LLMs to deliver a specific repeatable outcome, rather than human editable output(text, code), you know the models still aren't there yet and the whole product experience feels hacky

This doesn’t surprise me.

Set up whatever online portal or app you want. I’d prefer to just sign in and click a button.

If I’m picking up the phone or sending an email it’s because I have a weird problem and I need to talk to a person.

It's interesting that I had to close 3 popups just to read this article, two of them were for tracking cookies

Do we think that as a society there exists some meaningful lever by which we can apply force against this type of carelessness and exploitation?

I think we need to take a good hard look at what we want out of this world and ask ourselves. Do we really need to automate this? Like really. What benefit does the world receive having an AI chatbot tell grandma the wrong doage for her meds? This is the naked extraction of value from a service that's already been offshored. Companies like these are gleeful to make their service as shitty as they can because they know their users are doing so against their will. Infact, that's become the goal.

Related: A pharmacy chain in Vermont implemented AI for efficiency (11 days ago, 40 points) https://news.ycombinator.com/item?id=49105190

wonder how much of the ai CX hate is availability bias where people really notice it when it breaks but you never see the whole picture on total impact

it seems very implementation dependant, I know a startup with incredibly realistic voice agents that most people don’t detect is an ai that outperforms humans. But ai for CX seems like an easy project to do for companies looking to cut costs so the average agent a user encounters in the wild is poor quality

How open source makes it less private?!?

> wrong dosages, and missed prescription notifications.

BS. How is the AI touching that information?

Prescription notification is outbound communication. It’s almost a recorded message. A little text to speech if it is supposed to read the drug name.

Wrong dosages?? This must be handled by the “enroll a new scrip” functionality, right? So basically B2B doctor interaction. You let AI touch that? Aren’t the vast majority of the costs from customers not doctors? None of this makes sense.

The desired speed of AI adoption is what is hindering AI adoption. Big companies are trying to sell this as magic, and it's not. It requires proper use to get anything useful out of this system. But that takes away the magic of it and the investors can't have that as the eventual share price only works if it is magic. Frontier labs will smother their product with their timelines.

I think of this as the kiosk economy. It's like how businesses introduce kiosks even when having a human employee would be more convenient, just for the sake of greater profit.

The biggest problem with AI customer service is that a human employee would've let a minor issue slide without escalating it. But a chatbot often inflames the situation, and by the time the customer reaches a human agent, they're already furious.

Most people aren't rational or logical. Non verbal feedback, like acknowledging someone's anger and showing empathy, is incredibly important.

As someone who's been doing AI call centres for a while, I'm not surprised. It's really, really hard to build voice AI that works. There are no open source solutions that work out of the box.

But the biggest problem is ASR. WER is still atrocious even with SOTA models. When you add drug names and regional accents, it's a recipe for disaster.

I happened to be in Vermont last week and was unaware of this change. I heard complaints from three separate people unprompted. It was a total dumpster fire put on top of a critical service.

> Marraffa says Burt is fully HIPAA compliant and that the AI is not open-source and does not generate or manipulate data.

We need to fight the talking point that open source is bad!

[dead]

It's important to note while people are pumping the AI bubble that a near-moron with a checklist can do this job. They replaced AI with a regular old phone tree.

AI couldn't do this job with a checklist. It would forget the entire checklist after two responses, then get fixated on the words that the customer used to complain about how dumb the AI was and loop until they hung up and called back to hopefully clear the context.

edit: Unless Kinney had been sold a product that used caller ID to carry over context, then the customer's best chances would be to scream "Agent!" over and over again (to get put on hold for 45 minutes, then transferred to the most screamed-at Philippine minimum-wage gig worker on the planet), or to show up to the drugstore and start screaming at people.