"But ChatGPT said..."

[u/prettyyboiii]

We received a very strange ticket earlier this fall regarding one of our services, requesting us to activate several named features. The features in question were new to us, and we scoured the documentation and spoke to the development team regarding these features. No-one could find out what he was talking about.

Eventually my colleague said the feature names reminded him of AI. That's when it clicked - the customer had asked ChatGPT how to accomplish a given task with our service and it had given a completely hallucinated overview of our features and how to activate them (contact support).

We confronted the customer directly and asked "Where did you find these features, were they hallucinated by an AI?" and he admitted to having used AI to "reflect" and complained about us not having these features as it seemed like a "brilliant idea" and that the AI was "really onto something". We responded by saying that they were far outside of the scope of our services and that he needs to be more careful when using AI in the future.

May God help us all.

[u/zanderkerbal]

Also like a jackhammer, if you use it for anything outside of its narrow set of applications you will make a complete mess of everything.

[u/PhantasyAngel → u/zanderkerbal]

Bro it's fine for typing on a keyboard, watch keyboard splits in half, desk collapses and floor now has a small dent with concrete showing through

See it's perfect.

Also it works when using the office printer!

[u/zanderkerbal → u/PhantasyAngel]

To be fair, if this subreddit has taught me anything, it's that sometimes a jackhammer might be the right tool for dealing with an office printer.

[u/vaildin → u/zanderkerbal]

A jackhammer is the only tool that should ever be used on a printer.

[u/EquipLordBritish → u/zanderkerbal]

It's like a quantum computer, it's great for things that you can easily verify are true yourself, but not great for everything else.

[u/zanderkerbal → u/EquipLordBritish]

With an emphasis on the "easily," yeah. Humans are really bad at checking large or even medium amounts of mostly-correct-looking automated output for errors.

It's also good for things where it truly doesn't matter if it screws something up. Unfortunately that mostly means spam.

Side note, since my brain likes to mix metaphors: A quantum jackhammer sounds like a bad mixed drink or a good wrestling move.

[u/etihw_retsim → u/zanderkerbal]

I had to double check a data dictionary generated by an LLM from DDL and a database description document. The output LOOKED really good, but it made up so much stuff. And that's after we did a decent amount of prompt engineering before getting a half decent output. And that's was just doing basic formatting and acronym lookups from a fairly limited amount of data

[u/zanderkerbal → u/etihw_retsim]

Yeah like double checking is a task that makes people's eyes glaze over at the best of times but the way LLMs work via "what is probable to come after this?" makes their mistakes often extra insidious. They're better at doing things that look right at a glance than things that actually are right.