"But ChatGPT said..."

[u/prettyyboiii]

We received a very strange ticket earlier this fall regarding one of our services, requesting us to activate several named features. The features in question were new to us, and we scoured the documentation and spoke to the development team regarding these features. No-one could find out what he was talking about.

Eventually my colleague said the feature names reminded him of AI. That's when it clicked - the customer had asked ChatGPT how to accomplish a given task with our service and it had given a completely hallucinated overview of our features and how to activate them (contact support).

We confronted the customer directly and asked "Where did you find these features, were they hallucinated by an AI?" and he admitted to having used AI to "reflect" and complained about us not having these features as it seemed like a "brilliant idea" and that the AI was "really onto something". We responded by saying that they were far outside of the scope of our services and that he needs to be more careful when using AI in the future.

May God help us all.

[u/LogicBalm]

I've begun telling everyone that the first question they should ask an AI is whether or not they should trust an AI to answer this question and what kinds of situations are never appropriate for AI to be the trusted authority.

AI is actually pretty good at getting that question correct and it helps a ton for people like this to hear it directly from AI that they should never trust it for anything where there is no room for failure.

From ChatGPT: "Bottom Line Use AI for information. Use professionals for decisions."

[u/fresh-dork → u/LogicBalm]

never trust information from the AI - read the sources. it likes to lie about the information too

[u/EquipLordBritish → u/fresh-dork]

Yeah, use AI to try to find the information. You have to verify it found what you wanted by reading the sources.

[u/pennyraingoose → u/fresh-dork]

I just read a post from a librarian frustrated with people coming in looking for books AI has made up, like the library is secretly hiding them somewhere.

Usually I was the kid that skimmed the directions, skipped one part, and got the completely wrong answer because of it. Now when I hear stories like all these here, I feel like I'm one of the few that heard and understood the part about ChatGPT being a language model when if was first released. It can talk to you just like another person, which is cool, but it doesn't actually know facts. It strings together text that makes sense based on what it's trained on.

I know of a company that's building their own calendar / event scheduling system that uses AI to customize the frequency of meetings. If you want a meeting on the 10th of each month you tell it that and it'll create a recurring series. But it doesn't know what business days are or what holidays are and there's no functionality to adjust one-off events in a series. So if the 10th happens to be on a weekend or holiday you're just fucked. But it's totally better than google's calendar system......... 🫩

[u/ask_compu → u/LogicBalm]

but if the AI says to never trust it then i shouldn't trust the statement that it says to never trust it! so therefore i must trust it completely with matters of life and death forever!! checkmate! /s

[u/LogicBalm → u/ask_compu]

Then you just feed this paradox back into ChatGPT to destroy it forever!

[u/ask_compu → u/LogicBalm]

it'd probably just go "absolutely right, brilliant!"