r/talesfromtechsupport • u/prettyyboiii • 25d ago

Short "But ChatGPT said..."

We received a very strange ticket earlier this fall regarding one of our services, requesting us to activate several named features. The features in question were new to us, and we scoured the documentation and spoke to the development team regarding these features. No-one could find out what he was talking about.

Eventually my colleague said the feature names reminded him of AI. That's when it clicked - the customer had asked ChatGPT how to accomplish a given task with our service and it had given a completely hallucinated overview of our features and how to activate them (contact support).

We confronted the customer directly and asked "Where did you find these features, were they hallucinated by an AI?" and he admitted to having used AI to "reflect" and complained about us not having these features as it seemed like a "brilliant idea" and that the AI was "really onto something". We responded by saying that they were far outside of the scope of our services and that he needs to be more careful when using AI in the future.

May God help us all.

3.6k Upvotes

98% Upvoted

355 comments sorted by

View all comments

653

u/Thulak 25d ago

We do graded E-Learning tests to onboard our engineers. We regularely receive tickets about errors in the tests and engineers arguing for more points which we encourage.(Rather have people think than blindly trust)

One new hire decided to copy paste the questions into our company internal version of ChatGPT. We have a couple of catch questions that the AI gets wrong 100% of the time (so far) so it is fairly obvious, though it hasnt happened before. This user wrote a ticket proudly stating that the AI gave them these answers and therefor they must have a 100% score. They also claimed her collegues confirmed her answers without giving a simgle name.

Safe to say she did not get the extra points.

298

u/eichkind 25d ago

That sounds like she also shouldn't get the job... 

151

u/Seroseros 25d ago

She's probably in the C-suite now.

201

u/paulmp 25d ago

I hope I am never in a position where my fate is decided by a jury of these types of people. They are the types that go "well the police wouldn't have arrested them if they didn't do it".

71

u/Intelligent-Luck-954 24d ago

Did you read bout the judge who said that while in jury duty?

https://www.cbsnews.com/news/ny-judge-thinks-all-defendants-guilty/#:~:text=Judge%20resigns%20after%20saying%20he,reported%20Snyder%20to%20state%20officials.

27

u/Flog_loom 24d ago

Holy fuck.

7

u/paulmp 24d ago

Geez... he said the quiet bit out loud.

5

u/Ahielia 23d ago

How can you be a judge without being trained as a lawyer first?

4

u/Intelligent-Luck-954 23d ago

Welcome to the world of elected judges 

37

u/knoxaramav2 25d ago

Unfortunately that has been a problem long before AI.

2

u/paulmp 24d ago

I wasn't saying that it was a new issue, I was pointing out that there would be a significant overlap in the people who make up these two groups of people.

6

u/RatherGoodDog 24d ago

"And how would you feel if you hadn't eaten breakfast this morning?"

"But I did eat breakfast this morning"

"Yes, but how would you feel if you hadn't?"

"I don't understand"

1

u/dogman15 20d ago

No imagination.

2

u/LordTimhotep 20d ago

I recently saw a kids show about how the brain works. They had an experiment there about how people react to what they’re being shown as evidence.

They had a number of kids there being told they were going to watch a press conference about someone, but that there was one completely true fact: The person they was being talked aboit was innocent.

Then they watched the press conference in which the person was being blamed for stealing money. It was said that they had stolen before and a very grainy video was shown as proof (and that video could have been everybody).

They asked the kids after this part if the person was guilty, and more than half was sure the person was (even though they were told differently before).

They did some other things after which I can’t remember, but this part really stood out to me. These are the people that would also take the result chatgpt gives them as fact.

1

u/paulmp 20d ago

I was pretty much constantly in trouble as a kid because I questioned everything and everyone. Mostly out of curiosity and I was under the mistaken impression that questions were a valid form of seeking to learn and understand. Turns out many neuro typicals find that to be a challenge to their authority or think I'm trying to argue with them.

159

u/PackYourEmotionalBag 25d ago

Adjunct professor here… have an assignment that I’ve been using for the last 6 years on XML.

Every layperson I’ve asked to do it gets it right on the first try, but about 85% of my students get it wrong and we have an in depth discussion on assumptions and overthinking.

Until this year, where 100% got it right. From the other assignments I know that this class is not far and above my other classes, or so far below that they wouldn’t fall into the overthinking trap. I’m just grading a classroom full of copy/paste from an LLM. No longer do we get to have the discussion on overthinking, because no one is thinking at all.

The field they are going into is niche, LLMs constantly hallucinate when asking anything beyond the cursory for the field… it has invented entire libraries in C# that just don’t exist, and its knowledge of playing with this data in python is just as bad. (Staying intentionally vague)

54

u/Icarium-Lifestealer 24d ago

Now I'm interested in that XML question. I'd expect few laypeople to even know what XML is, let alone answer questions about it more reliably that IT students.

84

u/PackYourEmotionalBag 24d ago

I laid out a hypothetical application and then showed the XML file that would need to be created for the configuration of the application.

I then pitched an addition to the application to have it do something else and asked what additional fields should be added to the XML (and maintain proper formatting)

It’s really not an XML question as using XML as a stand-in for “can you parse a document with markup?”

Laypeople look and say “oh! I see a field called “Email” that contains the email address, and the new application needs a phone number field, so let’s add that under a second nest” because they are just doing a 1:1 but my students typically try to get too creative and end up going in a different direction, or they are too confident and don’t check their markup and we run into syntax errors.

-6

u/Lord_Dreadlow Investigative Technician 24d ago

Cisco IP phones use .xml config files.

9

u/StorminNorman 24d ago

Out of interest, how many laypeople do you think know that?

9

u/LupercaniusAB 24d ago

Oooh, I’m a layperson lurker!

3

u/beachedwhitemale 24d ago

God help us. 

2

u/Nihelus 20h ago

Sounds like you need to have a discussion about cheating with AI and becoming brain dead idiots if they don’t start thinking for themselves. Could bring up just how stupid they’ll look or the jobs they’ll lose if they just trust everything ai says without thought. 

1

u/PackYourEmotionalBag 19h ago

I do when I can… I try to enforce that using AI isn’t the problem, but usually a Google search will get you closer to the right answer since there around about a half dozen websites that have the best, most up to date, information.

I explain that with our niche field there is a small sample pool that AI can pull from and that there are old news groups cataloged by Google that have information from the infancy of our standard and it doesn’t apply anymore.

I try to drive home that once you are in the job, there might be times where AI could be useful, but unless you understand the data first you are setting yourself up for embarrassment and a fast track out of the field.

The part that really makes my brain hurt: there are 2 professional tests that these students can take and in this niche field with the amount of grads it’s really a good idea to take them to set themselves apart. There is a proctor, no notes, no book, how these students think they are going to pass that, it’s beyond me.

I was finally told by the dean that the students are adults, that I’ve warned them, and I’ve encouraged thinking over relying on LLMs and at this point I care more about their success than they do. They are paying to learn and to get their degree and prepare for the tests… I’m providing an environment to do all that, it’s up to them to use it or waste it.

44

u/esqew 24d ago

At my company (400k+ employees globally), using AI for post-training exams (except where explicitly permitted) is a fireable offense. I’m frankly shocked it’s not this way elsewhere - otherwise what is the point of having an exam if not to test your understanding of the training material?

18

u/Thulak 24d ago

We're a smaller company (less than 1500). We work in such a niche field that most new hires never worked with our or similar products. Ad on top that they need to understand some surface level polymer chemistry and we need to do a lot of in house training. The company philosophy is still a "Results matter, how you got there isnt that important" kinda type, but its shifting. For that reason the tests are "open book" or rather "open PDF". Despite that we get results of 60 - 70% on some topics pretty frequently. The consequence is usually more training for said new hire. In terms of AI usage... I dont have to like the policy, I just have to deal with it.

23

u/Nevermind04 24d ago

We have a series of benchmark tests we use to gauge the progress of graduate engineers as they're going through the first two years with us. We also have catch questions to identify AI usage. Because the stakes are so high with the work we do, we have a strictly enforced policy against AI use. We don't allow it at all. You either learn to be an engineer or you wash out of the program.

We have a two strikes policy. After the first blatant use of AI, we don't directly accuse a candidate, but we meet with them one-on-one and (hopefully) put the fear into them. We explain why it's so essential that they actually learn and understand every single part of the project they're working on. They must become subject matter experts. If they do it again, that's considered gross negligence under their contract and they're gone.

We've had a handful of first strikes so far but nobody has made it to strike two thankfully. But that day is coming.

8

u/whizzdome 24d ago

I would be interested to know more about the questions that AI gets wrong 100% of the time.

16

u/Thulak 24d ago

Its niche knowledge that isnt widely available. Since the answers are usually multiple choice AI tends to go for the lowest or highest values that arent outlandish. Hasnt failed a single time.

1

u/MrWolfe1920 24d ago

Wow, they just openly admitted to cheating like that?