In my experience, that’s being incredibly generous. We evaluated the ‘big names’ in AI chatbots for a project recently, and their biggest portfolio examples (not on their site; ones we went through with their consultants) trip up over depressingly trivial but relevant for the business questions. It might have been a surprise for the chatbot but, for instance, at one point the question asked was in the faq of the company but by the way we formulated it, it didn’t understand it was the same thing. FAQs are made by people who understand the business, a question is distilled from many questions that lead to the same answer; regular users won’t ask the question exactly or at all like that, which is where the AI/nlp is supposed to help. In my professional and personal experience it often does not.
Anecdotally, no one I personally know has ever been helped by a chatbot, but then again, no one I know would click on chat unless all other options (searching google) are exhausted.
Taking a few steps back to your automated phone menu; most systems that I call now use speech recognition to, for instance, have you tell it your account number, shipping code, order number etc. Even this never works for me; it has never understood what I read to it. And these are many different systems. Now sure; this might be because I talk funny or something (although my wife has the same experience and she was a radio and tv journalist), however, the human that comes after that has no issues at all understanding me reading the same information to them.
> no one I personally know has ever been helped by a chatbot
I treat chatbot like stalebot on bug reports list: as someone wishing to tell "go away, you are not wanted here" but not having enough honesty to do it openly.
I fully agree. It also seems like these chatbots are not even as good as the latest large language models. Either that or the language model conversations are very handpicked.
> "or the language model conversations are very handpicked."
This is very much the case. Experimenting with recent large models shows them to be undoubtedly powerful, but still easily distinguishable from intelligence. They regularly contradict themselves and spout unlikely narratives. Asking relatively simple questions often show a lack of understanding.
That being said, it would not surprise me if the chatbots are nevertheless worse than recent large language models.
For what it's worth, those examples are illustrative, but the best way to get a feel with them (how how handpicked examples may or may not be) is experimenting yourself with prompts.
> Experimenting with recent large models shows them to be undoubtedly powerful, but still easily distinguishable from intelligence. They regularly contradict themselves and spout unlikely narratives.
An entire generation of dreamers trying to brute force P=NP. Non-AGI AI tech can get 80-95% of the way there but it can never get to 100% (or even reasonably close) because the nature of the problem is unsolvable.
Everyone working on self driving cars is just patching infinity.
In my experience, that’s being incredibly generous. We evaluated the ‘big names’ in AI chatbots for a project recently, and their biggest portfolio examples (not on their site; ones we went through with their consultants) trip up over depressingly trivial but relevant for the business questions. It might have been a surprise for the chatbot but, for instance, at one point the question asked was in the faq of the company but by the way we formulated it, it didn’t understand it was the same thing. FAQs are made by people who understand the business, a question is distilled from many questions that lead to the same answer; regular users won’t ask the question exactly or at all like that, which is where the AI/nlp is supposed to help. In my professional and personal experience it often does not.
Anecdotally, no one I personally know has ever been helped by a chatbot, but then again, no one I know would click on chat unless all other options (searching google) are exhausted.
Taking a few steps back to your automated phone menu; most systems that I call now use speech recognition to, for instance, have you tell it your account number, shipping code, order number etc. Even this never works for me; it has never understood what I read to it. And these are many different systems. Now sure; this might be because I talk funny or something (although my wife has the same experience and she was a radio and tv journalist), however, the human that comes after that has no issues at all understanding me reading the same information to them.