Example of LLM confusion over a word with multiple meanings

In our system we can “hide” a user conversation in two ways.

  • There are internal “service” conversations that are technically owned by a user, but are not supposed to be visible on the main screen. E.g. automatically scheduled jobs. They still can be examined in more detailed screens.
  • There is a security system where user A can make a conversation “public” (visible for all) or “hide” it, i.e. make it visible only for that user.

Even though in both cases we can talk about “hiding” a conversation, these are two completely different things and humans never confuse these.
The LLM, however, was not as smart: I was talking to claude-4.5-opus-high about service conversation for some time, but when I asked how can I create a hidden conversation in the UI, it referred me to the button that operates on the security setting, because it happens to use the word “hide”. Rather disappointing.

Posted in

Leave a Reply

Your email address will not be published. Required fields are marked *