Can AI Read a Hebrew WhatsApp Message?
“אהלן זה רוני יש מקום למניקור מחר בבוקר”, and then a phone number with no dashes and a praying-hands emoji. Most Israeli small businesses get messages like that all day. No punctuation, a spelling mistake or two, and English dropped in the middle (“can i book”, “thx”, “plz”).
When the business owner reads it, it’s obvious: Roni wants a manicure tomorrow morning. The question I wanted answered is whether an AI model reads it the same way, because more Israeli businesses are now letting one answer for them.
I build Onimli (עונים לי), a Hebrew answering service for calls and WhatsApp messages. Its WhatsApp bot runs on one of OpenAI’s models today. So I’m not a neutral party here. I’m also the one paying when the model gets it wrong, and that is why I ran the test.
The test
I wrote 200 WhatsApp-style messages to imaginary small businesses: hair salons, a garage, a private maths tutor, a vet, an air-conditioner technician. Every message is fictional. No real customer conversation was used or paraphrased, and every phone number was made up for the test.
The messages were built the way Israelis really type. Openers like “שלומות” and “היי מותק”. Phone numbers written five different ways, from 0521234567 to +972-52-…. Days and times the way people say them: “after 5”, “next Sunday”, “a quarter to six in the evening”, “the 12th of the month”. Misspellings, emoji, and English mixed in.
There were five kinds of request: booking a new appointment (70), cancelling (35), moving an existing one (30), asking about price (35) and other questions such as “do you have parking?” (30).
Each model got the same instructions: read the message and return the intent, the customer’s first name, the phone number, the requested date and the requested time. The instructions also said today was Sunday, 1 November 2026, so “tomorrow” had one correct answer.
Scoring was automatic, against an answer key. No AI judge was used, so nothing depends on another model’s opinion.
I tested the current flagships from the three big labs, as listed in their official documentation on 27 October: OpenAI’s GPT-6 Astra, Google’s Gemini 4 Argon and Anthropic’s Claude Opus 5.5. I added GPT-5.6 Luna, the cheaper model that Onimli’s WhatsApp bot actually uses.
What happened
Every model understood every request. All four scored 100% on intent: none of them mixed up a cancellation with a reschedule or took a price question for a booking. All four also pulled out the phone number correctly every time, whatever its format.
Three of the four solved the whole message, every field correct, at least 98% of the time: Claude Opus 5.5 at 99.5%, Gemini 4 Argon at 99% and GPT-5.6 Luna at 98%.
The fourth result surprised me. GPT-6 Astra, OpenAI’s most expensive model, fully solved only 53.5% of the messages. It got names, phones, intents and times right without a single slip. Every one of its 93 mistakes was a date.
The test ran on 27 October, but the instructions said “today is 1 November”. Astra counted “tomorrow” and “on Thursday” from the real calendar instead of from the date it was given. “Tomorrow” became 28 October, and “Thursday” became the Thursday of the week the test ran.
In a live system the two dates are normally the same, so a business might never see the problem. It still matters. Any time the date the system hands the model differs from the model’s own idea of the date (a message read hours later, a booking system in another time zone, a test like this one), that model will quietly book the wrong day. And a wrong day is the worst kind of error, because the customer doesn’t show up and nobody knows why.
The small mistakes
The other three models made only a handful of errors, and they were very Hebrew ones.
GPT-5.6 Luna missed the name Itai twice. In “מדברת איתי”, the name איתי is spelled exactly like the Hebrew word for “with me”. A human reads it as a name without thinking. The model sometimes read it as a word. It also once read “at 11” as 11 at night.
Gemini attached a date to a question that had none. Asked “אתם עובדים בשבת?” (are you open on Saturday?), it filled in the date of the coming Saturday, as if the customer had asked to book.
Claude missed one name: “Yossi Biton” opened a message with no “this is” before it, and the model left the name field empty.
These are the mistakes you only find by testing in Hebrew. Benchmarks in English don’t have a word that is also a common first name.
What a business owner should take from it
First, for the job of understanding what a customer wants, the leading models are now good enough in everyday Hebrew. That was not obvious two years ago.
Second, the price tag doesn’t tell you which model is best for your job. The most expensive model in this test scored lowest, and the model I already pay a fraction for scored within two points of the top. Small, boutique studios such as Achiya Automation see this often when moving a client’s bot to a newer model: newer is not automatically better at a narrow task.
Third, dates are where things break. If you run an AI that books appointments, test it on “tomorrow”, “next Sunday” and “the 12th” before you trust it, and make sure the system tells the model today’s date in every message.
The limits
These were 200 messages assembled from parts I wrote, which makes them cleaner than real life. Real customers send voice notes, ask three things at once and change their minds halfway through. Because the top three scores are so close, the gaps between them are not meaningful. It was one run with one set of instructions; different wording, or the date written into the message itself, might change Astra’s result. I tested understanding only, not how well the bot replies, how fast it is or what it costs.
The full set of 200 messages, with the answer key, is free to download on Onimli’s website so anyone can repeat the test or point out a mistake in my key.
I’d like to hear from anyone running an AI on Israeli customer messages: what’s the strangest Hebrew message it got wrong, and what was it?
