Persian ی and ک in Arabic text: why search misses them

Same shape, different characters

Unicode gives every letter a fixed number and does not care what it looks like. Arabic yeh and Persian yeh are two different characters, and so are Arabic kaf and Persian keheh. Software compares those numbers, not shapes, so «علي» and «علی» are two different words and a search for one never finds the other.

According to the Unicode Standard, Farsi yeh has two dots below in its initial and medial forms and no dots in its final and isolated forms. It looks exactly like ي at the start or in the middle of a word and exactly like ى at the end. So an Arabic reader sees «علی» as the preposition «على» (on), even when the writer meant the name «علي» (Ali).

Keheh is also known in Unicode as kaf mashkula. At the start and in the middle of a word it looks like Arabic kaf in most fonts; the difference usually shows at the end, where Arabic kaf carries a small inner mark and keheh does not. In some fonts the two are close enough that you cannot tell them apart by eye.

Persian digits vs Arabic-Indic digits

Digits come in two separate sets as well:

• Arabic-Indic digits ٠١٢٣٤٥٦٧٨٩, U+0660 to U+0669. Unicode notes that these are used with Arabic.
• Extended (Eastern) Arabic-Indic digits ۰۱۲۳۴۵۶۷۸۹, U+06F0 to U+06F9, used for Arabic-script languages of Iran, Pakistan and India such as Persian, Urdu and Sindhi.

Most digits look the same in both sets. The visible difference is usually in 4, 5 and 6: «٤ ٥ ٦» in Arabic, «۴ ۵ ۶» in Persian. Unicode also notes that Urdu draws some of these digits differently from Persian while using the same code points. So «۲۰۲۶» can look identical to «٢٠٢٦» in a given font, yet the two strings do not match in search or sorting, and some systems may not treat Persian digits as numbers at all.

Where mixed text comes from

• Persian and Urdu keyboard layouts. Microsoft’s Windows keyboard layout pages show that the Persian, Persian (Standard) and Urdu layouts type Farsi yeh U+06CC and keheh U+06A9 on their base keys, and that Persian (Standard) types ۰–۹ on the number row. Anyone writing Arabic on one of these layouts gets Persian letters without seeing any difference.
• Text copied from Persian or Urdu sources. Names, terms and titles taken from Persian websites or files keep their original letters.
• Fonts. Many Arabic fonts draw the Persian letters almost exactly like their Arabic look-alikes, so nothing looks wrong on screen.
• Data merged from several systems. The same name arrives in two or three spellings, and the same customer appears twice in a list.

How to tell whether your text has Persian letters

Do not trust your eyes. Use one of these instead:

1. Name every character. Paste the text into Invisible text and tick Show all characters (first 300). You get a list of each character with its position, code point and official name.
2. Look up one letter. Paste the suspect letter into Search a character, name or U+0627 in Characters to see its name, code point and joining behaviour from the Unicode data.
3. Search for it directly. Copy «ی» and «ک» from this page and search your file or sheet for them. Any hit inside Arabic text means Persian letters are there.

علي علی

The two words look alike, but the third character is ARABIC LETTER YEH in the first and ARABIC LETTER FARSI YEH in the second.

Unicode normalization will not fix this. NFC and NFKC unify equivalent encodings of the same character, and Farsi yeh is not another encoding of Arabic yeh; it is a separate letter with no decomposition. Text goes through Normalization unchanged:

علی کتاب

علی کتاب

U+0639 U+0644 U+06CC U+0020 U+06A9 U+062A U+0627 U+0628

U+0639 U+0644 U+06CC U+0020 U+06A9 U+062A U+0627 U+0628

Convert ی and ک to ي and ك, step by step

1. Open the Cleaner and paste your text into Original text.
2. Open More controls at the bottom of Transform options and find Normalize letters for search.
3. Tick Persian ی ک → Arabic ي ك. It changes exactly two letters: ی becomes ي and ک becomes ك. It leaves alef maksura «ى» alone, and other Persian letters such as پ چ ژ گ too.
4. Check the Change summary for the number of Persian letters changed, then Compare changes to see each old letter struck through next to the new one.
5. Copy or download the result, and keep the original if it has to go back to its source.

اشترى علی کتابا من المکتبة.

اشترى علي كتابا من المكتبة.

Three letters changed: the yeh in «علی» and the kaf in «کتابا» and «المکتبة». «اشترى» kept its alef maksura because it was typed with Arabic ى in the first place.

Watch the yeh at the end of words

A Persian keyboard has one main yeh key, and it types «ی». At the end of a word «ی» looks exactly like «ى», so someone writing Arabic on that layout types «علی» and «إلی» and sees nothing wrong. The Cleaner cannot know what the writer meant, so it turns every ی into ي:

سافر إلی دبی وجلس علی الکرسی.

سافر إلي دبي وجلس علي الكرسي.

«دبي» and «الكرسي» come out right, but «إلي» and «علي» are wrong here; the sentence needs «إلى» and «على». After converting, review words that end in ي, especially ones normally written with alef maksura such as على، إلى، حتى، لدى، مصطفى and موسى. Fix them by hand, or use the Cleaner’s Find & replace for a phrase you are sure of.

If all you need is a search key, you do not have to decide. Tick Persian ی ک → Arabic ي ك together with ى → ي and all three spellings become one key:

علی
على
علي

علي
علي
علي

That copy is for indexing and matching, not for publishing. See normalizing Arabic for search for the full approach.

Converting Persian digits

In the Cleaner, open Convert digits and pick the target. It converts every digit in the text, whether Western, Arabic-Indic or Persian, to one set:

رقم الطلب ۴۵۶۲۷ والمبلغ ۱۲٫۵۰۰ د.ك

رقم الطلب ٤٥٦٢٧ والمبلغ ١٢٫٥٠٠ د.ك

Choose Western — 0123456789 if your audience writes 123. The Arabic decimal separator «٫» is left as it is either way, so after converting to Western digits, replace it with a full stop yourself:

رقم الطلب ۴۵۶۲۷ والمبلغ ۱۲٫۵۰۰ د.ك

رقم الطلب 45627 والمبلغ 12٫500 د.ك

If the text contains links or email addresses with digits in them, tick Protect detected links & emails during digit conversion. World numerals does the same conversion, with Protect links and email on by default:

هاتف: ۰۹۱۲ ۳۴۵ ۶۷۸۹ - https://example.com/p/۱۲

هاتف: ٠٩١٢ ٣٤٥ ٦٧٨٩ - https://example.com/p/۱۲

World numerals also shows the Default system and Native system for each locale from CLDR data. Pick fa and you will see Persian defaults to arabext; pick ur and you will see Urdu defaults to latn, with arabext as its native system. For more on digit shapes, read the Arabic numerals guide.

When not to convert

Farsi yeh and keheh are not errors in themselves. They are the correct letters for Persian, Urdu and several other languages. Convert them only in Arabic text that picked them up by accident.

این کتاب خیلی خوب است.

اين كتاب خيلي خوب است.

That was a correct Persian sentence; after conversion it is spelled with letters Persian does not use. So do not convert:

• Persian or Urdu text, including quotations from it inside Arabic text.
• Persian or Urdu proper names that must keep their original spelling, as in references and official documents.
• Mixed-language text without review. Convert the Arabic part only and leave the rest as it was.
• A file that goes back to a Persian or Urdu source. Make a search copy and keep the original.

Urdu also uses letters this option does not touch, such as ے ہ ھ. Converting does not make Urdu text Arabic; it leaves a mix of both spellings.