Hidden characters in Arabic text: how to find and remove them

Signs that your text is hiding something

An invisible character has no shape, so you notice it only by what it does. These are the usual symptoms:

• Punctuation or numbers on the wrong side. A full stop, bracket or number lands at the wrong end of the line, or a phone number displays in a strange order. Direction marks and embeddings move neutral characters such as punctuation and spaces.
• A space that will not delete. Backspace seems to do nothing, because the first press removed an invisible character. Or a double space survives collapsing, because one of the two is a no-break space.
• Identical words that do not match. A search, an Excel lookup or a duplicate check treats two copies of a name as different, because one carries a character you cannot see.
• A username, email address or file name that gets rejected even though it looks exactly right.
• Pasted text that behaves oddly compared with the same words typed by hand, after copying from a chat app, a web page or a document.

Unicode designed direction controls to affect display, not comparison. But many search boxes, spreadsheets and login forms compare the stored characters one by one, so one extra character makes two strings different. If the only problem is order, the cause may be the paragraph direction rather than a hidden character; the first section of the reversed Arabic guide shows how to tell.

The hidden characters you will meet in Arabic text

These are the characters that turn up most in Arabic text. Each has a fixed code point, which is what the inspector reports.

UAX #9, the Unicode Bidirectional Algorithm, explains the table. The marks are never displayed, yet they affect ordering as a letter would. The standard encourages isolates over embeddings in new documents, asks for overrides to be avoided wherever possible for security reasons, and prefers markup such as the HTML dir attribute on web pages.

Find them first: the Invisible text inspector

1. Open Invisible text and paste the suspicious text, starting with one short line.
2. Read the status line: Code points: … · characters to inspect: …. Ordinary spaces, tabs and line breaks are not counted there.
3. Look at the preview. Each hidden character is replaced by a badge such as RLM, NBSP or ZWSP, so you can see where it sits.
4. Check the table. It gives each character's position, code point, official name, bidi class and joining type. Positions start at 1, and code point and UTF-16 positions are listed separately, because one emoji can take two UTF-16 units.
5. Tick Show all characters (first 300) to see everything, or open Copyable text report to share the findings.

RLMأحمدNBSPعليZWSP

This name, copied from a chat, looks ordinary but carries three characters you cannot see: a right-to-left mark at position 1, a no-break space instead of a normal space at position 6, and a zero-width space at position 10. Any one of them makes the name fail an exact match against «أحمد علي» typed by hand.

To read more about a code point, search for it in Characters, for example U+200F, or pick the Invisible & controls category. The inspector never changes your text.

Not every invisible character is junk

Before deleting anything, ask whether it was put there on purpose. It must stay in these cases:

• ZWNJ in Persian. Persian writes «می‌خواهم» and plurals such as «کتاب‌ها» with a ZWNJ that keeps the two parts unjoined without adding a space. The Unicode Standard cites the Persian plural suffix as an example and warns that dropping these characters changes the meaning. Urdu, Kurdish and other Arabic-script languages rely on it too.
• ZWJ in emoji. 👩🏽‍💻 is a woman, a skin tone, a ZWJ and a laptop. Without the ZWJ, the parts show as separate emoji.
• ZWJ in teaching material. It lets a letter show its joined form with no neighbour, as in alphabet lessons that display medial heh on its own.
• Deliberate direction marks. An editor may add an RLM to a subtitle or an app string to pin punctuation in place. If a line looks worse after removal, go back to the original.

میZWNJخواهم 👩🏽ZWJ💻

The Cleaner's direction-control removal and its zero-width space option both leave ZWNJ and ZWJ alone. Below, only the two direction marks go; the non-joiner in the Persian word and the joiner in the emoji stay:

RLMمیZWNJخواهم 👩🏽ZWJ💻RLM

میZWNJخواهم 👩🏽ZWJ💻

Remove only what you need, one option at a time

The Cleaner has a separate option for each family. Turn on only the one that matches what the inspector found.

RLMأحمدNBSPعليZWSP

أحمد علي

The name is now plain text with an ordinary space. Paste the result into the inspector again: it should read characters to inspect: 0.

The Inspect text direction panel also shows Hidden direction controls as badges, and its Preview direction setting changes only the display, adding nothing to copied text. After removing direction marks, reread any line that mixes Arabic, English, numbers and brackets.

Common cases and what to do

A login form rejects an email address or username

An address or username copied from a message or a web page may carry a direction mark at the start or a zero-width space inside. The form may refuse those characters, or treat them as part of the text and fail to find your account. Remove both and paste the result:

LRMahmed.aliZWSP@example.com

ahmed.ali@example.com

Excel will not match two identical names

Microsoft notes that TRIM was designed for the ordinary space (code 32) and does not, by itself, remove the non-breaking space (code 160). CLEAN removes only control codes 0 to 31, so direction marks and zero-width spaces stay. Clean the column in the Cleaner before pasting it into the sheet, or check one cell in the inspector.

The first column of a CSV file cannot be found

If a program reads a UTF-8 CSV saved with a BOM without expecting one, the first column name starts with U+FEFF and a lookup for «الاسم» fails. The Unicode FAQ notes that some recipients of UTF-8 data do not expect a BOM, and that two fields can hold the same content without being binary-equal. Remove zero-width space & BOM deletes it from pasted text.

BOMالاسم,المدينة

A double space that will not go away

Collapse spaces and tabs handles ordinary spaces and tabs only, so a no-break space keeps the gap open. Add Convert non-breaking spaces:

السعرNBSP 25NBSPد.ك

السعر 25 د.ك

Punctuation is still on the wrong side

If the inspector finds nothing, the paragraph direction is the problem. Set the paragraph to right-to-left in your app, or compare with Preview direction in the Inspect text direction panel. For the full routine after pasting, read cleaning copied Arabic text.