How to remove invisible characters from text

· 7 min read

Removal is not one operation. Unusual spaces should become an ordinary space so words do not merge; true invisibles should be deleted; script-meaningful characters and private-use glyphs should be reported rather than removed. Getting these distinctions wrong is how cleaners corrupt legitimate multilingual text.

The four actions

ActionApplies toWhy
Replace with a spaceU+00A0, U+202F, U+2007, U+3000, U+2800 and the rest of the space familyDeleting them merges the words they separate
DeleteZero-width characters, Unicode Tags, reserved default-ignorables, noncharactersNothing legitimate renders from them
Convert to a newlineU+2028, U+2029, and the VT, FF and NEL controlsThey separate lines or records; deleting them joins unrelated content
Report, do not removePrivate-use characters, script format characters, homoglyphsThey are usually visible and often correct; the decision belongs to the reader

Context rules that prevent damage

  • Keep a zero-width joiner between two emoji, or between letters of Arabic, Persian or an Indic script.
  • Keep a zero-width non-joiner between letters of a joining script — Persian orthography depends on it.
  • Keep a joiner that follows a virama, which is how Malayalam chillu and the Marathi eyelash-ra are written.
  • Keep a variation selector after a base that legitimately takes one: an emoji, a CJK ideograph, or a Mongolian letter.
  • Keep tag characters that form one of the three real subdivision flag emoji.
  • Never treat a run of two or more chained invisibles as legitimate — no correct usage produces one.

Typography is a separate decision

Smart quotes, em dashes and ellipses are visible characters, not hidden ones. Converting them to ASCII is useful when text is heading for a plain-text field, a config file or a diff, and destructive when it is heading for publication.

That makes it a separate switch from invisible-character removal. It is still worth reporting what is there even when the switch is off, so nobody is told their document is clean when it still carries characters they cared about.

Doing it at scale

For a document, paste it into a tool and read the report. For a pipeline, sanitize at the boundary where untrusted text enters your system and keep the findings — the fact that a payload was present is worth more than the fact that it was removed.

curl -X POST https://api.zerotraceai.net/v1/clean \
  -H "Authorization: Bearer $ZEROTRACE_KEY" \
  -H "Content-Type: application/json" \
  -d '{"text": "...", "options": {"profile": "safe"}}'

Frequently asked

Will cleaning break Turkish characters?
It should not. Turkish letters such as ı, İ, ğ, ş, ç, ö and ü are ordinary letters with no invisible component, and a correct cleaner never touches them. The same applies to accented Latin, Greek and Cyrillic text.
Does removing invisible characters change how the text looks?
Almost never — that is what makes them invisible. The exceptions are unusual spaces, which become ordinary spaces of a slightly different width, and emoji sequences if you clean aggressively rather than safely.

Keep reading