Invisible Characters: What Is Hiding in Your Text

Invisible characters are Unicode characters that occupy a position in your text and draw nothing. Most arrive by accident, carried along when you copy from one app to another, and most are harmless right up until one lands inside a hashtag.

They are not all the same thing, and the difference decides whether you should care.

Zero-width characters draw nothing at all. The zero-width space is the common one; the byte-order mark, the word joiner and the soft hyphen behave similarly. A run of them in ordinary text is padding somebody else added.

Direction controls tell a renderer which way the text runs. Harmless alone, and the override versions can visually reorder text on screen without changing what it says — which is why some moderation systems treat them as an evasion signal.

Unusual spaces are the sneakiest, because they are not invisible so much as indistinguishable. A non-breaking space looks exactly like an ordinary one and is not one. Thin spaces, hair spaces and the full-width ideographic space all read as a space to you and as something else to a search index.

Substituted punctuation is the visible member of the family. Curly quotes, en dashes, the ellipsis character — a word processor swapped them in as you typed, and they read better than the straight versions right up until the text is compared exactly.

If the same character keeps appearing, that table usually tells you which app in your workflow to stop copying out of.

The remover names every character it finds rather than quietly deleting them, because "what was in there" is the answer that changes your workflow. Deleting them only fixes today's caption.

Three failures, in descending order of how often they bite.

Hashtags and handles stop matching. A zero-width space inside a tag makes it a different string. The tag looks perfect in the caption and reaches nobody, and no platform tells you. This is the expensive one; the mechanism is covered in the zero-width space.

Search stops finding things. A non-breaking space where a word break should be joins two words into one token. Your own site search, a platform's search, and anybody using find-in-page all miss it.

Counts go wrong. Every one of these characters spends a character against your limit. A caption that mysteriously will not fit is sometimes carrying thirty invisible passengers. Run it through the character counter before and after cleaning and the difference is the answer.

A zero-width joiner is invisible, and stripping every one of them is what naive cleaners do. Inside a multi-person emoji the joiners are the emoji: they are what glues four people into one family picture. Remove them and you get four separate people, three characters longer.

What happens to emoji on other people's devices is a separate problem with its own causes — that is emoji not showing.

Two settings are worth thinking about rather than accepting.

Straightening punctuation is usually wrong. Curly quotes read better than straight ones. Straighten them only when the text is going into a URL, a filename, a code block, or anywhere it is compared exactly.

Normalising spaces is usually right. There is almost never a reason for a hair space in a caption, and a non-breaking space in the middle of a sentence is nearly always an accident.

The Unicode Consortium publishes the character database all of this comes from, including which characters have zero width and which are formatting controls. The wider set of things that break between your draft and the feed is in caption formatting.

Paste anything you are about to publish into the invisible character remover and see what came along with it.