Non-ASCII URL Example
The hurricane carried my letter to Ceylon and brought it back with an accent no European quill can reproduce.
The short answer
- What it is
- A non-ASCII URL has a character outside the basic English set in its path or query, either raw or percent-encoded as %80 and above. An accented letter in a path is the usual example.
- Why it matters
- These URLs are valid and useful for many languages, but they can be encoded in more than one way. A mismatch can create duplicates or broken links.
- How a crawler should report it
- Report a notice that shows both the decoded form and the exact bytes requested. Flag raw and percent-encoded forms alike, and warn more strongly if two encodings of one path both appear in the crawl.
- Live examples
/lab/non-ascii-urls/kav%C3%A1rna/(positive)/lab/non-ascii-urls/near-miss/(near miss)
Accented Letters and Percent Encoding in Paths
A non-ASCII URL carries letters beyond plain English characters, such as the á in kavárna. On the wire, a URL can only use a small set of ASCII characters. So the letter is converted to UTF-8 bytes and written as percent escapes. The path becomes /kav%C3%A1rna/. Host names follow a different system called IDNA, which turns a name into a string that starts with xn--.
Non-English sites use these URLs for good reasons. A slug in the reader's own language is clear and honest. Trouble starts when one system handles the text in a new way. An old app may encode text in Latin-1, giving %E1 where UTF-8 gives %C3%A1. A slug can be encoded twice, which turns %C3 into %25C3. Unicode also has two ways to write some accented letters, and each form gives different bytes.
Browsers show the readable form in the address bar but send the encoded form. Copying the link often pastes the long encoded text, which looks messy in email or chat. Google can crawl and index these URLs, and it recommends UTF-8 encoding for them. It decodes them for display in results. Each byte form is still a distinct URL, so two encodings of the same word can act as duplicates.
To detect the issue, list any URL with bytes above the ASCII range or escapes from %80 up. Decode each one and check that it is valid UTF-8. Group URLs by their decoded text to catch twins. To fix a mismatch, choose one Unicode form, usually NFC, and one encoding, UTF-8. Redirect the others to it with a 301. Encode URLs in your sitemap. If your readers use English only, a plain ASCII slug is simpler.