Unicode and text encodings

Text in any language

xTalk text is Unicode: it can hold every writing system in use, and emoji. Chunk expressions work on characters as a reader sees them, whatever the language:

put "héllo wörld" into tText
put the number of chars of tText
put toUpper(tText)
put "日本語のテキスト" into tJapanese
put the number of chars of tJapanese
put char 1 to 3 of tJapanese

11
HÉLLO WÖRLD
8
日本語

Codepoints

Unicode gives every character a number, its codepoint. codepointToNum and numToCodepoint go between the two:

put codepointToNum("A") && codepointToNum("é") && codepointToNum("€")
put numToCodepoint(9731)
put numToCodepoint(128512)

65 233 8364
☃
😀

Characters, codepoints and code units

What a reader sees as one character can be more than one codepoint: an e followed by a combining accent is one character, é, made of two codepoints. And inside the engine some codepoints take two code units. Usually you want chars; the others are there when you need them:

put "e" & numToCodepoint(769) into tAccented
put the number of chars of tAccented
put the number of codepoints of tAccented
put numToCodepoint(128512) into tSmile
put the number of chars of tSmile
put the number of codeunits of tSmile

1
2
1
2

Comparing accented text

The same word can be stored in two ways: with é as one codepoint, or as e plus an accent. They look identical, and xTalk treats them as the same, as it ignores case: is, offset and array keys all see one word. But their bytes differ, so once the text leaves xTalk, written to a file, hashed or sent to a server, they are two different words. Set the formSensitive to true to compare the forms exactly, and use normalizeText to put text into one standard form, NFC:

put "caf" & "e" & numToCodepoint(769) into tDecomposed
put tDecomposed is "café"
put the number of bytes of textEncode(tDecomposed, "UTF-8") && the number of bytes of textEncode("café", "UTF-8")
set the formSensitive to true
put tDecomposed is "café"
put normalizeText(tDecomposed, "NFC") is "café"

true
6 5
false
true

Text and bytes: encodings

Files and networks carry bytes, not characters. An encoding is the rule that turns one into the other, and today's standard is UTF-8, which uses one byte for plain English letters and two to four for the rest. textEncode turns text into bytes, and textDecode turns bytes back into text:

put "naïve café" into tText
put textEncode(tText, "UTF-8") into tBytes
put the number of chars of tText
put the number of bytes of tBytes
put textDecode(tBytes, "UTF-8") is tText

10
12
true

Ten characters made twelve bytes: ï and é take two each. Other encodings make other bytes from the same text:

put "café" into tText
put the number of bytes of textEncode(tText, "UTF-8")
put the number of bytes of textEncode(tText, "ISO-8859-1")
put the number of bytes of textEncode(tText, "UTF-16")

5
4
8

The rule to remember: decode as text comes in, encode as it goes out. Read a file's bytes and textDecode them; textEncode text before you write it to a file or send it. The next lessons, Reading and writing files and Files a piece at a time, do exactly that.

function codepointList pText
   put empty into tList
   repeat for each codepoint tPoint in pText
      put format("U+%04X", codepointToNum(tPoint)) & space after tList
   end repeat
   return word 1 to -1 of tList
end codepointList

put codepointList("Añ€")

U+0041 U+00F1 U+20AC