Binary data

Bytes, not characters

A picture, a sound, a zip file or a network message is binary data: a row of bytes, each a number from 0 to 255, that is not text and must not be treated as text. xTalk keeps such data as it is, and chunk expressions with byte work on it. byteToNum and numToByte go between a byte and its number:

put numToByte(72) & numToByte(105) & numToByte(0) & numToByte(255) into tData
put the number of bytes of tData
put byteToNum(byte 1 of tData) && byteToNum(byte -1 of tData)
put textDecode(byte 1 to 2 of tData, "UTF-8")

4
72 255
Hi

Text becomes bytes through an encoding, and bytes become text again only through one (Unicode). Read a binary file with URL "binfile:", never URL "file:", which changes line endings and so changes the bytes (Reading and writing files).

Seeing bytes: hexadecimal

Bytes are easiest to read in hexadecimal, two digits a byte from 00 to ff. binaryDecode with the format H* turns data into hexadecimal, and binaryEncode with H* turns it back:

put textEncode("Hi!", "UTF-8") into tData
get binaryDecode("H*", tData, tHex)
put tHex
put textDecode(binaryEncode("H*", "4f58542d4265796f6e64"), "UTF-8")

486921
OXT-Beyond

binaryDecode puts its results into the variables named after the data, and returns how many it filled, here into it.

Packing numbers: binaryEncode and binaryDecode

Binary formats store numbers in a fixed number of bytes. binaryEncode packs values into bytes by a format, a letter for each value, and binaryDecode unpacks them with the same format. The letters you will use most:

put binaryEncode("CnNa3", 1, 513, 1000000, "abc") into tData
put the number of bytes of tData
get binaryDecode("H*", tData, tHex)
put tHex
get binaryDecode("CnNa3", tData, tOne, tTwo, tFour, tText)
put tOne && tTwo && tFour && tText

10
010201000f4240616263
1 513 1000000 abc

In the hexadecimal you can see each value: 01, then 513 as 0201 (2 × 256 + 1), then a million as 000f4240, then the codes of abc.

Reading a file's header

Every file format begins with a header that says what it is. A PNG picture begins with the byte 137 and the letters PNG, and keeps its width and height as two 4-byte numbers from its 17th byte. This example makes a picture, saves it as PNG, and reads its size back from the bytes:

put specialFolderPath("temporary") & "/oxt-lesson.png" into tPath
create graphic "Box"
set the rect of it to 0,0,30,20
export snapshot from graphic "Box" to file tPath as PNG

put URL ("binfile:" & tPath) into tData
put byteToNum(byte 1 of tData) && byte 2 to 4 of tData
get binaryDecode("NN", byte 17 to 24 of tData, tWidth, tHeight)
put tWidth && "by" && tHeight
delete file tPath

137 PNG
30 by 20

base64: bytes as text

Some places only take text: an email, a JSON document, a URL, a field. base64Encode writes any data as plain letters, digits, + and /, a third longer than the data, and base64Decode gives the bytes back exactly:

put numToByte(0) & numToByte(1) & numToByte(254) & numToByte(255) into tData
put base64Encode(tData) into tText
put tText
put base64Decode(tText) is tData
put base64Encode("hello, world")

AAH+/w==
true
aGVsbG8sIHdvcmxk

Compression

compress squeezes data with gzip, the same compression as a .gz file, and decompress gives it back. Text that repeats itself shrinks a lot; data that is already compressed (a PNG, a zip) hardly at all:

repeat 100 times
   put "the same words again and again " after tText
end repeat
put compress(tText) into tSmall
put the number of bytes of tText && "bytes became" && the number of bytes of tSmall
put decompress(tSmall) is tText

3100 bytes became 68
true

Bits

Inside a byte are 8 bits, each 0 or 1. Formats often pack several yes-or-no flags into one number, a bit each. bitAnd, bitOr and bitXor work on numbers bit by bit, and baseConvert shows a number in base 2 (or any base up to 36):

put baseConvert(12, 10, 2) && baseConvert(10, 10, 2)
put (12 bitAnd 10) && (12 bitOr 10) && (12 bitXor 10)
put baseConvert(255, 10, 16) && baseConvert("1010", 2, 10)

-- flags: 1 is bold, 2 is italic, 4 is underlined
put 1 bitOr 4 into tStyle
put "underlined?" && ((tStyle bitAnd 4) is not 0)
put "italic?" && ((tStyle bitAnd 2) is not 0)

1100 1010
8 14 6
FF 10
underlined? true
italic? false

function packPlayer pName, pLevel, pScore
   local tName
   put textEncode(pName, "UTF-8") into tName
   return binaryEncode("C", the number of bytes of tName) & tName & binaryEncode("nN", pLevel, pScore)
end packPlayer

function unpackPlayer pData
   local tLength, tPlayer, tLevel, tScore
   put byteToNum(byte 1 of pData) into tLength
   put textDecode(byte 2 to tLength + 1 of pData, "UTF-8") into tPlayer["name"]
   get binaryDecode("nN", byte tLength + 2 to -1 of pData, tLevel, tScore)
   put tLevel into tPlayer["level"]
   put tScore into tPlayer["score"]
   return tPlayer
end unpackPlayer

put packPlayer("Zoë", 7, 123456) into tSave
put the number of bytes of tSave && "bytes"
get binaryDecode("H*", tSave, tHex)
put tHex
put unpackPlayer(tSave) into tPlayer
put tPlayer["name"] && tPlayer["level"] && tPlayer["score"]

11 bytes
045a6fc3ab00070001e240
Zoë 7 123456