External Term Format

If any program needs to transfer data to an erlang server, this data must first be serialized. That is, converted to a binary form, so that the Erlang can then unpack them back. This is usually done using ASN.1, google protobuf, thrift, etc. All this is without a doubt worthy products.

As an alternative, you can consider the use of Erlang external term format. Erlang’s executable system has two functions - term_to_binary () and binary_to_term () , which can efficiently and quickly pack / unpack any values ​​into this format, and the format itself is well described in the documentation - www.erlang.org/doc/apps/erts/ erl_ext_dist.html

How it all works.

')
Ext. The format is very simple in structure. Typically, the data in it have the form "tag, data" or "tag, length, data." The tag describes what type of data is packed.

For basic data types, tags are

• Tuple - 104, number of elements (1 byte), all elements
• List - 108, number of elements (4 bytes), all elements, 106
• Atom - 115, length (1 byte), text of an atom
• Integer - 98, value (4 bytes), or for a short integer (less than 255) - 97, value (1 byte)
• Line - 107, length (2 bytes), text. In principle, a string can be encoded as a list of integers, but if you need to transfer a short string with 8-bit text, this tag is what the doctor ordered

All the packed data should be tagged 131. This is the version number of the current ext. term format.

That is, the Erlang value [{banknote, 100, rub}] will be packaged in the structure:



As a proof of concept, we will write, for example, on a python a simple procedure that packs python structures in ext term format, and unpack the result on an erlang ...

We will pack a Python list into an Erlang list, a Python tuple into an Erlang tuple, integers, Python strings. For atoms ... well ... for example, we will create an heir from the string in python so that they can be easily distinguished in the packing procedure from ordinary strings.

It turns out something like:

from types import IntType, StringType, TupleType, ListType from struct import pack import socket class atom(str): pass def _eterm(x,accum): if type(x) is IntType: accum.append(pack('>Bi',98,x)) return if type(x) is StringType: accum.append(pack(">BH",107,len(x))) accum.append(x) return if type(x) is TupleType: accum.append(pack("BB",104,len(x))) for term in x: _eterm(term,accum) return if type(x) is ListType: accum.append(pack(">BI",108,len(x))) for term in x:_eterm(term,accum) accum.append(chr(106)) return if isinstance(x,atom): accum.append(pack("BB",115,len(x))) accum.append(x) return raise AssertionError("Cannot convert that type to erlang term %s"%(x)) def binary(X): accum = [chr(131)] _eterm(X,accum) return "".join(accum) 

Fine. Now we will pack some complicated structure and transfer it to Erlang in the simplest way.

 pterm = (atom("vcard"),[(atom("firstname"),"Odobenus"), (atom("lastname"),"Rosmarus"), (atom("age"),48), (atom("children"),[ ("Dimon",1988), ("Natashka",1990), ("Katka",2000), ("Anka",2003)] ) erlterm = binary(pterm) sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) sock.sendto(erlterm,("localhost",10000)) 

That is, they packed and sent a udp packet. In erlang we accept and decode:

 1> gen_udp:open(10000,[binary]). {ok,#Port<0.585>} 2> R=receive {udp,_ ,_,_,Bin} -> Bin end. <<131,104,2,115,5,118,99,97,114,100,108,0,0,0,4,104,2,115, 9,102,105,114,115,116,110,97,109,101,107,...>> 3> binary_to_term( R ). {vcard,[{firstname,"Odobenus"}, {lastname,"Rosmarus"}, {age,48}, {children,[{"Dimon",1988}, {"Natashka",1990}, {"Katka",2000}, {"Anka",2003}]}]} 

findings

We have transferred (by small blood) a complex structure from a python to an erlang. The advantages of this approach -

• Flexibility. You can pack anything.
• Performance. On the erlang binary_to_term () and term_to_binary () are implemented as BIF, in C, and well optimized for speed and memory ...

Lack of ext. Term format-a: Any flexibility has a downside. In the case of hand curves or stupid programming, you can pack structures with which the Erlang server just won't know what to do.

The formats of double numbers, dimensionless integers, transfer of compressed data, binary packaging into this format, etc. are outside the article. etc. But all this is well described in the documentation.

The main thing to understand the idea.