Changelog¶
Unreleased¶
Documentation rebuilt (Sphinx, Furo, MyST) and rewritten for the current code: data types, codec options, limits; every example on the pages is run by the tests. It moves to https://pyrlang.clau.se/term/; generated HTML is no longer kept in the repository.
The native codec costs less per call: the term classes are imported once per process instead of once per call, and options
Noneor{}are not parsed at all. Decoding a small term such as a distribution control message takes a third of the time it did, encoding one about three quarters.The native codec decodes and encodes
STRING_EXTstrings about ten times faster than before, and no longer slower than the Python codec.The native encoder picks the encoding by the value’s type, as the Python codec does, not by the type’s name: a user class named
Pidorintis an object with no encoding of its own (catch_all,__etf__or the generic(b'ClassName', {...})form), and a subclass ofPid,Atom, … encodes as that term. Encoding is 2-4 times faster for pids, references, integers and bignums.The native encoder writes binaries of 4 KiB or more straight into the result: a 1 MB binary encodes as fast as OTP does it (15 times faster than before on Linux, twice on Windows). A binary of 4 GiB or more raises
PyCodecErrorinstead of being written with a wrong length.The native decoder builds lists and tuples with less overhead: deeply nested and long lists decode up to twice as fast.
Both decoders refuse a map whose keys are different in Erlang and equal in Python (
#{1 => a, 1.0 => b},trueand1,0.0and-0.0, an atom and a string of the same text, tuples of these) withPyCodecError. Such a map used to decode to a dict with fewer entries, without an error.New decode option
"atom_cache": a list or tuple of atom texts thatATOM_CACHE_REF(tag 82) refers to, for terms that follow a distribution header. Without the option tag 82 raisesPyCodecError.term.codecgainsbinary_to_term_2andterm_to_binary_2for terms without the 131 header byte.Decoding options (
atom,atom_call,byte_string) anddecode_hookapply to every nested term in both codecs. The Python codec used to apply them to the outermost term only.atom_calltakes precedence overatom.atom: "bytes"gives the UTF-8 text in both codecs, also for Latin-1 atoms. A non-callableatom_callor adecode_hookthat is not a dict raisesPyCodecError.Node names of decoded pids and references are always
term.Atom, whatever the options; hooks do not see them. A pid or reference whose node is not an atom raisesPyCodecError.The native codec decodes a zero-length
LIST_EXTwith a tailTasT, like the Python codec and OTP.Integers and strings encode as OTP writes them, in both codecs:
INTEGER_EXTfor the whole signed 32-bit range (the Python codec raisedstruct.errorfor 2^31..2^32-1, the native one wrote -2^31 as a bignum); bignums without a leading zero byte; strings of code points 0..255 and at most 65535 characters as Latin-1STRING_EXT(both codecs wrote UTF-8 bytes, so “é” arrived as two characters);""and[]asNIL_EXT; a 255-element tuple asSMALL_TUPLE_EXT(native). The native decoder readsSTRING_EXTas Latin-1 (it failed on bytes >= 128).New decode option
"bitstring": "tuple" | "BitString"(default"tuple", unchanged). With"BitString"a decoded bitstring encodes back to the same bitstring.BitStringhas equality, hash and repr. Both codecs reject a bitstring with 0 or more than 8 bits in the last byte, or with no bytes; 8 bits make a plain binary (bytes), and unused low bits of the last byte are cleared on decode and encode, as OTP does.Decoding malformed input raises only
PyCodecErrorin both codecs. The Python codec raisedIndexError,struct.error,UnicodeDecodeError,AssertionError,TypeErrororzlib.error, or silently decoded short reads. An exception raised bydecode_hookoratom_call, and theTypeErrorof an unhashable map key, become the__cause__of thePyCodecError(the native codec used to drop them); the message of the latter names the key’s type.ImproperList(elements, tail)keeps a falsy tail (0,None,"");[1|0]and[1|undefined]decoded as lists with the wrong tail.Resource limits on decoding, the same in both codecs. Decoding is iterative, so deep nesting no longer overflows the native stack (the native codec crashed the interpreter below 1000 levels in a debug build) or hits Python’s recursion limit (about 350 levels). New options:
max_depth(default 4096 nested compound terms) andmax_size(default 256 MiB, the most a compressed term may declare). Counts in the input are checked against the bytes that remain before anything is allocated (the native codec aborted on a list or tuple claiming 2^31 elements). A compressed term is inflated incrementally and only up to its declared size, and its zlib stream must be complete: a cut-off checksum used to be accepted.Encoding is iterative too, with the same
max_depthoption (default 4096): deeper values raisePyCodecErrorin both codecs instead ofRecursionError(Python, about 350 levels) or a native stack overflow (below 1000 levels in a debug build). A value that contains itself, and hooks (catch_all,__etf__) that keep returning values with no encoding, raisePyCodecErrorinstead of recursing until the stack runs out. Dicts are encoded from a snapshot of their items in the Python codec too.New
term.Port(node_name, id, creation). Both codecs decodePORT_EXT,NEW_PORT_EXTandV4_PORT_EXT(64-bit ids, mandatory between OTP 26+ nodes) and encode everyPortasV4_PORT_EXT, as OTP does. A port from an OTP node used to raiseUnknown tag.PORT_EXTwith a creation above 3 raisesPyCodecError, as in OTP.References: both codecs decode
REFERENCE_EXT(one word) and reject what OTP 29 rejects: more than 5 ID words,NEW_REFERENCE_EXTwith no words or a first word of more than 18 bits, an 8-bit creation above 3 inPID_EXT,REFERENCE_EXTandNEW_REFERENCE_EXT. AnNEWER_REFERENCE_EXTwith no ID words is rejected too (OTP 29 accepts it, reading past the term). Encoding aReferencewhose ID is not 1..5 whole words, or aPidfield beyond 32 bits, raisesPyCodecError(the Python codec raisedstruct.error` or wrote a broken term).Reference.createnumbers references with a per-process counter plus 32 random bits, so two created references are never equal (it used a clock value plus a random number).New
term.Export(module, function, arity)forfun M:F/A. Both codecs decodeEXPORT_EXTand encodeExportas OTP does; it used to raiseUnknown tag. As in OTP 29, the arity may arrive in any integer encoding but must be a non-negative small integer, and module and function stay atoms whatever the atom options (fun true:map/2keepsAtom('true')). Error messages for a node name that is not an atom are unchanged.Local funs (
Fun) encode asNEW_FUN_EXTin both codecs, with the rightSize, so a fun from Erlang can be sent back and called there. Before, the Python codec sent a(b'Fun', {...})tuple and the native codec the same. Decoding ignoresSize(as OTP does) and rejects what OTP rejects: a module that is not an atom,OldIndex/OldUniqthat are not small integers, a creator that is not a pid, more than 255 free variables. Module,OldIndex,OldUniqand the pid no longer pass through the decode hooks or atom options; free variables are alistin both codecs (native gave atuple).Funhas equality (the creator pid does not count, as in Erlang), hash and repr.Both codecs decode
FLOAT_EXT(term_to_binary(F, [{minor_version, 0}])) and accept and reject its text as OTP 29 does; floats are still encoded only asNEW_FLOAT_EXT. A NaN or infiniteNEW_FLOAT_EXTraisesPyCodecError(both codecs returnednan).Atoms have at most 255 characters, as in OTP: longer ones raise
PyCodecErroron decoding and encoding, in both codecs, wherever they appear (node names, modules too). A Latin-1 atom tag may hold at most 255 bytes, a UTF-8 one up to 1020. Both codecs used to accept up to 65535 bytes. An atom text that is not valid Unicode raisesPyCodecErrorin the Python encoder (wasUnicodeEncodeError).LOCAL_EXT(term_to_binary(T, [local]), OTP 26+) raisesPyCodecErrorsaying that only the node that encoded it can decode it, instead ofUnknown tag 121.New
term.Record(module, name, exported, fields)for OTP 29 native records (RECORD_EXT, EEP-79, experimental in OTP 29/30). Both codecs decode and encode it as OTP does: fields in definition order, flags 0 or 1 (other bits reserved), no field named twice. Module, name and field names are alwaysAtom; field values follow the decode options and hooks.Packaging:
pyproject.tomlis the only build configuration (setup.py,build-wheels.shand the stalerequirements*.txtare gone) and holds the one version;Cargo.tomlcarries it in semver form. Wheels areabi3: one per platform for every CPython >= 3.11. Without a Rust toolchain a source install now really falls back to the Python codec (the old fallback insetup.pynever ran). License metadata is the SPDXApache-2.0.CI: lint, type check and tests on Python 3.11-3.14 on Linux, Windows and macOS; OTP 25-29 judge the encoders; abi3 wheels for Linux (x86_64, aarch64), Windows and macOS (x86_64, arm64), each tested in a clean interpreter.