PACK和UNPACK的实现细节

在字节操作部分中迈克尔逊白色文件,其中提到:

" PACK:将数据序列化为其优化的二进制表示形式."

我想知道有关如何实现PACK和UNPACK以及二进制表示形式的更多细节的信息.我查看了源代码,实现中没有太多注释.

编辑:

用户catsigma给出的详细信息非常有用,但是我仍然希望能够跟踪序列化在源代码中的工作方式以及对二进制代码映射的操作来自何处.

我在这里找到interp函数和Pack的实现:

src/proto_alpha/lib_protocol/src/script_interpreter.ml#L697

 let rec interp =
...
| Pack t, Item (value, rest) ->
    Script_ir_translator.pack_data ctxt t value >>=? fun (bytes, ctxt) ->
    logged_return (Item (bytes, rest), ctxt)
 

pack_data在这里定义:

src/proto_alpha/lib_protocol/src/script_ir_translator.ml#L3162

 let pack_data ctxt typ data =
  unparse_data ctxt Optimized typ data >>=? fun (data, ctxt) ->
  let unparsed = strip_annotations @@ data in
  let bytes = Data_encoding.Binary.to_bytes_exn expr_encoding (Micheline.strip_locations unparsed) in
  Lwt.return @@ Gas.consume ctxt (Script.serialized_cost bytes) >>=? fun ctxt ->
  let bytes = MBytes.concat "" [ MBytes.of_string "\005" ; bytes ] in
  Lwt.return @@ Gas.consume ctxt (Script.serialized_cost bytes) >>=? fun ctxt ->
  return (bytes, ctxt)
 

Data_encoding.Binary.to_bytes_exn看起来像键函数.

src/lib_data_encoding/binary_writer.ml#L338

 let to_bytes_exn e v =
  match Encoding.classify e with
  | `Fixed n -> begin
      (* Preallocate the complete buffer *)
      let state = { buffer = MBytes.create n ;
                    offset = 0 ; allowed_bytes = Some n } in
      write_rec e state v ;
      state.buffer
    end
  | `Dynamic | `Variable ->
      (* Preallocate a minimal buffer and let's not hardcode a
         limit to its extension. *)
      let state = { buffer = MBytes.create 4096 ;
                    offset = 0 ; allowed_bytes = None } in
      write_rec e state v ;
      MBytes.sub state.buffer 0 state.offset
 

现在让我们尝试查找False的编码.这是重新整理的更多代码 src/lib_data_encoding/binary_writer.ml :

 let rec write_rec : type a. a Encoding.t -> state -> a -> unit =
  fun e state value ->
    let open Encoding in
    match e.encoding with
    | Bool -> Atom.bool state value
    ...

module Atom = struct
  let bool state v = uint8 state (if v then 255 else 0)
  let uint8 = int `Uint8
  let int kind state v =
    check_int_range (Binary_size.min_int kind) v (Binary_size.max_int kind) ;
    let ofs = state.offset in
    may_resize state (Binary_size.integer_to_size kind) ;
    set_int kind state.buffer ofs v

  let check_int_range min v max =
    if (v < min || max < v) then
      raise (Invalid_int { min ; v ; max })

  let set_int kind buffer ofs v =
    match kind with
    ...
    | `Uint8 -> MBytes.set_int8 buffer ofs v
end
 

现在,我们需要找到MBytes.set_int8.我只找到类型签名文件.

src/lib_protocol_environment/sigs/v1/mBytes.mli

mBytes看起来像是一个外部库,但我没有找到名称.另外,我看不到它与tezbridge-crypto中提到的op_mapping如何匹配.

In the Operations on bytes section of the Michelson white document, it is mentioned that:

"PACK: Serializes a piece of data to its optimized binary representation."

I am curious to know more details of how PACK and UNPACK are implemented and what the binary representation is. I have looked through the source code and there are not any many comments in the implementation.

Edit:

The details that user catsigma have given are very useful, but I would still like to be able to follow how serialization works in the source code and where the operation to binary code mappings come from.

I found the interp function here and the implementation of Pack:

src/proto_alpha/lib_protocol/src/script_interpreter.ml#L697

let rec interp =
...
| Pack t, Item (value, rest) ->
    Script_ir_translator.pack_data ctxt t value >>=? fun (bytes, ctxt) ->
    logged_return (Item (bytes, rest), ctxt)

pack_data is defined here:

src/proto_alpha/lib_protocol/src/script_ir_translator.ml#L3162

let pack_data ctxt typ data =
  unparse_data ctxt Optimized typ data >>=? fun (data, ctxt) ->
  let unparsed = strip_annotations @@ data in
  let bytes = Data_encoding.Binary.to_bytes_exn expr_encoding (Micheline.strip_locations unparsed) in
  Lwt.return @@ Gas.consume ctxt (Script.serialized_cost bytes) >>=? fun ctxt ->
  let bytes = MBytes.concat "" [ MBytes.of_string "\005" ; bytes ] in
  Lwt.return @@ Gas.consume ctxt (Script.serialized_cost bytes) >>=? fun ctxt ->
  return (bytes, ctxt)

Data_encoding.Binary.to_bytes_exn looks like the key function.

src/lib_data_encoding/binary_writer.ml#L338

let to_bytes_exn e v =
  match Encoding.classify e with
  | `Fixed n -> begin
      (* Preallocate the complete buffer *)
      let state = { buffer = MBytes.create n ;
                    offset = 0 ; allowed_bytes = Some n } in
      write_rec e state v ;
      state.buffer
    end
  | `Dynamic | `Variable ->
      (* Preallocate a minimal buffer and let's not hardcode a
         limit to its extension. *)
      let state = { buffer = MBytes.create 4096 ;
                    offset = 0 ; allowed_bytes = None } in
      write_rec e state v ;
      MBytes.sub state.buffer 0 state.offset

Now let's try to find what the encoding of False would be. Here is some more code reorganized src/lib_data_encoding/binary_writer.ml:

let rec write_rec : type a. a Encoding.t -> state -> a -> unit =
  fun e state value ->
    let open Encoding in
    match e.encoding with
    | Bool -> Atom.bool state value
    ...

module Atom = struct
  let bool state v = uint8 state (if v then 255 else 0)
  let uint8 = int `Uint8
  let int kind state v =
    check_int_range (Binary_size.min_int kind) v (Binary_size.max_int kind) ;
    let ofs = state.offset in
    may_resize state (Binary_size.integer_to_size kind) ;
    set_int kind state.buffer ofs v

  let check_int_range min v max =
    if (v < min || max < v) then
      raise (Invalid_int { min ; v ; max })

  let set_int kind buffer ofs v =
    match kind with
    ...
    | `Uint8 -> MBytes.set_int8 buffer ofs v
end

Now we need to find MBytes.set_int8. I have only found the type signature file.

src/lib_protocol_environment/sigs/v1/mBytes.mli

mBytes looks like it is an external library but I haven't found the name. Also, I am not seeing how this matches up with the op_mapping as mentioned in tezbridge-crypto.