From 9facb66ee8da7ab102be8d68bcfbec9da2d0a42c Mon Sep 17 00:00:00 2001 From: Herwin Bozet Date: Tue, 3 Nov 2026 16:51:37 +0100 Subject: [PATCH] Added examples, Added performance tests, Updated readme Co-Authored-By: Claude --- Claude.MD | 173 ++++++++-- Examples/BasicArrays/BasicArrays.pb | 102 ++++++ Examples/BasicBinary/BasicBinary.pb | 97 ++++++ .../BasicExplicitTyping.pb | 138 ++++++++ Examples/BasicExtensions/BasicExtensions.pb | 98 ++++++ .../BasicImplicitTyping.pb | 8 +- Examples/BasicMaps/BasicMaps.pb | 170 +++++++++ Examples/BasicStrings/BasicStrings.pb | 92 +++++ .../DumpExplicitTypes/DumpExplicitTypes.pb | 322 ++++++++++++++++++ .../DumpImplicitTypes/DumpImplicitTypes.pb | 253 ++++++++++++++ PerfTests/DeepNesting/PerfDeepNesting.pb | 172 ++++++++++ PerfTests/DeepNesting/RunPerfTests.ps1 | 15 + PerfTests/FlatMaps/DotNet/DotNetPerf.csproj | 13 + PerfTests/FlatMaps/DotNet/Program.cs | 48 +++ PerfTests/FlatMaps/PerfFlatMaps.pb | 111 ++++++ PerfTests/FlatMaps/PerfPython.py | 37 ++ PerfTests/FlatMaps/RunPerfTests.ps1 | 18 + PerfTests/FlatMaps/_Workload.pbi | 15 + PerfTests/Investigations.md | 59 ++++ .../DotNet/DotNetPerf.csproj | 13 + PerfTests/MassiveReadAmount/DotNet/Program.cs | 62 ++++ PerfTests/MassiveReadAmount/PerfExplicit.pb | 111 ++++++ PerfTests/MassiveReadAmount/PerfImplicit.pb | 132 +++++++ PerfTests/MassiveReadAmount/PerfPython.py | 54 +++ PerfTests/MassiveReadAmount/RunPerfTests.ps1 | 18 + PerfTests/MassiveReadAmount/_Workload.pbi | 15 + PerfTests/NullChecks/PerfNullChecksOff.pb | 6 + PerfTests/NullChecks/PerfNullChecksOn.pb | 6 + PerfTests/NullChecks/RunPerfTests.ps1 | 15 + PerfTests/NullChecks/_Scenarios.pbi | 96 ++++++ .../StringsAndBinary/DotNet/DotNetPerf.csproj | 13 + PerfTests/StringsAndBinary/DotNet/Program.cs | 77 +++++ PerfTests/StringsAndBinary/PerfPython.py | 42 +++ .../StringsAndBinary/PerfStringsAndBinary.pb | 174 ++++++++++ PerfTests/StringsAndBinary/RunPerfTests.ps1 | 18 + PerfTests/StringsAndBinary/_Workload.pbi | 14 + PerfTests/_PerfTesting.cs | 23 ++ PerfTests/_PerfTesting.pbi | 52 +++ PerfTests/_PerfTesting.ps1 | 132 +++++++ PerfTests/_PerfTesting.py | 17 + PerfTests/readme.md | 117 +++++++ .../requirements.txt | 0 RunTests.ps1 | 96 ++++++ readme.md | 83 ++++- 44 files changed, 3295 insertions(+), 32 deletions(-) create mode 100644 Examples/BasicArrays/BasicArrays.pb create mode 100644 Examples/BasicBinary/BasicBinary.pb create mode 100644 Examples/BasicExplicitTyping/BasicExplicitTyping.pb create mode 100644 Examples/BasicExtensions/BasicExtensions.pb create mode 100644 Examples/BasicMaps/BasicMaps.pb create mode 100644 Examples/BasicStrings/BasicStrings.pb create mode 100644 Examples/DumpExplicitTypes/DumpExplicitTypes.pb create mode 100644 Examples/DumpImplicitTypes/DumpImplicitTypes.pb create mode 100644 PerfTests/DeepNesting/PerfDeepNesting.pb create mode 100644 PerfTests/DeepNesting/RunPerfTests.ps1 create mode 100644 PerfTests/FlatMaps/DotNet/DotNetPerf.csproj create mode 100644 PerfTests/FlatMaps/DotNet/Program.cs create mode 100644 PerfTests/FlatMaps/PerfFlatMaps.pb create mode 100644 PerfTests/FlatMaps/PerfPython.py create mode 100644 PerfTests/FlatMaps/RunPerfTests.ps1 create mode 100644 PerfTests/FlatMaps/_Workload.pbi create mode 100644 PerfTests/Investigations.md create mode 100644 PerfTests/MassiveReadAmount/DotNet/DotNetPerf.csproj create mode 100644 PerfTests/MassiveReadAmount/DotNet/Program.cs create mode 100644 PerfTests/MassiveReadAmount/PerfExplicit.pb create mode 100644 PerfTests/MassiveReadAmount/PerfImplicit.pb create mode 100644 PerfTests/MassiveReadAmount/PerfPython.py create mode 100644 PerfTests/MassiveReadAmount/RunPerfTests.ps1 create mode 100644 PerfTests/MassiveReadAmount/_Workload.pbi create mode 100644 PerfTests/NullChecks/PerfNullChecksOff.pb create mode 100644 PerfTests/NullChecks/PerfNullChecksOn.pb create mode 100644 PerfTests/NullChecks/RunPerfTests.ps1 create mode 100644 PerfTests/NullChecks/_Scenarios.pbi create mode 100644 PerfTests/StringsAndBinary/DotNet/DotNetPerf.csproj create mode 100644 PerfTests/StringsAndBinary/DotNet/Program.cs create mode 100644 PerfTests/StringsAndBinary/PerfPython.py create mode 100644 PerfTests/StringsAndBinary/PerfStringsAndBinary.pb create mode 100644 PerfTests/StringsAndBinary/RunPerfTests.ps1 create mode 100644 PerfTests/StringsAndBinary/_Workload.pbi create mode 100644 PerfTests/_PerfTesting.cs create mode 100644 PerfTests/_PerfTesting.pbi create mode 100644 PerfTests/_PerfTesting.ps1 create mode 100644 PerfTests/_PerfTesting.py create mode 100644 PerfTests/readme.md rename requirements.txt => PerfTests/requirements.txt (100%) create mode 100644 RunTests.ps1 diff --git a/Claude.MD b/Claude.MD index 7fd5704..09b9f39 100644 --- a/Claude.MD +++ b/Claude.MD @@ -1,17 +1,24 @@ # MessagePack.pbi PureBasic include for creating/reading MessagePack data. -**When integrating, do not read any `.pbi` files to save on tokens, use the API below.** +**When integrating, do not read any `.pbi` files to save on tokens, use the API below.** \ +When working on the include itself, see [Working on this repository](#working-on-this-repository). ## APIs ### Behaviour -* ??? +* All the procedures take a `*MsgPackData` from `MsgPackCreate`, `MsgPackWrapMsgBuffer` or `MsgPackExamine`. +* Errors are reported through the internal error code, see `MsgPackLastError`. + * Writes and other procedures returning a `.MsgPack_ErrorCode` also return it directly. + * Reads returning a value give `0`, `""`, `#False`, `#Null` or `#MsgPack_InvalidSize` on errors, + which can also be valid values, so check `MsgPackLastError` after them. + * A successful read or write resets it to `#MsgPack_Error_Success`. +* Failed reads and writes don't move the internal offset, and write nothing. * The internal offset is auto-incremented on all complete reads and writes. - * Does not apply to `MsgPackGetFormatCode`. + * Does not apply to `MsgPackGetFormatCode`, `MsgPackGetExtensionType`, `MsgPackGetEntryCount` and `MsgPackGetFieldFullSize`. * Can be disabled with the internal `#MsgPack_DataFlags_MoveOffsetAfterRead/Write` flags, honored by every read and write. -* The internal buffer is automatically resized as needed. +* The internal buffer is automatically grown as needed, unless it is owned by the caller. * Invalid or truncated data never raises a `DebuggerError`, only programming errors do. (e.g.: `#Null` pointers) * Views from `MsgPackExamine` are read-only, writes return `#MsgPack_Error_BufferIsReadOnly`. @@ -20,7 +27,7 @@ PureBasic include for creating/reading MessagePack data. * `.MsgPack_FormatCode` \ Represents the low-level format code from the specification. \ - When [interacting] with fix* types, the value in it will be set to 0. + Fix* types are returned without the value/size bits they embed, e.g. `$A0` for any FixStr and `0` for positive fixints. * `#MsgPack_FormatCode_Null` * `#MsgPack_FormatCode_False` * `#MsgPack_FormatCode_True` @@ -63,7 +70,7 @@ PureBasic include for creating/reading MessagePack data. * `.MsgPack_DataType` \ High-level representation of a format code. \ Should be used when iterating on complex objects to detect the members' general type. - * `#MsgPack_DataTypes_Unknown` - The `0x??` format code. + * `#MsgPack_DataTypes_Unknown` - For `#MsgPack_FormatCode_Invalid`. * `#MsgPack_DataTypes_Null` * `#MsgPack_DataTypes_Boolean` * `#MsgPack_DataTypes_Integer` @@ -72,13 +79,14 @@ PureBasic include for creating/reading MessagePack data. * `#MsgPack_DataTypes_Binary` * `#MsgPack_DataTypes_Array` * `#MsgPack_DataTypes_Map` - * `#MsgPack_DataTypes_Ext` * `#MsgPack_DataTypes_Extension` * `.MsgPack_ErrorCode` \ * `#MsgPack_Error_Success = 0` * `#MsgPack_Error_NullPointerGiven` * `#MsgPack_Error_BufferTooSmall` + * `#MsgPack_Error_InvalidValue` + * `#MsgPack_Error_ValueOverflow` * `#MsgPack_Error_FailedToGrowBuffer` * `#MsgPack_Error_AtOrPastTheEndOfBuffer` * `#MsgPack_Error_ReadPastEndOfBuffer` @@ -91,27 +99,63 @@ PureBasic include for creating/reading MessagePack data. * `#MsgPack_Error_InvalidStringSize` * `#MsgPack_Error_TooManyEntries` * `#MsgPack_Error_OddMapFieldCount` + * `#MsgPack_Error_MaxDepthExceeded` * `#MsgPack_Error_CannotGrowBufferDueToConfig` * `#MsgPack_Error_BufferIsReadOnly` * `.MsgPack_DataFlag` \ Internal flags, do not touch. +* `.MsgPack_ExaminationOption` \ + Options for `MsgPackExamine`. + * `#MsgPack_ExaminationOptions_Unsafe` -### Commons + +### Constants & compiler options +* `#MsgPack_InvalidSize = -1` \ + Returned by procedures that return a size, on errors. + +* `#MsgPack_MaxDepth` \ + Maximum nesting depth of arrays and maps, a top-level array or map being at depth 1. \ + Defaults to `128`, can be defined before the inclusion, from `1` to `4096`. \ + Deeper data is rejected with `#MsgPack_Error_MaxDepthExceeded`. + +* `#MsgPack_DisableNullChecks` \ + Disables the `#Null` pointer checks when defined as `#True` before the inclusion. \ + Defaults to `#False`. + + +### Errors & format codes * `MsgPackLastError.MsgPack_ErrorCode(*MsgPackData.MsgPackData)` \ - ... + Returns the error code set by the last read, write or navigation on `*MsgPackData`, `#MsgPack_Error_Success` if it succeeded. \ + Returns `#MsgPack_Error_NullPointerGiven` if `*MsgPackData` is `#Null`. + +* `MsgPackGetFormatCode.MsgPack_FormatCode(*MsgPackData.MsgPackData)` \ + Returns the format code of the field at the current internal offset, without moving it or setting the internal error code. \ + Returns `#MsgPack_FormatCode_Invalid` at the end of the buffer, or for the `0xC1` byte. + +* `MsgPackFormatCodeToDataType.MsgPack_DataType(FormatCode.MsgPack_FormatCode)` \ + Returns the data type of a format code from `MsgPackGetFormatCode`. \ + Returns `#MsgPack_DataTypes_Unknown` for `#MsgPack_FormatCode_Invalid`. -### Thingies -* `MsgPackCreate.i(InitialSize.i)` \ +### Allocation & ownership +* `MsgPackCreate.i(InitialSize.i, GrowthIncrement.i = 1)` \ Prepares an internal structure for a buffer of a given size `InitialSize.i`. \ + The buffer grows in multiples of `GrowthIncrement` bytes, `1` growing it by the exact size needed by each write. \ Returns `#Null` on errors. -* `MsgPackWrapMsgBuffer.i(*Buffer, BufferSize.i)` \ - Wraps a given buffer `*Buffer` of a given size `InitialSize.i` into an internal structure. \ +* `MsgPackWrapMsgBuffer.i(*Buffer, BufferSize.i, GiveOwnership.b = #False, GrowthIncrement.i = 1)` \ + Wraps a given buffer `*Buffer` of a given size `BufferSize.i` into an internal structure, without copying it. \ + If `GiveOwnership` is `#False`, the buffer stays owned by the caller, and is never grown or freed. \ + If `GiveOwnership` is `#True`, `*Buffer` must come from `AllocateMemory()`, it can be grown, and freed with `MsgPackFree`. \ Returns `#Null` on errors. +* `MsgPackGrow.MsgPack_ErrorCode(*MsgPackData.MsgPackData, DesiredSizeIncrease.q)` \ + Grows the buffer by `DesiredSizeIncrease` bytes, rounded up to a multiple of its `GrowthIncrement`. \ + Called by the writes as needed, there's no need to call it beforehand. \ + Returns `#MsgPack_Error_Success` on success, something else otherwise. (e.g.: `#MsgPack_Error_CannotGrowBufferDueToConfig` for buffers owned by the caller) + * `MsgPackFree.b(*MsgPackData.MsgPackData, FreeDataBuffer.b)` \ Frees a given internal structure `*MsgPackData` and its associated buffer if `FreeDataBuffer` is `#True`. \ `FreeDataBuffer` is mandatory, use `#False` for buffers wrapped with `MsgPackWrapMsgBuffer` that the caller still owns. \ @@ -126,29 +170,28 @@ PureBasic include for creating/reading MessagePack data. ### Literals -* `MsgPackReadBoolean.b(*MsgPackData.MsgPackData)` +* `MsgPackReadBoolean.b(*MsgPackData.MsgPackData)` \ Attempts to read the boolean at the current internal offset. \ - Returns `#True` or `#False`. \ - May set the internal error code if any happens. \ + Returns `#True` or `#False`, `#False` on errors too. \ See: `MsgPackLastError` -* `MsgPackWriteNull.MsgPack_ErrorCode(*MsgPackData.MsgPackData)` - Attempts to write `#Null` at the current internal offset. \ +* `MsgPackWriteNull.MsgPack_ErrorCode(*MsgPackData.MsgPackData)` \ + Attempts to write a nil at the current internal offset. \ Returns `#MsgPack_Error_Success` on success, something else otherwise. -* `MsgPackWriteTrue.MsgPack_ErrorCode(*MsgPackData.MsgPackData)` +* `MsgPackWriteTrue.MsgPack_ErrorCode(*MsgPackData.MsgPackData)` \ Attempts to write `#True` at the current internal offset. \ Returns `#MsgPack_Error_Success` on success, something else otherwise. -* `MsgPackWriteFalse.MsgPack_ErrorCode(*MsgPackData.MsgPackData)` +* `MsgPackWriteFalse.MsgPack_ErrorCode(*MsgPackData.MsgPackData)` \ Attempts to write `#False` at the current internal offset. \ Returns `#MsgPack_Error_Success` on success, something else otherwise. -* `MsgPackWriteBoolean.MsgPack_ErrorCode(*MsgPackData.MsgPackData, Boolean.b)` +* `MsgPackWriteBoolean.MsgPack_ErrorCode(*MsgPackData.MsgPackData, Boolean.b)` \ Writes `#True` or `#False` depending on `Boolean`. \ Returns `#MsgPack_Error_Success` on success, something else otherwise. -* `MsgPackReadNull.MsgPack_ErrorCode(*MsgPackData.MsgPackData)` +* `MsgPackReadNull.MsgPack_ErrorCode(*MsgPackData.MsgPackData)` \ Attempts to read a nil at the current internal offset. \ Returns `#MsgPack_Error_Success` on success, something else otherwise, in which case the offset isn't moved. @@ -316,6 +359,49 @@ EndIf Returns `#MsgPack_Error_TooManyEntries` if the entries don't fit in it. +## Usage patterns + +### Writing, exporting and reading +```purebasic +Define *Writer = MsgPackCreate(64) +If MsgPackWriteInteger(*Writer, 42) <> #MsgPack_Error_Success + ; ... +EndIf + +; Frees `*Writer`, the caller now owns `*Buffer`. +Define *Buffer = MsgPackFinalize(*Writer) + +; Gives `*Buffer` back to a structure, freed along with it. +Define *Reader = MsgPackWrapMsgBuffer(*Buffer, MemorySize(*Buffer), #True) +Define Value.q = MsgPackReadInteger(*Reader) +If MsgPackLastError(*Reader) <> #MsgPack_Error_Success + ; ... +EndIf + +MsgPackFree(*Reader, #True) +``` + +### Building arrays & maps +Their content is written in a separate structure, then copied in with its fields counted. +```purebasic +Define *Fields = MsgPackCreate(32) +MsgPackWriteString(*Fields, "Key") +MsgPackWriteInteger(*Fields, 1) +MsgPackWriteMapFrom(*Writer, *Fields) +MsgPackFree(*Fields, #True) +``` +Forcing their format code requires the content as a buffer, e.g. from `MsgPackFinalize`, for `MsgPackWriteFixArray/Array16/...`. + +### Iterating over a buffer +The end of a buffer can't be queried, so either wrap the data in a top-level array/map, +or subtract each top-level field's `MsgPackGetFieldFullSize` from the buffer's size until it reaches `0`. + +### Missing or asymmetric procedures +* Floats have no implicitly typed setter, use `MsgPackWriteFloat32/64`. +* Strings have no explicitly typed getters, `MsgPackReadString` reads all str format codes. +* Fixints have no explicitly typed getter, `MsgPackReadInteger` reads them. + + ## Remarks * Never touch the `MsgPackData` structure unless you **absolutely** have to. @@ -323,3 +409,46 @@ EndIf ## Support - Both the ASM and C backends - x86, x64, arm64 + + +## Working on this repository + +### Layout +* `Includes/MessagePack.pbi` - Main file, only includes the others. +* `Includes/_*.pbi` - One file per family. (Allocators, Literals, Primitives, Strings, ...) +* `Includes/PB-Endianness/` - Git submodule, don't edit it from here. +* `UnitTests/` - One folder per family, numbered like the includes' order. +* `Examples/` - One folder per example, with a `.pb` of the same name. +* `PerfTests/` - Benchmarks, one folder per workload, each with a `RunPerfTests.ps1`. +* `SecurityTests/` - Private, git-ignored, leave it alone unless asked. + +### Conventions +* `.pb`/`.pbi` files use CRLF line endings and ASCII only, `Chr()` for anything else. +* Includes use tabs, examples use 4 spaces. +* Private procedures and macros are prefixed with `_MsgPack`, public ones with `MsgPack`. +* Every public procedure is documented above its definition, and in this file. +* `#Null` pointers raise a `DebuggerError` behind `CompilerIf Not #MsgPack_DisableNullChecks`, invalid data only uses `Debug`. + +### Unit tests +* Run with `RunTests.ps1`, which builds each test without the debugger, so `Debug` & co are compiled out. +* Each test includes `../_UnitTesting.pbi`, uses its `Assert*` procedures, and ends with `End 1` if `FailedUnitTests` isn't `0`. +* `*FatalErrors.pb` tests call `DebuggerError` on purpose, and only pass without the debugger. +* No PureBasic compiler is available in the cloud sandbox, ask the user to run the tests. + +### Performance tests +* Run with the `RunPerfTests.ps1` of each folder, Windows only. + * They dot-source `PerfTests/_PerfTesting.ps1`, which builds the folder's `.pb` files without the debugger, with `/OPTIMIZER` on the C backend, + runs `PerfPython.py` and `DotNet/` if present, and saves the results in `PerfTests/Results/__-.csv`. + * PureBasic 6.41 and 6.21 are configured in `$Compilers`, 6.21 being kept to compare the string handling changes of 6.40. +* Each implementation times its scenarios with the shared `_PerfTesting.pbi`/`.py`/`.cs`: one warm-up run, then the median of 5. +* Each program prints `Implementation,Scenario,MedianMs,NsPerOp,Check` lines on stdout, errors go to stderr. +* `Check` must be identical for a scenario across implementations, the scripts report mismatches. +* Each folder's workload is described in its `_Workload.pbi`, keep its `PerfPython.py` and `DotNet/Program.cs` in sync with it. +* Python packages are pinned in `PerfTests/requirements.txt`, MessagePack-CSharp in each `DotNetPerf.csproj`. +* The results tables are in `PerfTests/readme.md`, the conclusions in the main readme's "Performances" section, keep both in sync with new results. +* Open performance questions and their leads are in `PerfTests/Investigations.md`, update it when one is looked into. + +### Examples +* Start with a `;- Notes` section, then `;- Example`. +* Check every result, and use a different `End` code for each failure. +* Link new examples in the readme's "Examples" section. diff --git a/Examples/BasicArrays/BasicArrays.pb b/Examples/BasicArrays/BasicArrays.pb new file mode 100644 index 0000000..f4e25b7 --- /dev/null +++ b/Examples/BasicArrays/BasicArrays.pb @@ -0,0 +1,102 @@ +; ------------------------------------------------------------------------------ +;- Notes + +; This example shows you how to write an array from entries written in a +; separate writer, and how to iterate on it. + + +; ------------------------------------------------------------------------------ +;- Example + +EnableExplicit + +XIncludeFile "../../Includes/MessagePack.pbi" + +; Obscure structures that hold info about the MsgPack data +Define *Writer, *Entries, *Reader, *Iterator + +; MsgPack-encoded buffer that will get exported midway in this example. +Define *Buffer + +Define i.i, Value.q + + + +; Writing the entries in their own writer. +*Entries = MsgPackCreate(64) +If Not *Entries + Debug "Failed to create the entries writer !" + End 1 +EndIf + +For i = 1 To 5 + If MsgPackWriteInteger(*Entries, i * 10) <> #MsgPack_Error_Success + Debug "Failed to write entry #" + Str(i) + " !" + End 2 + EndIf +Next + + +; Writing the array, the entries are counted and copied into it. +*Writer = MsgPackCreate(64) +If Not *Writer + Debug "Failed to create the writer !" + End 3 +EndIf + +If MsgPackWriteArrayFrom(*Writer, *Entries) <> #MsgPack_Error_Success + Debug "Failed to write the array !" + End 4 +EndIf + +; The entries writer is left untouched, so it must be freed. +MsgPackFree(*Entries, #True) + + +; Exporting the data and giving it to a reader. +*Buffer = MsgPackFinalize(*Writer) +If Not *Buffer + Debug "Failed to finalize the writer !" + End 5 +EndIf + +Debug "Encoded size: " + Str(MemorySize(*Buffer)) + +*Reader = MsgPackWrapMsgBuffer(*Buffer, MemorySize(*Buffer), #True) +If Not *Reader + Debug "Failed to wrap the buffer !" + End 6 +EndIf + + +; Iterating on the array. +Debug "Entry count: " + Str(MsgPackGetEntryCount(*Reader)) + +*Iterator = MsgPackExamine(*Reader) +If Not *Iterator + Debug "Failed to examine the array !" + End 7 +EndIf + +While MsgPackNextEntry(*Iterator) + Value = MsgPackReadInteger(*Iterator) + If MsgPackLastError(*Iterator) <> #MsgPack_Error_Success + Debug "Failed to read an entry !" + End 8 + EndIf + + Debug "Entry: " + Str(Value) +Wend + +; `MsgPackNextEntry` also returns `#False` on errors. +If MsgPackLastError(*Iterator) <> #MsgPack_Error_Success + Debug "Failed to iterate on the array !" + End 9 +EndIf + + +; Freeing the view first, it reads from the reader's buffer. +MsgPackFinish(*Iterator) +MsgPackFree(*Reader, #True) + +End 0 diff --git a/Examples/BasicBinary/BasicBinary.pb b/Examples/BasicBinary/BasicBinary.pb new file mode 100644 index 0000000..34dd789 --- /dev/null +++ b/Examples/BasicBinary/BasicBinary.pb @@ -0,0 +1,97 @@ +; ------------------------------------------------------------------------------ +;- Notes + +; This example shows you how to write and read raw binary data. +; It can either be copied into a buffer you own, or into a new one. + + +; ------------------------------------------------------------------------------ +;- Example + +EnableExplicit + +XIncludeFile "../../Includes/MessagePack.pbi" + +; Obscure structures that hold info about the MsgPack data +Define *Writer, *Reader + +; MsgPack-encoded buffer that will get exported midway in this example. +Define *Buffer + +; Raw data to encode, and buffers that will receive the decoded one. +Define *Input = AllocateMemory(4) +Define *Output = AllocateMemory(16) +Define *Copy +Define BytesRead.q + +If Not *Input Or Not *Output + Debug "Failed to allocate the buffers !" + End 1 +EndIf + +PokeL(*Input, $DEADBEEF) + + + +; Writing the same data twice into the buffer. +*Writer = MsgPackCreate(64) +If Not *Writer + Debug "Failed to create the writer !" + End 2 +EndIf + +If MsgPackWriteBinary(*Writer, *Input, MemorySize(*Input)) <> #MsgPack_Error_Success + Debug "Failed to write the first binary field !" + End 3 +EndIf + +If MsgPackWriteBinary(*Writer, *Input, MemorySize(*Input)) <> #MsgPack_Error_Success + Debug "Failed to write the second binary field !" + End 4 +EndIf + + +; Exporting the data and giving it to a reader. +*Buffer = MsgPackFinalize(*Writer) +If Not *Buffer + Debug "Failed to finalize the writer !" + End 5 +EndIf + +Debug "Encoded size: " + Str(MemorySize(*Buffer)) + +*Reader = MsgPackWrapMsgBuffer(*Buffer, MemorySize(*Buffer), #True) +If Not *Reader + Debug "Failed to wrap the buffer !" + End 6 +EndIf + + +; Copying the first field into a buffer we own. +; It fails if the buffer is too small, without moving the offset. +BytesRead = MsgPackReadBinary(*Reader, *Output, MemorySize(*Output)) +If BytesRead = #MsgPack_InvalidSize + Debug "Failed to read the first binary field !" + End 7 +EndIf + +Debug "Read " + Str(BytesRead) + " bytes: $" + Hex(PeekL(*Output), #PB_Long) + + +; Copying the second field into a new buffer of the right size. +*Copy = MsgPackCopyBinary(*Reader) +If Not *Copy + Debug "Failed to copy the second binary field !" + End 8 +EndIf + +Debug "Copied " + Str(MemorySize(*Copy)) + " bytes: $" + Hex(PeekL(*Copy), #PB_Long) + + +; Freeing everything. +FreeMemory(*Copy) +FreeMemory(*Output) +FreeMemory(*Input) +MsgPackFree(*Reader, #True) + +End 0 diff --git a/Examples/BasicExplicitTyping/BasicExplicitTyping.pb b/Examples/BasicExplicitTyping/BasicExplicitTyping.pb new file mode 100644 index 0000000..0b8052f --- /dev/null +++ b/Examples/BasicExplicitTyping/BasicExplicitTyping.pb @@ -0,0 +1,138 @@ +; ------------------------------------------------------------------------------ +;- Notes + +; This example shows you how to write, export and read a MessagePack buffer +; with explicitly typed variables whose format code is forced by the +; procedure used to write them, even if a smaller one would fit. + + +; ------------------------------------------------------------------------------ +;- Example + +EnableExplicit + +XIncludeFile "../../Includes/MessagePack.pbi" + +Structure MyDataPacket + Id.q + UserName.s +EndStructure + +Procedure.b EncodeMyDataPacket(*MsgPackData, *Input.MyDataPacket) + ; Always written as an Int64, 9 bytes in total. + If MsgPackWriteInt64(*MsgPackData, *Input\Id) <> #MsgPack_Error_Success + ProcedureReturn #False + EndIf + + ; Always written as a Str8, which only fits up to 255 bytes. + If MsgPackWriteStr8(*MsgPackData, *Input\UserName) <> #MsgPack_Error_Success + ProcedureReturn #False + EndIf + + ProcedureReturn #True +EndProcedure + +Procedure.b DecodeMyDataPacket(*MsgPackData, *Output.MyDataPacket) + ; Fails if the field isn't an Int64. + *Output\Id = MsgPackReadInt64(*MsgPackData) + If MsgPackLastError(*MsgPackData) <> #MsgPack_Error_Success + ProcedureReturn #False + EndIf + + ; Strings don't have explicit getters, any str format code is accepted. + *Output\UserName = MsgPackReadString(*MsgPackData) + If MsgPackLastError(*MsgPackData) <> #MsgPack_Error_Success + ProcedureReturn #False + EndIf + + ProcedureReturn #True +EndProcedure + + + +; Obscure structures that hold info about the MsgPack data +Define *Writer, *Reader + +; MsgPack-encoded buffer that will get exported midway in this example. +Define *Buffer + +; Structure that will hold the data to encode and the decoded one. +; You could replace this with whatever you want in your use cases. +Define InputStruct.MyDataPacket, OutputStruct.MyDataPacket + + + +; Preparing the input structure. +With InputStruct + \Id = 67 + \UserName = "Herwin" +EndWith + + +; Writing the data into the buffer. +*Writer = MsgPackCreate(64) +If Not *Writer + Debug "Failed to create the writer !" + End 1 +EndIf + +If Not EncodeMyDataPacket(*Writer, @InputStruct) + Debug "Failed to encode the packet !" + End 2 +EndIf + + +; Exporting the data. +; Doing this frees the obscure structure in `*Writer`. +*Buffer = MsgPackFinalize(*Writer) +If Not *Buffer + Debug "Failed to finalize the writer !" + End 3 +EndIf + +; 9 bytes for the Int64, 2+6 bytes for the Str8. +Debug "Encoded size: " + Str(MemorySize(*Buffer)) + + +; Giving ownership of the buffer to the obscure structure. +*Reader = MsgPackWrapMsgBuffer(*Buffer, MemorySize(*Buffer), #True) +If Not *Reader + Debug "Failed to wrap the buffer !" + End 4 +EndIf + + +; Getting data out of the MsgPack buffer. +If Not DecodeMyDataPacket(*Reader, @OutputStruct) + Debug "Failed to decode the packet !" + End 5 +EndIf + +Debug "Id: " + Str(OutputStruct\Id) +Debug "UserName: " + OutputStruct\UserName + +; Freeing the obscure structure and the buffer it owns. +MsgPackFree(*Reader, #True) + + +; Forcing a format code that is too small for the data fails. +; Str8 only fits up to 255 bytes, so a 256 bytes name can't be written. +InputStruct\UserName = LSet("", 256, "A") + +*Writer = MsgPackCreate(64) +If Not *Writer + Debug "Failed to create the writer !" + End 6 +EndIf + +If EncodeMyDataPacket(*Writer, @InputStruct) + Debug "Encoded a 256 bytes name as Str8, this shouldn't happen !" + End 7 +EndIf + +Debug "Failed to encode a 256 bytes name as Str8, as expected." + +; Nothing was exported, so the writer still owns its buffer. +MsgPackFree(*Writer, #True) + +End 0 diff --git a/Examples/BasicExtensions/BasicExtensions.pb b/Examples/BasicExtensions/BasicExtensions.pb new file mode 100644 index 0000000..5572ddd --- /dev/null +++ b/Examples/BasicExtensions/BasicExtensions.pb @@ -0,0 +1,98 @@ +; ------------------------------------------------------------------------------ +;- Notes + +; This example shows you how to write and read extensions, which are binary +; data tagged with an application-defined type, here a 2D vector. +; Types from 0 to 127 are free to use, -128 to -1 are reserved by the spec. + + +; ------------------------------------------------------------------------------ +;- Example + +EnableExplicit + +XIncludeFile "../../Includes/MessagePack.pbi" + +#ExtType_Vector2 = 1 + +Structure Vector2 + X.f + Y.f +EndStructure + +; Obscure structures that hold info about the MsgPack data +Define *Writer, *Reader + +; MsgPack-encoded buffer that will get exported midway in this example. +Define *Buffer + +; Structure that will hold the data to encode and the decoded one. +Define InputVector.Vector2, OutputVector.Vector2 +Define ExtType.b + + + +; Preparing the input structure. +With InputVector + \X = 1.5 + \Y = -2.25 +EndWith + + +; Writing the vector into the buffer. +; It is 8 bytes long, so it is written as a FixExt8. +*Writer = MsgPackCreate(64) +If Not *Writer + Debug "Failed to create the writer !" + End 1 +EndIf + +If MsgPackWriteExtension(*Writer, #ExtType_Vector2, @InputVector, SizeOf(Vector2)) <> #MsgPack_Error_Success + Debug "Failed to write the extension !" + End 2 +EndIf + + +; Exporting the data and giving it to a reader. +*Buffer = MsgPackFinalize(*Writer) +If Not *Buffer + Debug "Failed to finalize the writer !" + End 3 +EndIf + +Debug "Encoded size: " + Str(MemorySize(*Buffer)) + +*Reader = MsgPackWrapMsgBuffer(*Buffer, MemorySize(*Buffer), #True) +If Not *Reader + Debug "Failed to wrap the buffer !" + End 4 +EndIf + + +; Checking the type first, it doesn't move the offset. +ExtType = MsgPackGetExtensionType(*Reader) +If MsgPackLastError(*Reader) <> #MsgPack_Error_Success + Debug "Failed to get the extension type !" + End 5 +EndIf + +If ExtType <> #ExtType_Vector2 + Debug "Unexpected extension type: " + Str(ExtType) + End 6 +EndIf + + +; Copying its data into the output structure. +If MsgPackReadExtension(*Reader, @OutputVector, SizeOf(Vector2)) <> SizeOf(Vector2) + Debug "Failed to read the extension !" + End 7 +EndIf + +Debug "X: " + StrF(OutputVector\X) +Debug "Y: " + StrF(OutputVector\Y) + + +; Freeing the obscure structure and the buffer it owns. +MsgPackFree(*Reader, #True) + +End 0 diff --git a/Examples/BasicImplicitTyping/BasicImplicitTyping.pb b/Examples/BasicImplicitTyping/BasicImplicitTyping.pb index dbfd0e7..37948de 100644 --- a/Examples/BasicImplicitTyping/BasicImplicitTyping.pb +++ b/Examples/BasicImplicitTyping/BasicImplicitTyping.pb @@ -1,8 +1,8 @@ ; ------------------------------------------------------------------------------ ;- Notes -; This example show you how to write, export and read a MessagePack buffer -; with implicitely typed variables whose format code is automatically +; This example shows you how to write, export and read a MessagePack buffer +; with implicitly typed variables whose format code is automatically ; resolved based on the given data. @@ -92,8 +92,7 @@ EndIf Debug "Encoded size: " + Str(MemorySize(*Buffer)) -; It also gives ownership of the buffer to the obscure structure. -; And it is marked internally as read-only. +; Giving ownership of the buffer to the obscure structure. *Reader = MsgPackWrapMsgBuffer(*Buffer, MemorySize(*Buffer), #True) If Not *Reader Debug "Failed to wrap the buffer !" @@ -102,7 +101,6 @@ EndIf ; Getting data out of the MsgPack buffer. -OutputStruct.MyDataPacket If Not DecodeMyDataPacket(*Reader, @OutputStruct) Debug "Failed to decode the packet !" End 5 diff --git a/Examples/BasicMaps/BasicMaps.pb b/Examples/BasicMaps/BasicMaps.pb new file mode 100644 index 0000000..5b0804e --- /dev/null +++ b/Examples/BasicMaps/BasicMaps.pb @@ -0,0 +1,170 @@ +; ------------------------------------------------------------------------------ +;- Notes + +; This example shows you how to write a structure as a map whose keys are +; its fields' names, and how to read it back by iterating on it. +; Unknown keys are skipped, and the keys' order doesn't matter. + + +; ------------------------------------------------------------------------------ +;- Example + +EnableExplicit + +XIncludeFile "../../Includes/MessagePack.pbi" + +Structure MyDataPacket + Id.q + UserName.s + IsAdmin.b +EndStructure + +; Writes the keys and values one after the other, alternating them. +Procedure.b EncodeMyDataPacketFields(*MsgPackData, *Input.MyDataPacket) + If MsgPackWriteString(*MsgPackData, "Id") <> #MsgPack_Error_Success + ProcedureReturn #False + EndIf + If MsgPackWriteInteger(*MsgPackData, *Input\Id) <> #MsgPack_Error_Success + ProcedureReturn #False + EndIf + + If MsgPackWriteString(*MsgPackData, "UserName") <> #MsgPack_Error_Success + ProcedureReturn #False + EndIf + If MsgPackWriteString(*MsgPackData, *Input\UserName) <> #MsgPack_Error_Success + ProcedureReturn #False + EndIf + + If MsgPackWriteString(*MsgPackData, "IsAdmin") <> #MsgPack_Error_Success + ProcedureReturn #False + EndIf + If MsgPackWriteBoolean(*MsgPackData, *Input\IsAdmin) <> #MsgPack_Error_Success + ProcedureReturn #False + EndIf + + ProcedureReturn #True +EndProcedure + +Procedure.b EncodeMyDataPacket(*MsgPackData, *Input.MyDataPacket) + Protected *Fields + Protected Success.b = #False + + ; Writing the keys and values in their own writer. + *Fields = MsgPackCreate(64) + If Not *Fields + ProcedureReturn #False + EndIf + + ; Writing the map, the fields are counted and copied into it. + If EncodeMyDataPacketFields(*Fields, *Input) + Success = Bool(MsgPackWriteMapFrom(*MsgPackData, *Fields) = #MsgPack_Error_Success) + EndIf + + ; The fields writer is left untouched, so it must be freed. + MsgPackFree(*Fields, #True) + + ProcedureReturn Success +EndProcedure + +Procedure.b DecodeMyDataPacket(*MsgPackData, *Output.MyDataPacket) + Protected *Iterator + Protected Key.s + + *Iterator = MsgPackExamine(*MsgPackData) + If Not *Iterator + ProcedureReturn #False + EndIf + + ; Each entry is read key first, then value. + While MsgPackNextEntry(*Iterator) + Key = MsgPackReadString(*Iterator) + + Select Key + Case "Id" + *Output\Id = MsgPackReadInteger(*Iterator) + Case "UserName" + *Output\UserName = MsgPackReadString(*Iterator) + Case "IsAdmin" + *Output\IsAdmin = MsgPackReadBoolean(*Iterator) + EndSelect + + If MsgPackLastError(*Iterator) <> #MsgPack_Error_Success + Break + EndIf + Wend + + ; `MsgPackNextEntry` also returns `#False` on errors. + If MsgPackLastError(*Iterator) <> #MsgPack_Error_Success + MsgPackFinish(*Iterator) + ProcedureReturn #False + EndIf + + MsgPackFinish(*Iterator) + ProcedureReturn #True +EndProcedure + + + +; Obscure structures that hold info about the MsgPack data +Define *Writer, *Reader + +; MsgPack-encoded buffer that will get exported midway in this example. +Define *Buffer + +; Structure that will hold the data to encode and the decoded one. +Define InputStruct.MyDataPacket, OutputStruct.MyDataPacket + + + +; Preparing the input structure. +With InputStruct + \Id = 67 + \UserName = "Herwin" + \IsAdmin = #True +EndWith + + +; Writing the data into the buffer. +*Writer = MsgPackCreate(64) +If Not *Writer + Debug "Failed to create the writer !" + End 1 +EndIf + +If Not EncodeMyDataPacket(*Writer, @InputStruct) + Debug "Failed to encode the packet !" + End 2 +EndIf + + +; Exporting the data and giving it to a reader. +*Buffer = MsgPackFinalize(*Writer) +If Not *Buffer + Debug "Failed to finalize the writer !" + End 3 +EndIf + +Debug "Encoded size: " + Str(MemorySize(*Buffer)) + +*Reader = MsgPackWrapMsgBuffer(*Buffer, MemorySize(*Buffer), #True) +If Not *Reader + Debug "Failed to wrap the buffer !" + End 4 +EndIf + + +; Getting data out of the MsgPack buffer. +If Not DecodeMyDataPacket(*Reader, @OutputStruct) + Debug "Failed to decode the packet !" + End 5 +EndIf + +Debug "Id: " + Str(OutputStruct\Id) +Debug "UserName: " + OutputStruct\UserName +Debug "IsAdmin: " + Str(OutputStruct\IsAdmin) + + +; Freeing the obscure structure and the buffer it owns. +MsgPackFree(*Reader, #True) + +End 0 diff --git a/Examples/BasicStrings/BasicStrings.pb b/Examples/BasicStrings/BasicStrings.pb new file mode 100644 index 0000000..5ad0781 --- /dev/null +++ b/Examples/BasicStrings/BasicStrings.pb @@ -0,0 +1,92 @@ +; ------------------------------------------------------------------------------ +;- Notes + +; This example shows you how to write and read strings in the supported +; encodings, UTF-8 being the default one and the one used by the spec. +; The string must be read with the encoding it was written with. + + +; ------------------------------------------------------------------------------ +;- Example + +EnableExplicit + +XIncludeFile "../../Includes/MessagePack.pbi" + +; Obscure structures that hold info about the MsgPack data +Define *Writer, *Reader + +; MsgPack-encoded buffer that will get exported midway in this example. +Define *Buffer + +; "Cafe" with an accented "e", 4 characters but 5 bytes in UTF-8. +Define Text.s = "Caf" + Chr($E9) + + + +; Writing the strings into the buffer. +*Writer = MsgPackCreate(64) +If Not *Writer + Debug "Failed to create the writer !" + End 1 +EndIf + +; UTF-8 by default. +If MsgPackWriteString(*Writer, Text) <> #MsgPack_Error_Success + Debug "Failed to write the UTF-8 string !" + End 2 +EndIf + +; UTF-16 LE, 2 bytes per character here. +If MsgPackWriteString(*Writer, Text, #PB_Unicode) <> #MsgPack_Error_Success + Debug "Failed to write the UTF-16 string !" + End 3 +EndIf + +; Empty strings are valid too. +If MsgPackWriteString(*Writer, "") <> #MsgPack_Error_Success + Debug "Failed to write the empty string !" + End 4 +EndIf + + +; Exporting the data and giving it to a reader. +*Buffer = MsgPackFinalize(*Writer) +If Not *Buffer + Debug "Failed to finalize the writer !" + End 5 +EndIf + +Debug "Encoded size: " + Str(MemorySize(*Buffer)) + +*Reader = MsgPackWrapMsgBuffer(*Buffer, MemorySize(*Buffer), #True) +If Not *Reader + Debug "Failed to wrap the buffer !" + End 6 +EndIf + + +; Reading the strings back, in the same order and encodings. +Debug "UTF-8: " + MsgPackReadString(*Reader) +If MsgPackLastError(*Reader) <> #MsgPack_Error_Success + Debug "Failed to read the UTF-8 string !" + End 7 +EndIf + +Debug "UTF-16: " + MsgPackReadString(*Reader, #PB_Unicode) +If MsgPackLastError(*Reader) <> #MsgPack_Error_Success + Debug "Failed to read the UTF-16 string !" + End 8 +EndIf + +Debug "Empty: '" + MsgPackReadString(*Reader) + "'" +If MsgPackLastError(*Reader) <> #MsgPack_Error_Success + Debug "Failed to read the empty string !" + End 9 +EndIf + + +; Freeing the obscure structure and the buffer it owns. +MsgPackFree(*Reader, #True) + +End 0 diff --git a/Examples/DumpExplicitTypes/DumpExplicitTypes.pb b/Examples/DumpExplicitTypes/DumpExplicitTypes.pb new file mode 100644 index 0000000..d1c1e84 --- /dev/null +++ b/Examples/DumpExplicitTypes/DumpExplicitTypes.pb @@ -0,0 +1,322 @@ +; ------------------------------------------------------------------------------ +;- Notes + +; This example shows you how to iterate over a MessagePack buffer and dump +; every field to the console with its exact format code. +; Each field is read with the procedure matching its format code. +; Arrays and maps are dumped recursively, map entries key first. + + +; ------------------------------------------------------------------------------ +;- Example + +EnableExplicit + +XIncludeFile "../../Includes/MessagePack.pbi" + + +; Ends the program if a write failed. +Procedure CheckWrite(ErrorCode.MsgPack_ErrorCode, What.s) + If ErrorCode <> #MsgPack_Error_Success + PrintN("Failed to write " + What + " ! (Error " + Str(ErrorCode) + ")") + End 1 + EndIf +EndProcedure + +; Ends the program if a structure or buffer couldn't be prepared. +Procedure CheckPointer(*Pointer, What.s) + If Not *Pointer + PrintN("Failed to prepare " + What + " !") + End 1 + EndIf +EndProcedure + + +; Returns the bytes of a buffer from `MsgPackCopyBinary/Extension` as hex. +; Empty bin and ext fields give a `#Null` buffer. +Procedure.s HexDump(*Data) + Protected Text.s = "" + Protected i.i + + If Not *Data + ProcedureReturn "" + EndIf + + For i = 0 To MemorySize(*Data) - 1 + Text = Text + RSet(Hex(PeekA(*Data + i)), 2, "0") + " " + Next + + ProcedureReturn RTrim(Text) +EndProcedure + + +Procedure.s GetFormatCodeName(FormatCode.MsgPack_FormatCode) + Select FormatCode + Case #MsgPack_FormatCode_Null : ProcedureReturn "Null" + Case #MsgPack_FormatCode_False : ProcedureReturn "False" + Case #MsgPack_FormatCode_True : ProcedureReturn "True" + Case #MsgPack_FormatCode_FixIntPositive : ProcedureReturn "FixIntPositive" + Case #MsgPack_FormatCode_FixIntNegative : ProcedureReturn "FixIntNegative" + Case #MsgPack_FormatCode_UInt8 : ProcedureReturn "UInt8" + Case #MsgPack_FormatCode_UInt16 : ProcedureReturn "UInt16" + Case #MsgPack_FormatCode_UInt32 : ProcedureReturn "UInt32" + Case #MsgPack_FormatCode_UInt64 : ProcedureReturn "UInt64" + Case #MsgPack_FormatCode_Int8 : ProcedureReturn "Int8" + Case #MsgPack_FormatCode_Int16 : ProcedureReturn "Int16" + Case #MsgPack_FormatCode_Int32 : ProcedureReturn "Int32" + Case #MsgPack_FormatCode_Int64 : ProcedureReturn "Int64" + Case #MsgPack_FormatCode_Float32 : ProcedureReturn "Float32" + Case #MsgPack_FormatCode_Float64 : ProcedureReturn "Float64" + Case #MsgPack_FormatCode_FixStr : ProcedureReturn "FixStr" + Case #MsgPack_FormatCode_Str8 : ProcedureReturn "Str8" + Case #MsgPack_FormatCode_Str16 : ProcedureReturn "Str16" + Case #MsgPack_FormatCode_Str32 : ProcedureReturn "Str32" + Case #MsgPack_FormatCode_Bin8 : ProcedureReturn "Bin8" + Case #MsgPack_FormatCode_Bin16 : ProcedureReturn "Bin16" + Case #MsgPack_FormatCode_Bin32 : ProcedureReturn "Bin32" + Case #MsgPack_FormatCode_FixArray : ProcedureReturn "FixArray" + Case #MsgPack_FormatCode_Array16 : ProcedureReturn "Array16" + Case #MsgPack_FormatCode_Array32 : ProcedureReturn "Array32" + Case #MsgPack_FormatCode_FixMap : ProcedureReturn "FixMap" + Case #MsgPack_FormatCode_Map16 : ProcedureReturn "Map16" + Case #MsgPack_FormatCode_Map32 : ProcedureReturn "Map32" + Case #MsgPack_FormatCode_FixExt1 : ProcedureReturn "FixExt1" + Case #MsgPack_FormatCode_FixExt2 : ProcedureReturn "FixExt2" + Case #MsgPack_FormatCode_FixExt4 : ProcedureReturn "FixExt4" + Case #MsgPack_FormatCode_FixExt8 : ProcedureReturn "FixExt8" + Case #MsgPack_FormatCode_FixExt16 : ProcedureReturn "FixExt16" + Case #MsgPack_FormatCode_Ext8 : ProcedureReturn "Ext8" + Case #MsgPack_FormatCode_Ext16 : ProcedureReturn "Ext16" + Case #MsgPack_FormatCode_Ext32 : ProcedureReturn "Ext32" + EndSelect + + ProcedureReturn "Invalid" +EndProcedure + + +; Prints the field at the current offset and moves past it. +; `Label` is printed before it, to tell array indexes and map keys/values apart. +Procedure.b DumpField(*MsgPackData, Indent.s, Label.s) + Protected FormatCode.MsgPack_FormatCode = MsgPackGetFormatCode(*MsgPackData) + Protected Text.s = GetFormatCodeName(FormatCode) + Protected *Iterator, *Data + Protected IsMap.b, Success.b + Protected EntryCount.q, Index.q + + Select FormatCode + Case #MsgPack_FormatCode_Null + MsgPackReadNull(*MsgPackData) + + Case #MsgPack_FormatCode_False, #MsgPack_FormatCode_True + ; The value is already given by the format code. + MsgPackReadBoolean(*MsgPackData) + + Case #MsgPack_FormatCode_FixIntPositive, #MsgPack_FormatCode_FixIntNegative + ; Fixints don't have their own getter, the generic one reads them. + Text = Text + ": " + Str(MsgPackReadInteger(*MsgPackData)) + + Case #MsgPack_FormatCode_UInt8 + Text = Text + ": " + Str(MsgPackReadUInt8(*MsgPackData)) + Case #MsgPack_FormatCode_UInt16 + Text = Text + ": " + Str(MsgPackReadUInt16(*MsgPackData)) + Case #MsgPack_FormatCode_UInt32 + ; Read as a `.l`, values above the signed max come back negative. + Text = Text + ": " + Str(MsgPackReadUInt32(*MsgPackData) & $FFFFFFFF) + Case #MsgPack_FormatCode_UInt64 + ; Read as a `.q`, values above the signed max come back negative. + Text = Text + ": " + StrU(MsgPackReadUInt64(*MsgPackData), #PB_Quad) + + Case #MsgPack_FormatCode_Int8 + Text = Text + ": " + Str(MsgPackReadInt8(*MsgPackData)) + Case #MsgPack_FormatCode_Int16 + Text = Text + ": " + Str(MsgPackReadInt16(*MsgPackData)) + Case #MsgPack_FormatCode_Int32 + Text = Text + ": " + Str(MsgPackReadInt32(*MsgPackData)) + Case #MsgPack_FormatCode_Int64 + Text = Text + ": " + Str(MsgPackReadInt64(*MsgPackData)) + + Case #MsgPack_FormatCode_Float32 + Text = Text + ": " + StrF(MsgPackReadFloat32(*MsgPackData)) + Case #MsgPack_FormatCode_Float64 + Text = Text + ": " + StrD(MsgPackReadFloat64(*MsgPackData)) + + Case #MsgPack_FormatCode_FixStr, #MsgPack_FormatCode_Str8, + #MsgPack_FormatCode_Str16, #MsgPack_FormatCode_Str32 + Text = Text + ": " + Chr(34) + MsgPackReadString(*MsgPackData) + Chr(34) + + Case #MsgPack_FormatCode_Bin8, #MsgPack_FormatCode_Bin16, #MsgPack_FormatCode_Bin32 + *Data = MsgPackCopyBinary(*MsgPackData) + Text = Text + ": " + HexDump(*Data) + + Case #MsgPack_FormatCode_FixExt1, #MsgPack_FormatCode_FixExt2, #MsgPack_FormatCode_FixExt4, + #MsgPack_FormatCode_FixExt8, #MsgPack_FormatCode_FixExt16, + #MsgPack_FormatCode_Ext8, #MsgPack_FormatCode_Ext16, #MsgPack_FormatCode_Ext32 + ; The type doesn't move the offset, the data is read right after. + Text = Text + " (Type " + Str(MsgPackGetExtensionType(*MsgPackData)) + ")" + If MsgPackLastError(*MsgPackData) <> #MsgPack_Error_Success + ProcedureReturn #False + EndIf + *Data = MsgPackCopyExtension(*MsgPackData) + Text = Text + ": " + HexDump(*Data) + + Case #MsgPack_FormatCode_FixArray, #MsgPack_FormatCode_Array16, #MsgPack_FormatCode_Array32, + #MsgPack_FormatCode_FixMap, #MsgPack_FormatCode_Map16, #MsgPack_FormatCode_Map32 + IsMap = Bool(MsgPackFormatCodeToDataType(FormatCode) = #MsgPack_DataTypes_Map) + + EntryCount = MsgPackGetEntryCount(*MsgPackData) + *Iterator = MsgPackExamine(*MsgPackData) + If Not *Iterator + ProcedureReturn #False + EndIf + + PrintN(Indent + Label + Text + " (" + Str(EntryCount) + " entries)") + + Success = #True + Index = 0 + While MsgPackNextEntry(*Iterator) + If IsMap + Success = DumpField(*Iterator, Indent + " ", "Key: ") + If Success + Success = DumpField(*Iterator, Indent + " ", "Value: ") + EndIf + Else + Success = DumpField(*Iterator, Indent + " ", "[" + Str(Index) + "] ") + EndIf + + If Not Success + Break + EndIf + Index = Index + 1 + Wend + + ; `MsgPackNextEntry` also returns `#False` on errors. + If MsgPackLastError(*Iterator) <> #MsgPack_Error_Success + Success = #False + EndIf + + MsgPackFinish(*Iterator) + ProcedureReturn Success + + Default + PrintN(Indent + Label + "Invalid format code !") + ProcedureReturn #False + EndSelect + + If *Data + FreeMemory(*Data) + EndIf + + If MsgPackLastError(*MsgPackData) <> #MsgPack_Error_Success + ProcedureReturn #False + EndIf + + PrintN(Indent + Label + Text) + ProcedureReturn #True +EndProcedure + + + +; Obscure structures that hold info about the MsgPack data +Define *Writer, *Entries, *Fields, *Reader + +; MsgPack-encoded buffers that will get exported in this example. +Define *Buffer, *EntriesBuffer, *FieldsBuffer + +; Raw data for the bin and ext fields. +Define Bytes.l = $44332211 + +Define FieldSize.q, BytesLeft.q + +OpenConsole() + + +; Writing a bit of everything, forcing each format code. +*Writer = MsgPackCreate(64) +CheckPointer(*Writer, "the writer") + +CheckWrite(MsgPackWriteNull(*Writer), "a nil") +CheckWrite(MsgPackWriteTrue(*Writer), "a true") +CheckWrite(MsgPackWriteFalse(*Writer), "a false") +CheckWrite(MsgPackWriteFixInt(*Writer, 42), "a positive fixint") +CheckWrite(MsgPackWriteFixInt(*Writer, -5), "a negative fixint") +CheckWrite(MsgPackWriteUInt8(*Writer, 200), "a UInt8") +CheckWrite(MsgPackWriteUInt32(*Writer, 70000), "a UInt32") +CheckWrite(MsgPackWriteInt16(*Writer, -1000), "an Int16") +CheckWrite(MsgPackWriteInt64(*Writer, 1), "an Int64") +CheckWrite(MsgPackWriteFloat32(*Writer, 1.5), "a Float32") +CheckWrite(MsgPackWriteFloat64(*Writer, 3.14159), "a Float64") +CheckWrite(MsgPackWriteFixStr(*Writer, "Hello"), "a FixStr") +CheckWrite(MsgPackWriteStr16(*Writer, "World"), "a Str16") +CheckWrite(MsgPackWriteBin8(*Writer, @Bytes, 4), "a Bin8") +CheckWrite(MsgPackWriteFixExt4(*Writer, 1, @Bytes), "a FixExt4") +CheckWrite(MsgPackWriteExt8(*Writer, 2, @Bytes, 3), "an Ext8") + + +; Forcing the format code of arrays and maps requires their content in a buffer. +; Writing the entries of an array in their own writer, then exporting them. +*Entries = MsgPackCreate(16) +CheckPointer(*Entries, "the entries writer") + +CheckWrite(MsgPackWriteUInt8(*Entries, 10), "an entry") +CheckWrite(MsgPackWriteUInt8(*Entries, 20), "an entry") +CheckWrite(MsgPackWriteUInt8(*Entries, 30), "an entry") + +*EntriesBuffer = MsgPackFinalize(*Entries) +CheckPointer(*EntriesBuffer, "the entries buffer") + + +; Writing the keys and values of a map, with the array as a value. +*Fields = MsgPackCreate(32) +CheckPointer(*Fields, "the fields writer") + +CheckWrite(MsgPackWriteFixStr(*Fields, "Name"), "a key") +CheckWrite(MsgPackWriteStr8(*Fields, "Herwin"), "a value") +CheckWrite(MsgPackWriteFixStr(*Fields, "Scores"), "a key") +CheckWrite(MsgPackWriteArray16(*Fields, *EntriesBuffer, MemorySize(*EntriesBuffer)), "an Array16") +FreeMemory(*EntriesBuffer) + +*FieldsBuffer = MsgPackFinalize(*Fields) +CheckPointer(*FieldsBuffer, "the fields buffer") + +CheckWrite(MsgPackWriteMap16(*Writer, *FieldsBuffer, MemorySize(*FieldsBuffer)), "a Map16") +FreeMemory(*FieldsBuffer) + + +; Exporting the data and giving it to a reader. +*Buffer = MsgPackFinalize(*Writer) +CheckPointer(*Buffer, "the final buffer") + +PrintN("Encoded size: " + Str(MemorySize(*Buffer))) +PrintN("") + +*Reader = MsgPackWrapMsgBuffer(*Buffer, MemorySize(*Buffer), #True) +CheckPointer(*Reader, "the reader") + + +; Dumping every top-level field until the whole buffer is read. +; Each field's full size is checked first, which also validates it. +BytesLeft = MemorySize(*Buffer) +While BytesLeft > 0 + FieldSize = MsgPackGetFieldFullSize(*Reader) + If FieldSize = #MsgPack_InvalidSize + PrintN("Invalid field ! (Error " + Str(MsgPackLastError(*Reader)) + ")") + End 2 + EndIf + + If Not DumpField(*Reader, "", "") + PrintN("Failed to dump a field !") + End 3 + EndIf + + BytesLeft = BytesLeft - FieldSize +Wend + + +; Freeing the obscure structure and the buffer it owns. +MsgPackFree(*Reader, #True) + +PrintN("") +PrintN("Press enter to exit.") +Input() + +End 0 diff --git a/Examples/DumpImplicitTypes/DumpImplicitTypes.pb b/Examples/DumpImplicitTypes/DumpImplicitTypes.pb new file mode 100644 index 0000000..44a25fe --- /dev/null +++ b/Examples/DumpImplicitTypes/DumpImplicitTypes.pb @@ -0,0 +1,253 @@ +; ------------------------------------------------------------------------------ +;- Notes + +; This example shows you how to iterate over a MessagePack buffer and dump +; every field to the console with its general data type. +; Each field is read with the implicitly typed procedure of its data type, +; which accepts all of the type's format codes. +; Arrays and maps are dumped recursively, map entries key first. + + +; ------------------------------------------------------------------------------ +;- Example + +EnableExplicit + +XIncludeFile "../../Includes/MessagePack.pbi" + + +; Ends the program if a write failed. +Procedure CheckWrite(ErrorCode.MsgPack_ErrorCode, What.s) + If ErrorCode <> #MsgPack_Error_Success + PrintN("Failed to write " + What + " ! (Error " + Str(ErrorCode) + ")") + End 1 + EndIf +EndProcedure + +; Ends the program if a structure or buffer couldn't be prepared. +Procedure CheckPointer(*Pointer, What.s) + If Not *Pointer + PrintN("Failed to prepare " + What + " !") + End 1 + EndIf +EndProcedure + + +; Returns the bytes of a buffer from `MsgPackCopyBinary/Extension` as hex. +; Empty bin and ext fields give a `#Null` buffer. +Procedure.s HexDump(*Data) + Protected Text.s = "" + Protected i.i + + If Not *Data + ProcedureReturn "" + EndIf + + For i = 0 To MemorySize(*Data) - 1 + Text = Text + RSet(Hex(PeekA(*Data + i)), 2, "0") + " " + Next + + ProcedureReturn RTrim(Text) +EndProcedure + + +; Prints the field at the current offset and moves past it. +; `Label` is printed before it, to tell array indexes and map keys/values apart. +Procedure.b DumpField(*MsgPackData, Indent.s, Label.s) + Protected DataType.MsgPack_DataType = MsgPackFormatCodeToDataType(MsgPackGetFormatCode(*MsgPackData)) + Protected Text.s + Protected *Iterator, *Data + Protected Success.b + Protected EntryCount.q, Index.q + + Select DataType + Case #MsgPack_DataTypes_Null + MsgPackReadNull(*MsgPackData) + Text = "Null" + + Case #MsgPack_DataTypes_Boolean + If MsgPackReadBoolean(*MsgPackData) + Text = "Boolean: True" + Else + Text = "Boolean: False" + EndIf + + Case #MsgPack_DataTypes_Integer + ; UInt64 values above `(2^63)-1` fail here, see `MsgPackReadUInt64`. + Text = "Integer: " + Str(MsgPackReadInteger(*MsgPackData)) + + Case #MsgPack_DataTypes_Float + ; Float32 values are returned as a `.d`. + Text = "Float: " + StrD(MsgPackReadFloat(*MsgPackData)) + + Case #MsgPack_DataTypes_String + Text = "String: " + Chr(34) + MsgPackReadString(*MsgPackData) + Chr(34) + + Case #MsgPack_DataTypes_Binary + *Data = MsgPackCopyBinary(*MsgPackData) + Text = "Binary: " + HexDump(*Data) + + Case #MsgPack_DataTypes_Extension + ; The type doesn't move the offset, the data is read right after. + Text = "Extension (Type " + Str(MsgPackGetExtensionType(*MsgPackData)) + ")" + If MsgPackLastError(*MsgPackData) <> #MsgPack_Error_Success + ProcedureReturn #False + EndIf + *Data = MsgPackCopyExtension(*MsgPackData) + Text = Text + ": " + HexDump(*Data) + + Case #MsgPack_DataTypes_Array, #MsgPack_DataTypes_Map + EntryCount = MsgPackGetEntryCount(*MsgPackData) + *Iterator = MsgPackExamine(*MsgPackData) + If Not *Iterator + ProcedureReturn #False + EndIf + + If DataType = #MsgPack_DataTypes_Map + PrintN(Indent + Label + "Map (" + Str(EntryCount) + " entries)") + Else + PrintN(Indent + Label + "Array (" + Str(EntryCount) + " entries)") + EndIf + + Success = #True + Index = 0 + While MsgPackNextEntry(*Iterator) + If DataType = #MsgPack_DataTypes_Map + Success = DumpField(*Iterator, Indent + " ", "Key: ") + If Success + Success = DumpField(*Iterator, Indent + " ", "Value: ") + EndIf + Else + Success = DumpField(*Iterator, Indent + " ", "[" + Str(Index) + "] ") + EndIf + + If Not Success + Break + EndIf + Index = Index + 1 + Wend + + ; `MsgPackNextEntry` also returns `#False` on errors. + If MsgPackLastError(*Iterator) <> #MsgPack_Error_Success + Success = #False + EndIf + + MsgPackFinish(*Iterator) + ProcedureReturn Success + + Default + PrintN(Indent + Label + "Unknown data type !") + ProcedureReturn #False + EndSelect + + If *Data + FreeMemory(*Data) + EndIf + + If MsgPackLastError(*MsgPackData) <> #MsgPack_Error_Success + ProcedureReturn #False + EndIf + + PrintN(Indent + Label + Text) + ProcedureReturn #True +EndProcedure + + + +; Obscure structures that hold info about the MsgPack data +Define *Writer, *Entries, *Fields, *Reader + +; MsgPack-encoded buffer that will get exported midway in this example. +Define *Buffer + +; Raw data for the bin and ext fields. +Define Bytes.l = $44332211 + +Define FieldSize.q, BytesLeft.q + +OpenConsole() + + +; Writing a bit of everything, the smallest format code that fits is used. +*Writer = MsgPackCreate(64) +CheckPointer(*Writer, "the writer") + +CheckWrite(MsgPackWriteNull(*Writer), "a nil") +CheckWrite(MsgPackWriteBoolean(*Writer, #True), "a boolean") +CheckWrite(MsgPackWriteBoolean(*Writer, #False), "a boolean") +CheckWrite(MsgPackWriteInteger(*Writer, 42), "an integer") +CheckWrite(MsgPackWriteInteger(*Writer, -5), "an integer") +CheckWrite(MsgPackWriteInteger(*Writer, 200), "an integer") +CheckWrite(MsgPackWriteInteger(*Writer, 70000), "an integer") +CheckWrite(MsgPackWriteInteger(*Writer, -1000), "an integer") +; Floats don't have an implicitly typed setter. +CheckWrite(MsgPackWriteFloat64(*Writer, 3.14159), "a float") +CheckWrite(MsgPackWriteString(*Writer, "Hello"), "a string") +CheckWrite(MsgPackWriteBinary(*Writer, @Bytes, 4), "a binary") +CheckWrite(MsgPackWriteExtension(*Writer, 1, @Bytes, 4), "an extension") +CheckWrite(MsgPackWriteExtension(*Writer, 2, @Bytes, 3), "an extension") + + +; Writing the entries of an array in their own writer. +*Entries = MsgPackCreate(16) +CheckPointer(*Entries, "the entries writer") + +CheckWrite(MsgPackWriteInteger(*Entries, 10), "an entry") +CheckWrite(MsgPackWriteInteger(*Entries, 20), "an entry") +CheckWrite(MsgPackWriteInteger(*Entries, 30), "an entry") + + +; Writing the keys and values of a map, with the array as a value. +*Fields = MsgPackCreate(32) +CheckPointer(*Fields, "the fields writer") + +CheckWrite(MsgPackWriteString(*Fields, "Name"), "a key") +CheckWrite(MsgPackWriteString(*Fields, "Herwin"), "a value") +CheckWrite(MsgPackWriteString(*Fields, "Scores"), "a key") +CheckWrite(MsgPackWriteArrayFrom(*Fields, *Entries), "an array") + +CheckWrite(MsgPackWriteMapFrom(*Writer, *Fields), "a map") + +; The entries and fields writers are left untouched, so they must be freed. +MsgPackFree(*Entries, #True) +MsgPackFree(*Fields, #True) + + +; Exporting the data and giving it to a reader. +*Buffer = MsgPackFinalize(*Writer) +CheckPointer(*Buffer, "the final buffer") + +PrintN("Encoded size: " + Str(MemorySize(*Buffer))) +PrintN("") + +*Reader = MsgPackWrapMsgBuffer(*Buffer, MemorySize(*Buffer), #True) +CheckPointer(*Reader, "the reader") + + +; Dumping every top-level field until the whole buffer is read. +; Each field's full size is checked first, which also validates it. +BytesLeft = MemorySize(*Buffer) +While BytesLeft > 0 + FieldSize = MsgPackGetFieldFullSize(*Reader) + If FieldSize = #MsgPack_InvalidSize + PrintN("Invalid field ! (Error " + Str(MsgPackLastError(*Reader)) + ")") + End 2 + EndIf + + If Not DumpField(*Reader, "", "") + PrintN("Failed to dump a field !") + End 3 + EndIf + + BytesLeft = BytesLeft - FieldSize +Wend + + +; Freeing the obscure structure and the buffer it owns. +MsgPackFree(*Reader, #True) + +PrintN("") +PrintN("Press enter to exit.") +Input() + +End 0 diff --git a/PerfTests/DeepNesting/PerfDeepNesting.pb b/PerfTests/DeepNesting/PerfDeepNesting.pb new file mode 100644 index 0000000..92a0030 --- /dev/null +++ b/PerfTests/DeepNesting/PerfDeepNesting.pb @@ -0,0 +1,172 @@ +EnableExplicit + +XIncludeFile "../../Includes/MessagePack.pbi" +XIncludeFile "../_PerfTesting.pbi" + + +; Builds and reads arrays nested `NestedDepth` levels deep, each one holding 3 integers before the next one. +; Each run handles `NestedRepeats` of them, and `NsPerOp` is the time per nested structure. +; The depth goes up to the default `#MsgPack_MaxDepth`. + +#NestedMaxDepth = 128 + +; Number of levels handled by each run, split between the repeats. +#NestedLevelsPerRun = 100000 + +Global NestedDepth.i +Global NestedRepeats.i +Global *NestedBuffer + + +Procedure WriteLevelIntegers(*Writer) + PerfCheckWrite(MsgPackWriteInteger(*Writer, 1)) + PerfCheckWrite(MsgPackWriteInteger(*Writer, 2)) + PerfCheckWrite(MsgPackWriteInteger(*Writer, 3)) +EndProcedure + +; Returns the finalized buffer, which must be freed with `FreeMemory()`. +; Each level's content is written in its own writer, then copied into its parent. +Procedure.i BuildNestedBuffer() + Protected *Content, *Parent, *Buffer + Protected Level.i + + *Content = MsgPackCreate(64) + If Not *Content + PerfFail("Failed to create a writer !") + EndIf + WriteLevelIntegers(*Content) + + For Level = 2 To NestedDepth + *Parent = MsgPackCreate(64) + If Not *Parent + PerfFail("Failed to create a writer !") + EndIf + WriteLevelIntegers(*Parent) + PerfCheckWrite(MsgPackWriteArrayFrom(*Parent, *Content)) + MsgPackFree(*Content, #True) + *Content = *Parent + Next + + ; The top-level array. + *Parent = MsgPackCreate(64) + If Not *Parent + PerfFail("Failed to create a writer !") + EndIf + PerfCheckWrite(MsgPackWriteArrayFrom(*Parent, *Content)) + MsgPackFree(*Content, #True) + + *Buffer = MsgPackFinalize(*Parent) + If Not *Buffer + PerfFail("Failed to finalize the writer !") + EndIf + + ProcedureReturn *Buffer +EndProcedure + +; Examines every level, reading its integers, and returns their sum for one structure. +Procedure.q ExamineNested(ExaminationOptions.MsgPack_ExaminationOption) + Protected Dim Views.i(NestedDepth) + Protected Total.q + Protected Iteration.i, Level.i, Entry.i + + For Iteration = 1 To NestedRepeats + Total = 0 + + Views(0) = MsgPackWrapMsgBuffer(*NestedBuffer, MemorySize(*NestedBuffer)) + If Not Views(0) + PerfFail("Failed to wrap the buffer !") + EndIf + + For Level = 1 To NestedDepth + Views(Level) = MsgPackExamine(Views(Level - 1), ExaminationOptions) + If Not Views(Level) + PerfFail("Failed to examine level " + Str(Level) + " !") + EndIf + + For Entry = 1 To 3 + If Not MsgPackNextEntry(Views(Level)) + PerfFail("Missing integer at level " + Str(Level) + " !") + EndIf + Total + MsgPackReadInteger(Views(Level)) + Next + + ; Moving to the nested array, the innermost level doesn't have one. + If Level < NestedDepth + If Not MsgPackNextEntry(Views(Level)) + PerfFail("Missing array at level " + Str(Level) + " !") + EndIf + EndIf + Next + + For Level = NestedDepth To 1 Step -1 + MsgPackFinish(Views(Level)) + Next + MsgPackFree(Views(0), #False) + Next + + ProcedureReturn Total +EndProcedure + + +Procedure.q ScenarioBuild() + Protected *Buffer + Protected Size.q + Protected Iteration.i + + For Iteration = 1 To NestedRepeats + *Buffer = BuildNestedBuffer() + Size = MemorySize(*Buffer) + FreeMemory(*Buffer) + Next + + ProcedureReturn Size +EndProcedure + +Procedure.q ScenarioExamine() + ProcedureReturn ExamineNested(0) +EndProcedure + +Procedure.q ScenarioExamineUnsafe() + ProcedureReturn ExamineNested(#MsgPack_ExaminationOptions_Unsafe) +EndProcedure + +Procedure.q ScenarioFullSize() + Protected *Reader + Protected Size.q + Protected Iteration.i + + For Iteration = 1 To NestedRepeats + *Reader = MsgPackWrapMsgBuffer(*NestedBuffer, MemorySize(*NestedBuffer)) + If Not *Reader + PerfFail("Failed to wrap the buffer !") + EndIf + + Size = MsgPackGetFieldFullSize(*Reader) + If Size = #MsgPack_InvalidSize + PerfFail("Failed to get the full size !") + EndIf + + MsgPackFree(*Reader, #False) + Next + + ProcedureReturn Size +EndProcedure + + +OpenConsole() + +NestedDepth = 1 +While NestedDepth <= #NestedMaxDepth + NestedRepeats = #NestedLevelsPerRun / NestedDepth + *NestedBuffer = BuildNestedBuffer() + + PerfRun("pb", "nested-" + Str(NestedDepth) + "-build", @ScenarioBuild(), NestedRepeats) + PerfRun("pb", "nested-" + Str(NestedDepth) + "-examine", @ScenarioExamine(), NestedRepeats) + PerfRun("pb", "nested-" + Str(NestedDepth) + "-examine-unsafe", @ScenarioExamineUnsafe(), NestedRepeats) + PerfRun("pb", "nested-" + Str(NestedDepth) + "-fullsize", @ScenarioFullSize(), NestedRepeats) + + FreeMemory(*NestedBuffer) + NestedDepth * 2 +Wend + +End 0 diff --git a/PerfTests/DeepNesting/RunPerfTests.ps1 b/PerfTests/DeepNesting/RunPerfTests.ps1 new file mode 100644 index 0000000..d2d651a --- /dev/null +++ b/PerfTests/DeepNesting/RunPerfTests.ps1 @@ -0,0 +1,15 @@ +$ErrorActionPreference = "Stop" + +# Shows the error instead of closing the window right away. +trap { + Write-Host $_ -ForegroundColor Red + Pause + exit 1 +} + +. (Join-Path $PSScriptRoot "..\_PerfTesting.ps1") + +$Rows = @(Invoke-PureBasicPerfTests $PSScriptRoot) + +Show-PerfResults $Rows $PSScriptRoot +Pause diff --git a/PerfTests/FlatMaps/DotNet/DotNetPerf.csproj b/PerfTests/FlatMaps/DotNet/DotNetPerf.csproj new file mode 100644 index 0000000..c5de0f4 --- /dev/null +++ b/PerfTests/FlatMaps/DotNet/DotNetPerf.csproj @@ -0,0 +1,13 @@ + + + Exe + net10.0 + enable + enable + true + + + + + + diff --git a/PerfTests/FlatMaps/DotNet/Program.cs b/PerfTests/FlatMaps/DotNet/Program.cs new file mode 100644 index 0000000..9a1dea6 --- /dev/null +++ b/PerfTests/FlatMaps/DotNet/Program.cs @@ -0,0 +1,48 @@ +// Writes and reads the workload shared with the PureBasic tests, see `../_Workload.pbi`. +// Uses the low-level MessagePackWriter/MessagePackReader from MessagePack-CSharp, pinned in the .csproj. +using System.Buffers; +using MessagePack; + +const int Maps = 100_000; +const int Entries = 16; + +string[] keys = Enumerable.Range(0, Entries).Select(i => $"Key{i:D2}").ToArray(); +byte[] encoded = Write(keys); + +Perf.Run("write", Maps, () => Write(keys).Length); +Perf.Run("read", Maps, () => Read(encoded)); +return 0; + + +static byte[] Write(string[] keys) +{ + var buffer = new ArrayBufferWriter(); + var writer = new MessagePackWriter(buffer); + for (int i = 0; i < Maps; i++) + { + writer.WriteMapHeader(Entries); + for (int j = 0; j < Entries; j++) + { + writer.Write(keys[j]); + writer.Write(j); + } + } + writer.Flush(); + return buffer.WrittenSpan.ToArray(); +} + +static long Read(byte[] encoded) +{ + var reader = new MessagePackReader(encoded); + long total = 0; + for (int i = 0; i < Maps; i++) + { + int count = reader.ReadMapHeader(); + for (int j = 0; j < count; j++) + { + total += reader.ReadString()!.Length; + total += reader.ReadInt64(); + } + } + return total; +} diff --git a/PerfTests/FlatMaps/PerfFlatMaps.pb b/PerfTests/FlatMaps/PerfFlatMaps.pb new file mode 100644 index 0000000..ba7b76a --- /dev/null +++ b/PerfTests/FlatMaps/PerfFlatMaps.pb @@ -0,0 +1,111 @@ +EnableExplicit + +XIncludeFile "../../Includes/MessagePack.pbi" +XIncludeFile "../_PerfTesting.pbi" +XIncludeFile "_Workload.pbi" + + +; Writes and reads the shared workload, see `_Workload.pbi`. +; Each map's fields are written in their own writer, then copied in with `MsgPackWriteMapFrom`. + +Global Dim Keys.s(#Perf_Entries - 1) +Global *Encoded + + +Procedure WriteMap(*Writer) + Protected *Fields = MsgPackCreate(128) + Protected i.i + + If Not *Fields + PerfFail("Failed to create the fields writer !") + EndIf + + For i = 0 To #Perf_Entries - 1 + PerfCheckWrite(MsgPackWriteString(*Fields, Keys(i))) + PerfCheckWrite(MsgPackWriteInteger(*Fields, i)) + Next + + PerfCheckWrite(MsgPackWriteMapFrom(*Writer, *Fields)) + MsgPackFree(*Fields, #True) +EndProcedure + +; Returns the finalized buffer, which must be freed with `FreeMemory()`. +Procedure.i EncodeMaps() + Protected *Writer = MsgPackCreate(#Perf_EncodedSize) + Protected *Buffer + Protected i.i + + If Not *Writer + PerfFail("Failed to create the writer !") + EndIf + + For i = 1 To #Perf_Maps + WriteMap(*Writer) + Next + + *Buffer = MsgPackFinalize(*Writer) + If Not *Buffer + PerfFail("Failed to finalize the writer !") + EndIf + + ProcedureReturn *Buffer +EndProcedure + + +Procedure.q ScenarioWrite() + Protected *Buffer = EncodeMaps() + Protected Size.q = MemorySize(*Buffer) + + FreeMemory(*Buffer) + ProcedureReturn Size +EndProcedure + +Procedure.q ScenarioRead() + Protected *Reader = MsgPackWrapMsgBuffer(*Encoded, MemorySize(*Encoded)) + Protected *Iterator + Protected Total.q + Protected i.i + + If Not *Reader + PerfFail("Failed to wrap the buffer !") + EndIf + + For i = 1 To #Perf_Maps + *Iterator = MsgPackExamine(*Reader) + If Not *Iterator + PerfFail("Failed to examine a map !") + EndIf + + While MsgPackNextEntry(*Iterator) + Total + Len(MsgPackReadString(*Iterator)) + Total + MsgPackReadInteger(*Iterator) + Wend + + ; `MsgPackNextEntry` also returns `#False` on errors, and failed reads break the next ones. + If MsgPackLastError(*Iterator) <> #MsgPack_Error_Success + PerfFail("A read failed !") + EndIf + + MsgPackFinish(*Iterator) + Next + + MsgPackFree(*Reader, #False) + ProcedureReturn Total +EndProcedure + + +OpenConsole() + +Define i.i +For i = 0 To #Perf_Entries - 1 + Keys(i) = "Key" + RSet(Str(i), 2, "0") +Next + +*Encoded = EncodeMaps() + +PerfRun("pb", "write", @ScenarioWrite(), #Perf_Maps) +PerfRun("pb", "read", @ScenarioRead(), #Perf_Maps) + +FreeMemory(*Encoded) + +End 0 diff --git a/PerfTests/FlatMaps/PerfPython.py b/PerfTests/FlatMaps/PerfPython.py new file mode 100644 index 0000000..79de8df --- /dev/null +++ b/PerfTests/FlatMaps/PerfPython.py @@ -0,0 +1,37 @@ +# Writes and reads the workload shared with the PureBasic tests, see `_Workload.pbi`. +# The maps are packed from and unpacked to dicts, how they're usually handled in Python. +import sys +from pathlib import Path + +import msgpack + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) +from _PerfTesting import run + +MAPS = 100_000 +MAP = {f"Key{i:02}": i for i in range(16)} + + +def write(): + packer = msgpack.Packer(autoreset=False) + pack = packer.pack + for _ in range(MAPS): + pack(MAP) + return packer.bytes() + + +def read(): + unpacker = msgpack.Unpacker(raw=False) + unpacker.feed(ENCODED) + unpack = unpacker.unpack + total = 0 + for _ in range(MAPS): + for key, value in unpack().items(): + total += len(key) + value + return total + + +ENCODED = write() + +run("write", MAPS, lambda: len(write())) +run("read", MAPS, read) diff --git a/PerfTests/FlatMaps/RunPerfTests.ps1 b/PerfTests/FlatMaps/RunPerfTests.ps1 new file mode 100644 index 0000000..800d2c4 --- /dev/null +++ b/PerfTests/FlatMaps/RunPerfTests.ps1 @@ -0,0 +1,18 @@ +$ErrorActionPreference = "Stop" + +# Shows the error instead of closing the window right away. +trap { + Write-Host $_ -ForegroundColor Red + Pause + exit 1 +} + +. (Join-Path $PSScriptRoot "..\_PerfTesting.ps1") + +# Requires Python with the packages from 'PerfTests\requirements.txt', and the .NET 10 SDK. +$Rows = @(Invoke-PureBasicPerfTests $PSScriptRoot) +$Rows += Invoke-PythonPerfTest (Join-Path $PSScriptRoot "PerfPython.py") +$Rows += Invoke-DotNetPerfTest (Join-Path $PSScriptRoot "DotNet") + +Show-PerfResults $Rows $PSScriptRoot +Pause diff --git a/PerfTests/FlatMaps/_Workload.pbi b/PerfTests/FlatMaps/_Workload.pbi new file mode 100644 index 0000000..417cc6d --- /dev/null +++ b/PerfTests/FlatMaps/_Workload.pbi @@ -0,0 +1,15 @@ +; Workload shared by every implementation in this folder, keep `PerfPython.py` and `DotNet/Program.cs` in sync. +; +; `#Perf_Maps` top-level maps are written, then read, each holding `#Perf_Entries` entries: +; "Key00" to "Key15" (FixStr), mapped to 0 to 15 (FixInt), in that order. +; 16 entries make it a Map16, the smallest format code that fits. +; `NsPerOp` is the time per map. +; +; Write check: the encoded size. +; Read check: the sum of the keys' length and of the values. + +#Perf_Maps = 100000 +#Perf_Entries = 16 + +; 3 bytes of header, then 6 bytes per key and 1 per value. +#Perf_EncodedSize = #Perf_Maps * (3 + #Perf_Entries * 7) diff --git a/PerfTests/Investigations.md b/PerfTests/Investigations.md new file mode 100644 index 0000000..a1be611 --- /dev/null +++ b/PerfTests/Investigations.md @@ -0,0 +1,59 @@ +# PureBasic - MsgPack - Performance investigations +Open questions raised by the [results](readme.md#results), with leads to check. \ +Nothing below is confirmed yet, update or remove each entry once it is looked into. + +Measured on 2026-11-03, see the results' CSVs in `Results/`. + + +## Strings are slow compared to binary data +x64 with the C backend, PureBasic 6.41, ns per field: + +| Payload | Binary write | String write | Binary read | String read | +|:--|--:|--:|--:|--:| +| 1 KB, UTF-8 | 190.6 | 720.9 (3.8x) | 50.2 | 570.0 (11.4x) | +| 1 KB, UTF-16 (2 KB) | - | 526.0 | - | 282.7 | +| 64 KB vs 1 MB | ~5.0 GB/s | ~1.4 GB/s | ~19 GB/s | ~2.1 GB/s | + +* UTF-16 is faster than UTF-8 while being twice as big, so the encoding itself costs a lot. +* MessagePack-CSharp reads 1 KB strings in 150 ns, and writes 8 B strings in 13 ns against 60 ns here. +* PureBasic 6.21 is another 2.5 to 4 times slower, see [PureBasic 6.21 vs 6.41](readme.md#purebasic-621-vs-641). + +### Leads in `Includes/_Strings.pbi` +* `String.s` is passed by value through `MsgPackWriteString`, `_MsgPackGetEncodedStringSize` and `_MsgPackWriteStringField`, + which may copy it at each call, and would explain why 6.40's string handling helps that much. \ + Passing `@String` and its length to the private helpers would avoid it. +* Each write goes over the string at least twice, with `StringByteLength()` then `PokeS()`. \ + For `#PB_Unicode` and `#PB_Ascii`, the size could come from `Len()` instead, if it doesn't scan the string. +* `PokeS()` is given `-1` as its length, so it may also look for the null terminator on its own. +* `MsgPackReadString` builds the string with `PeekS()`, then returns it, which may copy it again. + +### Leads in the benchmark +* `EncodeStrings()` preallocates `FieldSize * 2 + 5` bytes per field, 64 MB in total, against 32 MB for binaries. \ + The allocation is timed, see the next section, and `MsgPackFinalize` then shrinks it by half. +* `ScenarioStringRead()` calls `Len()` on each string, which may scan it on 6.21 and not on 6.41. \ + Python's `len()` and .NET's `Length` don't scan. + +### Experiments to run +1. Time raw `PokeS()`/`PeekS()` loops with the same sizes, without MsgPack, to split PureBasic's own cost from the include's. +2. Size the strings writer like the binaries one, and compare. +3. Try the `@String` + length helpers, and `Len()` for the fixed-width encodings. +4. Run 6.40 too, to check that the gain comes from 6.40 and not 6.41. + + +## Buffers are zero-filled +`MsgPackCreate` uses `AllocateMemory()` and `MsgPackGrow` uses `ReAllocateMemory()`, both without `#PB_Memory_NoClear`, +so every buffer is zeroed before being written over. +* That cost is in every write scenario with a preallocated buffer, more so for the big ones. +* With a `GrowthIncrement` of `1`, small writes take ~440 ns against ~12 ns preallocated, + the reallocation and copy probably cost more than the zeroing, but it adds up. + +To try: `#PB_Memory_NoClear` on both, as only the written bytes are ever read, then re-run every folder. + + +## Smaller oddities +* **Maps:** writing them is 8.6 times slower than MessagePack-CSharp and 1.9 times slower than msgpack. \ + Each map is written in its own structure, then checked, counted and copied by `MsgPackWriteMapFrom`, see the main readme. + Its 16 keys are strings too, see above. +* **ASM backend:** skipping small fields is 3.2 times slower than with the C backend (57.9 vs 18.3 ns), against 1 to 2 times for the rest. +* **6.21 faster than 6.41:** skip (14.7 vs 18.3 ns) and 1 KB binaries (0.8 to 0.9x). \ + Those runs only last 1 to 2 ms, so it may be noise, re-run them with more iterations before looking further. diff --git a/PerfTests/MassiveReadAmount/DotNet/DotNetPerf.csproj b/PerfTests/MassiveReadAmount/DotNet/DotNetPerf.csproj new file mode 100644 index 0000000..c5de0f4 --- /dev/null +++ b/PerfTests/MassiveReadAmount/DotNet/DotNetPerf.csproj @@ -0,0 +1,13 @@ + + + Exe + net10.0 + enable + enable + true + + + + + + diff --git a/PerfTests/MassiveReadAmount/DotNet/Program.cs b/PerfTests/MassiveReadAmount/DotNet/Program.cs new file mode 100644 index 0000000..3a82671 --- /dev/null +++ b/PerfTests/MassiveReadAmount/DotNet/Program.cs @@ -0,0 +1,62 @@ +// Writes, reads and skips the workload shared with the PureBasic tests, see `../_Workload.pbi`. +// Uses the low-level MessagePackWriter/MessagePackReader from MessagePack-CSharp, pinned in the .csproj. +using System.Buffers; +using MessagePack; + +const int Records = 125_000; +const int Fields = Records * 8; + +byte[] encoded = Write(); + +Perf.Run("write", Fields, () => Write().Length); +Perf.Run("read", Fields, () => Read(encoded)); +Perf.Run("skip", Fields, () => Skip(encoded)); +return 0; + + +static byte[] Write() +{ + var buffer = new ArrayBufferWriter(); + var writer = new MessagePackWriter(buffer); + for (int i = 0; i < Records; i++) + { + writer.Write(42); + writer.Write(-5); + writer.Write(200); + writer.Write(-1000); + writer.Write(70000); + writer.Write(-1099511627776L); + writer.Write(3.14); + writer.Write("Hello"); + } + writer.Flush(); + return buffer.WrittenSpan.ToArray(); +} + +static long Read(byte[] data) +{ + var reader = new MessagePackReader(data); + long total = 0; + for (int i = 0; i < Records; i++) + { + total += reader.ReadInt64(); + total += reader.ReadInt64(); + total += reader.ReadInt64(); + total += reader.ReadInt64(); + total += reader.ReadInt64(); + total += reader.ReadInt64(); + total += (long)reader.ReadDouble(); + total += reader.ReadString()!.Length; + } + return total; +} + +static long Skip(byte[] data) +{ + var reader = new MessagePackReader(data); + for (int i = 0; i < Fields; i++) + { + reader.Skip(); + } + return Fields; +} diff --git a/PerfTests/MassiveReadAmount/PerfExplicit.pb b/PerfTests/MassiveReadAmount/PerfExplicit.pb new file mode 100644 index 0000000..6cb99ab --- /dev/null +++ b/PerfTests/MassiveReadAmount/PerfExplicit.pb @@ -0,0 +1,111 @@ +EnableExplicit + +XIncludeFile "../../Includes/MessagePack.pbi" +XIncludeFile "../_PerfTesting.pbi" +XIncludeFile "_Workload.pbi" + + +; Writes and reads the shared workload, see `_Workload.pbi`, forcing each field's format code. + +Global *Encoded + + +Procedure WriteRecords(*Writer) + Protected i.i + + For i = 1 To #Perf_Records + PerfCheckWrite(MsgPackWriteFixInt(*Writer, 42)) + PerfCheckWrite(MsgPackWriteFixInt(*Writer, -5)) + PerfCheckWrite(MsgPackWriteUInt8(*Writer, 200)) + PerfCheckWrite(MsgPackWriteInt16(*Writer, -1000)) + PerfCheckWrite(MsgPackWriteUInt32(*Writer, 70000)) + PerfCheckWrite(MsgPackWriteInt64(*Writer, -1099511627776)) + PerfCheckWrite(MsgPackWriteFloat64(*Writer, 3.14)) + PerfCheckWrite(MsgPackWriteFixStr(*Writer, "Hello")) + Next +EndProcedure + +; Returns the finalized buffer, which must be freed with `FreeMemory()`. +Procedure.i Encode(InitialSize.i, GrowthIncrement.i) + Protected *Writer = MsgPackCreate(InitialSize, GrowthIncrement) + Protected *Buffer + + If Not *Writer + PerfFail("Failed to create the writer !") + EndIf + + WriteRecords(*Writer) + + *Buffer = MsgPackFinalize(*Writer) + If Not *Buffer + PerfFail("Failed to finalize the writer !") + EndIf + + ProcedureReturn *Buffer +EndProcedure + +Procedure.q EncodeAndFree(InitialSize.i, GrowthIncrement.i) + Protected *Buffer = Encode(InitialSize, GrowthIncrement) + Protected Size.q = MemorySize(*Buffer) + + FreeMemory(*Buffer) + ProcedureReturn Size +EndProcedure + + +Procedure.q ScenarioWrite() + ProcedureReturn EncodeAndFree(64, 1) +EndProcedure + +Procedure.q ScenarioWriteGrow64K() + ProcedureReturn EncodeAndFree(64, 65536) +EndProcedure + +Procedure.q ScenarioWritePrealloc() + ProcedureReturn EncodeAndFree(#Perf_EncodedSize, 1) +EndProcedure + +Procedure.q ScenarioRead() + Protected *Reader = MsgPackWrapMsgBuffer(*Encoded, MemorySize(*Encoded)) + Protected Total.q + Protected i.i + + If Not *Reader + PerfFail("Failed to wrap the buffer !") + EndIf + + For i = 1 To #Perf_Records + ; Fixints don't have an explicitly typed getter. + Total + MsgPackReadInteger(*Reader) + Total + MsgPackReadInteger(*Reader) + Total + MsgPackReadUInt8(*Reader) + Total + MsgPackReadInt16(*Reader) + Total + (MsgPackReadUInt32(*Reader) & $FFFFFFFF) + Total + MsgPackReadInt64(*Reader) + Total + Int(MsgPackReadFloat64(*Reader)) + Total + Len(MsgPackReadString(*Reader)) + Next + + ; Failed reads don't move the offset, so a failure breaks the next reads too. + If MsgPackLastError(*Reader) <> #MsgPack_Error_Success + PerfFail("A read failed !") + EndIf + + MsgPackFree(*Reader, #False) + ProcedureReturn Total +EndProcedure + + +OpenConsole() + +*Encoded = Encode(#Perf_EncodedSize, 1) + +PerfRun("pb-explicit", "read", @ScenarioRead(), #Perf_Fields) +PerfRun("pb-explicit", "write-prealloc", @ScenarioWritePrealloc(), #Perf_Fields) +PerfRun("pb-explicit", "write-grow64k", @ScenarioWriteGrow64K(), #Perf_Fields) +; Last since it reallocates the buffer on every write, which can be very slow. +PerfRun("pb-explicit", "write", @ScenarioWrite(), #Perf_Fields) + +FreeMemory(*Encoded) + +End 0 diff --git a/PerfTests/MassiveReadAmount/PerfImplicit.pb b/PerfTests/MassiveReadAmount/PerfImplicit.pb new file mode 100644 index 0000000..f725508 --- /dev/null +++ b/PerfTests/MassiveReadAmount/PerfImplicit.pb @@ -0,0 +1,132 @@ +EnableExplicit + +XIncludeFile "../../Includes/MessagePack.pbi" +XIncludeFile "../_PerfTesting.pbi" +XIncludeFile "_Workload.pbi" + + +; Writes and reads the shared workload, see `_Workload.pbi`, letting the library pick each field's format code. + +Global *Encoded + + +Procedure WriteRecords(*Writer) + Protected i.i + + For i = 1 To #Perf_Records + PerfCheckWrite(MsgPackWriteInteger(*Writer, 42)) + PerfCheckWrite(MsgPackWriteInteger(*Writer, -5)) + PerfCheckWrite(MsgPackWriteInteger(*Writer, 200)) + PerfCheckWrite(MsgPackWriteInteger(*Writer, -1000)) + PerfCheckWrite(MsgPackWriteInteger(*Writer, 70000)) + PerfCheckWrite(MsgPackWriteInteger(*Writer, -1099511627776)) + ; Floats don't have an implicitly typed setter. + PerfCheckWrite(MsgPackWriteFloat64(*Writer, 3.14)) + PerfCheckWrite(MsgPackWriteString(*Writer, "Hello")) + Next +EndProcedure + +; Returns the finalized buffer, which must be freed with `FreeMemory()`. +Procedure.i Encode(InitialSize.i, GrowthIncrement.i) + Protected *Writer = MsgPackCreate(InitialSize, GrowthIncrement) + Protected *Buffer + + If Not *Writer + PerfFail("Failed to create the writer !") + EndIf + + WriteRecords(*Writer) + + *Buffer = MsgPackFinalize(*Writer) + If Not *Buffer + PerfFail("Failed to finalize the writer !") + EndIf + + ProcedureReturn *Buffer +EndProcedure + +Procedure.q EncodeAndFree(InitialSize.i, GrowthIncrement.i) + Protected *Buffer = Encode(InitialSize, GrowthIncrement) + Protected Size.q = MemorySize(*Buffer) + + FreeMemory(*Buffer) + ProcedureReturn Size +EndProcedure + + +Procedure.q ScenarioWrite() + ProcedureReturn EncodeAndFree(64, 1) +EndProcedure + +Procedure.q ScenarioWriteGrow64K() + ProcedureReturn EncodeAndFree(64, 65536) +EndProcedure + +Procedure.q ScenarioWritePrealloc() + ProcedureReturn EncodeAndFree(#Perf_EncodedSize, 1) +EndProcedure + +Procedure.q ScenarioRead() + Protected *Reader = MsgPackWrapMsgBuffer(*Encoded, MemorySize(*Encoded)) + Protected Total.q + Protected i.i + + If Not *Reader + PerfFail("Failed to wrap the buffer !") + EndIf + + For i = 1 To #Perf_Records + Total + MsgPackReadInteger(*Reader) + Total + MsgPackReadInteger(*Reader) + Total + MsgPackReadInteger(*Reader) + Total + MsgPackReadInteger(*Reader) + Total + MsgPackReadInteger(*Reader) + Total + MsgPackReadInteger(*Reader) + Total + Int(MsgPackReadFloat(*Reader)) + Total + Len(MsgPackReadString(*Reader)) + Next + + ; Failed reads don't move the offset, so a failure breaks the next reads too. + If MsgPackLastError(*Reader) <> #MsgPack_Error_Success + PerfFail("A read failed !") + EndIf + + MsgPackFree(*Reader, #False) + ProcedureReturn Total +EndProcedure + +; Doesn't depend on the typing, so it's only measured here. +; Skips each field separately, like the Python and .NET implementations. +Procedure.q ScenarioSkip() + Protected *Reader = MsgPackWrapMsgBuffer(*Encoded, MemorySize(*Encoded)) + Protected i.i + + If Not *Reader + PerfFail("Failed to wrap the buffer !") + EndIf + + For i = 1 To #Perf_Fields + If MsgPackSkip(*Reader) <> #MsgPack_Error_Success + PerfFail("A skip failed !") + EndIf + Next + + MsgPackFree(*Reader, #False) + ProcedureReturn #Perf_Fields +EndProcedure + + +OpenConsole() + +*Encoded = Encode(#Perf_EncodedSize, 1) + +PerfRun("pb-implicit", "read", @ScenarioRead(), #Perf_Fields) +PerfRun("pb-implicit", "skip", @ScenarioSkip(), #Perf_Fields) +PerfRun("pb-implicit", "write-prealloc", @ScenarioWritePrealloc(), #Perf_Fields) +PerfRun("pb-implicit", "write-grow64k", @ScenarioWriteGrow64K(), #Perf_Fields) +; Last since it reallocates the buffer on every write, which can be very slow. +PerfRun("pb-implicit", "write", @ScenarioWrite(), #Perf_Fields) + +FreeMemory(*Encoded) + +End 0 diff --git a/PerfTests/MassiveReadAmount/PerfPython.py b/PerfTests/MassiveReadAmount/PerfPython.py new file mode 100644 index 0000000..5330be2 --- /dev/null +++ b/PerfTests/MassiveReadAmount/PerfPython.py @@ -0,0 +1,54 @@ +# Writes, reads and skips the workload shared with the PureBasic tests, see `_Workload.pbi`. +import sys +from pathlib import Path + +import msgpack + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) +from _PerfTesting import run + +RECORDS = 125_000 +FIELDS = RECORDS * 8 + + +def write(): + packer = msgpack.Packer(autoreset=False) + pack = packer.pack + for _ in range(RECORDS): + pack(42) + pack(-5) + pack(200) + pack(-1000) + pack(70000) + pack(-1099511627776) + pack(3.14) + pack("Hello") + return packer.bytes() + + +def read(): + unpacker = msgpack.Unpacker(raw=False) + unpacker.feed(ENCODED) + unpack = unpacker.unpack + total = 0 + for _ in range(RECORDS): + total += unpack() + unpack() + unpack() + unpack() + unpack() + unpack() + total += int(unpack()) + total += len(unpack()) + return total + + +def skip(): + unpacker = msgpack.Unpacker(raw=False) + unpacker.feed(ENCODED) + skip_field = unpacker.skip + for _ in range(FIELDS): + skip_field() + return FIELDS + + +ENCODED = write() + +run("write", FIELDS, lambda: len(write())) +run("read", FIELDS, read) +run("skip", FIELDS, skip) diff --git a/PerfTests/MassiveReadAmount/RunPerfTests.ps1 b/PerfTests/MassiveReadAmount/RunPerfTests.ps1 new file mode 100644 index 0000000..800d2c4 --- /dev/null +++ b/PerfTests/MassiveReadAmount/RunPerfTests.ps1 @@ -0,0 +1,18 @@ +$ErrorActionPreference = "Stop" + +# Shows the error instead of closing the window right away. +trap { + Write-Host $_ -ForegroundColor Red + Pause + exit 1 +} + +. (Join-Path $PSScriptRoot "..\_PerfTesting.ps1") + +# Requires Python with the packages from 'PerfTests\requirements.txt', and the .NET 10 SDK. +$Rows = @(Invoke-PureBasicPerfTests $PSScriptRoot) +$Rows += Invoke-PythonPerfTest (Join-Path $PSScriptRoot "PerfPython.py") +$Rows += Invoke-DotNetPerfTest (Join-Path $PSScriptRoot "DotNet") + +Show-PerfResults $Rows $PSScriptRoot +Pause diff --git a/PerfTests/MassiveReadAmount/_Workload.pbi b/PerfTests/MassiveReadAmount/_Workload.pbi new file mode 100644 index 0000000..cf0ac32 --- /dev/null +++ b/PerfTests/MassiveReadAmount/_Workload.pbi @@ -0,0 +1,15 @@ +; Workload shared by every implementation in this folder, keep `PerfPython.py` and `DotNet/Program.cs` in sync. +; +; `#Perf_Records` records of 8 fields, 1M fields in total, each record being: +; 42 (FixInt), -5 (FixInt), 200 (UInt8), -1000 (Int16), 70000 (UInt32), +; -1099511627776 (Int64), 3.14 (Float64), "Hello" (FixStr) +; +; Write check: the encoded size. +; Read check: the sum of the integers, of `Int()` of the float, and of the string's length. +; Skip check: the number of fields skipped. + +#Perf_Records = 125000 +#Perf_Fields = #Perf_Records * 8 + +; 1 + 1 + 2 + 3 + 5 + 9 + 9 + 6 bytes per record. +#Perf_EncodedSize = #Perf_Records * 36 diff --git a/PerfTests/NullChecks/PerfNullChecksOff.pb b/PerfTests/NullChecks/PerfNullChecksOff.pb new file mode 100644 index 0000000..547cd44 --- /dev/null +++ b/PerfTests/NullChecks/PerfNullChecksOff.pb @@ -0,0 +1,6 @@ +EnableExplicit + +#MsgPack_DisableNullChecks = #True +#Perf_Implementation = "pb-nullchecks-off" + +XIncludeFile "_Scenarios.pbi" diff --git a/PerfTests/NullChecks/PerfNullChecksOn.pb b/PerfTests/NullChecks/PerfNullChecksOn.pb new file mode 100644 index 0000000..7d72873 --- /dev/null +++ b/PerfTests/NullChecks/PerfNullChecksOn.pb @@ -0,0 +1,6 @@ +EnableExplicit + +; Default behaviour, `#Null` pointers are checked by every procedure. +#Perf_Implementation = "pb-nullchecks-on" + +XIncludeFile "_Scenarios.pbi" diff --git a/PerfTests/NullChecks/RunPerfTests.ps1 b/PerfTests/NullChecks/RunPerfTests.ps1 new file mode 100644 index 0000000..d2d651a --- /dev/null +++ b/PerfTests/NullChecks/RunPerfTests.ps1 @@ -0,0 +1,15 @@ +$ErrorActionPreference = "Stop" + +# Shows the error instead of closing the window right away. +trap { + Write-Host $_ -ForegroundColor Red + Pause + exit 1 +} + +. (Join-Path $PSScriptRoot "..\_PerfTesting.ps1") + +$Rows = @(Invoke-PureBasicPerfTests $PSScriptRoot) + +Show-PerfResults $Rows $PSScriptRoot +Pause diff --git a/PerfTests/NullChecks/_Scenarios.pbi b/PerfTests/NullChecks/_Scenarios.pbi new file mode 100644 index 0000000..17ee49d --- /dev/null +++ b/PerfTests/NullChecks/_Scenarios.pbi @@ -0,0 +1,96 @@ +; Shared by `PerfNullChecksOn.pb` and `PerfNullChecksOff.pb`, which only differ by `#MsgPack_DisableNullChecks`. +; Both must define `#Perf_Implementation` before including this file, and `#MsgPack_DisableNullChecks` for the "off" one. +; +; Writes and reads the "MassiveReadAmount" workload with implicit typing, into a buffer of the right size. +; Only the scenarios where the checks are the main overhead are measured. + +XIncludeFile "../../Includes/MessagePack.pbi" +XIncludeFile "../_PerfTesting.pbi" +XIncludeFile "../MassiveReadAmount/_Workload.pbi" + +Global *Encoded + + +Procedure WriteRecords(*Writer) + Protected i.i + + For i = 1 To #Perf_Records + PerfCheckWrite(MsgPackWriteInteger(*Writer, 42)) + PerfCheckWrite(MsgPackWriteInteger(*Writer, -5)) + PerfCheckWrite(MsgPackWriteInteger(*Writer, 200)) + PerfCheckWrite(MsgPackWriteInteger(*Writer, -1000)) + PerfCheckWrite(MsgPackWriteInteger(*Writer, 70000)) + PerfCheckWrite(MsgPackWriteInteger(*Writer, -1099511627776)) + PerfCheckWrite(MsgPackWriteFloat64(*Writer, 3.14)) + PerfCheckWrite(MsgPackWriteString(*Writer, "Hello")) + Next +EndProcedure + +; Returns the finalized buffer, which must be freed with `FreeMemory()`. +Procedure.i Encode() + Protected *Writer = MsgPackCreate(#Perf_EncodedSize) + Protected *Buffer + + If Not *Writer + PerfFail("Failed to create the writer !") + EndIf + + WriteRecords(*Writer) + + *Buffer = MsgPackFinalize(*Writer) + If Not *Buffer + PerfFail("Failed to finalize the writer !") + EndIf + + ProcedureReturn *Buffer +EndProcedure + + +Procedure.q ScenarioWritePrealloc() + Protected *Buffer = Encode() + Protected Size.q = MemorySize(*Buffer) + + FreeMemory(*Buffer) + ProcedureReturn Size +EndProcedure + +Procedure.q ScenarioRead() + Protected *Reader = MsgPackWrapMsgBuffer(*Encoded, MemorySize(*Encoded)) + Protected Total.q + Protected i.i + + If Not *Reader + PerfFail("Failed to wrap the buffer !") + EndIf + + For i = 1 To #Perf_Records + Total + MsgPackReadInteger(*Reader) + Total + MsgPackReadInteger(*Reader) + Total + MsgPackReadInteger(*Reader) + Total + MsgPackReadInteger(*Reader) + Total + MsgPackReadInteger(*Reader) + Total + MsgPackReadInteger(*Reader) + Total + Int(MsgPackReadFloat(*Reader)) + Total + Len(MsgPackReadString(*Reader)) + Next + + ; Failed reads don't move the offset, so a failure breaks the next reads too. + If MsgPackLastError(*Reader) <> #MsgPack_Error_Success + PerfFail("A read failed !") + EndIf + + MsgPackFree(*Reader, #False) + ProcedureReturn Total +EndProcedure + + +OpenConsole() + +*Encoded = Encode() + +PerfRun(#Perf_Implementation, "read", @ScenarioRead(), #Perf_Fields) +PerfRun(#Perf_Implementation, "write-prealloc", @ScenarioWritePrealloc(), #Perf_Fields) + +FreeMemory(*Encoded) + +End 0 diff --git a/PerfTests/StringsAndBinary/DotNet/DotNetPerf.csproj b/PerfTests/StringsAndBinary/DotNet/DotNetPerf.csproj new file mode 100644 index 0000000..c5de0f4 --- /dev/null +++ b/PerfTests/StringsAndBinary/DotNet/DotNetPerf.csproj @@ -0,0 +1,13 @@ + + + Exe + net10.0 + enable + enable + true + + + + + + diff --git a/PerfTests/StringsAndBinary/DotNet/Program.cs b/PerfTests/StringsAndBinary/DotNet/Program.cs new file mode 100644 index 0000000..f6a6636 --- /dev/null +++ b/PerfTests/StringsAndBinary/DotNet/Program.cs @@ -0,0 +1,77 @@ +// Writes and reads the workload shared with the PureBasic tests, see `../_Workload.pbi`. +// Uses the low-level MessagePackWriter/MessagePackReader from MessagePack-CSharp, pinned in the .csproj. +using System.Buffers; +using MessagePack; + +const int TotalSize = 33_554_432; + +foreach (int size in new[] { 8, 1024, 65536 }) +{ + int count = TotalSize / size; + string text = new string('a', size); + byte[] encoded = WriteStrings(text, count); + Perf.Run($"str-{size}-write", count, () => WriteStrings(text, count).Length); + Perf.Run($"str-{size}-read", count, () => ReadStrings(encoded, count)); +} + +foreach (int size in new[] { 1024, 1048576 }) +{ + int count = TotalSize / size; + byte[] data = new byte[size]; + Array.Fill(data, (byte)0xAB); + byte[] output = new byte[size]; + byte[] encoded = WriteBinaries(data, count); + Perf.Run($"bin-{size}-write", count, () => WriteBinaries(data, count).Length); + Perf.Run($"bin-{size}-read", count, () => ReadBinaries(encoded, count, output)); +} +return 0; + + +static byte[] WriteStrings(string text, int count) +{ + var buffer = new ArrayBufferWriter(); + var writer = new MessagePackWriter(buffer); + for (int i = 0; i < count; i++) + { + writer.Write(text); + } + writer.Flush(); + return buffer.WrittenSpan.ToArray(); +} + +static long ReadStrings(byte[] encoded, int count) +{ + var reader = new MessagePackReader(encoded); + long total = 0; + for (int i = 0; i < count; i++) + { + total += reader.ReadString()!.Length; + } + return total; +} + +static byte[] WriteBinaries(byte[] data, int count) +{ + var buffer = new ArrayBufferWriter(); + var writer = new MessagePackWriter(buffer); + for (int i = 0; i < count; i++) + { + writer.Write(data); + } + writer.Flush(); + return buffer.WrittenSpan.ToArray(); +} + +// Copies each field into `output`, like the PureBasic test. +static long ReadBinaries(byte[] encoded, int count, byte[] output) +{ + var reader = new MessagePackReader(encoded); + long total = 0; + for (int i = 0; i < count; i++) + { + var bytes = reader.ReadBytes()!.Value; + bytes.CopyTo(output); + total += bytes.Length; + } + return total; +} diff --git a/PerfTests/StringsAndBinary/PerfPython.py b/PerfTests/StringsAndBinary/PerfPython.py new file mode 100644 index 0000000..6199f86 --- /dev/null +++ b/PerfTests/StringsAndBinary/PerfPython.py @@ -0,0 +1,42 @@ +# Writes and reads the workload shared with the PureBasic tests, see `_Workload.pbi`. +import sys +from pathlib import Path + +import msgpack + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) +from _PerfTesting import run + +TOTAL_SIZE = 33_554_432 + + +def write(value, count): + packer = msgpack.Packer(autoreset=False) + pack = packer.pack + for _ in range(count): + pack(value) + return packer.bytes() + + +def read(encoded, count): + unpacker = msgpack.Unpacker(raw=False, max_buffer_size=len(encoded)) + unpacker.feed(encoded) + unpack = unpacker.unpack + total = 0 + for _ in range(count): + total += len(unpack()) + return total + + +def run_scenarios(name, value): + count = TOTAL_SIZE // len(value) + encoded = write(value, count) + run(f"{name}-{len(value)}-write", count, lambda: len(write(value, count))) + run(f"{name}-{len(value)}-read", count, lambda: read(encoded, count)) + + +for size in (8, 1024, 65536): + run_scenarios("str", "a" * size) + +for size in (1024, 1048576): + run_scenarios("bin", b"\xab" * size) diff --git a/PerfTests/StringsAndBinary/PerfStringsAndBinary.pb b/PerfTests/StringsAndBinary/PerfStringsAndBinary.pb new file mode 100644 index 0000000..bbee104 --- /dev/null +++ b/PerfTests/StringsAndBinary/PerfStringsAndBinary.pb @@ -0,0 +1,174 @@ +EnableExplicit + +XIncludeFile "../../Includes/MessagePack.pbi" +XIncludeFile "../_PerfTesting.pbi" +XIncludeFile "_Workload.pbi" + + +; Writes and reads the shared workload, see `_Workload.pbi`. + +; Current size, number of fields and string encoding. +Global FieldSize.i +Global FieldCount.i +Global Encoding.i + +; Data written in each field. +Global Text.s +Global *Data + +; Encoded fields, prepared once for the read scenarios. +Global *Encoded + +; Receives each binary field when reading. +Global *Output + + +; Returns the finalized buffer, which must be freed with `FreeMemory()`. +; The writer is big enough for all the fields, the reallocations are measured in "MassiveReadAmount". +Procedure.i EncodeStrings() + Protected *Writer = MsgPackCreate(FieldCount * (FieldSize * 2 + 5)) + Protected *Buffer + Protected i.i + + If Not *Writer + PerfFail("Failed to create the writer !") + EndIf + + For i = 1 To FieldCount + PerfCheckWrite(MsgPackWriteString(*Writer, Text, Encoding)) + Next + + *Buffer = MsgPackFinalize(*Writer) + If Not *Buffer + PerfFail("Failed to finalize the writer !") + EndIf + + ProcedureReturn *Buffer +EndProcedure + +; Returns the finalized buffer, which must be freed with `FreeMemory()`. +Procedure.i EncodeBinaries() + Protected *Writer = MsgPackCreate(FieldCount * (FieldSize + 5)) + Protected *Buffer + Protected i.i + + If Not *Writer + PerfFail("Failed to create the writer !") + EndIf + + For i = 1 To FieldCount + PerfCheckWrite(MsgPackWriteBinary(*Writer, *Data, FieldSize)) + Next + + *Buffer = MsgPackFinalize(*Writer) + If Not *Buffer + PerfFail("Failed to finalize the writer !") + EndIf + + ProcedureReturn *Buffer +EndProcedure + +Procedure.q BufferSizeAndFree(*Buffer) + Protected Size.q = MemorySize(*Buffer) + + FreeMemory(*Buffer) + ProcedureReturn Size +EndProcedure + + +Procedure.q ScenarioStringWrite() + ProcedureReturn BufferSizeAndFree(EncodeStrings()) +EndProcedure + +Procedure.q ScenarioStringRead() + Protected *Reader = MsgPackWrapMsgBuffer(*Encoded, MemorySize(*Encoded)) + Protected Total.q + Protected i.i + + If Not *Reader + PerfFail("Failed to wrap the buffer !") + EndIf + + For i = 1 To FieldCount + Total + Len(MsgPackReadString(*Reader, Encoding)) + Next + + ; Failed reads don't move the offset, so a failure breaks the next reads too. + If MsgPackLastError(*Reader) <> #MsgPack_Error_Success + PerfFail("A read failed !") + EndIf + + MsgPackFree(*Reader, #False) + ProcedureReturn Total +EndProcedure + +Procedure.q ScenarioBinaryWrite() + ProcedureReturn BufferSizeAndFree(EncodeBinaries()) +EndProcedure + +Procedure.q ScenarioBinaryRead() + Protected *Reader = MsgPackWrapMsgBuffer(*Encoded, MemorySize(*Encoded)) + Protected Total.q, BytesRead.q + Protected i.i + + If Not *Reader + PerfFail("Failed to wrap the buffer !") + EndIf + + For i = 1 To FieldCount + BytesRead = MsgPackReadBinary(*Reader, *Output, FieldSize) + If BytesRead = #MsgPack_InvalidSize + PerfFail("A read failed !") + EndIf + Total + BytesRead + Next + + MsgPackFree(*Reader, #False) + ProcedureReturn Total +EndProcedure + + +Procedure RunStringScenarios(Size.i, StringEncoding.i, Name.s) + FieldSize = Size + FieldCount = #Perf_TotalSize / Size + Encoding = StringEncoding + Text = LSet("", Size, "a") + *Encoded = EncodeStrings() + + PerfRun("pb", Name + "-" + Str(Size) + "-write", @ScenarioStringWrite(), FieldCount) + PerfRun("pb", Name + "-" + Str(Size) + "-read", @ScenarioStringRead(), FieldCount) + + FreeMemory(*Encoded) +EndProcedure + +Procedure RunBinaryScenarios(Size.i) + FieldSize = Size + FieldCount = #Perf_TotalSize / Size + *Data = AllocateMemory(Size) + *Output = AllocateMemory(Size) + If Not *Data Or Not *Output + PerfFail("Failed to allocate the buffers !") + EndIf + FillMemory(*Data, Size, $AB) + *Encoded = EncodeBinaries() + + PerfRun("pb", "bin-" + Str(Size) + "-write", @ScenarioBinaryWrite(), FieldCount) + PerfRun("pb", "bin-" + Str(Size) + "-read", @ScenarioBinaryRead(), FieldCount) + + FreeMemory(*Encoded) + FreeMemory(*Output) + FreeMemory(*Data) +EndProcedure + + +OpenConsole() + +RunStringScenarios(8, #PB_UTF8, "str") +RunStringScenarios(1024, #PB_UTF8, "str") +RunStringScenarios(65536, #PB_UTF8, "str") +RunStringScenarios(1024, #PB_Unicode, "str-utf16") + +RunBinaryScenarios(1024) +RunBinaryScenarios(1048576) + +End 0 diff --git a/PerfTests/StringsAndBinary/RunPerfTests.ps1 b/PerfTests/StringsAndBinary/RunPerfTests.ps1 new file mode 100644 index 0000000..800d2c4 --- /dev/null +++ b/PerfTests/StringsAndBinary/RunPerfTests.ps1 @@ -0,0 +1,18 @@ +$ErrorActionPreference = "Stop" + +# Shows the error instead of closing the window right away. +trap { + Write-Host $_ -ForegroundColor Red + Pause + exit 1 +} + +. (Join-Path $PSScriptRoot "..\_PerfTesting.ps1") + +# Requires Python with the packages from 'PerfTests\requirements.txt', and the .NET 10 SDK. +$Rows = @(Invoke-PureBasicPerfTests $PSScriptRoot) +$Rows += Invoke-PythonPerfTest (Join-Path $PSScriptRoot "PerfPython.py") +$Rows += Invoke-DotNetPerfTest (Join-Path $PSScriptRoot "DotNet") + +Show-PerfResults $Rows $PSScriptRoot +Pause diff --git a/PerfTests/StringsAndBinary/_Workload.pbi b/PerfTests/StringsAndBinary/_Workload.pbi new file mode 100644 index 0000000..f7e6b5e --- /dev/null +++ b/PerfTests/StringsAndBinary/_Workload.pbi @@ -0,0 +1,14 @@ +; Workload shared by every implementation in this folder, keep `PerfPython.py` and `DotNet/Program.cs` in sync. +; +; For each size, `#Perf_TotalSize / Size` fields of `Size` bytes are written, then read: +; * Strings of 8, 1024 and 65536 "a" characters, encoded as UTF-8. +; * Binaries of 1024 and 1048576 bytes set to `$AB`. +; The PureBasic test also writes and reads strings of 1024 characters as UTF-16, which the others don't support. +; +; Scenarios are named `str--write`, `str--read`, `bin--write` and `bin--read`. +; `NsPerOp` is the time per field. +; +; Write check: the encoded size. +; Read check: the total number of characters or bytes read. + +#Perf_TotalSize = 33554432 diff --git a/PerfTests/_PerfTesting.cs b/PerfTests/_PerfTesting.cs new file mode 100644 index 0000000..2cb6905 --- /dev/null +++ b/PerfTests/_PerfTesting.cs @@ -0,0 +1,23 @@ +// Shared by the .NET performance tests, see `_PerfTesting.pbi` for the output format. +using System.Diagnostics; + +static class Perf +{ + const int Runs = 5; + + // Runs `function` once to warm up, then `Runs` times, and prints its median. + public static void Run(string scenario, long operations, Func function) + { + long check = function(); + var times = new double[Runs]; + for (int i = 0; i < Runs; i++) + { + var stopwatch = Stopwatch.StartNew(); + check = function(); + times[i] = stopwatch.Elapsed.TotalMilliseconds; + } + Array.Sort(times); + double median = times[Runs / 2]; + Console.WriteLine($"dotnet,{scenario},{median:F1},{median * 1_000_000 / operations:F1},{check}"); + } +} diff --git a/PerfTests/_PerfTesting.pbi b/PerfTests/_PerfTesting.pbi new file mode 100644 index 0000000..3b65f9d --- /dev/null +++ b/PerfTests/_PerfTesting.pbi @@ -0,0 +1,52 @@ +; Shared by the PureBasic performance tests. +; +; Each scenario is run once to warm up, then `#Perf_Runs` times, and its median is printed as: +; `Implementation,Scenario,MedianMs,NsPerOp,Check` +; `Check` is returned by the scenario, e.g. an encoded size or a checksum, to compare implementations. + +#Perf_Runs = 5 + +; Times are measured with `QueryPerformanceCounter_()`, `ElapsedMilliseconds()` is too coarse. +CompilerIf #PB_Compiler_OS <> #PB_OS_Windows + CompilerError "The performance tests only support Windows !" +CompilerEndIf + +Prototype.q PerfScenario() + + +; Errors go to stderr, stdout only holds the results. +Procedure PerfFail(Message.s) + ConsoleError("FAILED: " + Message) + End 1 +EndProcedure + +Macro PerfCheckWrite(ErrorCode) + If ErrorCode <> #MsgPack_Error_Success + PerfFail("A write failed !") + EndIf +EndMacro + + +; `Operations` is the number of operations done by one run of `Function`, used for `NsPerOp`. +Procedure PerfRun(Implementation.s, Scenario.s, Function.PerfScenario, Operations.q) + Protected Dim Times.d(#Perf_Runs - 1) + Protected Check.q, Frequency.q, Start.q, Stop.q + Protected Median.d + Protected i.i + + QueryPerformanceFrequency_(@Frequency) + + Check = Function() + + For i = 0 To #Perf_Runs - 1 + QueryPerformanceCounter_(@Start) + Check = Function() + QueryPerformanceCounter_(@Stop) + Times(i) = (Stop - Start) * 1000.0 / Frequency + Next + + SortArray(Times(), #PB_Sort_Ascending) + Median = Times(#Perf_Runs / 2) + + PrintN(Implementation + "," + Scenario + "," + StrD(Median, 1) + "," + StrD(Median * 1000000.0 / Operations, 1) + "," + Str(Check)) +EndProcedure diff --git a/PerfTests/_PerfTesting.ps1 b/PerfTests/_PerfTesting.ps1 new file mode 100644 index 0000000..778e08a --- /dev/null +++ b/PerfTests/_PerfTesting.ps1 @@ -0,0 +1,132 @@ +# Shared by the 'RunPerfTests.ps1' scripts, dot-source it. + +# PureBasic versions, with the compiler executable of each architecture and backend. +# 6.21 is kept to compare with the string handling changes of 6.40. +# No x86 ASM backend, its 'FAsm.exe' triggers the AV and crashes anyway. +$Compilers = @{ + "6.41" = @{ + "x86" = @{ Path = "C:\Program Files (x86)\PureBasic_641"; Backends = @{ "C" = "pbcompilerc.exe" } } + "x64" = @{ Path = "C:\Program Files\PureBasic_641"; Backends = @{ "ASM" = "pbcompiler.exe"; "C" = "pbcompilerc.exe" } } + "arm64" = @{ Path = "C:\Program Files\PureBasic_641_arm64\PureBasic"; Backends = @{ "C" = "pbcompiler.exe" } } + } + "6.21" = @{ + "x86" = @{ Path = "C:\Program Files (x86)\PureBasic_621"; Backends = @{ "C" = "pbcompilerc.exe" } } + "x64" = @{ Path = "C:\Program Files\PureBasic_621"; Backends = @{ "ASM" = "pbcompiler.exe"; "C" = "pbcompilerc.exe" } } + } +} + +$PerfHeader = @("Implementation", "Scenario", "MedianMs", "NsPerOp", "Check") + +# Where `Show-PerfResults` saves the results. +$PerfResultsDir = Join-Path $PSScriptRoot "Results" + +# Prints each result line as it arrives, and passes it along. +filter Show-PerfLine { + Write-Host " $_" + $_ +} + +# Prompts for a choice and used `&` for the hotkey. +function Select-Choice([string]$Title, [string[]]$Labels) { + $Choices = $Labels | ForEach-Object { New-Object System.Management.Automation.Host.ChoiceDescription $_ } + $Index = $Host.UI.PromptForChoice($Title, "", $Choices, 0) + return $Labels[$Index].Replace("&", "") +} + +# Prompts for a PureBasic version, architecture and backend, then builds and runs every '.pb' file in '$Directory'. +# They're built without the debugger, and with the C optimizer, like a release would be. +# Returns the parsed result lines. +function Invoke-PureBasicPerfTests([string]$Directory) { + $Version = Select-Choice "Version" @("6.&41", "6.&21") + $Architecture = Select-Choice "Architecture" @("x&86", "x&64", "&arm64") + $Backend = Select-Choice "Backend" @("&ASM", "&C") + + # Used to name the results file. + $script:PerfTarget = "${Version}_$Architecture-$Backend" + + $Compiler = $Compilers[$Version][$Architecture] + if (-not $Compiler -or -not $Compiler.Backends.Contains($Backend)) { + throw "No compiler configured for PureBasic $Version on $Architecture with the $Backend backend." + } + $CompilerExe = Join-Path $Compiler.Path "Compilers\$($Compiler.Backends[$Backend])" + if (-not (Test-Path $CompilerExe)) { + throw "Compiler not found: $CompilerExe" + } + + $CompilerFlags = @("/CONSOLE") + if ($Backend -eq "C") { + $CompilerFlags += "/OPTIMIZER" + } + + $Rows = @() + + # Built in the repo, outside of any temp folder, and deleted afterward. + $BuildDir = Join-Path $Directory "Build" + New-Item -ItemType Directory -Force -Path $BuildDir | Out-Null + + try { + foreach ($PerfTest in Get-ChildItem -Path $Directory -Filter "*.pb") { + $Exe = Join-Path $BuildDir "$($PerfTest.BaseName).exe" + + $CompilerOutput = & $CompilerExe $PerfTest.FullName /EXE $Exe @CompilerFlags | Out-String + if ($LASTEXITCODE -ne 0) { + throw "Compilation failed: $($PerfTest.Name)`n$CompilerOutput" + } + + Write-Host "> $($PerfTest.Name) - PureBasic $Version - $Architecture - $Backend" + $Rows += & $Exe | Show-PerfLine | ConvertFrom-Csv -Header $PerfHeader + if ($LASTEXITCODE -ne 0) { + throw "Failed: $($PerfTest.Name) (exit code $LASTEXITCODE)" + } + } + } finally { + Remove-Item -Recurse -Force $BuildDir + } + + return $Rows +} + +# Runs a Python perf test, requires the packages from 'PerfTests\requirements.txt'. +# Returns the parsed result lines. +function Invoke-PythonPerfTest([string]$Script) { + Write-Host "> $(Split-Path $Script -Leaf)" + $Rows = @(& python $Script | Show-PerfLine | ConvertFrom-Csv -Header $PerfHeader) + if ($LASTEXITCODE -ne 0) { + throw "Failed: $Script, install its packages with 'pip install -r PerfTests\requirements.txt'" + } + return $Rows +} + +# Builds and runs a .NET perf test project, requires the .NET 10 SDK. +# Returns the parsed result lines. +function Invoke-DotNetPerfTest([string]$Project) { + Write-Host "> $(Split-Path $Project -Leaf)" + & dotnet build $Project -c Release --nologo -v quiet | Out-Host + if ($LASTEXITCODE -ne 0) { + throw "Compilation failed: $Project" + } + $Rows = @(& dotnet run --project $Project -c Release --no-build | Show-PerfLine | ConvertFrom-Csv -Header $PerfHeader) + if ($LASTEXITCODE -ne 0) { + throw "Failed: $Project (exit code $LASTEXITCODE)" + } + return $Rows +} + +# Prints the results, reports scenarios whose checks differ between implementations, +# and saves them as 'Results\__-.csv', replacing the previous ones. +function Show-PerfResults([object[]]$Rows, [string]$Directory) { + $Rows | Format-Table -AutoSize | Out-Host + + # Every implementation must have encoded the same bytes and read the same values. + foreach ($Group in $Rows | Group-Object Scenario) { + $Checks = @($Group.Group | ForEach-Object { $_.Check } | Select-Object -Unique) + if ($Checks.Count -ne 1) { + Write-Host "Mismatching checks for '$($Group.Name)': $($Checks -join ', ')" -ForegroundColor Red + } + } + + New-Item -ItemType Directory -Force -Path $PerfResultsDir | Out-Null + $ResultsFile = Join-Path $PerfResultsDir "$(Split-Path $Directory -Leaf)_$PerfTarget.csv" + $Rows | Export-Csv -Path $ResultsFile -NoTypeInformation + Write-Host "Results saved to $ResultsFile" +} diff --git a/PerfTests/_PerfTesting.py b/PerfTests/_PerfTesting.py new file mode 100644 index 0000000..03e19d2 --- /dev/null +++ b/PerfTests/_PerfTesting.py @@ -0,0 +1,17 @@ +# Shared by the Python performance tests, see `_PerfTesting.pbi` for the output format. +import statistics +import time + +RUNS = 5 + + +def run(scenario, operations, function): + """Runs `function` once to warm up, then `RUNS` times, and prints its median.""" + check = function() + times = [] + for _ in range(RUNS): + start = time.perf_counter() + check = function() + times.append((time.perf_counter() - start) * 1000) + median = statistics.median(times) + print(f"python,{scenario},{median:.1f},{median * 1_000_000 / operations:.1f},{check}", flush=True) diff --git a/PerfTests/readme.md b/PerfTests/readme.md new file mode 100644 index 0000000..299bae4 --- /dev/null +++ b/PerfTests/readme.md @@ -0,0 +1,117 @@ +# PureBasic - MsgPack - Performances +Performance benchmarks for the [PB-MessagePack](../readme.md) include. + +Please refer to the [Performances](../readme.md#performances) section for the warnings and conclusions. \ +Open questions raised by these results are tracked in [Investigations.md](Investigations.md). + + +## Workloads +Each folder has a `RunPerfTests.ps1` script that runs its benchmarks: +* `MassiveReadAmount/` - 1M small fields written, read and skipped, with explicit and implicit typing. +* `StringsAndBinary/` - Strings of 8 B to 64 KB, and binaries of 1 KB and 1 MB, written and read. +* `FlatMaps/` - 100K maps of 16 entries written and read. +* `DeepNesting/` - Arrays nested 1 to 128 levels deep, built, examined and measured. (PureBasic only) +* `NullChecks/` - Small fields written and read with and without `#MsgPack_DisableNullChecks`. (PureBasic only) + +The first three are compared to [msgpack](https://pypi.org/project/msgpack/) (Python) +and [MessagePack-CSharp](https://github.com/MessagePack-CSharp/MessagePack-CSharp) (.NET). + +The scripts prompt for the PureBasic version, architecture and backend. \ +The results are saved in `Results/`, as one CSV per folder, PureBasic version, architecture and backend. + + +## Results +Hardware: +* Proxmox VM running Windows 10 +* 10 Intel Core i5-13500 cores **(No E/P core pinning)** +* PureBasic 6.41 +* Python 3.14 with [msgpack](https://pypi.org/project/msgpack/) 1.2.3 +* .NET 10 with [MessagePack-CSharp](https://github.com/MessagePack-CSharp/MessagePack-CSharp) 3.1.12 + +Times are in nanoseconds per field, per map or per nested structure, lower is better. \ +Every implementation produced the same bytes and read back the same values. \ +PureBasic 6.41 is used everywhere, except in the [last table](#purebasic-621-vs-641). + +### Compared to other implementations +x64 with the C backend, the fastest of each row in bold. + +| Workload | Op | PureBasic | Python | .NET | +|:--|:--|--:|--:|--:| +| 1M small fields 1 | Write | 11.9 | 50.9 | **5.4** | +| | Read | 12.7 | 52.4 | **10.6** | +| | Skip | **18.3** | 40.2 | 29.4 | +| Strings, 8 B | Write | 59.6 | 62.0 | **13.2** | +| | Read | 46.6 | 73.4 | **16.2** | +| Strings, 1 KB | Write | 720.9 | **556.0** | 804.8 | +| | Read | 570.0 | 341.3 | **150.0** | +| Strings, 64 KB | Write | 45,917 | **33,932** | 39,848 | +| | Read | 31,473 | **16,390** | 23,045 | +| Binary, 1 KB | Write | **190.6** | 556.6 | 576.1 | +| | Read | **50.2** | 311.3 | 204.0 | +| Binary, 1 MB | Write | **207,744** | 574,825 | 438,025 | +| | Read | **54,812** | 465,075 | 59,459 | +| Maps of 16 entries | Write | 1,793 | 927.2 | **209.2** | +| | Read | 1,858 | 1,962 | **292.1** | + +1 1M fields of mixed integers, floats and short strings, with implicit typing. PureBasic writes into a preallocated buffer. + +### Architectures & backends +PureBasic only. + +| Scenario | x64 C | x64 ASM | x86 C | +|:--|--:|--:|--:| +| Small fields, explicit write | 11.0 | 18.0 | 13.9 | +| Small fields, explicit read | 9.8 | 15.9 | 8.8 | +| Small fields, implicit write | 11.9 | 25.4 | 14.2 | +| Small fields, implicit read | 12.7 | 25.3 | 11.5 | +| Small fields, skip | 18.3 | 57.9 | 23.2 | +| Small fields, write, 64 KB growth 2 | 37.9 | 51.5 | 44.7 | +| Small fields, write, exact growth 2 | 441.0 | 461.6 | 503.3 | +| Small fields, read, null checks off 3 | 11.4 | 19.5 | 11.4 | +| Small fields, read, null checks on 3 | 12.2 | 20.2 | 11.4 | +| Strings, 1 KB UTF-16, write | 526.0 | 539.1 | 599.0 | +| Strings, 1 KB UTF-16, read | 282.7 | 298.8 | 289.1 | +| Maps of 16 entries, write | 1,793 | 3,472 | 2,189 | +| Maps of 16 entries, read | 1,858 | 3,232 | 1,782 | + +2 The buffer starts at 64 bytes and grows in multiples of 64 KB, or by the exact size of each write. (`GrowthIncrement` of `1`) \ +3 With and without `#MsgPack_DisableNullChecks`. + +### Deep nesting +x64 with the C backend, arrays of 3 integers nested `N` levels deep. + +| Depth | Build | Examine | Examine, unsafe | `MsgPackGetFieldFullSize` | +|--:|--:|--:|--:|--:| +| 1 | 283.3 | 177.8 | 134.9 | 80.4 | +| 2 | 453.4 | 363.2 | 261.4 | 115.8 | +| 4 | 1,002 | 825.2 | 464.1 | 177.3 | +| 8 | 2,226 | 2,154 | 906.9 | 308.9 | +| 16 | 6,414 | 6,155 | 1,807 | 546.5 | +| 32 | 21,888 | 21,134 | 4,367 | 985.9 | +| 64 | 76,370 | 72,715 | 7,774 | 1,997 | +| 128 | 278,388 | 269,656 | 15,803 | 3,812 | + +### PureBasic 6.21 vs 6.41 +x64 with the C backend, the new string handling of 6.40 being the main difference. + +| Scenario | 6.21 | 6.41 | Speedup | +|:--|--:|--:|--:| +| Small fields, implicit write 4 | 23.0 | 11.9 | 1.9x | +| Small fields, implicit read 4 | 26.4 | 12.7 | 2.1x | +| Small fields, skip | 14.7 | 18.3 | 0.8x | +| Strings, 8 B, write | 162.8 | 59.6 | 2.7x | +| Strings, 8 B, read | 164.5 | 46.6 | 3.5x | +| Strings, 1 KB, write | 1,971 | 720.9 | 2.7x | +| Strings, 1 KB, read | 1,412 | 570.0 | 2.5x | +| Strings, 64 KB, write | 113,872 | 45,917 | 2.5x | +| Strings, 64 KB, read | 77,056 | 31,473 | 2.4x | +| Strings, 1 KB UTF-16, write | 2,026 | 526.0 | 3.9x | +| Strings, 1 KB UTF-16, read | 1,065 | 282.7 | 3.8x | +| Binary, 1 KB, write | 173.4 | 190.6 | 0.9x | +| Binary, 1 KB, read | 41.2 | 50.2 | 0.8x | +| Maps of 16 entries, write 4 | 3,221 | 1,793 | 1.8x | +| Maps of 16 entries, read 4 | 3,478 | 1,858 | 1.9x | +| Nested 128 levels, examine | 263,158 | 269,656 | 1.0x | + +4 Includes strings, one per record of 8 fields, and one per map key. \ +The x64 ASM and x86 C results show the same 2 to 3 times speedup on strings. diff --git a/requirements.txt b/PerfTests/requirements.txt similarity index 100% rename from requirements.txt rename to PerfTests/requirements.txt diff --git a/RunTests.ps1 b/RunTests.ps1 new file mode 100644 index 0000000..ea115d4 --- /dev/null +++ b/RunTests.ps1 @@ -0,0 +1,96 @@ +$ErrorActionPreference = "Stop" + + +# PureBasic versions used for unit tests, with the compiler executable of each backend. +$Compilers = [ordered]@{ + "x86" = @( + @{ Name = "5.73"; Path = "C:\Program Files (x86)\PureBasic_573"; Backends = @{ "ASM" = "pbcompiler.exe" } }, + @{ Name = "6.21"; Path = "C:\Program Files (x86)\PureBasic_621"; Backends = @{ "ASM" = "pbcompiler.exe"; "C" = "pbcompilerc.exe" } }, + @{ Name = "6.40"; Path = "C:\Program Files (x86)\PureBasic_640"; Backends = @{ "ASM" = "pbcompiler.exe"; "C" = "pbcompilerc.exe" } }, + # No ASM backend, its 'FAsm.exe' triggers the AV and crashes anyway. + @{ Name = "6.41"; Path = "C:\Program Files (x86)\PureBasic_641"; Backends = @{ "C" = "pbcompilerc.exe" } } + ) + "x64" = @( + @{ Name = "5.73"; Path = "C:\Program Files\PureBasic_573"; Backends = @{ "ASM" = "pbcompiler.exe" } }, + @{ Name = "6.21"; Path = "C:\Program Files\PureBasic_621"; Backends = @{ "ASM" = "pbcompiler.exe"; "C" = "pbcompilerc.exe" } }, + @{ Name = "6.40"; Path = "C:\Program Files\PureBasic_640"; Backends = @{ "ASM" = "pbcompiler.exe"; "C" = "pbcompilerc.exe" } }, + @{ Name = "6.41"; Path = "C:\Program Files\PureBasic_641"; Backends = @{ "ASM" = "pbcompiler.exe"; "C" = "pbcompilerc.exe" } } + ) + "arm64" = @( + @{ Name = "6.21"; Path = "C:\Program Files\PureBasic_621_arm64"; Backends = @{ "C" = "pbcompiler.exe" } }, + @{ Name = "6.41"; Path = "C:\Program Files\PureBasic_641_arm64\PureBasic"; Backends = @{ "C" = "pbcompiler.exe" } } + ) +} + +# Prompts for a choice and used `&` for the hotkey. +function Select-Choice([string]$Title, [string[]]$Labels) { + $Choices = $Labels | ForEach-Object { New-Object System.Management.Automation.Host.ChoiceDescription $_ } + $Index = $Host.UI.PromptForChoice($Title, "", $Choices, 0) + return $Labels[$Index].Replace("&", "") +} + + +$Architecture = Select-Choice "Architecture" @("x&86", "x&64", "&arm64") +$Backend = Select-Choice "Backend" @("&ASM", "&C") + +# Sets '#Endianness_UseGccBuiltins' in the PB-Endianness include, only used by the C backend. +$UseGccBuiltins = "False" +if ($Backend -eq "C") { + $UseGccBuiltins = Select-Choice "Use GCC builtins" @("&False", "&True") +} +$UseGccBuiltinsValue = @{ "False" = 0; "True" = 1 }[$UseGccBuiltins] + +$SelectedCompilers = @($Compilers[$Architecture] | Where-Object { $_.Backends.Contains($Backend) }) +if ($SelectedCompilers.Count -eq 0) { + throw "No compilers configured for $Architecture with the $Backend backend." +} + +$UnitTests = Get-ChildItem -Path (Join-Path $PSScriptRoot "UnitTests") -Recurse -Filter "*.pb" + +# Built in the repo, outside of any temp folder, and deleted afterward. +$BuildDir = Join-Path $PSScriptRoot "Build" +New-Item -ItemType Directory -Force -Path $BuildDir | Out-Null + +$FailedCount = 0 + +try { + foreach ($Compiler in $SelectedCompilers) { + $CompilerExe = Join-Path $Compiler.Path "Compilers\$($Compiler.Backends[$Backend])" + if (-not (Test-Path $CompilerExe)) { + throw "Compiler not found: $CompilerExe" + } + + Write-Host "`n=== PureBasic $($Compiler.Name) - $Architecture - $Backend - GCC builtins: $UseGccBuiltins ===" -ForegroundColor Cyan + + foreach ($UnitTest in $UnitTests) { + $Exe = Join-Path $BuildDir "$($UnitTest.BaseName).exe" + + # Built without the debugger, the 'FatalErrors' tests rely on it. + $CompilerOutput = & $CompilerExe $UnitTest.FullName /CONSOLE /EXE $Exe /CONSTANT "Endianness_UseGccBuiltins=$UseGccBuiltinsValue" | Out-String + if ($LASTEXITCODE -ne 0) { + throw "Compilation failed: $($UnitTest.Name) with PureBasic $($Compiler.Name)`n$CompilerOutput" + } + + Write-Host "> $($UnitTest.Directory.Name)\$($UnitTest.Name)" + & $Exe + if ($LASTEXITCODE -ne 0) { + Write-Host "FAILED (exit code $LASTEXITCODE)" -ForegroundColor Red + $FailedCount++ + } else { + Write-Host "PASSED" -ForegroundColor Green + } + + Remove-Item -Force $Exe + } + } +} finally { + Remove-Item -Recurse -Force $BuildDir +} + +Write-Host "" +if ($FailedCount -ne 0) { + Write-Host "$FailedCount test(s) failed." -ForegroundColor Red +} else { + Write-Host "All tests passed." -ForegroundColor Green +} +Pause diff --git a/readme.md b/readme.md index 616e9e1..ca1498e 100644 --- a/readme.md +++ b/readme.md @@ -14,15 +14,19 @@ Based on the [msgpack/msgpack](https://github.com/msgpack/msgpack) specification * Explicit or implicit data formats when reading/writing * *ReadInteger vs ReadI8/ReadU8/...* * ~~Fully documented~~ +* Supports all modern PureBasic setups + * x86/x64/arm64 + * asm/c backends + * 5.73 to 6.41 ## Limitations * No custom `malloc`/`realloc`/`free` callbacks * More involved Map and Array interactions * *No dynamic typing* - * *Building arrays/maps requires special insertion in parent MsgPack ~~(See examples)~~* + * *Building arrays/maps requires special insertion in parent MsgPack (See [examples](#arrays--maps))* * *Lots of internal checks for boundaries and counts before any reading can be done* - * *[TODO: Note about deep nesting and performances]* + * *Deep nesting costs more time, see [Performances](#performances)* * Usage of larger `.q` type for field sizes * *Helps prevent unsigned numbers causing issues on x86* * Particular handling of signaling `NaN` for *Float32* @@ -45,11 +49,67 @@ Pull latest submodule version: git submodule update --init --recursive ``` -TODO: Add note about handling pre-inclusion of PB-Endianness - ### Inclusion -... +Include the main file, it pulls in everything else, [PB-Endianness](https://github.com/aziascreations/PB-Endianness) included: +```purebasic +XIncludeFile "Includes/PB-MessagePack/Includes/MessagePack.pbi" +``` + +The options below can be defined before the inclusion: +* `#MsgPack_MaxDepth` - Maximum nesting depth of arrays and maps, from `1` to `4096`. (Default: `128`) +* `#MsgPack_DisableNullChecks` - Skips the `#Null` pointer checks when set to `#True`. (Default: `#False`) + +### Double import safeguard +`Endianness.pbi` raises a `CompilerError` if it is included twice from different paths, +to prevent two versions of it from being mixed. + +If your project, or another include, already uses its own copy of PB-Endianness, +include it **before** `MessagePack.pbi`, which will then skip its own copy: +```purebasic +XIncludeFile "Includes/PB-Endianness/Includes/Endianness.pbi" +XIncludeFile "Includes/PB-MessagePack/Includes/MessagePack.pbi" +``` + +Including it afterward will trigger the error, unless `#Endianness_BypassIncludeCheck` is defined. + + +## Performances + +> [!WARNING] +> Parts of the library can be much slower than C# and Python. +> +> This is partially due to many security checks, and the rest is actively being investigated. +> We're also in the early stages of development, so there's also that... + +### Security trade-offs +Arrays and maps are always validated before being used: +* `MsgPackExamine` checks the whole array/map before returning a view, so its content is read twice. \ + Examining a nested array/map checks its content again, so data nested `N` levels deep can be read up to `N+1` times. +* `MsgPackWriteArray/Map` and their variants check and count the given fields, then copy them. \ + Building nested arrays/maps copies the inner data once per level. +* `MsgPackGetFieldFullSize`, `MsgPackSkip` and `MsgPackNextEntry` walk through every contained field, they don't jump over them. + +This is mostly noticeable with deeply nested data, which is why `#MsgPack_MaxDepth` defaults to `128` to prevent wasting time on hostile data. + +`#MsgPack_ExaminationOptions_Unsafe` skips the check done by `MsgPackExamine`, and it can be used on non-root iterable values. \ +However, using on an iterable element at the root means invalid data is only detected while iterating ! + + +### Current status +* Binary data and skipping fields is faster than in msgpack and MessagePack-CSharp. +* Small fields, short strings and maps are slower than in MessagePack-CSharp. +* Writing maps is slower that C# and Python because of the checks mentionned above. +* Strings are about 4 times slower to write and up to 11 times slower to read than binary data of the same size. + * *Currently being [investigated](PerfTests/Investigations.md).* +* The new strings library in PureBasic 6.40+ is 2.5 to 4 times faster than older versions. +* Using the C backend instead of the ASM one is up to 3 times faster. +* Preallocate the buffer or using a bigger `GrowthIncrement` than the default of `1` makes small writes up to 35 times faster. +* `#MsgPack_DisableNullChecks` only saves about 5%. +* Examining nested data gets quadratically slower with the depth + * *It is up to 17 times slower when not using `#MsgPack_ExaminationOptions_Unsafe` at 128 levels.* + +The detailed results, workloads and how to run them are in [PerfTests/readme.md](PerfTests/readme.md). ## Examples @@ -58,6 +118,19 @@ TODO: Add note about handling pre-inclusion of PB-Endianness * [BasicExplicitTyping.pb](Examples/BasicExplicitTyping/BasicExplicitTyping.pb) * [BasicImplicitTyping.pb](Examples/BasicImplicitTyping/BasicImplicitTyping.pb) +### Data types +* [BasicStrings.pb](Examples/BasicStrings/BasicStrings.pb) +* [BasicBinary.pb](Examples/BasicBinary/BasicBinary.pb) +* [BasicExtensions.pb](Examples/BasicExtensions/BasicExtensions.pb) + +### Arrays & Maps +* [BasicArrays.pb](Examples/BasicArrays/BasicArrays.pb) +* [BasicMaps.pb](Examples/BasicMaps/BasicMaps.pb) + +### Dumping a buffer +* [DumpExplicitTypes.pb](Examples/DumpExplicitTypes/DumpExplicitTypes.pb) - Exact format code of each field +* [DumpImplicitTypes.pb](Examples/DumpImplicitTypes/DumpImplicitTypes.pb) - General data type of each field + ## Cloning Use this command to clone the repository and its submodules: