mirror of
https://github.com/aziascreations/PB-MessagePack.git
synced 2026-11-03 17:01:24 +01:00
Added examples, Added performance tests, Updated readme
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -1,17 +1,24 @@
|
||||
# MessagePack.pbi
|
||||
PureBasic include for creating/reading MessagePack data.
|
||||
|
||||
**When integrating, do not read any `.pbi` files to save on tokens, use the API below.**
|
||||
**When integrating, do not read any `.pbi` files to save on tokens, use the API below.** \
|
||||
When working on the include itself, see [Working on this repository](#working-on-this-repository).
|
||||
|
||||
|
||||
## APIs
|
||||
|
||||
### Behaviour
|
||||
* ???
|
||||
* All the procedures take a `*MsgPackData` from `MsgPackCreate`, `MsgPackWrapMsgBuffer` or `MsgPackExamine`.
|
||||
* Errors are reported through the internal error code, see `MsgPackLastError`.
|
||||
* Writes and other procedures returning a `.MsgPack_ErrorCode` also return it directly.
|
||||
* Reads returning a value give `0`, `""`, `#False`, `#Null` or `#MsgPack_InvalidSize` on errors,
|
||||
which can also be valid values, so check `MsgPackLastError` after them.
|
||||
* A successful read or write resets it to `#MsgPack_Error_Success`.
|
||||
* Failed reads and writes don't move the internal offset, and write nothing.
|
||||
* The internal offset is auto-incremented on all complete reads and writes.
|
||||
* Does not apply to `MsgPackGetFormatCode`.
|
||||
* Does not apply to `MsgPackGetFormatCode`, `MsgPackGetExtensionType`, `MsgPackGetEntryCount` and `MsgPackGetFieldFullSize`.
|
||||
* Can be disabled with the internal `#MsgPack_DataFlags_MoveOffsetAfterRead/Write` flags, honored by every read and write.
|
||||
* The internal buffer is automatically resized as needed.
|
||||
* The internal buffer is automatically grown as needed, unless it is owned by the caller.
|
||||
* Invalid or truncated data never raises a `DebuggerError`, only programming errors do. (e.g.: `#Null` pointers)
|
||||
* Views from `MsgPackExamine` are read-only, writes return `#MsgPack_Error_BufferIsReadOnly`.
|
||||
|
||||
@@ -20,7 +27,7 @@ PureBasic include for creating/reading MessagePack data.
|
||||
|
||||
* `.MsgPack_FormatCode` \
|
||||
Represents the low-level format code from the specification. \
|
||||
When [interacting] with fix* types, the value in it will be set to 0.
|
||||
Fix* types are returned without the value/size bits they embed, e.g. `$A0` for any FixStr and `0` for positive fixints.
|
||||
* `#MsgPack_FormatCode_Null`
|
||||
* `#MsgPack_FormatCode_False`
|
||||
* `#MsgPack_FormatCode_True`
|
||||
@@ -63,7 +70,7 @@ PureBasic include for creating/reading MessagePack data.
|
||||
* `.MsgPack_DataType` \
|
||||
High-level representation of a format code. \
|
||||
Should be used when iterating on complex objects to detect the members' general type.
|
||||
* `#MsgPack_DataTypes_Unknown` - The `0x??` format code.
|
||||
* `#MsgPack_DataTypes_Unknown` - For `#MsgPack_FormatCode_Invalid`.
|
||||
* `#MsgPack_DataTypes_Null`
|
||||
* `#MsgPack_DataTypes_Boolean`
|
||||
* `#MsgPack_DataTypes_Integer`
|
||||
@@ -72,13 +79,14 @@ PureBasic include for creating/reading MessagePack data.
|
||||
* `#MsgPack_DataTypes_Binary`
|
||||
* `#MsgPack_DataTypes_Array`
|
||||
* `#MsgPack_DataTypes_Map`
|
||||
* `#MsgPack_DataTypes_Ext`
|
||||
* `#MsgPack_DataTypes_Extension`
|
||||
|
||||
* `.MsgPack_ErrorCode` \
|
||||
* `#MsgPack_Error_Success = 0`
|
||||
* `#MsgPack_Error_NullPointerGiven`
|
||||
* `#MsgPack_Error_BufferTooSmall`
|
||||
* `#MsgPack_Error_InvalidValue`
|
||||
* `#MsgPack_Error_ValueOverflow`
|
||||
* `#MsgPack_Error_FailedToGrowBuffer`
|
||||
* `#MsgPack_Error_AtOrPastTheEndOfBuffer`
|
||||
* `#MsgPack_Error_ReadPastEndOfBuffer`
|
||||
@@ -91,27 +99,63 @@ PureBasic include for creating/reading MessagePack data.
|
||||
* `#MsgPack_Error_InvalidStringSize`
|
||||
* `#MsgPack_Error_TooManyEntries`
|
||||
* `#MsgPack_Error_OddMapFieldCount`
|
||||
* `#MsgPack_Error_MaxDepthExceeded`
|
||||
* `#MsgPack_Error_CannotGrowBufferDueToConfig`
|
||||
* `#MsgPack_Error_BufferIsReadOnly`
|
||||
|
||||
* `.MsgPack_DataFlag` \
|
||||
Internal flags, do not touch.
|
||||
|
||||
* `.MsgPack_ExaminationOption` \
|
||||
Options for `MsgPackExamine`.
|
||||
* `#MsgPack_ExaminationOptions_Unsafe`
|
||||
|
||||
### Commons
|
||||
|
||||
### Constants & compiler options
|
||||
* `#MsgPack_InvalidSize = -1` \
|
||||
Returned by procedures that return a size, on errors.
|
||||
|
||||
* `#MsgPack_MaxDepth` \
|
||||
Maximum nesting depth of arrays and maps, a top-level array or map being at depth 1. \
|
||||
Defaults to `128`, can be defined before the inclusion, from `1` to `4096`. \
|
||||
Deeper data is rejected with `#MsgPack_Error_MaxDepthExceeded`.
|
||||
|
||||
* `#MsgPack_DisableNullChecks` \
|
||||
Disables the `#Null` pointer checks when defined as `#True` before the inclusion. \
|
||||
Defaults to `#False`.
|
||||
|
||||
|
||||
### Errors & format codes
|
||||
* `MsgPackLastError.MsgPack_ErrorCode(*MsgPackData.MsgPackData)` \
|
||||
...
|
||||
Returns the error code set by the last read, write or navigation on `*MsgPackData`, `#MsgPack_Error_Success` if it succeeded. \
|
||||
Returns `#MsgPack_Error_NullPointerGiven` if `*MsgPackData` is `#Null`.
|
||||
|
||||
* `MsgPackGetFormatCode.MsgPack_FormatCode(*MsgPackData.MsgPackData)` \
|
||||
Returns the format code of the field at the current internal offset, without moving it or setting the internal error code. \
|
||||
Returns `#MsgPack_FormatCode_Invalid` at the end of the buffer, or for the `0xC1` byte.
|
||||
|
||||
* `MsgPackFormatCodeToDataType.MsgPack_DataType(FormatCode.MsgPack_FormatCode)` \
|
||||
Returns the data type of a format code from `MsgPackGetFormatCode`. \
|
||||
Returns `#MsgPack_DataTypes_Unknown` for `#MsgPack_FormatCode_Invalid`.
|
||||
|
||||
|
||||
### Thingies
|
||||
* `MsgPackCreate.i(InitialSize.i)` \
|
||||
### Allocation & ownership
|
||||
* `MsgPackCreate.i(InitialSize.i, GrowthIncrement.i = 1)` \
|
||||
Prepares an internal structure for a buffer of a given size `InitialSize.i`. \
|
||||
The buffer grows in multiples of `GrowthIncrement` bytes, `1` growing it by the exact size needed by each write. \
|
||||
Returns `#Null` on errors.
|
||||
|
||||
* `MsgPackWrapMsgBuffer.i(*Buffer, BufferSize.i)` \
|
||||
Wraps a given buffer `*Buffer` of a given size `InitialSize.i` into an internal structure. \
|
||||
* `MsgPackWrapMsgBuffer.i(*Buffer, BufferSize.i, GiveOwnership.b = #False, GrowthIncrement.i = 1)` \
|
||||
Wraps a given buffer `*Buffer` of a given size `BufferSize.i` into an internal structure, without copying it. \
|
||||
If `GiveOwnership` is `#False`, the buffer stays owned by the caller, and is never grown or freed. \
|
||||
If `GiveOwnership` is `#True`, `*Buffer` must come from `AllocateMemory()`, it can be grown, and freed with `MsgPackFree`. \
|
||||
Returns `#Null` on errors.
|
||||
|
||||
* `MsgPackGrow.MsgPack_ErrorCode(*MsgPackData.MsgPackData, DesiredSizeIncrease.q)` \
|
||||
Grows the buffer by `DesiredSizeIncrease` bytes, rounded up to a multiple of its `GrowthIncrement`. \
|
||||
Called by the writes as needed, there's no need to call it beforehand. \
|
||||
Returns `#MsgPack_Error_Success` on success, something else otherwise. (e.g.: `#MsgPack_Error_CannotGrowBufferDueToConfig` for buffers owned by the caller)
|
||||
|
||||
* `MsgPackFree.b(*MsgPackData.MsgPackData, FreeDataBuffer.b)` \
|
||||
Frees a given internal structure `*MsgPackData` and its associated buffer if `FreeDataBuffer` is `#True`. \
|
||||
`FreeDataBuffer` is mandatory, use `#False` for buffers wrapped with `MsgPackWrapMsgBuffer` that the caller still owns. \
|
||||
@@ -126,29 +170,28 @@ PureBasic include for creating/reading MessagePack data.
|
||||
|
||||
|
||||
### Literals
|
||||
* `MsgPackReadBoolean.b(*MsgPackData.MsgPackData)`
|
||||
* `MsgPackReadBoolean.b(*MsgPackData.MsgPackData)` \
|
||||
Attempts to read the boolean at the current internal offset. \
|
||||
Returns `#True` or `#False`. \
|
||||
May set the internal error code if any happens. \
|
||||
Returns `#True` or `#False`, `#False` on errors too. \
|
||||
See: `MsgPackLastError`
|
||||
|
||||
* `MsgPackWriteNull.MsgPack_ErrorCode(*MsgPackData.MsgPackData)`
|
||||
Attempts to write `#Null` at the current internal offset. \
|
||||
* `MsgPackWriteNull.MsgPack_ErrorCode(*MsgPackData.MsgPackData)` \
|
||||
Attempts to write a nil at the current internal offset. \
|
||||
Returns `#MsgPack_Error_Success` on success, something else otherwise.
|
||||
|
||||
* `MsgPackWriteTrue.MsgPack_ErrorCode(*MsgPackData.MsgPackData)`
|
||||
* `MsgPackWriteTrue.MsgPack_ErrorCode(*MsgPackData.MsgPackData)` \
|
||||
Attempts to write `#True` at the current internal offset. \
|
||||
Returns `#MsgPack_Error_Success` on success, something else otherwise.
|
||||
|
||||
* `MsgPackWriteFalse.MsgPack_ErrorCode(*MsgPackData.MsgPackData)`
|
||||
* `MsgPackWriteFalse.MsgPack_ErrorCode(*MsgPackData.MsgPackData)` \
|
||||
Attempts to write `#False` at the current internal offset. \
|
||||
Returns `#MsgPack_Error_Success` on success, something else otherwise.
|
||||
|
||||
* `MsgPackWriteBoolean.MsgPack_ErrorCode(*MsgPackData.MsgPackData, Boolean.b)`
|
||||
* `MsgPackWriteBoolean.MsgPack_ErrorCode(*MsgPackData.MsgPackData, Boolean.b)` \
|
||||
Writes `#True` or `#False` depending on `Boolean`. \
|
||||
Returns `#MsgPack_Error_Success` on success, something else otherwise.
|
||||
|
||||
* `MsgPackReadNull.MsgPack_ErrorCode(*MsgPackData.MsgPackData)`
|
||||
* `MsgPackReadNull.MsgPack_ErrorCode(*MsgPackData.MsgPackData)` \
|
||||
Attempts to read a nil at the current internal offset. \
|
||||
Returns `#MsgPack_Error_Success` on success, something else otherwise, in which case the offset isn't moved.
|
||||
|
||||
@@ -316,6 +359,49 @@ EndIf
|
||||
Returns `#MsgPack_Error_TooManyEntries` if the entries don't fit in it.
|
||||
|
||||
|
||||
## Usage patterns
|
||||
|
||||
### Writing, exporting and reading
|
||||
```purebasic
|
||||
Define *Writer = MsgPackCreate(64)
|
||||
If MsgPackWriteInteger(*Writer, 42) <> #MsgPack_Error_Success
|
||||
; ...
|
||||
EndIf
|
||||
|
||||
; Frees `*Writer`, the caller now owns `*Buffer`.
|
||||
Define *Buffer = MsgPackFinalize(*Writer)
|
||||
|
||||
; Gives `*Buffer` back to a structure, freed along with it.
|
||||
Define *Reader = MsgPackWrapMsgBuffer(*Buffer, MemorySize(*Buffer), #True)
|
||||
Define Value.q = MsgPackReadInteger(*Reader)
|
||||
If MsgPackLastError(*Reader) <> #MsgPack_Error_Success
|
||||
; ...
|
||||
EndIf
|
||||
|
||||
MsgPackFree(*Reader, #True)
|
||||
```
|
||||
|
||||
### Building arrays & maps
|
||||
Their content is written in a separate structure, then copied in with its fields counted.
|
||||
```purebasic
|
||||
Define *Fields = MsgPackCreate(32)
|
||||
MsgPackWriteString(*Fields, "Key")
|
||||
MsgPackWriteInteger(*Fields, 1)
|
||||
MsgPackWriteMapFrom(*Writer, *Fields)
|
||||
MsgPackFree(*Fields, #True)
|
||||
```
|
||||
Forcing their format code requires the content as a buffer, e.g. from `MsgPackFinalize`, for `MsgPackWriteFixArray/Array16/...`.
|
||||
|
||||
### Iterating over a buffer
|
||||
The end of a buffer can't be queried, so either wrap the data in a top-level array/map,
|
||||
or subtract each top-level field's `MsgPackGetFieldFullSize` from the buffer's size until it reaches `0`.
|
||||
|
||||
### Missing or asymmetric procedures
|
||||
* Floats have no implicitly typed setter, use `MsgPackWriteFloat32/64`.
|
||||
* Strings have no explicitly typed getters, `MsgPackReadString` reads all str format codes.
|
||||
* Fixints have no explicitly typed getter, `MsgPackReadInteger` reads them.
|
||||
|
||||
|
||||
## Remarks
|
||||
* Never touch the `MsgPackData` structure unless you **absolutely** have to.
|
||||
|
||||
@@ -323,3 +409,46 @@ EndIf
|
||||
## Support
|
||||
- Both the ASM and C backends
|
||||
- x86, x64, arm64
|
||||
|
||||
|
||||
## Working on this repository
|
||||
|
||||
### Layout
|
||||
* `Includes/MessagePack.pbi` - Main file, only includes the others.
|
||||
* `Includes/_*.pbi` - One file per family. (Allocators, Literals, Primitives, Strings, ...)
|
||||
* `Includes/PB-Endianness/` - Git submodule, don't edit it from here.
|
||||
* `UnitTests/` - One folder per family, numbered like the includes' order.
|
||||
* `Examples/` - One folder per example, with a `.pb` of the same name.
|
||||
* `PerfTests/` - Benchmarks, one folder per workload, each with a `RunPerfTests.ps1`.
|
||||
* `SecurityTests/` - Private, git-ignored, leave it alone unless asked.
|
||||
|
||||
### Conventions
|
||||
* `.pb`/`.pbi` files use CRLF line endings and ASCII only, `Chr()` for anything else.
|
||||
* Includes use tabs, examples use 4 spaces.
|
||||
* Private procedures and macros are prefixed with `_MsgPack`, public ones with `MsgPack`.
|
||||
* Every public procedure is documented above its definition, and in this file.
|
||||
* `#Null` pointers raise a `DebuggerError` behind `CompilerIf Not #MsgPack_DisableNullChecks`, invalid data only uses `Debug`.
|
||||
|
||||
### Unit tests
|
||||
* Run with `RunTests.ps1`, which builds each test without the debugger, so `Debug` & co are compiled out.
|
||||
* Each test includes `../_UnitTesting.pbi`, uses its `Assert*` procedures, and ends with `End 1` if `FailedUnitTests` isn't `0`.
|
||||
* `*FatalErrors.pb` tests call `DebuggerError` on purpose, and only pass without the debugger.
|
||||
* No PureBasic compiler is available in the cloud sandbox, ask the user to run the tests.
|
||||
|
||||
### Performance tests
|
||||
* Run with the `RunPerfTests.ps1` of each folder, Windows only.
|
||||
* They dot-source `PerfTests/_PerfTesting.ps1`, which builds the folder's `.pb` files without the debugger, with `/OPTIMIZER` on the C backend,
|
||||
runs `PerfPython.py` and `DotNet/` if present, and saves the results in `PerfTests/Results/<Folder>_<Version>_<Architecture>-<Backend>.csv`.
|
||||
* PureBasic 6.41 and 6.21 are configured in `$Compilers`, 6.21 being kept to compare the string handling changes of 6.40.
|
||||
* Each implementation times its scenarios with the shared `_PerfTesting.pbi`/`.py`/`.cs`: one warm-up run, then the median of 5.
|
||||
* Each program prints `Implementation,Scenario,MedianMs,NsPerOp,Check` lines on stdout, errors go to stderr.
|
||||
* `Check` must be identical for a scenario across implementations, the scripts report mismatches.
|
||||
* Each folder's workload is described in its `_Workload.pbi`, keep its `PerfPython.py` and `DotNet/Program.cs` in sync with it.
|
||||
* Python packages are pinned in `PerfTests/requirements.txt`, MessagePack-CSharp in each `DotNetPerf.csproj`.
|
||||
* The results tables are in `PerfTests/readme.md`, the conclusions in the main readme's "Performances" section, keep both in sync with new results.
|
||||
* Open performance questions and their leads are in `PerfTests/Investigations.md`, update it when one is looked into.
|
||||
|
||||
### Examples
|
||||
* Start with a `;- Notes` section, then `;- Example`.
|
||||
* Check every result, and use a different `End` code for each failure.
|
||||
* Link new examples in the readme's "Examples" section.
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
; ------------------------------------------------------------------------------
|
||||
;- Notes
|
||||
|
||||
; This example shows you how to write an array from entries written in a
|
||||
; separate writer, and how to iterate on it.
|
||||
|
||||
|
||||
; ------------------------------------------------------------------------------
|
||||
;- Example
|
||||
|
||||
EnableExplicit
|
||||
|
||||
XIncludeFile "../../Includes/MessagePack.pbi"
|
||||
|
||||
; Obscure structures that hold info about the MsgPack data
|
||||
Define *Writer, *Entries, *Reader, *Iterator
|
||||
|
||||
; MsgPack-encoded buffer that will get exported midway in this example.
|
||||
Define *Buffer
|
||||
|
||||
Define i.i, Value.q
|
||||
|
||||
|
||||
|
||||
; Writing the entries in their own writer.
|
||||
*Entries = MsgPackCreate(64)
|
||||
If Not *Entries
|
||||
Debug "Failed to create the entries writer !"
|
||||
End 1
|
||||
EndIf
|
||||
|
||||
For i = 1 To 5
|
||||
If MsgPackWriteInteger(*Entries, i * 10) <> #MsgPack_Error_Success
|
||||
Debug "Failed to write entry #" + Str(i) + " !"
|
||||
End 2
|
||||
EndIf
|
||||
Next
|
||||
|
||||
|
||||
; Writing the array, the entries are counted and copied into it.
|
||||
*Writer = MsgPackCreate(64)
|
||||
If Not *Writer
|
||||
Debug "Failed to create the writer !"
|
||||
End 3
|
||||
EndIf
|
||||
|
||||
If MsgPackWriteArrayFrom(*Writer, *Entries) <> #MsgPack_Error_Success
|
||||
Debug "Failed to write the array !"
|
||||
End 4
|
||||
EndIf
|
||||
|
||||
; The entries writer is left untouched, so it must be freed.
|
||||
MsgPackFree(*Entries, #True)
|
||||
|
||||
|
||||
; Exporting the data and giving it to a reader.
|
||||
*Buffer = MsgPackFinalize(*Writer)
|
||||
If Not *Buffer
|
||||
Debug "Failed to finalize the writer !"
|
||||
End 5
|
||||
EndIf
|
||||
|
||||
Debug "Encoded size: " + Str(MemorySize(*Buffer))
|
||||
|
||||
*Reader = MsgPackWrapMsgBuffer(*Buffer, MemorySize(*Buffer), #True)
|
||||
If Not *Reader
|
||||
Debug "Failed to wrap the buffer !"
|
||||
End 6
|
||||
EndIf
|
||||
|
||||
|
||||
; Iterating on the array.
|
||||
Debug "Entry count: " + Str(MsgPackGetEntryCount(*Reader))
|
||||
|
||||
*Iterator = MsgPackExamine(*Reader)
|
||||
If Not *Iterator
|
||||
Debug "Failed to examine the array !"
|
||||
End 7
|
||||
EndIf
|
||||
|
||||
While MsgPackNextEntry(*Iterator)
|
||||
Value = MsgPackReadInteger(*Iterator)
|
||||
If MsgPackLastError(*Iterator) <> #MsgPack_Error_Success
|
||||
Debug "Failed to read an entry !"
|
||||
End 8
|
||||
EndIf
|
||||
|
||||
Debug "Entry: " + Str(Value)
|
||||
Wend
|
||||
|
||||
; `MsgPackNextEntry` also returns `#False` on errors.
|
||||
If MsgPackLastError(*Iterator) <> #MsgPack_Error_Success
|
||||
Debug "Failed to iterate on the array !"
|
||||
End 9
|
||||
EndIf
|
||||
|
||||
|
||||
; Freeing the view first, it reads from the reader's buffer.
|
||||
MsgPackFinish(*Iterator)
|
||||
MsgPackFree(*Reader, #True)
|
||||
|
||||
End 0
|
||||
@@ -0,0 +1,97 @@
|
||||
; ------------------------------------------------------------------------------
|
||||
;- Notes
|
||||
|
||||
; This example shows you how to write and read raw binary data.
|
||||
; It can either be copied into a buffer you own, or into a new one.
|
||||
|
||||
|
||||
; ------------------------------------------------------------------------------
|
||||
;- Example
|
||||
|
||||
EnableExplicit
|
||||
|
||||
XIncludeFile "../../Includes/MessagePack.pbi"
|
||||
|
||||
; Obscure structures that hold info about the MsgPack data
|
||||
Define *Writer, *Reader
|
||||
|
||||
; MsgPack-encoded buffer that will get exported midway in this example.
|
||||
Define *Buffer
|
||||
|
||||
; Raw data to encode, and buffers that will receive the decoded one.
|
||||
Define *Input = AllocateMemory(4)
|
||||
Define *Output = AllocateMemory(16)
|
||||
Define *Copy
|
||||
Define BytesRead.q
|
||||
|
||||
If Not *Input Or Not *Output
|
||||
Debug "Failed to allocate the buffers !"
|
||||
End 1
|
||||
EndIf
|
||||
|
||||
PokeL(*Input, $DEADBEEF)
|
||||
|
||||
|
||||
|
||||
; Writing the same data twice into the buffer.
|
||||
*Writer = MsgPackCreate(64)
|
||||
If Not *Writer
|
||||
Debug "Failed to create the writer !"
|
||||
End 2
|
||||
EndIf
|
||||
|
||||
If MsgPackWriteBinary(*Writer, *Input, MemorySize(*Input)) <> #MsgPack_Error_Success
|
||||
Debug "Failed to write the first binary field !"
|
||||
End 3
|
||||
EndIf
|
||||
|
||||
If MsgPackWriteBinary(*Writer, *Input, MemorySize(*Input)) <> #MsgPack_Error_Success
|
||||
Debug "Failed to write the second binary field !"
|
||||
End 4
|
||||
EndIf
|
||||
|
||||
|
||||
; Exporting the data and giving it to a reader.
|
||||
*Buffer = MsgPackFinalize(*Writer)
|
||||
If Not *Buffer
|
||||
Debug "Failed to finalize the writer !"
|
||||
End 5
|
||||
EndIf
|
||||
|
||||
Debug "Encoded size: " + Str(MemorySize(*Buffer))
|
||||
|
||||
*Reader = MsgPackWrapMsgBuffer(*Buffer, MemorySize(*Buffer), #True)
|
||||
If Not *Reader
|
||||
Debug "Failed to wrap the buffer !"
|
||||
End 6
|
||||
EndIf
|
||||
|
||||
|
||||
; Copying the first field into a buffer we own.
|
||||
; It fails if the buffer is too small, without moving the offset.
|
||||
BytesRead = MsgPackReadBinary(*Reader, *Output, MemorySize(*Output))
|
||||
If BytesRead = #MsgPack_InvalidSize
|
||||
Debug "Failed to read the first binary field !"
|
||||
End 7
|
||||
EndIf
|
||||
|
||||
Debug "Read " + Str(BytesRead) + " bytes: $" + Hex(PeekL(*Output), #PB_Long)
|
||||
|
||||
|
||||
; Copying the second field into a new buffer of the right size.
|
||||
*Copy = MsgPackCopyBinary(*Reader)
|
||||
If Not *Copy
|
||||
Debug "Failed to copy the second binary field !"
|
||||
End 8
|
||||
EndIf
|
||||
|
||||
Debug "Copied " + Str(MemorySize(*Copy)) + " bytes: $" + Hex(PeekL(*Copy), #PB_Long)
|
||||
|
||||
|
||||
; Freeing everything.
|
||||
FreeMemory(*Copy)
|
||||
FreeMemory(*Output)
|
||||
FreeMemory(*Input)
|
||||
MsgPackFree(*Reader, #True)
|
||||
|
||||
End 0
|
||||
@@ -0,0 +1,138 @@
|
||||
; ------------------------------------------------------------------------------
|
||||
;- Notes
|
||||
|
||||
; This example shows you how to write, export and read a MessagePack buffer
|
||||
; with explicitly typed variables whose format code is forced by the
|
||||
; procedure used to write them, even if a smaller one would fit.
|
||||
|
||||
|
||||
; ------------------------------------------------------------------------------
|
||||
;- Example
|
||||
|
||||
EnableExplicit
|
||||
|
||||
XIncludeFile "../../Includes/MessagePack.pbi"
|
||||
|
||||
Structure MyDataPacket
|
||||
Id.q
|
||||
UserName.s
|
||||
EndStructure
|
||||
|
||||
Procedure.b EncodeMyDataPacket(*MsgPackData, *Input.MyDataPacket)
|
||||
; Always written as an Int64, 9 bytes in total.
|
||||
If MsgPackWriteInt64(*MsgPackData, *Input\Id) <> #MsgPack_Error_Success
|
||||
ProcedureReturn #False
|
||||
EndIf
|
||||
|
||||
; Always written as a Str8, which only fits up to 255 bytes.
|
||||
If MsgPackWriteStr8(*MsgPackData, *Input\UserName) <> #MsgPack_Error_Success
|
||||
ProcedureReturn #False
|
||||
EndIf
|
||||
|
||||
ProcedureReturn #True
|
||||
EndProcedure
|
||||
|
||||
Procedure.b DecodeMyDataPacket(*MsgPackData, *Output.MyDataPacket)
|
||||
; Fails if the field isn't an Int64.
|
||||
*Output\Id = MsgPackReadInt64(*MsgPackData)
|
||||
If MsgPackLastError(*MsgPackData) <> #MsgPack_Error_Success
|
||||
ProcedureReturn #False
|
||||
EndIf
|
||||
|
||||
; Strings don't have explicit getters, any str format code is accepted.
|
||||
*Output\UserName = MsgPackReadString(*MsgPackData)
|
||||
If MsgPackLastError(*MsgPackData) <> #MsgPack_Error_Success
|
||||
ProcedureReturn #False
|
||||
EndIf
|
||||
|
||||
ProcedureReturn #True
|
||||
EndProcedure
|
||||
|
||||
|
||||
|
||||
; Obscure structures that hold info about the MsgPack data
|
||||
Define *Writer, *Reader
|
||||
|
||||
; MsgPack-encoded buffer that will get exported midway in this example.
|
||||
Define *Buffer
|
||||
|
||||
; Structure that will hold the data to encode and the decoded one.
|
||||
; You could replace this with whatever you want in your use cases.
|
||||
Define InputStruct.MyDataPacket, OutputStruct.MyDataPacket
|
||||
|
||||
|
||||
|
||||
; Preparing the input structure.
|
||||
With InputStruct
|
||||
\Id = 67
|
||||
\UserName = "Herwin"
|
||||
EndWith
|
||||
|
||||
|
||||
; Writing the data into the buffer.
|
||||
*Writer = MsgPackCreate(64)
|
||||
If Not *Writer
|
||||
Debug "Failed to create the writer !"
|
||||
End 1
|
||||
EndIf
|
||||
|
||||
If Not EncodeMyDataPacket(*Writer, @InputStruct)
|
||||
Debug "Failed to encode the packet !"
|
||||
End 2
|
||||
EndIf
|
||||
|
||||
|
||||
; Exporting the data.
|
||||
; Doing this frees the obscure structure in `*Writer`.
|
||||
*Buffer = MsgPackFinalize(*Writer)
|
||||
If Not *Buffer
|
||||
Debug "Failed to finalize the writer !"
|
||||
End 3
|
||||
EndIf
|
||||
|
||||
; 9 bytes for the Int64, 2+6 bytes for the Str8.
|
||||
Debug "Encoded size: " + Str(MemorySize(*Buffer))
|
||||
|
||||
|
||||
; Giving ownership of the buffer to the obscure structure.
|
||||
*Reader = MsgPackWrapMsgBuffer(*Buffer, MemorySize(*Buffer), #True)
|
||||
If Not *Reader
|
||||
Debug "Failed to wrap the buffer !"
|
||||
End 4
|
||||
EndIf
|
||||
|
||||
|
||||
; Getting data out of the MsgPack buffer.
|
||||
If Not DecodeMyDataPacket(*Reader, @OutputStruct)
|
||||
Debug "Failed to decode the packet !"
|
||||
End 5
|
||||
EndIf
|
||||
|
||||
Debug "Id: " + Str(OutputStruct\Id)
|
||||
Debug "UserName: " + OutputStruct\UserName
|
||||
|
||||
; Freeing the obscure structure and the buffer it owns.
|
||||
MsgPackFree(*Reader, #True)
|
||||
|
||||
|
||||
; Forcing a format code that is too small for the data fails.
|
||||
; Str8 only fits up to 255 bytes, so a 256 bytes name can't be written.
|
||||
InputStruct\UserName = LSet("", 256, "A")
|
||||
|
||||
*Writer = MsgPackCreate(64)
|
||||
If Not *Writer
|
||||
Debug "Failed to create the writer !"
|
||||
End 6
|
||||
EndIf
|
||||
|
||||
If EncodeMyDataPacket(*Writer, @InputStruct)
|
||||
Debug "Encoded a 256 bytes name as Str8, this shouldn't happen !"
|
||||
End 7
|
||||
EndIf
|
||||
|
||||
Debug "Failed to encode a 256 bytes name as Str8, as expected."
|
||||
|
||||
; Nothing was exported, so the writer still owns its buffer.
|
||||
MsgPackFree(*Writer, #True)
|
||||
|
||||
End 0
|
||||
@@ -0,0 +1,98 @@
|
||||
; ------------------------------------------------------------------------------
|
||||
;- Notes
|
||||
|
||||
; This example shows you how to write and read extensions, which are binary
|
||||
; data tagged with an application-defined type, here a 2D vector.
|
||||
; Types from 0 to 127 are free to use, -128 to -1 are reserved by the spec.
|
||||
|
||||
|
||||
; ------------------------------------------------------------------------------
|
||||
;- Example
|
||||
|
||||
EnableExplicit
|
||||
|
||||
XIncludeFile "../../Includes/MessagePack.pbi"
|
||||
|
||||
#ExtType_Vector2 = 1
|
||||
|
||||
Structure Vector2
|
||||
X.f
|
||||
Y.f
|
||||
EndStructure
|
||||
|
||||
; Obscure structures that hold info about the MsgPack data
|
||||
Define *Writer, *Reader
|
||||
|
||||
; MsgPack-encoded buffer that will get exported midway in this example.
|
||||
Define *Buffer
|
||||
|
||||
; Structure that will hold the data to encode and the decoded one.
|
||||
Define InputVector.Vector2, OutputVector.Vector2
|
||||
Define ExtType.b
|
||||
|
||||
|
||||
|
||||
; Preparing the input structure.
|
||||
With InputVector
|
||||
\X = 1.5
|
||||
\Y = -2.25
|
||||
EndWith
|
||||
|
||||
|
||||
; Writing the vector into the buffer.
|
||||
; It is 8 bytes long, so it is written as a FixExt8.
|
||||
*Writer = MsgPackCreate(64)
|
||||
If Not *Writer
|
||||
Debug "Failed to create the writer !"
|
||||
End 1
|
||||
EndIf
|
||||
|
||||
If MsgPackWriteExtension(*Writer, #ExtType_Vector2, @InputVector, SizeOf(Vector2)) <> #MsgPack_Error_Success
|
||||
Debug "Failed to write the extension !"
|
||||
End 2
|
||||
EndIf
|
||||
|
||||
|
||||
; Exporting the data and giving it to a reader.
|
||||
*Buffer = MsgPackFinalize(*Writer)
|
||||
If Not *Buffer
|
||||
Debug "Failed to finalize the writer !"
|
||||
End 3
|
||||
EndIf
|
||||
|
||||
Debug "Encoded size: " + Str(MemorySize(*Buffer))
|
||||
|
||||
*Reader = MsgPackWrapMsgBuffer(*Buffer, MemorySize(*Buffer), #True)
|
||||
If Not *Reader
|
||||
Debug "Failed to wrap the buffer !"
|
||||
End 4
|
||||
EndIf
|
||||
|
||||
|
||||
; Checking the type first, it doesn't move the offset.
|
||||
ExtType = MsgPackGetExtensionType(*Reader)
|
||||
If MsgPackLastError(*Reader) <> #MsgPack_Error_Success
|
||||
Debug "Failed to get the extension type !"
|
||||
End 5
|
||||
EndIf
|
||||
|
||||
If ExtType <> #ExtType_Vector2
|
||||
Debug "Unexpected extension type: " + Str(ExtType)
|
||||
End 6
|
||||
EndIf
|
||||
|
||||
|
||||
; Copying its data into the output structure.
|
||||
If MsgPackReadExtension(*Reader, @OutputVector, SizeOf(Vector2)) <> SizeOf(Vector2)
|
||||
Debug "Failed to read the extension !"
|
||||
End 7
|
||||
EndIf
|
||||
|
||||
Debug "X: " + StrF(OutputVector\X)
|
||||
Debug "Y: " + StrF(OutputVector\Y)
|
||||
|
||||
|
||||
; Freeing the obscure structure and the buffer it owns.
|
||||
MsgPackFree(*Reader, #True)
|
||||
|
||||
End 0
|
||||
@@ -1,8 +1,8 @@
|
||||
; ------------------------------------------------------------------------------
|
||||
;- Notes
|
||||
|
||||
; This example show you how to write, export and read a MessagePack buffer
|
||||
; with implicitely typed variables whose format code is automatically
|
||||
; This example shows you how to write, export and read a MessagePack buffer
|
||||
; with implicitly typed variables whose format code is automatically
|
||||
; resolved based on the given data.
|
||||
|
||||
|
||||
@@ -92,8 +92,7 @@ EndIf
|
||||
Debug "Encoded size: " + Str(MemorySize(*Buffer))
|
||||
|
||||
|
||||
; It also gives ownership of the buffer to the obscure structure.
|
||||
; And it is marked internally as read-only.
|
||||
; Giving ownership of the buffer to the obscure structure.
|
||||
*Reader = MsgPackWrapMsgBuffer(*Buffer, MemorySize(*Buffer), #True)
|
||||
If Not *Reader
|
||||
Debug "Failed to wrap the buffer !"
|
||||
@@ -102,7 +101,6 @@ EndIf
|
||||
|
||||
|
||||
; Getting data out of the MsgPack buffer.
|
||||
OutputStruct.MyDataPacket
|
||||
If Not DecodeMyDataPacket(*Reader, @OutputStruct)
|
||||
Debug "Failed to decode the packet !"
|
||||
End 5
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
; ------------------------------------------------------------------------------
|
||||
;- Notes
|
||||
|
||||
; This example shows you how to write a structure as a map whose keys are
|
||||
; its fields' names, and how to read it back by iterating on it.
|
||||
; Unknown keys are skipped, and the keys' order doesn't matter.
|
||||
|
||||
|
||||
; ------------------------------------------------------------------------------
|
||||
;- Example
|
||||
|
||||
EnableExplicit
|
||||
|
||||
XIncludeFile "../../Includes/MessagePack.pbi"
|
||||
|
||||
Structure MyDataPacket
|
||||
Id.q
|
||||
UserName.s
|
||||
IsAdmin.b
|
||||
EndStructure
|
||||
|
||||
; Writes the keys and values one after the other, alternating them.
|
||||
Procedure.b EncodeMyDataPacketFields(*MsgPackData, *Input.MyDataPacket)
|
||||
If MsgPackWriteString(*MsgPackData, "Id") <> #MsgPack_Error_Success
|
||||
ProcedureReturn #False
|
||||
EndIf
|
||||
If MsgPackWriteInteger(*MsgPackData, *Input\Id) <> #MsgPack_Error_Success
|
||||
ProcedureReturn #False
|
||||
EndIf
|
||||
|
||||
If MsgPackWriteString(*MsgPackData, "UserName") <> #MsgPack_Error_Success
|
||||
ProcedureReturn #False
|
||||
EndIf
|
||||
If MsgPackWriteString(*MsgPackData, *Input\UserName) <> #MsgPack_Error_Success
|
||||
ProcedureReturn #False
|
||||
EndIf
|
||||
|
||||
If MsgPackWriteString(*MsgPackData, "IsAdmin") <> #MsgPack_Error_Success
|
||||
ProcedureReturn #False
|
||||
EndIf
|
||||
If MsgPackWriteBoolean(*MsgPackData, *Input\IsAdmin) <> #MsgPack_Error_Success
|
||||
ProcedureReturn #False
|
||||
EndIf
|
||||
|
||||
ProcedureReturn #True
|
||||
EndProcedure
|
||||
|
||||
Procedure.b EncodeMyDataPacket(*MsgPackData, *Input.MyDataPacket)
|
||||
Protected *Fields
|
||||
Protected Success.b = #False
|
||||
|
||||
; Writing the keys and values in their own writer.
|
||||
*Fields = MsgPackCreate(64)
|
||||
If Not *Fields
|
||||
ProcedureReturn #False
|
||||
EndIf
|
||||
|
||||
; Writing the map, the fields are counted and copied into it.
|
||||
If EncodeMyDataPacketFields(*Fields, *Input)
|
||||
Success = Bool(MsgPackWriteMapFrom(*MsgPackData, *Fields) = #MsgPack_Error_Success)
|
||||
EndIf
|
||||
|
||||
; The fields writer is left untouched, so it must be freed.
|
||||
MsgPackFree(*Fields, #True)
|
||||
|
||||
ProcedureReturn Success
|
||||
EndProcedure
|
||||
|
||||
Procedure.b DecodeMyDataPacket(*MsgPackData, *Output.MyDataPacket)
|
||||
Protected *Iterator
|
||||
Protected Key.s
|
||||
|
||||
*Iterator = MsgPackExamine(*MsgPackData)
|
||||
If Not *Iterator
|
||||
ProcedureReturn #False
|
||||
EndIf
|
||||
|
||||
; Each entry is read key first, then value.
|
||||
While MsgPackNextEntry(*Iterator)
|
||||
Key = MsgPackReadString(*Iterator)
|
||||
|
||||
Select Key
|
||||
Case "Id"
|
||||
*Output\Id = MsgPackReadInteger(*Iterator)
|
||||
Case "UserName"
|
||||
*Output\UserName = MsgPackReadString(*Iterator)
|
||||
Case "IsAdmin"
|
||||
*Output\IsAdmin = MsgPackReadBoolean(*Iterator)
|
||||
EndSelect
|
||||
|
||||
If MsgPackLastError(*Iterator) <> #MsgPack_Error_Success
|
||||
Break
|
||||
EndIf
|
||||
Wend
|
||||
|
||||
; `MsgPackNextEntry` also returns `#False` on errors.
|
||||
If MsgPackLastError(*Iterator) <> #MsgPack_Error_Success
|
||||
MsgPackFinish(*Iterator)
|
||||
ProcedureReturn #False
|
||||
EndIf
|
||||
|
||||
MsgPackFinish(*Iterator)
|
||||
ProcedureReturn #True
|
||||
EndProcedure
|
||||
|
||||
|
||||
|
||||
; Obscure structures that hold info about the MsgPack data
|
||||
Define *Writer, *Reader
|
||||
|
||||
; MsgPack-encoded buffer that will get exported midway in this example.
|
||||
Define *Buffer
|
||||
|
||||
; Structure that will hold the data to encode and the decoded one.
|
||||
Define InputStruct.MyDataPacket, OutputStruct.MyDataPacket
|
||||
|
||||
|
||||
|
||||
; Preparing the input structure.
|
||||
With InputStruct
|
||||
\Id = 67
|
||||
\UserName = "Herwin"
|
||||
\IsAdmin = #True
|
||||
EndWith
|
||||
|
||||
|
||||
; Writing the data into the buffer.
|
||||
*Writer = MsgPackCreate(64)
|
||||
If Not *Writer
|
||||
Debug "Failed to create the writer !"
|
||||
End 1
|
||||
EndIf
|
||||
|
||||
If Not EncodeMyDataPacket(*Writer, @InputStruct)
|
||||
Debug "Failed to encode the packet !"
|
||||
End 2
|
||||
EndIf
|
||||
|
||||
|
||||
; Exporting the data and giving it to a reader.
|
||||
*Buffer = MsgPackFinalize(*Writer)
|
||||
If Not *Buffer
|
||||
Debug "Failed to finalize the writer !"
|
||||
End 3
|
||||
EndIf
|
||||
|
||||
Debug "Encoded size: " + Str(MemorySize(*Buffer))
|
||||
|
||||
*Reader = MsgPackWrapMsgBuffer(*Buffer, MemorySize(*Buffer), #True)
|
||||
If Not *Reader
|
||||
Debug "Failed to wrap the buffer !"
|
||||
End 4
|
||||
EndIf
|
||||
|
||||
|
||||
; Getting data out of the MsgPack buffer.
|
||||
If Not DecodeMyDataPacket(*Reader, @OutputStruct)
|
||||
Debug "Failed to decode the packet !"
|
||||
End 5
|
||||
EndIf
|
||||
|
||||
Debug "Id: " + Str(OutputStruct\Id)
|
||||
Debug "UserName: " + OutputStruct\UserName
|
||||
Debug "IsAdmin: " + Str(OutputStruct\IsAdmin)
|
||||
|
||||
|
||||
; Freeing the obscure structure and the buffer it owns.
|
||||
MsgPackFree(*Reader, #True)
|
||||
|
||||
End 0
|
||||
@@ -0,0 +1,92 @@
|
||||
; ------------------------------------------------------------------------------
|
||||
;- Notes
|
||||
|
||||
; This example shows you how to write and read strings in the supported
|
||||
; encodings, UTF-8 being the default one and the one used by the spec.
|
||||
; The string must be read with the encoding it was written with.
|
||||
|
||||
|
||||
; ------------------------------------------------------------------------------
|
||||
;- Example
|
||||
|
||||
EnableExplicit
|
||||
|
||||
XIncludeFile "../../Includes/MessagePack.pbi"
|
||||
|
||||
; Obscure structures that hold info about the MsgPack data
|
||||
Define *Writer, *Reader
|
||||
|
||||
; MsgPack-encoded buffer that will get exported midway in this example.
|
||||
Define *Buffer
|
||||
|
||||
; "Cafe" with an accented "e", 4 characters but 5 bytes in UTF-8.
|
||||
Define Text.s = "Caf" + Chr($E9)
|
||||
|
||||
|
||||
|
||||
; Writing the strings into the buffer.
|
||||
*Writer = MsgPackCreate(64)
|
||||
If Not *Writer
|
||||
Debug "Failed to create the writer !"
|
||||
End 1
|
||||
EndIf
|
||||
|
||||
; UTF-8 by default.
|
||||
If MsgPackWriteString(*Writer, Text) <> #MsgPack_Error_Success
|
||||
Debug "Failed to write the UTF-8 string !"
|
||||
End 2
|
||||
EndIf
|
||||
|
||||
; UTF-16 LE, 2 bytes per character here.
|
||||
If MsgPackWriteString(*Writer, Text, #PB_Unicode) <> #MsgPack_Error_Success
|
||||
Debug "Failed to write the UTF-16 string !"
|
||||
End 3
|
||||
EndIf
|
||||
|
||||
; Empty strings are valid too.
|
||||
If MsgPackWriteString(*Writer, "") <> #MsgPack_Error_Success
|
||||
Debug "Failed to write the empty string !"
|
||||
End 4
|
||||
EndIf
|
||||
|
||||
|
||||
; Exporting the data and giving it to a reader.
|
||||
*Buffer = MsgPackFinalize(*Writer)
|
||||
If Not *Buffer
|
||||
Debug "Failed to finalize the writer !"
|
||||
End 5
|
||||
EndIf
|
||||
|
||||
Debug "Encoded size: " + Str(MemorySize(*Buffer))
|
||||
|
||||
*Reader = MsgPackWrapMsgBuffer(*Buffer, MemorySize(*Buffer), #True)
|
||||
If Not *Reader
|
||||
Debug "Failed to wrap the buffer !"
|
||||
End 6
|
||||
EndIf
|
||||
|
||||
|
||||
; Reading the strings back, in the same order and encodings.
|
||||
Debug "UTF-8: " + MsgPackReadString(*Reader)
|
||||
If MsgPackLastError(*Reader) <> #MsgPack_Error_Success
|
||||
Debug "Failed to read the UTF-8 string !"
|
||||
End 7
|
||||
EndIf
|
||||
|
||||
Debug "UTF-16: " + MsgPackReadString(*Reader, #PB_Unicode)
|
||||
If MsgPackLastError(*Reader) <> #MsgPack_Error_Success
|
||||
Debug "Failed to read the UTF-16 string !"
|
||||
End 8
|
||||
EndIf
|
||||
|
||||
Debug "Empty: '" + MsgPackReadString(*Reader) + "'"
|
||||
If MsgPackLastError(*Reader) <> #MsgPack_Error_Success
|
||||
Debug "Failed to read the empty string !"
|
||||
End 9
|
||||
EndIf
|
||||
|
||||
|
||||
; Freeing the obscure structure and the buffer it owns.
|
||||
MsgPackFree(*Reader, #True)
|
||||
|
||||
End 0
|
||||
@@ -0,0 +1,322 @@
|
||||
; ------------------------------------------------------------------------------
|
||||
;- Notes
|
||||
|
||||
; This example shows you how to iterate over a MessagePack buffer and dump
|
||||
; every field to the console with its exact format code.
|
||||
; Each field is read with the procedure matching its format code.
|
||||
; Arrays and maps are dumped recursively, map entries key first.
|
||||
|
||||
|
||||
; ------------------------------------------------------------------------------
|
||||
;- Example
|
||||
|
||||
EnableExplicit
|
||||
|
||||
XIncludeFile "../../Includes/MessagePack.pbi"
|
||||
|
||||
|
||||
; Ends the program if a write failed.
|
||||
Procedure CheckWrite(ErrorCode.MsgPack_ErrorCode, What.s)
|
||||
If ErrorCode <> #MsgPack_Error_Success
|
||||
PrintN("Failed to write " + What + " ! (Error " + Str(ErrorCode) + ")")
|
||||
End 1
|
||||
EndIf
|
||||
EndProcedure
|
||||
|
||||
; Ends the program if a structure or buffer couldn't be prepared.
|
||||
Procedure CheckPointer(*Pointer, What.s)
|
||||
If Not *Pointer
|
||||
PrintN("Failed to prepare " + What + " !")
|
||||
End 1
|
||||
EndIf
|
||||
EndProcedure
|
||||
|
||||
|
||||
; Returns the bytes of a buffer from `MsgPackCopyBinary/Extension` as hex.
|
||||
; Empty bin and ext fields give a `#Null` buffer.
|
||||
Procedure.s HexDump(*Data)
|
||||
Protected Text.s = ""
|
||||
Protected i.i
|
||||
|
||||
If Not *Data
|
||||
ProcedureReturn ""
|
||||
EndIf
|
||||
|
||||
For i = 0 To MemorySize(*Data) - 1
|
||||
Text = Text + RSet(Hex(PeekA(*Data + i)), 2, "0") + " "
|
||||
Next
|
||||
|
||||
ProcedureReturn RTrim(Text)
|
||||
EndProcedure
|
||||
|
||||
|
||||
Procedure.s GetFormatCodeName(FormatCode.MsgPack_FormatCode)
|
||||
Select FormatCode
|
||||
Case #MsgPack_FormatCode_Null : ProcedureReturn "Null"
|
||||
Case #MsgPack_FormatCode_False : ProcedureReturn "False"
|
||||
Case #MsgPack_FormatCode_True : ProcedureReturn "True"
|
||||
Case #MsgPack_FormatCode_FixIntPositive : ProcedureReturn "FixIntPositive"
|
||||
Case #MsgPack_FormatCode_FixIntNegative : ProcedureReturn "FixIntNegative"
|
||||
Case #MsgPack_FormatCode_UInt8 : ProcedureReturn "UInt8"
|
||||
Case #MsgPack_FormatCode_UInt16 : ProcedureReturn "UInt16"
|
||||
Case #MsgPack_FormatCode_UInt32 : ProcedureReturn "UInt32"
|
||||
Case #MsgPack_FormatCode_UInt64 : ProcedureReturn "UInt64"
|
||||
Case #MsgPack_FormatCode_Int8 : ProcedureReturn "Int8"
|
||||
Case #MsgPack_FormatCode_Int16 : ProcedureReturn "Int16"
|
||||
Case #MsgPack_FormatCode_Int32 : ProcedureReturn "Int32"
|
||||
Case #MsgPack_FormatCode_Int64 : ProcedureReturn "Int64"
|
||||
Case #MsgPack_FormatCode_Float32 : ProcedureReturn "Float32"
|
||||
Case #MsgPack_FormatCode_Float64 : ProcedureReturn "Float64"
|
||||
Case #MsgPack_FormatCode_FixStr : ProcedureReturn "FixStr"
|
||||
Case #MsgPack_FormatCode_Str8 : ProcedureReturn "Str8"
|
||||
Case #MsgPack_FormatCode_Str16 : ProcedureReturn "Str16"
|
||||
Case #MsgPack_FormatCode_Str32 : ProcedureReturn "Str32"
|
||||
Case #MsgPack_FormatCode_Bin8 : ProcedureReturn "Bin8"
|
||||
Case #MsgPack_FormatCode_Bin16 : ProcedureReturn "Bin16"
|
||||
Case #MsgPack_FormatCode_Bin32 : ProcedureReturn "Bin32"
|
||||
Case #MsgPack_FormatCode_FixArray : ProcedureReturn "FixArray"
|
||||
Case #MsgPack_FormatCode_Array16 : ProcedureReturn "Array16"
|
||||
Case #MsgPack_FormatCode_Array32 : ProcedureReturn "Array32"
|
||||
Case #MsgPack_FormatCode_FixMap : ProcedureReturn "FixMap"
|
||||
Case #MsgPack_FormatCode_Map16 : ProcedureReturn "Map16"
|
||||
Case #MsgPack_FormatCode_Map32 : ProcedureReturn "Map32"
|
||||
Case #MsgPack_FormatCode_FixExt1 : ProcedureReturn "FixExt1"
|
||||
Case #MsgPack_FormatCode_FixExt2 : ProcedureReturn "FixExt2"
|
||||
Case #MsgPack_FormatCode_FixExt4 : ProcedureReturn "FixExt4"
|
||||
Case #MsgPack_FormatCode_FixExt8 : ProcedureReturn "FixExt8"
|
||||
Case #MsgPack_FormatCode_FixExt16 : ProcedureReturn "FixExt16"
|
||||
Case #MsgPack_FormatCode_Ext8 : ProcedureReturn "Ext8"
|
||||
Case #MsgPack_FormatCode_Ext16 : ProcedureReturn "Ext16"
|
||||
Case #MsgPack_FormatCode_Ext32 : ProcedureReturn "Ext32"
|
||||
EndSelect
|
||||
|
||||
ProcedureReturn "Invalid"
|
||||
EndProcedure
|
||||
|
||||
|
||||
; Prints the field at the current offset and moves past it.
|
||||
; `Label` is printed before it, to tell array indexes and map keys/values apart.
|
||||
Procedure.b DumpField(*MsgPackData, Indent.s, Label.s)
|
||||
Protected FormatCode.MsgPack_FormatCode = MsgPackGetFormatCode(*MsgPackData)
|
||||
Protected Text.s = GetFormatCodeName(FormatCode)
|
||||
Protected *Iterator, *Data
|
||||
Protected IsMap.b, Success.b
|
||||
Protected EntryCount.q, Index.q
|
||||
|
||||
Select FormatCode
|
||||
Case #MsgPack_FormatCode_Null
|
||||
MsgPackReadNull(*MsgPackData)
|
||||
|
||||
Case #MsgPack_FormatCode_False, #MsgPack_FormatCode_True
|
||||
; The value is already given by the format code.
|
||||
MsgPackReadBoolean(*MsgPackData)
|
||||
|
||||
Case #MsgPack_FormatCode_FixIntPositive, #MsgPack_FormatCode_FixIntNegative
|
||||
; Fixints don't have their own getter, the generic one reads them.
|
||||
Text = Text + ": " + Str(MsgPackReadInteger(*MsgPackData))
|
||||
|
||||
Case #MsgPack_FormatCode_UInt8
|
||||
Text = Text + ": " + Str(MsgPackReadUInt8(*MsgPackData))
|
||||
Case #MsgPack_FormatCode_UInt16
|
||||
Text = Text + ": " + Str(MsgPackReadUInt16(*MsgPackData))
|
||||
Case #MsgPack_FormatCode_UInt32
|
||||
; Read as a `.l`, values above the signed max come back negative.
|
||||
Text = Text + ": " + Str(MsgPackReadUInt32(*MsgPackData) & $FFFFFFFF)
|
||||
Case #MsgPack_FormatCode_UInt64
|
||||
; Read as a `.q`, values above the signed max come back negative.
|
||||
Text = Text + ": " + StrU(MsgPackReadUInt64(*MsgPackData), #PB_Quad)
|
||||
|
||||
Case #MsgPack_FormatCode_Int8
|
||||
Text = Text + ": " + Str(MsgPackReadInt8(*MsgPackData))
|
||||
Case #MsgPack_FormatCode_Int16
|
||||
Text = Text + ": " + Str(MsgPackReadInt16(*MsgPackData))
|
||||
Case #MsgPack_FormatCode_Int32
|
||||
Text = Text + ": " + Str(MsgPackReadInt32(*MsgPackData))
|
||||
Case #MsgPack_FormatCode_Int64
|
||||
Text = Text + ": " + Str(MsgPackReadInt64(*MsgPackData))
|
||||
|
||||
Case #MsgPack_FormatCode_Float32
|
||||
Text = Text + ": " + StrF(MsgPackReadFloat32(*MsgPackData))
|
||||
Case #MsgPack_FormatCode_Float64
|
||||
Text = Text + ": " + StrD(MsgPackReadFloat64(*MsgPackData))
|
||||
|
||||
Case #MsgPack_FormatCode_FixStr, #MsgPack_FormatCode_Str8,
|
||||
#MsgPack_FormatCode_Str16, #MsgPack_FormatCode_Str32
|
||||
Text = Text + ": " + Chr(34) + MsgPackReadString(*MsgPackData) + Chr(34)
|
||||
|
||||
Case #MsgPack_FormatCode_Bin8, #MsgPack_FormatCode_Bin16, #MsgPack_FormatCode_Bin32
|
||||
*Data = MsgPackCopyBinary(*MsgPackData)
|
||||
Text = Text + ": " + HexDump(*Data)
|
||||
|
||||
Case #MsgPack_FormatCode_FixExt1, #MsgPack_FormatCode_FixExt2, #MsgPack_FormatCode_FixExt4,
|
||||
#MsgPack_FormatCode_FixExt8, #MsgPack_FormatCode_FixExt16,
|
||||
#MsgPack_FormatCode_Ext8, #MsgPack_FormatCode_Ext16, #MsgPack_FormatCode_Ext32
|
||||
; The type doesn't move the offset, the data is read right after.
|
||||
Text = Text + " (Type " + Str(MsgPackGetExtensionType(*MsgPackData)) + ")"
|
||||
If MsgPackLastError(*MsgPackData) <> #MsgPack_Error_Success
|
||||
ProcedureReturn #False
|
||||
EndIf
|
||||
*Data = MsgPackCopyExtension(*MsgPackData)
|
||||
Text = Text + ": " + HexDump(*Data)
|
||||
|
||||
Case #MsgPack_FormatCode_FixArray, #MsgPack_FormatCode_Array16, #MsgPack_FormatCode_Array32,
|
||||
#MsgPack_FormatCode_FixMap, #MsgPack_FormatCode_Map16, #MsgPack_FormatCode_Map32
|
||||
IsMap = Bool(MsgPackFormatCodeToDataType(FormatCode) = #MsgPack_DataTypes_Map)
|
||||
|
||||
EntryCount = MsgPackGetEntryCount(*MsgPackData)
|
||||
*Iterator = MsgPackExamine(*MsgPackData)
|
||||
If Not *Iterator
|
||||
ProcedureReturn #False
|
||||
EndIf
|
||||
|
||||
PrintN(Indent + Label + Text + " (" + Str(EntryCount) + " entries)")
|
||||
|
||||
Success = #True
|
||||
Index = 0
|
||||
While MsgPackNextEntry(*Iterator)
|
||||
If IsMap
|
||||
Success = DumpField(*Iterator, Indent + " ", "Key: ")
|
||||
If Success
|
||||
Success = DumpField(*Iterator, Indent + " ", "Value: ")
|
||||
EndIf
|
||||
Else
|
||||
Success = DumpField(*Iterator, Indent + " ", "[" + Str(Index) + "] ")
|
||||
EndIf
|
||||
|
||||
If Not Success
|
||||
Break
|
||||
EndIf
|
||||
Index = Index + 1
|
||||
Wend
|
||||
|
||||
; `MsgPackNextEntry` also returns `#False` on errors.
|
||||
If MsgPackLastError(*Iterator) <> #MsgPack_Error_Success
|
||||
Success = #False
|
||||
EndIf
|
||||
|
||||
MsgPackFinish(*Iterator)
|
||||
ProcedureReturn Success
|
||||
|
||||
Default
|
||||
PrintN(Indent + Label + "Invalid format code !")
|
||||
ProcedureReturn #False
|
||||
EndSelect
|
||||
|
||||
If *Data
|
||||
FreeMemory(*Data)
|
||||
EndIf
|
||||
|
||||
If MsgPackLastError(*MsgPackData) <> #MsgPack_Error_Success
|
||||
ProcedureReturn #False
|
||||
EndIf
|
||||
|
||||
PrintN(Indent + Label + Text)
|
||||
ProcedureReturn #True
|
||||
EndProcedure
|
||||
|
||||
|
||||
|
||||
; Obscure structures that hold info about the MsgPack data
|
||||
Define *Writer, *Entries, *Fields, *Reader
|
||||
|
||||
; MsgPack-encoded buffers that will get exported in this example.
|
||||
Define *Buffer, *EntriesBuffer, *FieldsBuffer
|
||||
|
||||
; Raw data for the bin and ext fields.
|
||||
Define Bytes.l = $44332211
|
||||
|
||||
Define FieldSize.q, BytesLeft.q
|
||||
|
||||
OpenConsole()
|
||||
|
||||
|
||||
; Writing a bit of everything, forcing each format code.
|
||||
*Writer = MsgPackCreate(64)
|
||||
CheckPointer(*Writer, "the writer")
|
||||
|
||||
CheckWrite(MsgPackWriteNull(*Writer), "a nil")
|
||||
CheckWrite(MsgPackWriteTrue(*Writer), "a true")
|
||||
CheckWrite(MsgPackWriteFalse(*Writer), "a false")
|
||||
CheckWrite(MsgPackWriteFixInt(*Writer, 42), "a positive fixint")
|
||||
CheckWrite(MsgPackWriteFixInt(*Writer, -5), "a negative fixint")
|
||||
CheckWrite(MsgPackWriteUInt8(*Writer, 200), "a UInt8")
|
||||
CheckWrite(MsgPackWriteUInt32(*Writer, 70000), "a UInt32")
|
||||
CheckWrite(MsgPackWriteInt16(*Writer, -1000), "an Int16")
|
||||
CheckWrite(MsgPackWriteInt64(*Writer, 1), "an Int64")
|
||||
CheckWrite(MsgPackWriteFloat32(*Writer, 1.5), "a Float32")
|
||||
CheckWrite(MsgPackWriteFloat64(*Writer, 3.14159), "a Float64")
|
||||
CheckWrite(MsgPackWriteFixStr(*Writer, "Hello"), "a FixStr")
|
||||
CheckWrite(MsgPackWriteStr16(*Writer, "World"), "a Str16")
|
||||
CheckWrite(MsgPackWriteBin8(*Writer, @Bytes, 4), "a Bin8")
|
||||
CheckWrite(MsgPackWriteFixExt4(*Writer, 1, @Bytes), "a FixExt4")
|
||||
CheckWrite(MsgPackWriteExt8(*Writer, 2, @Bytes, 3), "an Ext8")
|
||||
|
||||
|
||||
; Forcing the format code of arrays and maps requires their content in a buffer.
|
||||
; Writing the entries of an array in their own writer, then exporting them.
|
||||
*Entries = MsgPackCreate(16)
|
||||
CheckPointer(*Entries, "the entries writer")
|
||||
|
||||
CheckWrite(MsgPackWriteUInt8(*Entries, 10), "an entry")
|
||||
CheckWrite(MsgPackWriteUInt8(*Entries, 20), "an entry")
|
||||
CheckWrite(MsgPackWriteUInt8(*Entries, 30), "an entry")
|
||||
|
||||
*EntriesBuffer = MsgPackFinalize(*Entries)
|
||||
CheckPointer(*EntriesBuffer, "the entries buffer")
|
||||
|
||||
|
||||
; Writing the keys and values of a map, with the array as a value.
|
||||
*Fields = MsgPackCreate(32)
|
||||
CheckPointer(*Fields, "the fields writer")
|
||||
|
||||
CheckWrite(MsgPackWriteFixStr(*Fields, "Name"), "a key")
|
||||
CheckWrite(MsgPackWriteStr8(*Fields, "Herwin"), "a value")
|
||||
CheckWrite(MsgPackWriteFixStr(*Fields, "Scores"), "a key")
|
||||
CheckWrite(MsgPackWriteArray16(*Fields, *EntriesBuffer, MemorySize(*EntriesBuffer)), "an Array16")
|
||||
FreeMemory(*EntriesBuffer)
|
||||
|
||||
*FieldsBuffer = MsgPackFinalize(*Fields)
|
||||
CheckPointer(*FieldsBuffer, "the fields buffer")
|
||||
|
||||
CheckWrite(MsgPackWriteMap16(*Writer, *FieldsBuffer, MemorySize(*FieldsBuffer)), "a Map16")
|
||||
FreeMemory(*FieldsBuffer)
|
||||
|
||||
|
||||
; Exporting the data and giving it to a reader.
|
||||
*Buffer = MsgPackFinalize(*Writer)
|
||||
CheckPointer(*Buffer, "the final buffer")
|
||||
|
||||
PrintN("Encoded size: " + Str(MemorySize(*Buffer)))
|
||||
PrintN("")
|
||||
|
||||
*Reader = MsgPackWrapMsgBuffer(*Buffer, MemorySize(*Buffer), #True)
|
||||
CheckPointer(*Reader, "the reader")
|
||||
|
||||
|
||||
; Dumping every top-level field until the whole buffer is read.
|
||||
; Each field's full size is checked first, which also validates it.
|
||||
BytesLeft = MemorySize(*Buffer)
|
||||
While BytesLeft > 0
|
||||
FieldSize = MsgPackGetFieldFullSize(*Reader)
|
||||
If FieldSize = #MsgPack_InvalidSize
|
||||
PrintN("Invalid field ! (Error " + Str(MsgPackLastError(*Reader)) + ")")
|
||||
End 2
|
||||
EndIf
|
||||
|
||||
If Not DumpField(*Reader, "", "")
|
||||
PrintN("Failed to dump a field !")
|
||||
End 3
|
||||
EndIf
|
||||
|
||||
BytesLeft = BytesLeft - FieldSize
|
||||
Wend
|
||||
|
||||
|
||||
; Freeing the obscure structure and the buffer it owns.
|
||||
MsgPackFree(*Reader, #True)
|
||||
|
||||
PrintN("")
|
||||
PrintN("Press enter to exit.")
|
||||
Input()
|
||||
|
||||
End 0
|
||||
@@ -0,0 +1,253 @@
|
||||
; ------------------------------------------------------------------------------
|
||||
;- Notes
|
||||
|
||||
; This example shows you how to iterate over a MessagePack buffer and dump
|
||||
; every field to the console with its general data type.
|
||||
; Each field is read with the implicitly typed procedure of its data type,
|
||||
; which accepts all of the type's format codes.
|
||||
; Arrays and maps are dumped recursively, map entries key first.
|
||||
|
||||
|
||||
; ------------------------------------------------------------------------------
|
||||
;- Example
|
||||
|
||||
EnableExplicit
|
||||
|
||||
XIncludeFile "../../Includes/MessagePack.pbi"
|
||||
|
||||
|
||||
; Ends the program if a write failed.
|
||||
Procedure CheckWrite(ErrorCode.MsgPack_ErrorCode, What.s)
|
||||
If ErrorCode <> #MsgPack_Error_Success
|
||||
PrintN("Failed to write " + What + " ! (Error " + Str(ErrorCode) + ")")
|
||||
End 1
|
||||
EndIf
|
||||
EndProcedure
|
||||
|
||||
; Ends the program if a structure or buffer couldn't be prepared.
|
||||
Procedure CheckPointer(*Pointer, What.s)
|
||||
If Not *Pointer
|
||||
PrintN("Failed to prepare " + What + " !")
|
||||
End 1
|
||||
EndIf
|
||||
EndProcedure
|
||||
|
||||
|
||||
; Returns the bytes of a buffer from `MsgPackCopyBinary/Extension` as hex.
|
||||
; Empty bin and ext fields give a `#Null` buffer.
|
||||
Procedure.s HexDump(*Data)
|
||||
Protected Text.s = ""
|
||||
Protected i.i
|
||||
|
||||
If Not *Data
|
||||
ProcedureReturn ""
|
||||
EndIf
|
||||
|
||||
For i = 0 To MemorySize(*Data) - 1
|
||||
Text = Text + RSet(Hex(PeekA(*Data + i)), 2, "0") + " "
|
||||
Next
|
||||
|
||||
ProcedureReturn RTrim(Text)
|
||||
EndProcedure
|
||||
|
||||
|
||||
; Prints the field at the current offset and moves past it.
|
||||
; `Label` is printed before it, to tell array indexes and map keys/values apart.
|
||||
Procedure.b DumpField(*MsgPackData, Indent.s, Label.s)
|
||||
Protected DataType.MsgPack_DataType = MsgPackFormatCodeToDataType(MsgPackGetFormatCode(*MsgPackData))
|
||||
Protected Text.s
|
||||
Protected *Iterator, *Data
|
||||
Protected Success.b
|
||||
Protected EntryCount.q, Index.q
|
||||
|
||||
Select DataType
|
||||
Case #MsgPack_DataTypes_Null
|
||||
MsgPackReadNull(*MsgPackData)
|
||||
Text = "Null"
|
||||
|
||||
Case #MsgPack_DataTypes_Boolean
|
||||
If MsgPackReadBoolean(*MsgPackData)
|
||||
Text = "Boolean: True"
|
||||
Else
|
||||
Text = "Boolean: False"
|
||||
EndIf
|
||||
|
||||
Case #MsgPack_DataTypes_Integer
|
||||
; UInt64 values above `(2^63)-1` fail here, see `MsgPackReadUInt64`.
|
||||
Text = "Integer: " + Str(MsgPackReadInteger(*MsgPackData))
|
||||
|
||||
Case #MsgPack_DataTypes_Float
|
||||
; Float32 values are returned as a `.d`.
|
||||
Text = "Float: " + StrD(MsgPackReadFloat(*MsgPackData))
|
||||
|
||||
Case #MsgPack_DataTypes_String
|
||||
Text = "String: " + Chr(34) + MsgPackReadString(*MsgPackData) + Chr(34)
|
||||
|
||||
Case #MsgPack_DataTypes_Binary
|
||||
*Data = MsgPackCopyBinary(*MsgPackData)
|
||||
Text = "Binary: " + HexDump(*Data)
|
||||
|
||||
Case #MsgPack_DataTypes_Extension
|
||||
; The type doesn't move the offset, the data is read right after.
|
||||
Text = "Extension (Type " + Str(MsgPackGetExtensionType(*MsgPackData)) + ")"
|
||||
If MsgPackLastError(*MsgPackData) <> #MsgPack_Error_Success
|
||||
ProcedureReturn #False
|
||||
EndIf
|
||||
*Data = MsgPackCopyExtension(*MsgPackData)
|
||||
Text = Text + ": " + HexDump(*Data)
|
||||
|
||||
Case #MsgPack_DataTypes_Array, #MsgPack_DataTypes_Map
|
||||
EntryCount = MsgPackGetEntryCount(*MsgPackData)
|
||||
*Iterator = MsgPackExamine(*MsgPackData)
|
||||
If Not *Iterator
|
||||
ProcedureReturn #False
|
||||
EndIf
|
||||
|
||||
If DataType = #MsgPack_DataTypes_Map
|
||||
PrintN(Indent + Label + "Map (" + Str(EntryCount) + " entries)")
|
||||
Else
|
||||
PrintN(Indent + Label + "Array (" + Str(EntryCount) + " entries)")
|
||||
EndIf
|
||||
|
||||
Success = #True
|
||||
Index = 0
|
||||
While MsgPackNextEntry(*Iterator)
|
||||
If DataType = #MsgPack_DataTypes_Map
|
||||
Success = DumpField(*Iterator, Indent + " ", "Key: ")
|
||||
If Success
|
||||
Success = DumpField(*Iterator, Indent + " ", "Value: ")
|
||||
EndIf
|
||||
Else
|
||||
Success = DumpField(*Iterator, Indent + " ", "[" + Str(Index) + "] ")
|
||||
EndIf
|
||||
|
||||
If Not Success
|
||||
Break
|
||||
EndIf
|
||||
Index = Index + 1
|
||||
Wend
|
||||
|
||||
; `MsgPackNextEntry` also returns `#False` on errors.
|
||||
If MsgPackLastError(*Iterator) <> #MsgPack_Error_Success
|
||||
Success = #False
|
||||
EndIf
|
||||
|
||||
MsgPackFinish(*Iterator)
|
||||
ProcedureReturn Success
|
||||
|
||||
Default
|
||||
PrintN(Indent + Label + "Unknown data type !")
|
||||
ProcedureReturn #False
|
||||
EndSelect
|
||||
|
||||
If *Data
|
||||
FreeMemory(*Data)
|
||||
EndIf
|
||||
|
||||
If MsgPackLastError(*MsgPackData) <> #MsgPack_Error_Success
|
||||
ProcedureReturn #False
|
||||
EndIf
|
||||
|
||||
PrintN(Indent + Label + Text)
|
||||
ProcedureReturn #True
|
||||
EndProcedure
|
||||
|
||||
|
||||
|
||||
; Obscure structures that hold info about the MsgPack data
|
||||
Define *Writer, *Entries, *Fields, *Reader
|
||||
|
||||
; MsgPack-encoded buffer that will get exported midway in this example.
|
||||
Define *Buffer
|
||||
|
||||
; Raw data for the bin and ext fields.
|
||||
Define Bytes.l = $44332211
|
||||
|
||||
Define FieldSize.q, BytesLeft.q
|
||||
|
||||
OpenConsole()
|
||||
|
||||
|
||||
; Writing a bit of everything, the smallest format code that fits is used.
|
||||
*Writer = MsgPackCreate(64)
|
||||
CheckPointer(*Writer, "the writer")
|
||||
|
||||
CheckWrite(MsgPackWriteNull(*Writer), "a nil")
|
||||
CheckWrite(MsgPackWriteBoolean(*Writer, #True), "a boolean")
|
||||
CheckWrite(MsgPackWriteBoolean(*Writer, #False), "a boolean")
|
||||
CheckWrite(MsgPackWriteInteger(*Writer, 42), "an integer")
|
||||
CheckWrite(MsgPackWriteInteger(*Writer, -5), "an integer")
|
||||
CheckWrite(MsgPackWriteInteger(*Writer, 200), "an integer")
|
||||
CheckWrite(MsgPackWriteInteger(*Writer, 70000), "an integer")
|
||||
CheckWrite(MsgPackWriteInteger(*Writer, -1000), "an integer")
|
||||
; Floats don't have an implicitly typed setter.
|
||||
CheckWrite(MsgPackWriteFloat64(*Writer, 3.14159), "a float")
|
||||
CheckWrite(MsgPackWriteString(*Writer, "Hello"), "a string")
|
||||
CheckWrite(MsgPackWriteBinary(*Writer, @Bytes, 4), "a binary")
|
||||
CheckWrite(MsgPackWriteExtension(*Writer, 1, @Bytes, 4), "an extension")
|
||||
CheckWrite(MsgPackWriteExtension(*Writer, 2, @Bytes, 3), "an extension")
|
||||
|
||||
|
||||
; Writing the entries of an array in their own writer.
|
||||
*Entries = MsgPackCreate(16)
|
||||
CheckPointer(*Entries, "the entries writer")
|
||||
|
||||
CheckWrite(MsgPackWriteInteger(*Entries, 10), "an entry")
|
||||
CheckWrite(MsgPackWriteInteger(*Entries, 20), "an entry")
|
||||
CheckWrite(MsgPackWriteInteger(*Entries, 30), "an entry")
|
||||
|
||||
|
||||
; Writing the keys and values of a map, with the array as a value.
|
||||
*Fields = MsgPackCreate(32)
|
||||
CheckPointer(*Fields, "the fields writer")
|
||||
|
||||
CheckWrite(MsgPackWriteString(*Fields, "Name"), "a key")
|
||||
CheckWrite(MsgPackWriteString(*Fields, "Herwin"), "a value")
|
||||
CheckWrite(MsgPackWriteString(*Fields, "Scores"), "a key")
|
||||
CheckWrite(MsgPackWriteArrayFrom(*Fields, *Entries), "an array")
|
||||
|
||||
CheckWrite(MsgPackWriteMapFrom(*Writer, *Fields), "a map")
|
||||
|
||||
; The entries and fields writers are left untouched, so they must be freed.
|
||||
MsgPackFree(*Entries, #True)
|
||||
MsgPackFree(*Fields, #True)
|
||||
|
||||
|
||||
; Exporting the data and giving it to a reader.
|
||||
*Buffer = MsgPackFinalize(*Writer)
|
||||
CheckPointer(*Buffer, "the final buffer")
|
||||
|
||||
PrintN("Encoded size: " + Str(MemorySize(*Buffer)))
|
||||
PrintN("")
|
||||
|
||||
*Reader = MsgPackWrapMsgBuffer(*Buffer, MemorySize(*Buffer), #True)
|
||||
CheckPointer(*Reader, "the reader")
|
||||
|
||||
|
||||
; Dumping every top-level field until the whole buffer is read.
|
||||
; Each field's full size is checked first, which also validates it.
|
||||
BytesLeft = MemorySize(*Buffer)
|
||||
While BytesLeft > 0
|
||||
FieldSize = MsgPackGetFieldFullSize(*Reader)
|
||||
If FieldSize = #MsgPack_InvalidSize
|
||||
PrintN("Invalid field ! (Error " + Str(MsgPackLastError(*Reader)) + ")")
|
||||
End 2
|
||||
EndIf
|
||||
|
||||
If Not DumpField(*Reader, "", "")
|
||||
PrintN("Failed to dump a field !")
|
||||
End 3
|
||||
EndIf
|
||||
|
||||
BytesLeft = BytesLeft - FieldSize
|
||||
Wend
|
||||
|
||||
|
||||
; Freeing the obscure structure and the buffer it owns.
|
||||
MsgPackFree(*Reader, #True)
|
||||
|
||||
PrintN("")
|
||||
PrintN("Press enter to exit.")
|
||||
Input()
|
||||
|
||||
End 0
|
||||
@@ -0,0 +1,172 @@
|
||||
EnableExplicit
|
||||
|
||||
XIncludeFile "../../Includes/MessagePack.pbi"
|
||||
XIncludeFile "../_PerfTesting.pbi"
|
||||
|
||||
|
||||
; Builds and reads arrays nested `NestedDepth` levels deep, each one holding 3 integers before the next one.
|
||||
; Each run handles `NestedRepeats` of them, and `NsPerOp` is the time per nested structure.
|
||||
; The depth goes up to the default `#MsgPack_MaxDepth`.
|
||||
|
||||
#NestedMaxDepth = 128
|
||||
|
||||
; Number of levels handled by each run, split between the repeats.
|
||||
#NestedLevelsPerRun = 100000
|
||||
|
||||
Global NestedDepth.i
|
||||
Global NestedRepeats.i
|
||||
Global *NestedBuffer
|
||||
|
||||
|
||||
Procedure WriteLevelIntegers(*Writer)
|
||||
PerfCheckWrite(MsgPackWriteInteger(*Writer, 1))
|
||||
PerfCheckWrite(MsgPackWriteInteger(*Writer, 2))
|
||||
PerfCheckWrite(MsgPackWriteInteger(*Writer, 3))
|
||||
EndProcedure
|
||||
|
||||
; Returns the finalized buffer, which must be freed with `FreeMemory()`.
|
||||
; Each level's content is written in its own writer, then copied into its parent.
|
||||
Procedure.i BuildNestedBuffer()
|
||||
Protected *Content, *Parent, *Buffer
|
||||
Protected Level.i
|
||||
|
||||
*Content = MsgPackCreate(64)
|
||||
If Not *Content
|
||||
PerfFail("Failed to create a writer !")
|
||||
EndIf
|
||||
WriteLevelIntegers(*Content)
|
||||
|
||||
For Level = 2 To NestedDepth
|
||||
*Parent = MsgPackCreate(64)
|
||||
If Not *Parent
|
||||
PerfFail("Failed to create a writer !")
|
||||
EndIf
|
||||
WriteLevelIntegers(*Parent)
|
||||
PerfCheckWrite(MsgPackWriteArrayFrom(*Parent, *Content))
|
||||
MsgPackFree(*Content, #True)
|
||||
*Content = *Parent
|
||||
Next
|
||||
|
||||
; The top-level array.
|
||||
*Parent = MsgPackCreate(64)
|
||||
If Not *Parent
|
||||
PerfFail("Failed to create a writer !")
|
||||
EndIf
|
||||
PerfCheckWrite(MsgPackWriteArrayFrom(*Parent, *Content))
|
||||
MsgPackFree(*Content, #True)
|
||||
|
||||
*Buffer = MsgPackFinalize(*Parent)
|
||||
If Not *Buffer
|
||||
PerfFail("Failed to finalize the writer !")
|
||||
EndIf
|
||||
|
||||
ProcedureReturn *Buffer
|
||||
EndProcedure
|
||||
|
||||
; Examines every level, reading its integers, and returns their sum for one structure.
|
||||
Procedure.q ExamineNested(ExaminationOptions.MsgPack_ExaminationOption)
|
||||
Protected Dim Views.i(NestedDepth)
|
||||
Protected Total.q
|
||||
Protected Iteration.i, Level.i, Entry.i
|
||||
|
||||
For Iteration = 1 To NestedRepeats
|
||||
Total = 0
|
||||
|
||||
Views(0) = MsgPackWrapMsgBuffer(*NestedBuffer, MemorySize(*NestedBuffer))
|
||||
If Not Views(0)
|
||||
PerfFail("Failed to wrap the buffer !")
|
||||
EndIf
|
||||
|
||||
For Level = 1 To NestedDepth
|
||||
Views(Level) = MsgPackExamine(Views(Level - 1), ExaminationOptions)
|
||||
If Not Views(Level)
|
||||
PerfFail("Failed to examine level " + Str(Level) + " !")
|
||||
EndIf
|
||||
|
||||
For Entry = 1 To 3
|
||||
If Not MsgPackNextEntry(Views(Level))
|
||||
PerfFail("Missing integer at level " + Str(Level) + " !")
|
||||
EndIf
|
||||
Total + MsgPackReadInteger(Views(Level))
|
||||
Next
|
||||
|
||||
; Moving to the nested array, the innermost level doesn't have one.
|
||||
If Level < NestedDepth
|
||||
If Not MsgPackNextEntry(Views(Level))
|
||||
PerfFail("Missing array at level " + Str(Level) + " !")
|
||||
EndIf
|
||||
EndIf
|
||||
Next
|
||||
|
||||
For Level = NestedDepth To 1 Step -1
|
||||
MsgPackFinish(Views(Level))
|
||||
Next
|
||||
MsgPackFree(Views(0), #False)
|
||||
Next
|
||||
|
||||
ProcedureReturn Total
|
||||
EndProcedure
|
||||
|
||||
|
||||
Procedure.q ScenarioBuild()
|
||||
Protected *Buffer
|
||||
Protected Size.q
|
||||
Protected Iteration.i
|
||||
|
||||
For Iteration = 1 To NestedRepeats
|
||||
*Buffer = BuildNestedBuffer()
|
||||
Size = MemorySize(*Buffer)
|
||||
FreeMemory(*Buffer)
|
||||
Next
|
||||
|
||||
ProcedureReturn Size
|
||||
EndProcedure
|
||||
|
||||
Procedure.q ScenarioExamine()
|
||||
ProcedureReturn ExamineNested(0)
|
||||
EndProcedure
|
||||
|
||||
Procedure.q ScenarioExamineUnsafe()
|
||||
ProcedureReturn ExamineNested(#MsgPack_ExaminationOptions_Unsafe)
|
||||
EndProcedure
|
||||
|
||||
Procedure.q ScenarioFullSize()
|
||||
Protected *Reader
|
||||
Protected Size.q
|
||||
Protected Iteration.i
|
||||
|
||||
For Iteration = 1 To NestedRepeats
|
||||
*Reader = MsgPackWrapMsgBuffer(*NestedBuffer, MemorySize(*NestedBuffer))
|
||||
If Not *Reader
|
||||
PerfFail("Failed to wrap the buffer !")
|
||||
EndIf
|
||||
|
||||
Size = MsgPackGetFieldFullSize(*Reader)
|
||||
If Size = #MsgPack_InvalidSize
|
||||
PerfFail("Failed to get the full size !")
|
||||
EndIf
|
||||
|
||||
MsgPackFree(*Reader, #False)
|
||||
Next
|
||||
|
||||
ProcedureReturn Size
|
||||
EndProcedure
|
||||
|
||||
|
||||
OpenConsole()
|
||||
|
||||
NestedDepth = 1
|
||||
While NestedDepth <= #NestedMaxDepth
|
||||
NestedRepeats = #NestedLevelsPerRun / NestedDepth
|
||||
*NestedBuffer = BuildNestedBuffer()
|
||||
|
||||
PerfRun("pb", "nested-" + Str(NestedDepth) + "-build", @ScenarioBuild(), NestedRepeats)
|
||||
PerfRun("pb", "nested-" + Str(NestedDepth) + "-examine", @ScenarioExamine(), NestedRepeats)
|
||||
PerfRun("pb", "nested-" + Str(NestedDepth) + "-examine-unsafe", @ScenarioExamineUnsafe(), NestedRepeats)
|
||||
PerfRun("pb", "nested-" + Str(NestedDepth) + "-fullsize", @ScenarioFullSize(), NestedRepeats)
|
||||
|
||||
FreeMemory(*NestedBuffer)
|
||||
NestedDepth * 2
|
||||
Wend
|
||||
|
||||
End 0
|
||||
@@ -0,0 +1,15 @@
|
||||
$ErrorActionPreference = "Stop"
|
||||
|
||||
# Shows the error instead of closing the window right away.
|
||||
trap {
|
||||
Write-Host $_ -ForegroundColor Red
|
||||
Pause
|
||||
exit 1
|
||||
}
|
||||
|
||||
. (Join-Path $PSScriptRoot "..\_PerfTesting.ps1")
|
||||
|
||||
$Rows = @(Invoke-PureBasicPerfTests $PSScriptRoot)
|
||||
|
||||
Show-PerfResults $Rows $PSScriptRoot
|
||||
Pause
|
||||
@@ -0,0 +1,13 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<OutputType>Exe</OutputType>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<Nullable>enable</Nullable>
|
||||
<InvariantGlobalization>true</InvariantGlobalization>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="MessagePack" Version="3.1.12" />
|
||||
<Compile Include="..\..\_PerfTesting.cs" Link="_PerfTesting.cs" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
@@ -0,0 +1,48 @@
|
||||
// Writes and reads the workload shared with the PureBasic tests, see `../_Workload.pbi`.
|
||||
// Uses the low-level MessagePackWriter/MessagePackReader from MessagePack-CSharp, pinned in the .csproj.
|
||||
using System.Buffers;
|
||||
using MessagePack;
|
||||
|
||||
const int Maps = 100_000;
|
||||
const int Entries = 16;
|
||||
|
||||
string[] keys = Enumerable.Range(0, Entries).Select(i => $"Key{i:D2}").ToArray();
|
||||
byte[] encoded = Write(keys);
|
||||
|
||||
Perf.Run("write", Maps, () => Write(keys).Length);
|
||||
Perf.Run("read", Maps, () => Read(encoded));
|
||||
return 0;
|
||||
|
||||
|
||||
static byte[] Write(string[] keys)
|
||||
{
|
||||
var buffer = new ArrayBufferWriter<byte>();
|
||||
var writer = new MessagePackWriter(buffer);
|
||||
for (int i = 0; i < Maps; i++)
|
||||
{
|
||||
writer.WriteMapHeader(Entries);
|
||||
for (int j = 0; j < Entries; j++)
|
||||
{
|
||||
writer.Write(keys[j]);
|
||||
writer.Write(j);
|
||||
}
|
||||
}
|
||||
writer.Flush();
|
||||
return buffer.WrittenSpan.ToArray();
|
||||
}
|
||||
|
||||
static long Read(byte[] encoded)
|
||||
{
|
||||
var reader = new MessagePackReader(encoded);
|
||||
long total = 0;
|
||||
for (int i = 0; i < Maps; i++)
|
||||
{
|
||||
int count = reader.ReadMapHeader();
|
||||
for (int j = 0; j < count; j++)
|
||||
{
|
||||
total += reader.ReadString()!.Length;
|
||||
total += reader.ReadInt64();
|
||||
}
|
||||
}
|
||||
return total;
|
||||
}
|
||||
@@ -0,0 +1,111 @@
|
||||
EnableExplicit
|
||||
|
||||
XIncludeFile "../../Includes/MessagePack.pbi"
|
||||
XIncludeFile "../_PerfTesting.pbi"
|
||||
XIncludeFile "_Workload.pbi"
|
||||
|
||||
|
||||
; Writes and reads the shared workload, see `_Workload.pbi`.
|
||||
; Each map's fields are written in their own writer, then copied in with `MsgPackWriteMapFrom`.
|
||||
|
||||
Global Dim Keys.s(#Perf_Entries - 1)
|
||||
Global *Encoded
|
||||
|
||||
|
||||
Procedure WriteMap(*Writer)
|
||||
Protected *Fields = MsgPackCreate(128)
|
||||
Protected i.i
|
||||
|
||||
If Not *Fields
|
||||
PerfFail("Failed to create the fields writer !")
|
||||
EndIf
|
||||
|
||||
For i = 0 To #Perf_Entries - 1
|
||||
PerfCheckWrite(MsgPackWriteString(*Fields, Keys(i)))
|
||||
PerfCheckWrite(MsgPackWriteInteger(*Fields, i))
|
||||
Next
|
||||
|
||||
PerfCheckWrite(MsgPackWriteMapFrom(*Writer, *Fields))
|
||||
MsgPackFree(*Fields, #True)
|
||||
EndProcedure
|
||||
|
||||
; Returns the finalized buffer, which must be freed with `FreeMemory()`.
|
||||
Procedure.i EncodeMaps()
|
||||
Protected *Writer = MsgPackCreate(#Perf_EncodedSize)
|
||||
Protected *Buffer
|
||||
Protected i.i
|
||||
|
||||
If Not *Writer
|
||||
PerfFail("Failed to create the writer !")
|
||||
EndIf
|
||||
|
||||
For i = 1 To #Perf_Maps
|
||||
WriteMap(*Writer)
|
||||
Next
|
||||
|
||||
*Buffer = MsgPackFinalize(*Writer)
|
||||
If Not *Buffer
|
||||
PerfFail("Failed to finalize the writer !")
|
||||
EndIf
|
||||
|
||||
ProcedureReturn *Buffer
|
||||
EndProcedure
|
||||
|
||||
|
||||
Procedure.q ScenarioWrite()
|
||||
Protected *Buffer = EncodeMaps()
|
||||
Protected Size.q = MemorySize(*Buffer)
|
||||
|
||||
FreeMemory(*Buffer)
|
||||
ProcedureReturn Size
|
||||
EndProcedure
|
||||
|
||||
Procedure.q ScenarioRead()
|
||||
Protected *Reader = MsgPackWrapMsgBuffer(*Encoded, MemorySize(*Encoded))
|
||||
Protected *Iterator
|
||||
Protected Total.q
|
||||
Protected i.i
|
||||
|
||||
If Not *Reader
|
||||
PerfFail("Failed to wrap the buffer !")
|
||||
EndIf
|
||||
|
||||
For i = 1 To #Perf_Maps
|
||||
*Iterator = MsgPackExamine(*Reader)
|
||||
If Not *Iterator
|
||||
PerfFail("Failed to examine a map !")
|
||||
EndIf
|
||||
|
||||
While MsgPackNextEntry(*Iterator)
|
||||
Total + Len(MsgPackReadString(*Iterator))
|
||||
Total + MsgPackReadInteger(*Iterator)
|
||||
Wend
|
||||
|
||||
; `MsgPackNextEntry` also returns `#False` on errors, and failed reads break the next ones.
|
||||
If MsgPackLastError(*Iterator) <> #MsgPack_Error_Success
|
||||
PerfFail("A read failed !")
|
||||
EndIf
|
||||
|
||||
MsgPackFinish(*Iterator)
|
||||
Next
|
||||
|
||||
MsgPackFree(*Reader, #False)
|
||||
ProcedureReturn Total
|
||||
EndProcedure
|
||||
|
||||
|
||||
OpenConsole()
|
||||
|
||||
Define i.i
|
||||
For i = 0 To #Perf_Entries - 1
|
||||
Keys(i) = "Key" + RSet(Str(i), 2, "0")
|
||||
Next
|
||||
|
||||
*Encoded = EncodeMaps()
|
||||
|
||||
PerfRun("pb", "write", @ScenarioWrite(), #Perf_Maps)
|
||||
PerfRun("pb", "read", @ScenarioRead(), #Perf_Maps)
|
||||
|
||||
FreeMemory(*Encoded)
|
||||
|
||||
End 0
|
||||
@@ -0,0 +1,37 @@
|
||||
# Writes and reads the workload shared with the PureBasic tests, see `_Workload.pbi`.
|
||||
# The maps are packed from and unpacked to dicts, how they're usually handled in Python.
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import msgpack
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
||||
from _PerfTesting import run
|
||||
|
||||
MAPS = 100_000
|
||||
MAP = {f"Key{i:02}": i for i in range(16)}
|
||||
|
||||
|
||||
def write():
|
||||
packer = msgpack.Packer(autoreset=False)
|
||||
pack = packer.pack
|
||||
for _ in range(MAPS):
|
||||
pack(MAP)
|
||||
return packer.bytes()
|
||||
|
||||
|
||||
def read():
|
||||
unpacker = msgpack.Unpacker(raw=False)
|
||||
unpacker.feed(ENCODED)
|
||||
unpack = unpacker.unpack
|
||||
total = 0
|
||||
for _ in range(MAPS):
|
||||
for key, value in unpack().items():
|
||||
total += len(key) + value
|
||||
return total
|
||||
|
||||
|
||||
ENCODED = write()
|
||||
|
||||
run("write", MAPS, lambda: len(write()))
|
||||
run("read", MAPS, read)
|
||||
@@ -0,0 +1,18 @@
|
||||
$ErrorActionPreference = "Stop"
|
||||
|
||||
# Shows the error instead of closing the window right away.
|
||||
trap {
|
||||
Write-Host $_ -ForegroundColor Red
|
||||
Pause
|
||||
exit 1
|
||||
}
|
||||
|
||||
. (Join-Path $PSScriptRoot "..\_PerfTesting.ps1")
|
||||
|
||||
# Requires Python with the packages from 'PerfTests\requirements.txt', and the .NET 10 SDK.
|
||||
$Rows = @(Invoke-PureBasicPerfTests $PSScriptRoot)
|
||||
$Rows += Invoke-PythonPerfTest (Join-Path $PSScriptRoot "PerfPython.py")
|
||||
$Rows += Invoke-DotNetPerfTest (Join-Path $PSScriptRoot "DotNet")
|
||||
|
||||
Show-PerfResults $Rows $PSScriptRoot
|
||||
Pause
|
||||
@@ -0,0 +1,15 @@
|
||||
; Workload shared by every implementation in this folder, keep `PerfPython.py` and `DotNet/Program.cs` in sync.
|
||||
;
|
||||
; `#Perf_Maps` top-level maps are written, then read, each holding `#Perf_Entries` entries:
|
||||
; "Key00" to "Key15" (FixStr), mapped to 0 to 15 (FixInt), in that order.
|
||||
; 16 entries make it a Map16, the smallest format code that fits.
|
||||
; `NsPerOp` is the time per map.
|
||||
;
|
||||
; Write check: the encoded size.
|
||||
; Read check: the sum of the keys' length and of the values.
|
||||
|
||||
#Perf_Maps = 100000
|
||||
#Perf_Entries = 16
|
||||
|
||||
; 3 bytes of header, then 6 bytes per key and 1 per value.
|
||||
#Perf_EncodedSize = #Perf_Maps * (3 + #Perf_Entries * 7)
|
||||
@@ -0,0 +1,59 @@
|
||||
# PureBasic - MsgPack - Performance investigations
|
||||
Open questions raised by the [results](readme.md#results), with leads to check. \
|
||||
Nothing below is confirmed yet, update or remove each entry once it is looked into.
|
||||
|
||||
Measured on 2026-11-03, see the results' CSVs in `Results/`.
|
||||
|
||||
|
||||
## Strings are slow compared to binary data
|
||||
x64 with the C backend, PureBasic 6.41, ns per field:
|
||||
|
||||
| Payload | Binary write | String write | Binary read | String read |
|
||||
|:--|--:|--:|--:|--:|
|
||||
| 1 KB, UTF-8 | 190.6 | 720.9 (3.8x) | 50.2 | 570.0 (11.4x) |
|
||||
| 1 KB, UTF-16 (2 KB) | - | 526.0 | - | 282.7 |
|
||||
| 64 KB vs 1 MB | ~5.0 GB/s | ~1.4 GB/s | ~19 GB/s | ~2.1 GB/s |
|
||||
|
||||
* UTF-16 is faster than UTF-8 while being twice as big, so the encoding itself costs a lot.
|
||||
* MessagePack-CSharp reads 1 KB strings in 150 ns, and writes 8 B strings in 13 ns against 60 ns here.
|
||||
* PureBasic 6.21 is another 2.5 to 4 times slower, see [PureBasic 6.21 vs 6.41](readme.md#purebasic-621-vs-641).
|
||||
|
||||
### Leads in `Includes/_Strings.pbi`
|
||||
* `String.s` is passed by value through `MsgPackWriteString`, `_MsgPackGetEncodedStringSize` and `_MsgPackWriteStringField`,
|
||||
which may copy it at each call, and would explain why 6.40's string handling helps that much. \
|
||||
Passing `@String` and its length to the private helpers would avoid it.
|
||||
* Each write goes over the string at least twice, with `StringByteLength()` then `PokeS()`. \
|
||||
For `#PB_Unicode` and `#PB_Ascii`, the size could come from `Len()` instead, if it doesn't scan the string.
|
||||
* `PokeS()` is given `-1` as its length, so it may also look for the null terminator on its own.
|
||||
* `MsgPackReadString` builds the string with `PeekS()`, then returns it, which may copy it again.
|
||||
|
||||
### Leads in the benchmark
|
||||
* `EncodeStrings()` preallocates `FieldSize * 2 + 5` bytes per field, 64 MB in total, against 32 MB for binaries. \
|
||||
The allocation is timed, see the next section, and `MsgPackFinalize` then shrinks it by half.
|
||||
* `ScenarioStringRead()` calls `Len()` on each string, which may scan it on 6.21 and not on 6.41. \
|
||||
Python's `len()` and .NET's `Length` don't scan.
|
||||
|
||||
### Experiments to run
|
||||
1. Time raw `PokeS()`/`PeekS()` loops with the same sizes, without MsgPack, to split PureBasic's own cost from the include's.
|
||||
2. Size the strings writer like the binaries one, and compare.
|
||||
3. Try the `@String` + length helpers, and `Len()` for the fixed-width encodings.
|
||||
4. Run 6.40 too, to check that the gain comes from 6.40 and not 6.41.
|
||||
|
||||
|
||||
## Buffers are zero-filled
|
||||
`MsgPackCreate` uses `AllocateMemory()` and `MsgPackGrow` uses `ReAllocateMemory()`, both without `#PB_Memory_NoClear`,
|
||||
so every buffer is zeroed before being written over.
|
||||
* That cost is in every write scenario with a preallocated buffer, more so for the big ones.
|
||||
* With a `GrowthIncrement` of `1`, small writes take ~440 ns against ~12 ns preallocated,
|
||||
the reallocation and copy probably cost more than the zeroing, but it adds up.
|
||||
|
||||
To try: `#PB_Memory_NoClear` on both, as only the written bytes are ever read, then re-run every folder.
|
||||
|
||||
|
||||
## Smaller oddities
|
||||
* **Maps:** writing them is 8.6 times slower than MessagePack-CSharp and 1.9 times slower than msgpack. \
|
||||
Each map is written in its own structure, then checked, counted and copied by `MsgPackWriteMapFrom`, see the main readme.
|
||||
Its 16 keys are strings too, see above.
|
||||
* **ASM backend:** skipping small fields is 3.2 times slower than with the C backend (57.9 vs 18.3 ns), against 1 to 2 times for the rest.
|
||||
* **6.21 faster than 6.41:** skip (14.7 vs 18.3 ns) and 1 KB binaries (0.8 to 0.9x). \
|
||||
Those runs only last 1 to 2 ms, so it may be noise, re-run them with more iterations before looking further.
|
||||
@@ -0,0 +1,13 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<OutputType>Exe</OutputType>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<Nullable>enable</Nullable>
|
||||
<InvariantGlobalization>true</InvariantGlobalization>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="MessagePack" Version="3.1.12" />
|
||||
<Compile Include="..\..\_PerfTesting.cs" Link="_PerfTesting.cs" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
@@ -0,0 +1,62 @@
|
||||
// Writes, reads and skips the workload shared with the PureBasic tests, see `../_Workload.pbi`.
|
||||
// Uses the low-level MessagePackWriter/MessagePackReader from MessagePack-CSharp, pinned in the .csproj.
|
||||
using System.Buffers;
|
||||
using MessagePack;
|
||||
|
||||
const int Records = 125_000;
|
||||
const int Fields = Records * 8;
|
||||
|
||||
byte[] encoded = Write();
|
||||
|
||||
Perf.Run("write", Fields, () => Write().Length);
|
||||
Perf.Run("read", Fields, () => Read(encoded));
|
||||
Perf.Run("skip", Fields, () => Skip(encoded));
|
||||
return 0;
|
||||
|
||||
|
||||
static byte[] Write()
|
||||
{
|
||||
var buffer = new ArrayBufferWriter<byte>();
|
||||
var writer = new MessagePackWriter(buffer);
|
||||
for (int i = 0; i < Records; i++)
|
||||
{
|
||||
writer.Write(42);
|
||||
writer.Write(-5);
|
||||
writer.Write(200);
|
||||
writer.Write(-1000);
|
||||
writer.Write(70000);
|
||||
writer.Write(-1099511627776L);
|
||||
writer.Write(3.14);
|
||||
writer.Write("Hello");
|
||||
}
|
||||
writer.Flush();
|
||||
return buffer.WrittenSpan.ToArray();
|
||||
}
|
||||
|
||||
static long Read(byte[] data)
|
||||
{
|
||||
var reader = new MessagePackReader(data);
|
||||
long total = 0;
|
||||
for (int i = 0; i < Records; i++)
|
||||
{
|
||||
total += reader.ReadInt64();
|
||||
total += reader.ReadInt64();
|
||||
total += reader.ReadInt64();
|
||||
total += reader.ReadInt64();
|
||||
total += reader.ReadInt64();
|
||||
total += reader.ReadInt64();
|
||||
total += (long)reader.ReadDouble();
|
||||
total += reader.ReadString()!.Length;
|
||||
}
|
||||
return total;
|
||||
}
|
||||
|
||||
static long Skip(byte[] data)
|
||||
{
|
||||
var reader = new MessagePackReader(data);
|
||||
for (int i = 0; i < Fields; i++)
|
||||
{
|
||||
reader.Skip();
|
||||
}
|
||||
return Fields;
|
||||
}
|
||||
@@ -0,0 +1,111 @@
|
||||
EnableExplicit
|
||||
|
||||
XIncludeFile "../../Includes/MessagePack.pbi"
|
||||
XIncludeFile "../_PerfTesting.pbi"
|
||||
XIncludeFile "_Workload.pbi"
|
||||
|
||||
|
||||
; Writes and reads the shared workload, see `_Workload.pbi`, forcing each field's format code.
|
||||
|
||||
Global *Encoded
|
||||
|
||||
|
||||
Procedure WriteRecords(*Writer)
|
||||
Protected i.i
|
||||
|
||||
For i = 1 To #Perf_Records
|
||||
PerfCheckWrite(MsgPackWriteFixInt(*Writer, 42))
|
||||
PerfCheckWrite(MsgPackWriteFixInt(*Writer, -5))
|
||||
PerfCheckWrite(MsgPackWriteUInt8(*Writer, 200))
|
||||
PerfCheckWrite(MsgPackWriteInt16(*Writer, -1000))
|
||||
PerfCheckWrite(MsgPackWriteUInt32(*Writer, 70000))
|
||||
PerfCheckWrite(MsgPackWriteInt64(*Writer, -1099511627776))
|
||||
PerfCheckWrite(MsgPackWriteFloat64(*Writer, 3.14))
|
||||
PerfCheckWrite(MsgPackWriteFixStr(*Writer, "Hello"))
|
||||
Next
|
||||
EndProcedure
|
||||
|
||||
; Returns the finalized buffer, which must be freed with `FreeMemory()`.
|
||||
Procedure.i Encode(InitialSize.i, GrowthIncrement.i)
|
||||
Protected *Writer = MsgPackCreate(InitialSize, GrowthIncrement)
|
||||
Protected *Buffer
|
||||
|
||||
If Not *Writer
|
||||
PerfFail("Failed to create the writer !")
|
||||
EndIf
|
||||
|
||||
WriteRecords(*Writer)
|
||||
|
||||
*Buffer = MsgPackFinalize(*Writer)
|
||||
If Not *Buffer
|
||||
PerfFail("Failed to finalize the writer !")
|
||||
EndIf
|
||||
|
||||
ProcedureReturn *Buffer
|
||||
EndProcedure
|
||||
|
||||
Procedure.q EncodeAndFree(InitialSize.i, GrowthIncrement.i)
|
||||
Protected *Buffer = Encode(InitialSize, GrowthIncrement)
|
||||
Protected Size.q = MemorySize(*Buffer)
|
||||
|
||||
FreeMemory(*Buffer)
|
||||
ProcedureReturn Size
|
||||
EndProcedure
|
||||
|
||||
|
||||
Procedure.q ScenarioWrite()
|
||||
ProcedureReturn EncodeAndFree(64, 1)
|
||||
EndProcedure
|
||||
|
||||
Procedure.q ScenarioWriteGrow64K()
|
||||
ProcedureReturn EncodeAndFree(64, 65536)
|
||||
EndProcedure
|
||||
|
||||
Procedure.q ScenarioWritePrealloc()
|
||||
ProcedureReturn EncodeAndFree(#Perf_EncodedSize, 1)
|
||||
EndProcedure
|
||||
|
||||
Procedure.q ScenarioRead()
|
||||
Protected *Reader = MsgPackWrapMsgBuffer(*Encoded, MemorySize(*Encoded))
|
||||
Protected Total.q
|
||||
Protected i.i
|
||||
|
||||
If Not *Reader
|
||||
PerfFail("Failed to wrap the buffer !")
|
||||
EndIf
|
||||
|
||||
For i = 1 To #Perf_Records
|
||||
; Fixints don't have an explicitly typed getter.
|
||||
Total + MsgPackReadInteger(*Reader)
|
||||
Total + MsgPackReadInteger(*Reader)
|
||||
Total + MsgPackReadUInt8(*Reader)
|
||||
Total + MsgPackReadInt16(*Reader)
|
||||
Total + (MsgPackReadUInt32(*Reader) & $FFFFFFFF)
|
||||
Total + MsgPackReadInt64(*Reader)
|
||||
Total + Int(MsgPackReadFloat64(*Reader))
|
||||
Total + Len(MsgPackReadString(*Reader))
|
||||
Next
|
||||
|
||||
; Failed reads don't move the offset, so a failure breaks the next reads too.
|
||||
If MsgPackLastError(*Reader) <> #MsgPack_Error_Success
|
||||
PerfFail("A read failed !")
|
||||
EndIf
|
||||
|
||||
MsgPackFree(*Reader, #False)
|
||||
ProcedureReturn Total
|
||||
EndProcedure
|
||||
|
||||
|
||||
OpenConsole()
|
||||
|
||||
*Encoded = Encode(#Perf_EncodedSize, 1)
|
||||
|
||||
PerfRun("pb-explicit", "read", @ScenarioRead(), #Perf_Fields)
|
||||
PerfRun("pb-explicit", "write-prealloc", @ScenarioWritePrealloc(), #Perf_Fields)
|
||||
PerfRun("pb-explicit", "write-grow64k", @ScenarioWriteGrow64K(), #Perf_Fields)
|
||||
; Last since it reallocates the buffer on every write, which can be very slow.
|
||||
PerfRun("pb-explicit", "write", @ScenarioWrite(), #Perf_Fields)
|
||||
|
||||
FreeMemory(*Encoded)
|
||||
|
||||
End 0
|
||||
@@ -0,0 +1,132 @@
|
||||
EnableExplicit
|
||||
|
||||
XIncludeFile "../../Includes/MessagePack.pbi"
|
||||
XIncludeFile "../_PerfTesting.pbi"
|
||||
XIncludeFile "_Workload.pbi"
|
||||
|
||||
|
||||
; Writes and reads the shared workload, see `_Workload.pbi`, letting the library pick each field's format code.
|
||||
|
||||
Global *Encoded
|
||||
|
||||
|
||||
Procedure WriteRecords(*Writer)
|
||||
Protected i.i
|
||||
|
||||
For i = 1 To #Perf_Records
|
||||
PerfCheckWrite(MsgPackWriteInteger(*Writer, 42))
|
||||
PerfCheckWrite(MsgPackWriteInteger(*Writer, -5))
|
||||
PerfCheckWrite(MsgPackWriteInteger(*Writer, 200))
|
||||
PerfCheckWrite(MsgPackWriteInteger(*Writer, -1000))
|
||||
PerfCheckWrite(MsgPackWriteInteger(*Writer, 70000))
|
||||
PerfCheckWrite(MsgPackWriteInteger(*Writer, -1099511627776))
|
||||
; Floats don't have an implicitly typed setter.
|
||||
PerfCheckWrite(MsgPackWriteFloat64(*Writer, 3.14))
|
||||
PerfCheckWrite(MsgPackWriteString(*Writer, "Hello"))
|
||||
Next
|
||||
EndProcedure
|
||||
|
||||
; Returns the finalized buffer, which must be freed with `FreeMemory()`.
|
||||
Procedure.i Encode(InitialSize.i, GrowthIncrement.i)
|
||||
Protected *Writer = MsgPackCreate(InitialSize, GrowthIncrement)
|
||||
Protected *Buffer
|
||||
|
||||
If Not *Writer
|
||||
PerfFail("Failed to create the writer !")
|
||||
EndIf
|
||||
|
||||
WriteRecords(*Writer)
|
||||
|
||||
*Buffer = MsgPackFinalize(*Writer)
|
||||
If Not *Buffer
|
||||
PerfFail("Failed to finalize the writer !")
|
||||
EndIf
|
||||
|
||||
ProcedureReturn *Buffer
|
||||
EndProcedure
|
||||
|
||||
Procedure.q EncodeAndFree(InitialSize.i, GrowthIncrement.i)
|
||||
Protected *Buffer = Encode(InitialSize, GrowthIncrement)
|
||||
Protected Size.q = MemorySize(*Buffer)
|
||||
|
||||
FreeMemory(*Buffer)
|
||||
ProcedureReturn Size
|
||||
EndProcedure
|
||||
|
||||
|
||||
Procedure.q ScenarioWrite()
|
||||
ProcedureReturn EncodeAndFree(64, 1)
|
||||
EndProcedure
|
||||
|
||||
Procedure.q ScenarioWriteGrow64K()
|
||||
ProcedureReturn EncodeAndFree(64, 65536)
|
||||
EndProcedure
|
||||
|
||||
Procedure.q ScenarioWritePrealloc()
|
||||
ProcedureReturn EncodeAndFree(#Perf_EncodedSize, 1)
|
||||
EndProcedure
|
||||
|
||||
Procedure.q ScenarioRead()
|
||||
Protected *Reader = MsgPackWrapMsgBuffer(*Encoded, MemorySize(*Encoded))
|
||||
Protected Total.q
|
||||
Protected i.i
|
||||
|
||||
If Not *Reader
|
||||
PerfFail("Failed to wrap the buffer !")
|
||||
EndIf
|
||||
|
||||
For i = 1 To #Perf_Records
|
||||
Total + MsgPackReadInteger(*Reader)
|
||||
Total + MsgPackReadInteger(*Reader)
|
||||
Total + MsgPackReadInteger(*Reader)
|
||||
Total + MsgPackReadInteger(*Reader)
|
||||
Total + MsgPackReadInteger(*Reader)
|
||||
Total + MsgPackReadInteger(*Reader)
|
||||
Total + Int(MsgPackReadFloat(*Reader))
|
||||
Total + Len(MsgPackReadString(*Reader))
|
||||
Next
|
||||
|
||||
; Failed reads don't move the offset, so a failure breaks the next reads too.
|
||||
If MsgPackLastError(*Reader) <> #MsgPack_Error_Success
|
||||
PerfFail("A read failed !")
|
||||
EndIf
|
||||
|
||||
MsgPackFree(*Reader, #False)
|
||||
ProcedureReturn Total
|
||||
EndProcedure
|
||||
|
||||
; Doesn't depend on the typing, so it's only measured here.
|
||||
; Skips each field separately, like the Python and .NET implementations.
|
||||
Procedure.q ScenarioSkip()
|
||||
Protected *Reader = MsgPackWrapMsgBuffer(*Encoded, MemorySize(*Encoded))
|
||||
Protected i.i
|
||||
|
||||
If Not *Reader
|
||||
PerfFail("Failed to wrap the buffer !")
|
||||
EndIf
|
||||
|
||||
For i = 1 To #Perf_Fields
|
||||
If MsgPackSkip(*Reader) <> #MsgPack_Error_Success
|
||||
PerfFail("A skip failed !")
|
||||
EndIf
|
||||
Next
|
||||
|
||||
MsgPackFree(*Reader, #False)
|
||||
ProcedureReturn #Perf_Fields
|
||||
EndProcedure
|
||||
|
||||
|
||||
OpenConsole()
|
||||
|
||||
*Encoded = Encode(#Perf_EncodedSize, 1)
|
||||
|
||||
PerfRun("pb-implicit", "read", @ScenarioRead(), #Perf_Fields)
|
||||
PerfRun("pb-implicit", "skip", @ScenarioSkip(), #Perf_Fields)
|
||||
PerfRun("pb-implicit", "write-prealloc", @ScenarioWritePrealloc(), #Perf_Fields)
|
||||
PerfRun("pb-implicit", "write-grow64k", @ScenarioWriteGrow64K(), #Perf_Fields)
|
||||
; Last since it reallocates the buffer on every write, which can be very slow.
|
||||
PerfRun("pb-implicit", "write", @ScenarioWrite(), #Perf_Fields)
|
||||
|
||||
FreeMemory(*Encoded)
|
||||
|
||||
End 0
|
||||
@@ -0,0 +1,54 @@
|
||||
# Writes, reads and skips the workload shared with the PureBasic tests, see `_Workload.pbi`.
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import msgpack
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
||||
from _PerfTesting import run
|
||||
|
||||
RECORDS = 125_000
|
||||
FIELDS = RECORDS * 8
|
||||
|
||||
|
||||
def write():
|
||||
packer = msgpack.Packer(autoreset=False)
|
||||
pack = packer.pack
|
||||
for _ in range(RECORDS):
|
||||
pack(42)
|
||||
pack(-5)
|
||||
pack(200)
|
||||
pack(-1000)
|
||||
pack(70000)
|
||||
pack(-1099511627776)
|
||||
pack(3.14)
|
||||
pack("Hello")
|
||||
return packer.bytes()
|
||||
|
||||
|
||||
def read():
|
||||
unpacker = msgpack.Unpacker(raw=False)
|
||||
unpacker.feed(ENCODED)
|
||||
unpack = unpacker.unpack
|
||||
total = 0
|
||||
for _ in range(RECORDS):
|
||||
total += unpack() + unpack() + unpack() + unpack() + unpack() + unpack()
|
||||
total += int(unpack())
|
||||
total += len(unpack())
|
||||
return total
|
||||
|
||||
|
||||
def skip():
|
||||
unpacker = msgpack.Unpacker(raw=False)
|
||||
unpacker.feed(ENCODED)
|
||||
skip_field = unpacker.skip
|
||||
for _ in range(FIELDS):
|
||||
skip_field()
|
||||
return FIELDS
|
||||
|
||||
|
||||
ENCODED = write()
|
||||
|
||||
run("write", FIELDS, lambda: len(write()))
|
||||
run("read", FIELDS, read)
|
||||
run("skip", FIELDS, skip)
|
||||
@@ -0,0 +1,18 @@
|
||||
$ErrorActionPreference = "Stop"
|
||||
|
||||
# Shows the error instead of closing the window right away.
|
||||
trap {
|
||||
Write-Host $_ -ForegroundColor Red
|
||||
Pause
|
||||
exit 1
|
||||
}
|
||||
|
||||
. (Join-Path $PSScriptRoot "..\_PerfTesting.ps1")
|
||||
|
||||
# Requires Python with the packages from 'PerfTests\requirements.txt', and the .NET 10 SDK.
|
||||
$Rows = @(Invoke-PureBasicPerfTests $PSScriptRoot)
|
||||
$Rows += Invoke-PythonPerfTest (Join-Path $PSScriptRoot "PerfPython.py")
|
||||
$Rows += Invoke-DotNetPerfTest (Join-Path $PSScriptRoot "DotNet")
|
||||
|
||||
Show-PerfResults $Rows $PSScriptRoot
|
||||
Pause
|
||||
@@ -0,0 +1,15 @@
|
||||
; Workload shared by every implementation in this folder, keep `PerfPython.py` and `DotNet/Program.cs` in sync.
|
||||
;
|
||||
; `#Perf_Records` records of 8 fields, 1M fields in total, each record being:
|
||||
; 42 (FixInt), -5 (FixInt), 200 (UInt8), -1000 (Int16), 70000 (UInt32),
|
||||
; -1099511627776 (Int64), 3.14 (Float64), "Hello" (FixStr)
|
||||
;
|
||||
; Write check: the encoded size.
|
||||
; Read check: the sum of the integers, of `Int()` of the float, and of the string's length.
|
||||
; Skip check: the number of fields skipped.
|
||||
|
||||
#Perf_Records = 125000
|
||||
#Perf_Fields = #Perf_Records * 8
|
||||
|
||||
; 1 + 1 + 2 + 3 + 5 + 9 + 9 + 6 bytes per record.
|
||||
#Perf_EncodedSize = #Perf_Records * 36
|
||||
@@ -0,0 +1,6 @@
|
||||
EnableExplicit
|
||||
|
||||
#MsgPack_DisableNullChecks = #True
|
||||
#Perf_Implementation = "pb-nullchecks-off"
|
||||
|
||||
XIncludeFile "_Scenarios.pbi"
|
||||
@@ -0,0 +1,6 @@
|
||||
EnableExplicit
|
||||
|
||||
; Default behaviour, `#Null` pointers are checked by every procedure.
|
||||
#Perf_Implementation = "pb-nullchecks-on"
|
||||
|
||||
XIncludeFile "_Scenarios.pbi"
|
||||
@@ -0,0 +1,15 @@
|
||||
$ErrorActionPreference = "Stop"
|
||||
|
||||
# Shows the error instead of closing the window right away.
|
||||
trap {
|
||||
Write-Host $_ -ForegroundColor Red
|
||||
Pause
|
||||
exit 1
|
||||
}
|
||||
|
||||
. (Join-Path $PSScriptRoot "..\_PerfTesting.ps1")
|
||||
|
||||
$Rows = @(Invoke-PureBasicPerfTests $PSScriptRoot)
|
||||
|
||||
Show-PerfResults $Rows $PSScriptRoot
|
||||
Pause
|
||||
@@ -0,0 +1,96 @@
|
||||
; Shared by `PerfNullChecksOn.pb` and `PerfNullChecksOff.pb`, which only differ by `#MsgPack_DisableNullChecks`.
|
||||
; Both must define `#Perf_Implementation` before including this file, and `#MsgPack_DisableNullChecks` for the "off" one.
|
||||
;
|
||||
; Writes and reads the "MassiveReadAmount" workload with implicit typing, into a buffer of the right size.
|
||||
; Only the scenarios where the checks are the main overhead are measured.
|
||||
|
||||
XIncludeFile "../../Includes/MessagePack.pbi"
|
||||
XIncludeFile "../_PerfTesting.pbi"
|
||||
XIncludeFile "../MassiveReadAmount/_Workload.pbi"
|
||||
|
||||
Global *Encoded
|
||||
|
||||
|
||||
Procedure WriteRecords(*Writer)
|
||||
Protected i.i
|
||||
|
||||
For i = 1 To #Perf_Records
|
||||
PerfCheckWrite(MsgPackWriteInteger(*Writer, 42))
|
||||
PerfCheckWrite(MsgPackWriteInteger(*Writer, -5))
|
||||
PerfCheckWrite(MsgPackWriteInteger(*Writer, 200))
|
||||
PerfCheckWrite(MsgPackWriteInteger(*Writer, -1000))
|
||||
PerfCheckWrite(MsgPackWriteInteger(*Writer, 70000))
|
||||
PerfCheckWrite(MsgPackWriteInteger(*Writer, -1099511627776))
|
||||
PerfCheckWrite(MsgPackWriteFloat64(*Writer, 3.14))
|
||||
PerfCheckWrite(MsgPackWriteString(*Writer, "Hello"))
|
||||
Next
|
||||
EndProcedure
|
||||
|
||||
; Returns the finalized buffer, which must be freed with `FreeMemory()`.
|
||||
Procedure.i Encode()
|
||||
Protected *Writer = MsgPackCreate(#Perf_EncodedSize)
|
||||
Protected *Buffer
|
||||
|
||||
If Not *Writer
|
||||
PerfFail("Failed to create the writer !")
|
||||
EndIf
|
||||
|
||||
WriteRecords(*Writer)
|
||||
|
||||
*Buffer = MsgPackFinalize(*Writer)
|
||||
If Not *Buffer
|
||||
PerfFail("Failed to finalize the writer !")
|
||||
EndIf
|
||||
|
||||
ProcedureReturn *Buffer
|
||||
EndProcedure
|
||||
|
||||
|
||||
Procedure.q ScenarioWritePrealloc()
|
||||
Protected *Buffer = Encode()
|
||||
Protected Size.q = MemorySize(*Buffer)
|
||||
|
||||
FreeMemory(*Buffer)
|
||||
ProcedureReturn Size
|
||||
EndProcedure
|
||||
|
||||
Procedure.q ScenarioRead()
|
||||
Protected *Reader = MsgPackWrapMsgBuffer(*Encoded, MemorySize(*Encoded))
|
||||
Protected Total.q
|
||||
Protected i.i
|
||||
|
||||
If Not *Reader
|
||||
PerfFail("Failed to wrap the buffer !")
|
||||
EndIf
|
||||
|
||||
For i = 1 To #Perf_Records
|
||||
Total + MsgPackReadInteger(*Reader)
|
||||
Total + MsgPackReadInteger(*Reader)
|
||||
Total + MsgPackReadInteger(*Reader)
|
||||
Total + MsgPackReadInteger(*Reader)
|
||||
Total + MsgPackReadInteger(*Reader)
|
||||
Total + MsgPackReadInteger(*Reader)
|
||||
Total + Int(MsgPackReadFloat(*Reader))
|
||||
Total + Len(MsgPackReadString(*Reader))
|
||||
Next
|
||||
|
||||
; Failed reads don't move the offset, so a failure breaks the next reads too.
|
||||
If MsgPackLastError(*Reader) <> #MsgPack_Error_Success
|
||||
PerfFail("A read failed !")
|
||||
EndIf
|
||||
|
||||
MsgPackFree(*Reader, #False)
|
||||
ProcedureReturn Total
|
||||
EndProcedure
|
||||
|
||||
|
||||
OpenConsole()
|
||||
|
||||
*Encoded = Encode()
|
||||
|
||||
PerfRun(#Perf_Implementation, "read", @ScenarioRead(), #Perf_Fields)
|
||||
PerfRun(#Perf_Implementation, "write-prealloc", @ScenarioWritePrealloc(), #Perf_Fields)
|
||||
|
||||
FreeMemory(*Encoded)
|
||||
|
||||
End 0
|
||||
@@ -0,0 +1,13 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<OutputType>Exe</OutputType>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<Nullable>enable</Nullable>
|
||||
<InvariantGlobalization>true</InvariantGlobalization>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="MessagePack" Version="3.1.12" />
|
||||
<Compile Include="..\..\_PerfTesting.cs" Link="_PerfTesting.cs" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
@@ -0,0 +1,77 @@
|
||||
// Writes and reads the workload shared with the PureBasic tests, see `../_Workload.pbi`.
|
||||
// Uses the low-level MessagePackWriter/MessagePackReader from MessagePack-CSharp, pinned in the .csproj.
|
||||
using System.Buffers;
|
||||
using MessagePack;
|
||||
|
||||
const int TotalSize = 33_554_432;
|
||||
|
||||
foreach (int size in new[] { 8, 1024, 65536 })
|
||||
{
|
||||
int count = TotalSize / size;
|
||||
string text = new string('a', size);
|
||||
byte[] encoded = WriteStrings(text, count);
|
||||
Perf.Run($"str-{size}-write", count, () => WriteStrings(text, count).Length);
|
||||
Perf.Run($"str-{size}-read", count, () => ReadStrings(encoded, count));
|
||||
}
|
||||
|
||||
foreach (int size in new[] { 1024, 1048576 })
|
||||
{
|
||||
int count = TotalSize / size;
|
||||
byte[] data = new byte[size];
|
||||
Array.Fill(data, (byte)0xAB);
|
||||
byte[] output = new byte[size];
|
||||
byte[] encoded = WriteBinaries(data, count);
|
||||
Perf.Run($"bin-{size}-write", count, () => WriteBinaries(data, count).Length);
|
||||
Perf.Run($"bin-{size}-read", count, () => ReadBinaries(encoded, count, output));
|
||||
}
|
||||
return 0;
|
||||
|
||||
|
||||
static byte[] WriteStrings(string text, int count)
|
||||
{
|
||||
var buffer = new ArrayBufferWriter<byte>();
|
||||
var writer = new MessagePackWriter(buffer);
|
||||
for (int i = 0; i < count; i++)
|
||||
{
|
||||
writer.Write(text);
|
||||
}
|
||||
writer.Flush();
|
||||
return buffer.WrittenSpan.ToArray();
|
||||
}
|
||||
|
||||
static long ReadStrings(byte[] encoded, int count)
|
||||
{
|
||||
var reader = new MessagePackReader(encoded);
|
||||
long total = 0;
|
||||
for (int i = 0; i < count; i++)
|
||||
{
|
||||
total += reader.ReadString()!.Length;
|
||||
}
|
||||
return total;
|
||||
}
|
||||
|
||||
static byte[] WriteBinaries(byte[] data, int count)
|
||||
{
|
||||
var buffer = new ArrayBufferWriter<byte>();
|
||||
var writer = new MessagePackWriter(buffer);
|
||||
for (int i = 0; i < count; i++)
|
||||
{
|
||||
writer.Write(data);
|
||||
}
|
||||
writer.Flush();
|
||||
return buffer.WrittenSpan.ToArray();
|
||||
}
|
||||
|
||||
// Copies each field into `output`, like the PureBasic test.
|
||||
static long ReadBinaries(byte[] encoded, int count, byte[] output)
|
||||
{
|
||||
var reader = new MessagePackReader(encoded);
|
||||
long total = 0;
|
||||
for (int i = 0; i < count; i++)
|
||||
{
|
||||
var bytes = reader.ReadBytes()!.Value;
|
||||
bytes.CopyTo(output);
|
||||
total += bytes.Length;
|
||||
}
|
||||
return total;
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
# Writes and reads the workload shared with the PureBasic tests, see `_Workload.pbi`.
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import msgpack
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
||||
from _PerfTesting import run
|
||||
|
||||
TOTAL_SIZE = 33_554_432
|
||||
|
||||
|
||||
def write(value, count):
|
||||
packer = msgpack.Packer(autoreset=False)
|
||||
pack = packer.pack
|
||||
for _ in range(count):
|
||||
pack(value)
|
||||
return packer.bytes()
|
||||
|
||||
|
||||
def read(encoded, count):
|
||||
unpacker = msgpack.Unpacker(raw=False, max_buffer_size=len(encoded))
|
||||
unpacker.feed(encoded)
|
||||
unpack = unpacker.unpack
|
||||
total = 0
|
||||
for _ in range(count):
|
||||
total += len(unpack())
|
||||
return total
|
||||
|
||||
|
||||
def run_scenarios(name, value):
|
||||
count = TOTAL_SIZE // len(value)
|
||||
encoded = write(value, count)
|
||||
run(f"{name}-{len(value)}-write", count, lambda: len(write(value, count)))
|
||||
run(f"{name}-{len(value)}-read", count, lambda: read(encoded, count))
|
||||
|
||||
|
||||
for size in (8, 1024, 65536):
|
||||
run_scenarios("str", "a" * size)
|
||||
|
||||
for size in (1024, 1048576):
|
||||
run_scenarios("bin", b"\xab" * size)
|
||||
@@ -0,0 +1,174 @@
|
||||
EnableExplicit
|
||||
|
||||
XIncludeFile "../../Includes/MessagePack.pbi"
|
||||
XIncludeFile "../_PerfTesting.pbi"
|
||||
XIncludeFile "_Workload.pbi"
|
||||
|
||||
|
||||
; Writes and reads the shared workload, see `_Workload.pbi`.
|
||||
|
||||
; Current size, number of fields and string encoding.
|
||||
Global FieldSize.i
|
||||
Global FieldCount.i
|
||||
Global Encoding.i
|
||||
|
||||
; Data written in each field.
|
||||
Global Text.s
|
||||
Global *Data
|
||||
|
||||
; Encoded fields, prepared once for the read scenarios.
|
||||
Global *Encoded
|
||||
|
||||
; Receives each binary field when reading.
|
||||
Global *Output
|
||||
|
||||
|
||||
; Returns the finalized buffer, which must be freed with `FreeMemory()`.
|
||||
; The writer is big enough for all the fields, the reallocations are measured in "MassiveReadAmount".
|
||||
Procedure.i EncodeStrings()
|
||||
Protected *Writer = MsgPackCreate(FieldCount * (FieldSize * 2 + 5))
|
||||
Protected *Buffer
|
||||
Protected i.i
|
||||
|
||||
If Not *Writer
|
||||
PerfFail("Failed to create the writer !")
|
||||
EndIf
|
||||
|
||||
For i = 1 To FieldCount
|
||||
PerfCheckWrite(MsgPackWriteString(*Writer, Text, Encoding))
|
||||
Next
|
||||
|
||||
*Buffer = MsgPackFinalize(*Writer)
|
||||
If Not *Buffer
|
||||
PerfFail("Failed to finalize the writer !")
|
||||
EndIf
|
||||
|
||||
ProcedureReturn *Buffer
|
||||
EndProcedure
|
||||
|
||||
; Returns the finalized buffer, which must be freed with `FreeMemory()`.
|
||||
Procedure.i EncodeBinaries()
|
||||
Protected *Writer = MsgPackCreate(FieldCount * (FieldSize + 5))
|
||||
Protected *Buffer
|
||||
Protected i.i
|
||||
|
||||
If Not *Writer
|
||||
PerfFail("Failed to create the writer !")
|
||||
EndIf
|
||||
|
||||
For i = 1 To FieldCount
|
||||
PerfCheckWrite(MsgPackWriteBinary(*Writer, *Data, FieldSize))
|
||||
Next
|
||||
|
||||
*Buffer = MsgPackFinalize(*Writer)
|
||||
If Not *Buffer
|
||||
PerfFail("Failed to finalize the writer !")
|
||||
EndIf
|
||||
|
||||
ProcedureReturn *Buffer
|
||||
EndProcedure
|
||||
|
||||
Procedure.q BufferSizeAndFree(*Buffer)
|
||||
Protected Size.q = MemorySize(*Buffer)
|
||||
|
||||
FreeMemory(*Buffer)
|
||||
ProcedureReturn Size
|
||||
EndProcedure
|
||||
|
||||
|
||||
Procedure.q ScenarioStringWrite()
|
||||
ProcedureReturn BufferSizeAndFree(EncodeStrings())
|
||||
EndProcedure
|
||||
|
||||
Procedure.q ScenarioStringRead()
|
||||
Protected *Reader = MsgPackWrapMsgBuffer(*Encoded, MemorySize(*Encoded))
|
||||
Protected Total.q
|
||||
Protected i.i
|
||||
|
||||
If Not *Reader
|
||||
PerfFail("Failed to wrap the buffer !")
|
||||
EndIf
|
||||
|
||||
For i = 1 To FieldCount
|
||||
Total + Len(MsgPackReadString(*Reader, Encoding))
|
||||
Next
|
||||
|
||||
; Failed reads don't move the offset, so a failure breaks the next reads too.
|
||||
If MsgPackLastError(*Reader) <> #MsgPack_Error_Success
|
||||
PerfFail("A read failed !")
|
||||
EndIf
|
||||
|
||||
MsgPackFree(*Reader, #False)
|
||||
ProcedureReturn Total
|
||||
EndProcedure
|
||||
|
||||
Procedure.q ScenarioBinaryWrite()
|
||||
ProcedureReturn BufferSizeAndFree(EncodeBinaries())
|
||||
EndProcedure
|
||||
|
||||
Procedure.q ScenarioBinaryRead()
|
||||
Protected *Reader = MsgPackWrapMsgBuffer(*Encoded, MemorySize(*Encoded))
|
||||
Protected Total.q, BytesRead.q
|
||||
Protected i.i
|
||||
|
||||
If Not *Reader
|
||||
PerfFail("Failed to wrap the buffer !")
|
||||
EndIf
|
||||
|
||||
For i = 1 To FieldCount
|
||||
BytesRead = MsgPackReadBinary(*Reader, *Output, FieldSize)
|
||||
If BytesRead = #MsgPack_InvalidSize
|
||||
PerfFail("A read failed !")
|
||||
EndIf
|
||||
Total + BytesRead
|
||||
Next
|
||||
|
||||
MsgPackFree(*Reader, #False)
|
||||
ProcedureReturn Total
|
||||
EndProcedure
|
||||
|
||||
|
||||
Procedure RunStringScenarios(Size.i, StringEncoding.i, Name.s)
|
||||
FieldSize = Size
|
||||
FieldCount = #Perf_TotalSize / Size
|
||||
Encoding = StringEncoding
|
||||
Text = LSet("", Size, "a")
|
||||
*Encoded = EncodeStrings()
|
||||
|
||||
PerfRun("pb", Name + "-" + Str(Size) + "-write", @ScenarioStringWrite(), FieldCount)
|
||||
PerfRun("pb", Name + "-" + Str(Size) + "-read", @ScenarioStringRead(), FieldCount)
|
||||
|
||||
FreeMemory(*Encoded)
|
||||
EndProcedure
|
||||
|
||||
Procedure RunBinaryScenarios(Size.i)
|
||||
FieldSize = Size
|
||||
FieldCount = #Perf_TotalSize / Size
|
||||
*Data = AllocateMemory(Size)
|
||||
*Output = AllocateMemory(Size)
|
||||
If Not *Data Or Not *Output
|
||||
PerfFail("Failed to allocate the buffers !")
|
||||
EndIf
|
||||
FillMemory(*Data, Size, $AB)
|
||||
*Encoded = EncodeBinaries()
|
||||
|
||||
PerfRun("pb", "bin-" + Str(Size) + "-write", @ScenarioBinaryWrite(), FieldCount)
|
||||
PerfRun("pb", "bin-" + Str(Size) + "-read", @ScenarioBinaryRead(), FieldCount)
|
||||
|
||||
FreeMemory(*Encoded)
|
||||
FreeMemory(*Output)
|
||||
FreeMemory(*Data)
|
||||
EndProcedure
|
||||
|
||||
|
||||
OpenConsole()
|
||||
|
||||
RunStringScenarios(8, #PB_UTF8, "str")
|
||||
RunStringScenarios(1024, #PB_UTF8, "str")
|
||||
RunStringScenarios(65536, #PB_UTF8, "str")
|
||||
RunStringScenarios(1024, #PB_Unicode, "str-utf16")
|
||||
|
||||
RunBinaryScenarios(1024)
|
||||
RunBinaryScenarios(1048576)
|
||||
|
||||
End 0
|
||||
@@ -0,0 +1,18 @@
|
||||
$ErrorActionPreference = "Stop"
|
||||
|
||||
# Shows the error instead of closing the window right away.
|
||||
trap {
|
||||
Write-Host $_ -ForegroundColor Red
|
||||
Pause
|
||||
exit 1
|
||||
}
|
||||
|
||||
. (Join-Path $PSScriptRoot "..\_PerfTesting.ps1")
|
||||
|
||||
# Requires Python with the packages from 'PerfTests\requirements.txt', and the .NET 10 SDK.
|
||||
$Rows = @(Invoke-PureBasicPerfTests $PSScriptRoot)
|
||||
$Rows += Invoke-PythonPerfTest (Join-Path $PSScriptRoot "PerfPython.py")
|
||||
$Rows += Invoke-DotNetPerfTest (Join-Path $PSScriptRoot "DotNet")
|
||||
|
||||
Show-PerfResults $Rows $PSScriptRoot
|
||||
Pause
|
||||
@@ -0,0 +1,14 @@
|
||||
; Workload shared by every implementation in this folder, keep `PerfPython.py` and `DotNet/Program.cs` in sync.
|
||||
;
|
||||
; For each size, `#Perf_TotalSize / Size` fields of `Size` bytes are written, then read:
|
||||
; * Strings of 8, 1024 and 65536 "a" characters, encoded as UTF-8.
|
||||
; * Binaries of 1024 and 1048576 bytes set to `$AB`.
|
||||
; The PureBasic test also writes and reads strings of 1024 characters as UTF-16, which the others don't support.
|
||||
;
|
||||
; Scenarios are named `str-<Size>-write`, `str-<Size>-read`, `bin-<Size>-write` and `bin-<Size>-read`.
|
||||
; `NsPerOp` is the time per field.
|
||||
;
|
||||
; Write check: the encoded size.
|
||||
; Read check: the total number of characters or bytes read.
|
||||
|
||||
#Perf_TotalSize = 33554432
|
||||
@@ -0,0 +1,23 @@
|
||||
// Shared by the .NET performance tests, see `_PerfTesting.pbi` for the output format.
|
||||
using System.Diagnostics;
|
||||
|
||||
static class Perf
|
||||
{
|
||||
const int Runs = 5;
|
||||
|
||||
// Runs `function` once to warm up, then `Runs` times, and prints its median.
|
||||
public static void Run(string scenario, long operations, Func<long> function)
|
||||
{
|
||||
long check = function();
|
||||
var times = new double[Runs];
|
||||
for (int i = 0; i < Runs; i++)
|
||||
{
|
||||
var stopwatch = Stopwatch.StartNew();
|
||||
check = function();
|
||||
times[i] = stopwatch.Elapsed.TotalMilliseconds;
|
||||
}
|
||||
Array.Sort(times);
|
||||
double median = times[Runs / 2];
|
||||
Console.WriteLine($"dotnet,{scenario},{median:F1},{median * 1_000_000 / operations:F1},{check}");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
; Shared by the PureBasic performance tests.
|
||||
;
|
||||
; Each scenario is run once to warm up, then `#Perf_Runs` times, and its median is printed as:
|
||||
; `Implementation,Scenario,MedianMs,NsPerOp,Check`
|
||||
; `Check` is returned by the scenario, e.g. an encoded size or a checksum, to compare implementations.
|
||||
|
||||
#Perf_Runs = 5
|
||||
|
||||
; Times are measured with `QueryPerformanceCounter_()`, `ElapsedMilliseconds()` is too coarse.
|
||||
CompilerIf #PB_Compiler_OS <> #PB_OS_Windows
|
||||
CompilerError "The performance tests only support Windows !"
|
||||
CompilerEndIf
|
||||
|
||||
Prototype.q PerfScenario()
|
||||
|
||||
|
||||
; Errors go to stderr, stdout only holds the results.
|
||||
Procedure PerfFail(Message.s)
|
||||
ConsoleError("FAILED: " + Message)
|
||||
End 1
|
||||
EndProcedure
|
||||
|
||||
Macro PerfCheckWrite(ErrorCode)
|
||||
If ErrorCode <> #MsgPack_Error_Success
|
||||
PerfFail("A write failed !")
|
||||
EndIf
|
||||
EndMacro
|
||||
|
||||
|
||||
; `Operations` is the number of operations done by one run of `Function`, used for `NsPerOp`.
|
||||
Procedure PerfRun(Implementation.s, Scenario.s, Function.PerfScenario, Operations.q)
|
||||
Protected Dim Times.d(#Perf_Runs - 1)
|
||||
Protected Check.q, Frequency.q, Start.q, Stop.q
|
||||
Protected Median.d
|
||||
Protected i.i
|
||||
|
||||
QueryPerformanceFrequency_(@Frequency)
|
||||
|
||||
Check = Function()
|
||||
|
||||
For i = 0 To #Perf_Runs - 1
|
||||
QueryPerformanceCounter_(@Start)
|
||||
Check = Function()
|
||||
QueryPerformanceCounter_(@Stop)
|
||||
Times(i) = (Stop - Start) * 1000.0 / Frequency
|
||||
Next
|
||||
|
||||
SortArray(Times(), #PB_Sort_Ascending)
|
||||
Median = Times(#Perf_Runs / 2)
|
||||
|
||||
PrintN(Implementation + "," + Scenario + "," + StrD(Median, 1) + "," + StrD(Median * 1000000.0 / Operations, 1) + "," + Str(Check))
|
||||
EndProcedure
|
||||
@@ -0,0 +1,132 @@
|
||||
# Shared by the 'RunPerfTests.ps1' scripts, dot-source it.
|
||||
|
||||
# PureBasic versions, with the compiler executable of each architecture and backend.
|
||||
# 6.21 is kept to compare with the string handling changes of 6.40.
|
||||
# No x86 ASM backend, its 'FAsm.exe' triggers the AV and crashes anyway.
|
||||
$Compilers = @{
|
||||
"6.41" = @{
|
||||
"x86" = @{ Path = "C:\Program Files (x86)\PureBasic_641"; Backends = @{ "C" = "pbcompilerc.exe" } }
|
||||
"x64" = @{ Path = "C:\Program Files\PureBasic_641"; Backends = @{ "ASM" = "pbcompiler.exe"; "C" = "pbcompilerc.exe" } }
|
||||
"arm64" = @{ Path = "C:\Program Files\PureBasic_641_arm64\PureBasic"; Backends = @{ "C" = "pbcompiler.exe" } }
|
||||
}
|
||||
"6.21" = @{
|
||||
"x86" = @{ Path = "C:\Program Files (x86)\PureBasic_621"; Backends = @{ "C" = "pbcompilerc.exe" } }
|
||||
"x64" = @{ Path = "C:\Program Files\PureBasic_621"; Backends = @{ "ASM" = "pbcompiler.exe"; "C" = "pbcompilerc.exe" } }
|
||||
}
|
||||
}
|
||||
|
||||
$PerfHeader = @("Implementation", "Scenario", "MedianMs", "NsPerOp", "Check")
|
||||
|
||||
# Where `Show-PerfResults` saves the results.
|
||||
$PerfResultsDir = Join-Path $PSScriptRoot "Results"
|
||||
|
||||
# Prints each result line as it arrives, and passes it along.
|
||||
filter Show-PerfLine {
|
||||
Write-Host " $_"
|
||||
$_
|
||||
}
|
||||
|
||||
# Prompts for a choice and used `&` for the hotkey.
|
||||
function Select-Choice([string]$Title, [string[]]$Labels) {
|
||||
$Choices = $Labels | ForEach-Object { New-Object System.Management.Automation.Host.ChoiceDescription $_ }
|
||||
$Index = $Host.UI.PromptForChoice($Title, "", $Choices, 0)
|
||||
return $Labels[$Index].Replace("&", "")
|
||||
}
|
||||
|
||||
# Prompts for a PureBasic version, architecture and backend, then builds and runs every '.pb' file in '$Directory'.
|
||||
# They're built without the debugger, and with the C optimizer, like a release would be.
|
||||
# Returns the parsed result lines.
|
||||
function Invoke-PureBasicPerfTests([string]$Directory) {
|
||||
$Version = Select-Choice "Version" @("6.&41", "6.&21")
|
||||
$Architecture = Select-Choice "Architecture" @("x&86", "x&64", "&arm64")
|
||||
$Backend = Select-Choice "Backend" @("&ASM", "&C")
|
||||
|
||||
# Used to name the results file.
|
||||
$script:PerfTarget = "${Version}_$Architecture-$Backend"
|
||||
|
||||
$Compiler = $Compilers[$Version][$Architecture]
|
||||
if (-not $Compiler -or -not $Compiler.Backends.Contains($Backend)) {
|
||||
throw "No compiler configured for PureBasic $Version on $Architecture with the $Backend backend."
|
||||
}
|
||||
$CompilerExe = Join-Path $Compiler.Path "Compilers\$($Compiler.Backends[$Backend])"
|
||||
if (-not (Test-Path $CompilerExe)) {
|
||||
throw "Compiler not found: $CompilerExe"
|
||||
}
|
||||
|
||||
$CompilerFlags = @("/CONSOLE")
|
||||
if ($Backend -eq "C") {
|
||||
$CompilerFlags += "/OPTIMIZER"
|
||||
}
|
||||
|
||||
$Rows = @()
|
||||
|
||||
# Built in the repo, outside of any temp folder, and deleted afterward.
|
||||
$BuildDir = Join-Path $Directory "Build"
|
||||
New-Item -ItemType Directory -Force -Path $BuildDir | Out-Null
|
||||
|
||||
try {
|
||||
foreach ($PerfTest in Get-ChildItem -Path $Directory -Filter "*.pb") {
|
||||
$Exe = Join-Path $BuildDir "$($PerfTest.BaseName).exe"
|
||||
|
||||
$CompilerOutput = & $CompilerExe $PerfTest.FullName /EXE $Exe @CompilerFlags | Out-String
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "Compilation failed: $($PerfTest.Name)`n$CompilerOutput"
|
||||
}
|
||||
|
||||
Write-Host "> $($PerfTest.Name) - PureBasic $Version - $Architecture - $Backend"
|
||||
$Rows += & $Exe | Show-PerfLine | ConvertFrom-Csv -Header $PerfHeader
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "Failed: $($PerfTest.Name) (exit code $LASTEXITCODE)"
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
Remove-Item -Recurse -Force $BuildDir
|
||||
}
|
||||
|
||||
return $Rows
|
||||
}
|
||||
|
||||
# Runs a Python perf test, requires the packages from 'PerfTests\requirements.txt'.
|
||||
# Returns the parsed result lines.
|
||||
function Invoke-PythonPerfTest([string]$Script) {
|
||||
Write-Host "> $(Split-Path $Script -Leaf)"
|
||||
$Rows = @(& python $Script | Show-PerfLine | ConvertFrom-Csv -Header $PerfHeader)
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "Failed: $Script, install its packages with 'pip install -r PerfTests\requirements.txt'"
|
||||
}
|
||||
return $Rows
|
||||
}
|
||||
|
||||
# Builds and runs a .NET perf test project, requires the .NET 10 SDK.
|
||||
# Returns the parsed result lines.
|
||||
function Invoke-DotNetPerfTest([string]$Project) {
|
||||
Write-Host "> $(Split-Path $Project -Leaf)"
|
||||
& dotnet build $Project -c Release --nologo -v quiet | Out-Host
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "Compilation failed: $Project"
|
||||
}
|
||||
$Rows = @(& dotnet run --project $Project -c Release --no-build | Show-PerfLine | ConvertFrom-Csv -Header $PerfHeader)
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "Failed: $Project (exit code $LASTEXITCODE)"
|
||||
}
|
||||
return $Rows
|
||||
}
|
||||
|
||||
# Prints the results, reports scenarios whose checks differ between implementations,
|
||||
# and saves them as 'Results\<Directory's name>_<Version>_<Architecture>-<Backend>.csv', replacing the previous ones.
|
||||
function Show-PerfResults([object[]]$Rows, [string]$Directory) {
|
||||
$Rows | Format-Table -AutoSize | Out-Host
|
||||
|
||||
# Every implementation must have encoded the same bytes and read the same values.
|
||||
foreach ($Group in $Rows | Group-Object Scenario) {
|
||||
$Checks = @($Group.Group | ForEach-Object { $_.Check } | Select-Object -Unique)
|
||||
if ($Checks.Count -ne 1) {
|
||||
Write-Host "Mismatching checks for '$($Group.Name)': $($Checks -join ', ')" -ForegroundColor Red
|
||||
}
|
||||
}
|
||||
|
||||
New-Item -ItemType Directory -Force -Path $PerfResultsDir | Out-Null
|
||||
$ResultsFile = Join-Path $PerfResultsDir "$(Split-Path $Directory -Leaf)_$PerfTarget.csv"
|
||||
$Rows | Export-Csv -Path $ResultsFile -NoTypeInformation
|
||||
Write-Host "Results saved to $ResultsFile"
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
# Shared by the Python performance tests, see `_PerfTesting.pbi` for the output format.
|
||||
import statistics
|
||||
import time
|
||||
|
||||
RUNS = 5
|
||||
|
||||
|
||||
def run(scenario, operations, function):
|
||||
"""Runs `function` once to warm up, then `RUNS` times, and prints its median."""
|
||||
check = function()
|
||||
times = []
|
||||
for _ in range(RUNS):
|
||||
start = time.perf_counter()
|
||||
check = function()
|
||||
times.append((time.perf_counter() - start) * 1000)
|
||||
median = statistics.median(times)
|
||||
print(f"python,{scenario},{median:.1f},{median * 1_000_000 / operations:.1f},{check}", flush=True)
|
||||
@@ -0,0 +1,117 @@
|
||||
# PureBasic - MsgPack - Performances
|
||||
Performance benchmarks for the [PB-MessagePack](../readme.md) include.
|
||||
|
||||
Please refer to the [Performances](../readme.md#performances) section for the warnings and conclusions. \
|
||||
Open questions raised by these results are tracked in [Investigations.md](Investigations.md).
|
||||
|
||||
|
||||
## Workloads
|
||||
Each folder has a `RunPerfTests.ps1` script that runs its benchmarks:
|
||||
* `MassiveReadAmount/` - 1M small fields written, read and skipped, with explicit and implicit typing.
|
||||
* `StringsAndBinary/` - Strings of 8 B to 64 KB, and binaries of 1 KB and 1 MB, written and read.
|
||||
* `FlatMaps/` - 100K maps of 16 entries written and read.
|
||||
* `DeepNesting/` - Arrays nested 1 to 128 levels deep, built, examined and measured. (PureBasic only)
|
||||
* `NullChecks/` - Small fields written and read with and without `#MsgPack_DisableNullChecks`. (PureBasic only)
|
||||
|
||||
The first three are compared to [msgpack](https://pypi.org/project/msgpack/) (Python)
|
||||
and [MessagePack-CSharp](https://github.com/MessagePack-CSharp/MessagePack-CSharp) (.NET).
|
||||
|
||||
The scripts prompt for the PureBasic version, architecture and backend. \
|
||||
The results are saved in `Results/`, as one CSV per folder, PureBasic version, architecture and backend.
|
||||
|
||||
|
||||
## Results
|
||||
Hardware:
|
||||
* Proxmox VM running Windows 10
|
||||
* 10 Intel Core i5-13500 cores **(No E/P core pinning)**
|
||||
* PureBasic 6.41
|
||||
* Python 3.14 with [msgpack](https://pypi.org/project/msgpack/) 1.2.3
|
||||
* .NET 10 with [MessagePack-CSharp](https://github.com/MessagePack-CSharp/MessagePack-CSharp) 3.1.12
|
||||
|
||||
Times are in nanoseconds per field, per map or per nested structure, lower is better. \
|
||||
Every implementation produced the same bytes and read back the same values. \
|
||||
PureBasic 6.41 is used everywhere, except in the [last table](#purebasic-621-vs-641).
|
||||
|
||||
### Compared to other implementations
|
||||
x64 with the C backend, the fastest of each row in bold.
|
||||
|
||||
| Workload | Op | PureBasic | Python | .NET |
|
||||
|:--|:--|--:|--:|--:|
|
||||
| 1M small fields <sup>1</sup> | Write | 11.9 | 50.9 | **5.4** |
|
||||
| | Read | 12.7 | 52.4 | **10.6** |
|
||||
| | Skip | **18.3** | 40.2 | 29.4 |
|
||||
| Strings, 8 B | Write | 59.6 | 62.0 | **13.2** |
|
||||
| | Read | 46.6 | 73.4 | **16.2** |
|
||||
| Strings, 1 KB | Write | 720.9 | **556.0** | 804.8 |
|
||||
| | Read | 570.0 | 341.3 | **150.0** |
|
||||
| Strings, 64 KB | Write | 45,917 | **33,932** | 39,848 |
|
||||
| | Read | 31,473 | **16,390** | 23,045 |
|
||||
| Binary, 1 KB | Write | **190.6** | 556.6 | 576.1 |
|
||||
| | Read | **50.2** | 311.3 | 204.0 |
|
||||
| Binary, 1 MB | Write | **207,744** | 574,825 | 438,025 |
|
||||
| | Read | **54,812** | 465,075 | 59,459 |
|
||||
| Maps of 16 entries | Write | 1,793 | 927.2 | **209.2** |
|
||||
| | Read | 1,858 | 1,962 | **292.1** |
|
||||
|
||||
<sup>1</sup> 1M fields of mixed integers, floats and short strings, with implicit typing. PureBasic writes into a preallocated buffer.
|
||||
|
||||
### Architectures & backends
|
||||
PureBasic only.
|
||||
|
||||
| Scenario | x64 C | x64 ASM | x86 C |
|
||||
|:--|--:|--:|--:|
|
||||
| Small fields, explicit write | 11.0 | 18.0 | 13.9 |
|
||||
| Small fields, explicit read | 9.8 | 15.9 | 8.8 |
|
||||
| Small fields, implicit write | 11.9 | 25.4 | 14.2 |
|
||||
| Small fields, implicit read | 12.7 | 25.3 | 11.5 |
|
||||
| Small fields, skip | 18.3 | 57.9 | 23.2 |
|
||||
| Small fields, write, 64 KB growth <sup>2</sup> | 37.9 | 51.5 | 44.7 |
|
||||
| Small fields, write, exact growth <sup>2</sup> | 441.0 | 461.6 | 503.3 |
|
||||
| Small fields, read, null checks off <sup>3</sup> | 11.4 | 19.5 | 11.4 |
|
||||
| Small fields, read, null checks on <sup>3</sup> | 12.2 | 20.2 | 11.4 |
|
||||
| Strings, 1 KB UTF-16, write | 526.0 | 539.1 | 599.0 |
|
||||
| Strings, 1 KB UTF-16, read | 282.7 | 298.8 | 289.1 |
|
||||
| Maps of 16 entries, write | 1,793 | 3,472 | 2,189 |
|
||||
| Maps of 16 entries, read | 1,858 | 3,232 | 1,782 |
|
||||
|
||||
<sup>2</sup> The buffer starts at 64 bytes and grows in multiples of 64 KB, or by the exact size of each write. (`GrowthIncrement` of `1`) \
|
||||
<sup>3</sup> With and without `#MsgPack_DisableNullChecks`.
|
||||
|
||||
### Deep nesting
|
||||
x64 with the C backend, arrays of 3 integers nested `N` levels deep.
|
||||
|
||||
| Depth | Build | Examine | Examine, unsafe | `MsgPackGetFieldFullSize` |
|
||||
|--:|--:|--:|--:|--:|
|
||||
| 1 | 283.3 | 177.8 | 134.9 | 80.4 |
|
||||
| 2 | 453.4 | 363.2 | 261.4 | 115.8 |
|
||||
| 4 | 1,002 | 825.2 | 464.1 | 177.3 |
|
||||
| 8 | 2,226 | 2,154 | 906.9 | 308.9 |
|
||||
| 16 | 6,414 | 6,155 | 1,807 | 546.5 |
|
||||
| 32 | 21,888 | 21,134 | 4,367 | 985.9 |
|
||||
| 64 | 76,370 | 72,715 | 7,774 | 1,997 |
|
||||
| 128 | 278,388 | 269,656 | 15,803 | 3,812 |
|
||||
|
||||
### PureBasic 6.21 vs 6.41
|
||||
x64 with the C backend, the new string handling of 6.40 being the main difference.
|
||||
|
||||
| Scenario | 6.21 | 6.41 | Speedup |
|
||||
|:--|--:|--:|--:|
|
||||
| Small fields, implicit write <sup>4</sup> | 23.0 | 11.9 | 1.9x |
|
||||
| Small fields, implicit read <sup>4</sup> | 26.4 | 12.7 | 2.1x |
|
||||
| Small fields, skip | 14.7 | 18.3 | 0.8x |
|
||||
| Strings, 8 B, write | 162.8 | 59.6 | 2.7x |
|
||||
| Strings, 8 B, read | 164.5 | 46.6 | 3.5x |
|
||||
| Strings, 1 KB, write | 1,971 | 720.9 | 2.7x |
|
||||
| Strings, 1 KB, read | 1,412 | 570.0 | 2.5x |
|
||||
| Strings, 64 KB, write | 113,872 | 45,917 | 2.5x |
|
||||
| Strings, 64 KB, read | 77,056 | 31,473 | 2.4x |
|
||||
| Strings, 1 KB UTF-16, write | 2,026 | 526.0 | 3.9x |
|
||||
| Strings, 1 KB UTF-16, read | 1,065 | 282.7 | 3.8x |
|
||||
| Binary, 1 KB, write | 173.4 | 190.6 | 0.9x |
|
||||
| Binary, 1 KB, read | 41.2 | 50.2 | 0.8x |
|
||||
| Maps of 16 entries, write <sup>4</sup> | 3,221 | 1,793 | 1.8x |
|
||||
| Maps of 16 entries, read <sup>4</sup> | 3,478 | 1,858 | 1.9x |
|
||||
| Nested 128 levels, examine | 263,158 | 269,656 | 1.0x |
|
||||
|
||||
<sup>4</sup> Includes strings, one per record of 8 fields, and one per map key. \
|
||||
The x64 ASM and x86 C results show the same 2 to 3 times speedup on strings.
|
||||
@@ -0,0 +1,96 @@
|
||||
$ErrorActionPreference = "Stop"
|
||||
|
||||
|
||||
# PureBasic versions used for unit tests, with the compiler executable of each backend.
|
||||
$Compilers = [ordered]@{
|
||||
"x86" = @(
|
||||
@{ Name = "5.73"; Path = "C:\Program Files (x86)\PureBasic_573"; Backends = @{ "ASM" = "pbcompiler.exe" } },
|
||||
@{ Name = "6.21"; Path = "C:\Program Files (x86)\PureBasic_621"; Backends = @{ "ASM" = "pbcompiler.exe"; "C" = "pbcompilerc.exe" } },
|
||||
@{ Name = "6.40"; Path = "C:\Program Files (x86)\PureBasic_640"; Backends = @{ "ASM" = "pbcompiler.exe"; "C" = "pbcompilerc.exe" } },
|
||||
# No ASM backend, its 'FAsm.exe' triggers the AV and crashes anyway.
|
||||
@{ Name = "6.41"; Path = "C:\Program Files (x86)\PureBasic_641"; Backends = @{ "C" = "pbcompilerc.exe" } }
|
||||
)
|
||||
"x64" = @(
|
||||
@{ Name = "5.73"; Path = "C:\Program Files\PureBasic_573"; Backends = @{ "ASM" = "pbcompiler.exe" } },
|
||||
@{ Name = "6.21"; Path = "C:\Program Files\PureBasic_621"; Backends = @{ "ASM" = "pbcompiler.exe"; "C" = "pbcompilerc.exe" } },
|
||||
@{ Name = "6.40"; Path = "C:\Program Files\PureBasic_640"; Backends = @{ "ASM" = "pbcompiler.exe"; "C" = "pbcompilerc.exe" } },
|
||||
@{ Name = "6.41"; Path = "C:\Program Files\PureBasic_641"; Backends = @{ "ASM" = "pbcompiler.exe"; "C" = "pbcompilerc.exe" } }
|
||||
)
|
||||
"arm64" = @(
|
||||
@{ Name = "6.21"; Path = "C:\Program Files\PureBasic_621_arm64"; Backends = @{ "C" = "pbcompiler.exe" } },
|
||||
@{ Name = "6.41"; Path = "C:\Program Files\PureBasic_641_arm64\PureBasic"; Backends = @{ "C" = "pbcompiler.exe" } }
|
||||
)
|
||||
}
|
||||
|
||||
# Prompts for a choice and used `&` for the hotkey.
|
||||
function Select-Choice([string]$Title, [string[]]$Labels) {
|
||||
$Choices = $Labels | ForEach-Object { New-Object System.Management.Automation.Host.ChoiceDescription $_ }
|
||||
$Index = $Host.UI.PromptForChoice($Title, "", $Choices, 0)
|
||||
return $Labels[$Index].Replace("&", "")
|
||||
}
|
||||
|
||||
|
||||
$Architecture = Select-Choice "Architecture" @("x&86", "x&64", "&arm64")
|
||||
$Backend = Select-Choice "Backend" @("&ASM", "&C")
|
||||
|
||||
# Sets '#Endianness_UseGccBuiltins' in the PB-Endianness include, only used by the C backend.
|
||||
$UseGccBuiltins = "False"
|
||||
if ($Backend -eq "C") {
|
||||
$UseGccBuiltins = Select-Choice "Use GCC builtins" @("&False", "&True")
|
||||
}
|
||||
$UseGccBuiltinsValue = @{ "False" = 0; "True" = 1 }[$UseGccBuiltins]
|
||||
|
||||
$SelectedCompilers = @($Compilers[$Architecture] | Where-Object { $_.Backends.Contains($Backend) })
|
||||
if ($SelectedCompilers.Count -eq 0) {
|
||||
throw "No compilers configured for $Architecture with the $Backend backend."
|
||||
}
|
||||
|
||||
$UnitTests = Get-ChildItem -Path (Join-Path $PSScriptRoot "UnitTests") -Recurse -Filter "*.pb"
|
||||
|
||||
# Built in the repo, outside of any temp folder, and deleted afterward.
|
||||
$BuildDir = Join-Path $PSScriptRoot "Build"
|
||||
New-Item -ItemType Directory -Force -Path $BuildDir | Out-Null
|
||||
|
||||
$FailedCount = 0
|
||||
|
||||
try {
|
||||
foreach ($Compiler in $SelectedCompilers) {
|
||||
$CompilerExe = Join-Path $Compiler.Path "Compilers\$($Compiler.Backends[$Backend])"
|
||||
if (-not (Test-Path $CompilerExe)) {
|
||||
throw "Compiler not found: $CompilerExe"
|
||||
}
|
||||
|
||||
Write-Host "`n=== PureBasic $($Compiler.Name) - $Architecture - $Backend - GCC builtins: $UseGccBuiltins ===" -ForegroundColor Cyan
|
||||
|
||||
foreach ($UnitTest in $UnitTests) {
|
||||
$Exe = Join-Path $BuildDir "$($UnitTest.BaseName).exe"
|
||||
|
||||
# Built without the debugger, the 'FatalErrors' tests rely on it.
|
||||
$CompilerOutput = & $CompilerExe $UnitTest.FullName /CONSOLE /EXE $Exe /CONSTANT "Endianness_UseGccBuiltins=$UseGccBuiltinsValue" | Out-String
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "Compilation failed: $($UnitTest.Name) with PureBasic $($Compiler.Name)`n$CompilerOutput"
|
||||
}
|
||||
|
||||
Write-Host "> $($UnitTest.Directory.Name)\$($UnitTest.Name)"
|
||||
& $Exe
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
Write-Host "FAILED (exit code $LASTEXITCODE)" -ForegroundColor Red
|
||||
$FailedCount++
|
||||
} else {
|
||||
Write-Host "PASSED" -ForegroundColor Green
|
||||
}
|
||||
|
||||
Remove-Item -Force $Exe
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
Remove-Item -Recurse -Force $BuildDir
|
||||
}
|
||||
|
||||
Write-Host ""
|
||||
if ($FailedCount -ne 0) {
|
||||
Write-Host "$FailedCount test(s) failed." -ForegroundColor Red
|
||||
} else {
|
||||
Write-Host "All tests passed." -ForegroundColor Green
|
||||
}
|
||||
Pause
|
||||
@@ -14,15 +14,19 @@ Based on the [msgpack/msgpack](https://github.com/msgpack/msgpack) specification
|
||||
* Explicit or implicit data formats when reading/writing
|
||||
* *ReadInteger vs ReadI8/ReadU8/...*
|
||||
* ~~Fully documented~~
|
||||
* Supports all modern PureBasic setups
|
||||
* x86/x64/arm64
|
||||
* asm/c backends
|
||||
* 5.73 to 6.41
|
||||
|
||||
|
||||
## Limitations
|
||||
* No custom `malloc`/`realloc`/`free` callbacks
|
||||
* More involved Map and Array interactions
|
||||
* *No dynamic typing*
|
||||
* *Building arrays/maps requires special insertion in parent MsgPack ~~(See examples)~~*
|
||||
* *Building arrays/maps requires special insertion in parent MsgPack (See [examples](#arrays--maps))*
|
||||
* *Lots of internal checks for boundaries and counts before any reading can be done*
|
||||
* *[TODO: Note about deep nesting and performances]*
|
||||
* *Deep nesting costs more time, see [Performances](#performances)*
|
||||
* Usage of larger `.q` type for field sizes
|
||||
* *Helps prevent unsigned numbers causing issues on x86*
|
||||
* Particular handling of signaling `NaN` for *Float32*
|
||||
@@ -45,11 +49,67 @@ Pull latest submodule version:
|
||||
git submodule update --init --recursive
|
||||
```
|
||||
|
||||
TODO: Add note about handling pre-inclusion of PB-Endianness
|
||||
|
||||
|
||||
### Inclusion
|
||||
...
|
||||
Include the main file, it pulls in everything else, [PB-Endianness](https://github.com/aziascreations/PB-Endianness) included:
|
||||
```purebasic
|
||||
XIncludeFile "Includes/PB-MessagePack/Includes/MessagePack.pbi"
|
||||
```
|
||||
|
||||
The options below can be defined before the inclusion:
|
||||
* `#MsgPack_MaxDepth` - Maximum nesting depth of arrays and maps, from `1` to `4096`. (Default: `128`)
|
||||
* `#MsgPack_DisableNullChecks` - Skips the `#Null` pointer checks when set to `#True`. (Default: `#False`)
|
||||
|
||||
### Double import safeguard
|
||||
`Endianness.pbi` raises a `CompilerError` if it is included twice from different paths,
|
||||
to prevent two versions of it from being mixed.
|
||||
|
||||
If your project, or another include, already uses its own copy of PB-Endianness,
|
||||
include it **before** `MessagePack.pbi`, which will then skip its own copy:
|
||||
```purebasic
|
||||
XIncludeFile "Includes/PB-Endianness/Includes/Endianness.pbi"
|
||||
XIncludeFile "Includes/PB-MessagePack/Includes/MessagePack.pbi"
|
||||
```
|
||||
|
||||
Including it afterward will trigger the error, unless `#Endianness_BypassIncludeCheck` is defined.
|
||||
|
||||
|
||||
## Performances
|
||||
|
||||
> [!WARNING]
|
||||
> Parts of the library can be much slower than C# and Python.
|
||||
>
|
||||
> This is partially due to many security checks, and the rest is actively being investigated.
|
||||
> We're also in the early stages of development, so there's also that...
|
||||
|
||||
### Security trade-offs
|
||||
Arrays and maps are always validated before being used:
|
||||
* `MsgPackExamine` checks the whole array/map before returning a view, so its content is read twice. \
|
||||
Examining a nested array/map checks its content again, so data nested `N` levels deep can be read up to `N+1` times.
|
||||
* `MsgPackWriteArray/Map` and their variants check and count the given fields, then copy them. \
|
||||
Building nested arrays/maps copies the inner data once per level.
|
||||
* `MsgPackGetFieldFullSize`, `MsgPackSkip` and `MsgPackNextEntry` walk through every contained field, they don't jump over them.
|
||||
|
||||
This is mostly noticeable with deeply nested data, which is why `#MsgPack_MaxDepth` defaults to `128` to prevent wasting time on hostile data.
|
||||
|
||||
`#MsgPack_ExaminationOptions_Unsafe` skips the check done by `MsgPackExamine`, and it can be used on non-root iterable values. \
|
||||
However, using on an iterable element at the root means invalid data is only detected while iterating !
|
||||
|
||||
|
||||
### Current status
|
||||
* Binary data and skipping fields is faster than in msgpack and MessagePack-CSharp.
|
||||
* Small fields, short strings and maps are slower than in MessagePack-CSharp.
|
||||
* Writing maps is slower that C# and Python because of the checks mentionned above.
|
||||
* Strings are about 4 times slower to write and up to 11 times slower to read than binary data of the same size.
|
||||
* *Currently being [investigated](PerfTests/Investigations.md).*
|
||||
* The new strings library in PureBasic 6.40+ is 2.5 to 4 times faster than older versions.
|
||||
* Using the C backend instead of the ASM one is up to 3 times faster.
|
||||
* Preallocate the buffer or using a bigger `GrowthIncrement` than the default of `1` makes small writes up to 35 times faster.
|
||||
* `#MsgPack_DisableNullChecks` only saves about 5%.
|
||||
* Examining nested data gets quadratically slower with the depth
|
||||
* *It is up to 17 times slower when not using `#MsgPack_ExaminationOptions_Unsafe` at 128 levels.*
|
||||
|
||||
The detailed results, workloads and how to run them are in [PerfTests/readme.md](PerfTests/readme.md).
|
||||
|
||||
|
||||
## Examples
|
||||
@@ -58,6 +118,19 @@ TODO: Add note about handling pre-inclusion of PB-Endianness
|
||||
* [BasicExplicitTyping.pb](Examples/BasicExplicitTyping/BasicExplicitTyping.pb)
|
||||
* [BasicImplicitTyping.pb](Examples/BasicImplicitTyping/BasicImplicitTyping.pb)
|
||||
|
||||
### Data types
|
||||
* [BasicStrings.pb](Examples/BasicStrings/BasicStrings.pb)
|
||||
* [BasicBinary.pb](Examples/BasicBinary/BasicBinary.pb)
|
||||
* [BasicExtensions.pb](Examples/BasicExtensions/BasicExtensions.pb)
|
||||
|
||||
### Arrays & Maps
|
||||
* [BasicArrays.pb](Examples/BasicArrays/BasicArrays.pb)
|
||||
* [BasicMaps.pb](Examples/BasicMaps/BasicMaps.pb)
|
||||
|
||||
### Dumping a buffer
|
||||
* [DumpExplicitTypes.pb](Examples/DumpExplicitTypes/DumpExplicitTypes.pb) - Exact format code of each field
|
||||
* [DumpImplicitTypes.pb](Examples/DumpImplicitTypes/DumpImplicitTypes.pb) - General data type of each field
|
||||
|
||||
|
||||
## Cloning
|
||||
Use this command to clone the repository and its submodules:
|
||||
|
||||
Reference in New Issue
Block a user