Store
Encoding

Encoding

Store uses a custom encoding scheme to store data more compactly than Solidity. It is comparable to abi.encodePacked, but with some notable differences:

  • Array elements are tightly packed, without any padding. This might cause some elements to wrap around two storage slots, but saves a lot of space. For example, an address[3] array will only use 2 storage slots in MUD instead of 3 in Solidity, leading to a 33% reduction in storage costs.
  • Array lengths are packed into a single slot. Tables are limited to up to 5 dynamic fields, and the length of each field is limited to 240 (over a million millions), we can pack the lengths of all dynamic length fields of a table into a single storage slot. Compared to Solidity, this saves 1 storage slot per dynamic length element except for the first one.

This encoding scheme greatly reduces gas for storage operations on dynamic length fields and for emitting events. For events it reduces the payload size by over 80% compared to the Solidity default of abi.encode (depending on the table schema). Note that since the encoding happens at runtime and does not have access to compiler internals, there is some overhead for encoding and decoding that is not present in vanilla Solidity.

Schema

The schema defines the data type of every field in the record, whether it is part of the key used to look the record up, or the value that the table provides.

To save gas, onchain the schema definition is divided between the key schema and the value schema. Both definitions use the same data structure, a bytes32 value that encodes the total byte length of all static length fields, the number of static-length fields, the number of dynamic-length fields, and up to 28 column types.

Byte(s)ValueTypeConstraint
0-1Total length of static fieldsuint16
2Number of static length fieldsuint8≤ (28 - number of dynamic length fields)
3Number of dynamic length fieldsuint8For the key schema, 0
For the value schema, ≤5
4Type of first columnSchemaType
5Type of second columnSchemaType
.
nType of (n-3)'th columnSchemaType
.
31Type of 28th columnSchemaType

Types are represented via the SchemaType enum and encoded as one byte per type. The encoding is specified in SchemaType.sol (opens in a new tab)

Schema types
Value (hex)Value (dec)Type
0x00≤n≤0x1F0≤n≤31uint<8(n+1)>
0x20≤n≤0x3F32≤n≤63int<8(n-31)>
0x40≤n≤0x5F64≤n≤95bytes<n-63>
0x6096bool
0x6197address
0x62≤n≤0x8198≤n≤129uint<8(n-97)>[]
0x82≤n≤0xA1130≤n≤161int<8(n-129)>[]
0xA2≤n≤0xC1162≤n≤193bytes<8(n-161)>[]
0xC2194bool[]
0xC3195address[]
0xC4196bytes
0xC5197string

Note that the dynamic-length types must come after all the static-length types.

For example, let's interpret the schema 0x001c0303180001c5c48300000000000000000000000000000000000000000000.

Byte number0-12345678910-31
Schema data0x001c0303180001c5c4830...0
Meaning28 bytes of static datathree static length fieldsthree dynamic length fields1st field - 18 (uint200)2nd field - 00 (uint8)3rd field - 01 (uin16)4th field - c5 (string)5th field - c4 (bytes)6th field - 83 (int16[])No other fields

Events

The Store_ events have two topics:

  1. The event signature
  2. The ResourceId for the table being modified.

Store_SetRecord

This event (opens in a new tab) is emitted when a new record is created with all the data fields.

The fields are in the event definition. In Solidity, unindexed event fields are ABI encoded (opens in a new tab).

FieldEncoding
bytes32[] keyTupleStandard ABI encoding (32-byte fields)
bytes staticDataPacked ABI encoding of a tuple with the value's static fields
EncodedLengths encodedLengthsSee below.
bytes dynamicDataModified packed ABI encoding of data, use encodedLengths for length delimiters

encodedLengths is a 32-byte value that contains the length in bytes of all the dynamic data fields:

EncodedLengths = bytes[32] where:

bytes[0:6]   = len of all dynamic data
bytes[6:11]  = len of 1st dynamic field
bytes[11:16] = len of 2nd dynamic field
bytes[16:21] = len of 3rd dynamic field
bytes[21:26] = len of 4th dynamic field
bytes[26:31] = len of 5th dynamic field
Example

For example, we have a table with this definition:

    Complicated: {
       schema: {
         key1: "uint200",
         key2: "uint8",
         val1: "uint200",
         val2: "uint8",
         val3: "uint16",
         dyn1: "string",
         dyn2: "bytes",
         dyn3: "int16[]",
       },
       key: ["key1", "key2"],
    },

And we want to understand this data:

WordData
00x0000000000000000000000000000000000000000000000000000000000000080
10x00000000000000000000000000000000000000000000000000000000000000e0
20x0000000000000000000000000000060000000005000000000500000000000010
30x0000000000000000000000000000000000000000000000000000000000000120
40x0000000000000000000000000000000000000000000000000000000000000002
50x00000000000000000000000000000000000000000000000000000000000060a7
60x0000000000000000000000000000000000000000000000000000000000000002
70x000000000000000000000000000000000000000000000000000000000000001c
80x00000000000000000000000000000000000000000000000bad04600d00000000
90x0000000000000000000000000000000000000000000000000000000000000010
100x68656c6c6f776f726c6400010002000300000000000000000000000000000000

The first four 256-bit words correspond to the four unindexed event fields.

WordEvent fieldData
0bytes32[] keyTuple0x0...080
1bytes staticData0x0...0e0
2EncodedLengths encodedLengths0x0000000000000000000000000000060000000005000000000500000000000010
3bytes dynamicData0x0...0120

ABI encoding starts with one 32-byte word per field. If the field is static, the word contains that value. If the field is dynamic, the word contains an offset into where the field data starts.

Event fieldStarting byteWords in the data
bytes32[] keyTuple0x80 = 128 = 4*324-6
bytes staticData0xE0 = 224 = 7*327-8
bytes dynamicData0x120 = 288 = 9*329-10

bytes32[] keyTuple

WordData
40x0......02
50x0...060a7
60x0......02

This is a variable-length array. The first word is the array length, followed by the values. The key fields are treated internally as 32-byte values, so every value takes 32 bytes. This gives us

Record fieldValue
key10x60A7
key22

bytes staticData

This is the static data (the static fields that are not in the key).

WordData
70x0...01c
80x00000000000000000000000000000000000000000000000bad 04 600d 00000000

Because the type is an unspecified number of bytes, the first 32-byte word is the length. In this case, the length is 0x1C=28. This means that only the first (most significant) 28 bytes of word 8 are meaningful. The other four are zero.

Going back to the schema, we see that there are three static data fields:

         val1: "uint200",
         val2: "uint8",
         val3: "uint16",

The first one is 200 bits = 25 bytes, the second one is 8 bits = 1 byte, and the third is 16 bits = 2 bytes. They are written consecutively, so

Record fieldValue
val10xBAD
val24
val30x600D

EncodedLengths encodedLengths

This is the definition of the EncodedLengths type (opens in a new tab).

Bytes0x1F-0x1B0x1A-0x160x15-0x110x10-0x0C0x0B-0x070x06-0x00
Value0x0...00x0...00x0...060x0...050x0...050x0...010
Length (in bytes) of5th dynamic field4th dynamic field3rd dynamic field2nd dynamic field1st dynamic fieldall dynamic fields together

This is the dynamic field part of the schema:

         dyn1: "string",
         dyn2: "bytes",
         dyn3: "int16[]",

Here we see that the first five bytes of the dynamic data are going to be dyn1, the next five dyn2, and the final six dyn3. The total length is 0x10 = 16.

bytes dynamicData

WordData
90x0000000000000000000000000000000000000000000000000000000000000010
100x68656c6c6f 776f726c64 000100020003 00000000000000000000000000000000

This value is also bytes, so the first word is the length of the data, in this case 0x10=16. Combining this with the encoded lengths information, we get these values:

Record fieldValue
dyn10x68656c6c6f
dyn20x776f726c64
dyn30x00100020003

To interpret the values we need to get the types from the schema.

Record fieldField typeValue
dyn1stringhello
dyn2bytes0x776f726c64 (= world)
dyn3int16[][1,2,3]

Store_SpliceStaticData

This event (opens in a new tab) is emitted when a record's static fields are modified. Note that this event is also emitted if the record did not exist previously. Solidity does not distinguish between a record not existing, and a record existing with all the values set to zero.

FieldEncoding
bytes32[] keyTupleStandard ABI encoding (32-byte fields)
uint48 startThe starting byte of the segment of static data to change
bytes dataPacked ABI encoding of a tuple with the value's static fields
Example

Using the same table definition as with Store_SetRecord, we want to understand this data:

WordData
00x0000000000000000000000000000000000000000000000000000000000000060
10x0000000000000000000000000000000000000000000000000000000000000019
20x00000000000000000000000000000000000000000000000000000000000000c0
30x0000000000000000000000000000000000000000000000000000000000000002
40x00000000000000000000000000000000000000000000000000000000000060a7
50x0000000000000000000000000000000000000000000000000000000000000003
60x0000000000000000000000000000000000000000000000000000000000000001
70xff00000000000000000000000000000000000000000000000000000000000000

The first three 256-bit words correspond to the three unindexed event fields.

WordEvent fieldData
0bytes32[] keyTuple0x00...60
1uint48 start0x00...19
2bytes data0x00...C0

keyTuple is interpreted the same way it is in Store_SetRecord.

uint48 start

start is 0x19 = 25.

Going back to the schema, we see that there are three static data fields:

         val1: "uint200",
         val2: "uint8",
         val3: "uint16",

Starting from zero, this means that val1 is bytes 0-24, val2 is byte 25, and val3 is bytes 26-27. We are changing from byte 25, so val1's value stays the same.

bytes data

This data starts at word six (byte 0xC0 = 192 = 32*6).

WordData
60x0...01
70xff 0...0

The first word of this field, word 6, tells us that we are only changing one byte of the data, so we aren't changing val3 either. The first byte of word 7, the only one that matters, is 0xFF. This is the new value of val2.

Store_SpliceDynamicData

This event (opens in a new tab) is emitted when a record's dynamic fields are modified. Note that this event is also emitted if the record did not exist previously. Solidity does not distinguish between a record not existing, and a record existing with all the values set to zero.

FieldEncoding
bytes32[] keyTupleStandard ABI encoding (32-byte fields)
uint8 dynamicFieldIndexThe sequential number (counting from zero) of the dynamic field to change. 0 ≤ dynamicFieldIndex ≤ 4
uint48 startThe starting byte of the segment of the dynamicFieldIndex'th dynamic field to change
uint40 deleteCountThe number of bytes to delete from the dynamicFieldIndex'th dynamic field.
EncodedLengths encodedLengthsSimilar to encodedLengths in Store_SetRecord, except that only the size of the dynamicFieldIndex'th field is included
bytes dataBytes to add/change in the dynamicFieldIndex'th dynamic field.
Example: Appending and modifying data

Using the same table definition as with Store_SetRecord, we want to understand this data. Note that the log entry by itself does not tell us if we are appending data or modifying an existing value. It only tells us that we are changing the data in bytes six and seven, not whether there is already data there or not.

WordData
00x00000000000000000000000000000000000000000000000000000000000000c0
10x0000000000000000000000000000000000000000000000000000000000000002
20x0000000000000000000000000000000000000000000000000000000000000006
30x0000000000000000000000000000000000000000000000000000000000000000
40x0000000000000000000000000000080000000000000000000000000000000008
50x0000000000000000000000000000000000000000000000000000000000000120
60x0000000000000000000000000000000000000000000000000000000000000002
70x00000000000000000000000000000000000000000000000000000000000060a7
80x0000000000000000000000000000000000000000000000000000000000000005
90x0000000000000000000000000000000000000000000000000000000000000002
100x1234000000000000000000000000000000000000000000000000000000000000

The first six 256-bit words correspond to the unindexed event fields.

WordEvent fieldData
0bytes32[] keyTuple0x00...c0
1uint8 dynamicFieldIndex0x00...02
2uint48 start0x00...06
3uint40 deleteCount0x00...00
4EncodedLengths encodedLengths0x0000000000 0000000000 0000000008 0000000000 0000000000 00000000000008
5bytes data0x00...120

keyTuple is interpreted the same way it is in Store_SetRecord.

uint8 dynamicFieldIndex

The index of the dynamic field we are modifying. The value here is two. Counting from zero it is the third schema field.

         dyn3: "int16[]",

uint48 start

The byte at which we start the change. Here is it byte six. The effect on the length of the data depends on what we are doing. If we are appending data, we would have six bytes now (three values) and add to them. If we are modifying data, we should have at least eight bytes now (four values), and we modify the fourth value (dyn3[3]).

uint40 deleteCount

The number of bytes we delete. In this case, zero.

EncodedLengths encodedLengths

The encoded lengths in the data after the change (if they change). This value tells us that only the third field changes, and that it will be eight bytes long.

bytes data

This value starts at byte 0x120 = 288 = 9*32, or word nine.

WordData
90x0000000000000000000000000000000000000000000000000000000000000002
100x1234 000000000000000000000000000000000000000000000000000000000000

Again, the first word of the bytes value tells us how many bytes we have (two). The second gives us the value we are adding, in this case 0x1234.

Example: Deleting data

Using the same table definition as with Store_SetRecord, we want to understand this data:

WordData
00x00000000000000000000000000000000000000000000000000000000000000c0
10x0000000000000000000000000000000000000000000000000000000000000002
20x0000000000000000000000000000000000000000000000000000000000000006
30x0000000000000000000000000000000000000000000000000000000000000002
40x0000000000000000000000000000060000000000000000000000000000000006
50x0000000000000000000000000000000000000000000000000000000000000120
60x0000000000000000000000000000000000000000000000000000000000000002
70x00000000000000000000000000000000000000000000000000000000000060a7
80x0000000000000000000000000000000000000000000000000000000000000005
90x0000000000000000000000000000000000000000000000000000000000000000

The first six 256-bit words correspond to the unindexed event fields.

WordEvent fieldData
0bytes32[] keyTuple0x00...c0
1uint8 dynamicFieldIndex0x00...02
2uint48 start0x00...06
3uint40 deleteCount0x00...02
4EncodedLengths encodedLengths0x0000000000000000000000000000060000000000000000000000000000000006
5bytes data0x00...120

This is very similar to appending or modifying data, except that deleteCount has a value. This is the number of bytes we are deleting, two (a single array element). Combining this with the start value of six, we are deleting bytes six and seven.

The other difference is that word 9 is zero - there is a bytes data value, but it is empty.

Store_DeleteRecord

This event (opens in a new tab) is emitted when a record is deleted.

FieldEncoding
bytes32[] keyTupleStandard ABI encoding (32-byte fields)

The only unindexed event field is bytes32[] keyTuple, which contains the key. See the example in Store_SetRecord to interpret it.

Onchain storage layout

The data of each record of each table is stored in its own location in storage.

The location of the static-length data is determined by a hash of the table ID and the key tuple. Since data is tightly packed, fields may wrap around two storage slots.

function _getStaticDataLocation(ResourceId tableId, bytes32[] memory keyTuple) internal pure returns (uint256) {
  return uint256(SLOT ^ keccak256(abi.encodePacked(tableId, keyTuple)));
}

The encodedLengths field (the length of all dynamic-length fields) is stored in a single storage slot, which is determined by a hash of the table ID and the key tuple.

function _getDynamicDataLengthLocation(ResourceId tableId, bytes32[] memory keyTuple) internal pure returns (uint256) {
  return uint256(DYNAMIC_DATA_LENGTH_SLOT ^ keccak256(abi.encodePacked(tableId, keyTuple)));
}

The data for each dynamic-length field is stored in its own location in storage, which is determined by a hash of the table ID, the key tuple, and the field index. This allows MUD to change the length of a dynamic-length field without having to move the data of other fields.

function _getDynamicDataLocation(
  ResourceId tableId,
  bytes32[] memory keyTuple,
  uint8 dynamicFieldIndex
) internal pure returns (uint256) {
  return uint256(DYNAMIC_DATA_SLOT ^ bytes1(dynamicFieldIndex) ^ keccak256(abi.encodePacked(tableId, keyTuple)));
}