# Understanding the data storage of Solana Program Derived Addresses

Decentralized applications need to store data immutable onchain. Solana is no different, that’s why we get Program Derived Addresses (PDAs). Creating a PDA is done through `system_instruction::SystemInstruction::CreateAccountWithSeed` which derives the PDA public key from a base, mostly the program public key and a seed which is a string of Rust UTF8 characters.

#### How a Solana PDA initialized

The initialized storage is an array of the maximum memory size the data structure takes up in memory. Storing a data structure like a username as a Rust `String` would require `24 bytes` of memory allocated on the Solana blockchain. The bigger the data structure the higher the rent needed to pay for onchain storage. Calculating the maximum size a data structure would take up can be done using the Rust function `core::mem::size_of::<T>()` where `T` is the data structure.

```rust
core::mem::size_of::<String>(); //24 bytes
```

When `system_instruction::SystemInstruction::CreateAccountWithSeed` is called with the size of the String (24 bytes), will allocate an empty array of 24 bytes `[u8; 24]`.

```rust
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
```

For large data structures serialize/ deserialize libraries are used to handle the conversion of data structures to and from bytes.

---

> **A BIT OF GLASS:** Solana uses Rust for programming onchain programs and the most popular serialization framework is `serde`. Unfortunately, `serde` uses too many CPU cycles to serialize and deserialize making it expensive to convert data to and from bytes on Solana’s limited BPF smart contract execution stack. `borsh` crate is the go to serialization library for Solana.

---

Suppose we need to create an end-to-end encrypted messaging Dapp that stores a user’s Diffie-Hellman public keys onchain in the user’s PDA address. The data structure in Rust would look like this:

```rust
use borsh::{BorshSerialize, BorshDeserialize};
 
#[derive(Debug, BorshSerialize, BorshDeserialize, Default)]
pub struct MessagingAccount {
   username: String,
   dh_keys: Vec<[u8; 32]>,
}
```

Calculating the size using `core::mem::size_of::<MessagingAccount>()` will return a maximum size of `48 bytes` that can be stored in memory. Creating a PDA on Solana using `system_instruction::SystemInstruction::CreateAccountWithSeed` will initialize a zeroed array of 48 bytes (`[u8; 48]`) which can be represented as:

```rust
[0u8; core::mem::size_of::<MessagingAccount>()]
```

Using `solana_program::msg` to log the PDA would show the initialized account as

```rust
AccountInfo {
       key: pda_address,
       owner: program_id,
       is_signer: false,
       is_writable: true,
       executable: false,
       ..,
       data.len: 48,
       data: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000,
   .. }
```

The data part of the `AccountInfo` above shows that `000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000` which is hex for the 48 bytes of the zeroed storage initialized when the PDA account was created `[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]`

We can deserialize the data structure onchain as follows:

```rust
// – Code snippet –
entrypoint!(process_instruction);
 
   pub fn process_instruction(
       program_id: &Pubkey,
       accounts: &[AccountInfo],
       instruction_data: &[u8],
   ) -> ProgramResult {
       let accounts_iter = &mut accounts.iter();
 
       let pda_account = next_account_info(accounts_iter)?;
       let pda_data = MessagingAccount::try_from_slice(instruction_data)?;
   }
```

Calling the smart contract code will return a `borsh` error

```rust
Err(
    Custom {
        kind: InvalidData,
        error: "Not all bytes read",
    },
)
```

The error returned is because `borsh` tries to deserialize the PDA account based on the data size encoded in the `borsh specification - https://borsh.io/`, in our case `MessagingAccount::default` which is `[0, 0, 0, 0, 0, 0, 0, 0]` which is an array of `8 bytes` while the initialized PDA store is an array of `48 bytes`.

### What To Do?

A simple way to solve this issue is to pack and unpack bytes using a storage format. The storage format will contain a `marker` to indicate the length of the valid bytes. The marker is of type `usize` which has a maximum of `8 bytes` which can be calculated by `core::mem::size_of::<usize>().len()` The data storage format can be defined as:

```sh
MARKER (8 bytes) | DATA (variable byte length) | Zeros (filled with zeroes)
```

##### To Serialize the data into the storage format

1. calculate the size of the valid bytes serialized by borsh: `MessagingAccount::try_to_vec()?.len()`
    
2. Get the size of the storage length of the pda account: `pda_account.data.len()`
    
3. Add the `MARKER` length to the serialized data length: `8 + MessagingAccount::try_to_vec()?.len()`
    
4. Check if the length of the `MARKER + MessagingAccount::try_to_vec()?.len()` is greater than the length of the `pda_account` data storage.
    
5. If the size of the `MARKER + MessagingAccount::try_to_vec()?.len()` os greater, return an error informing the user that the data cannot be written to the Solana PDA storage because it exceeded the capacity of the PDA account.
    
6. If the data is less than or equal to the capacity of the PDA account storage, concate the `MARKER` with the bytes of the serialized data and then if the concatenated data is still less than the capacity of the PDA account storage, fill the remaining space with zeroes.
    
7. Write the data into the PDA storage
    

##### To deserialize the data storage format

1. Get the first 8 bytes and convert them to a `MARKER` : `usize::from_le_bytes(bytes_stored[0..8]`
    
2. Skip the first 8 bytes as indicated by the marker and then fetch the rest of the bytes up to the index indicated by the marker: `let data = bytes_stored.iter().skip(8).take(MARKER).collect::<Vec<u8>>();`
    
3. Deserialize the data using borsh: `MessagingAccount.try_from_slice(&data)?;`
    

##### Transforming the pseudocode above to code:

The code contains comments that explain each step.

* Create code to handle the errors
    

```rust
/// The result type that encompasses the `AccountStoreError`
pub type AccountStoreResult<T> = Result<T, AccountStoreError>;

#[derive(Debug)]
pub enum AccountStoreError {
    /// The buffer cannot acommodate the size of the `MARKER` which is `8 bytes`
    BufferTooSmallForMarker = 0,
    /// The buffer cannot acommodate the size of the data and the  `MARKER`
    BufferTooSmallForData = 1,
    /// The deserialized bytes do not contain a `MARKER`
    CorruptedMarker = 2,
    /// The data provided does not contain enough data length as specified by the `MARKER`
    CorruptedStorage = 3,
    /// The error provided is invalid
    InvalidError = 4,
}
```

* Create a data structure to handle these operations:
    

```rust
      pub const MARKER_SIZE: usize = 8;
      
      #[derive(Debug)]
      pub struct AccountStore<T> {
          pub data: T,
      }
```

* Create a method to calculate the size needed for a generic data structure to be stored onchain with our storage format. The `impl {}` block to take a generic parameter `T` that must implement `BorshSerialize`, `BorshDeserialize` and `core::fmt::Debug`.
    

```rust
impl<T> AccountStore<T>
where
    T: BorshDeserialize + BorshSerialize + Default + Sized,
{
    pub fn size_of() -> usize {
        core::mem::size_of::<T>() + MARKER_SIZE
    }
}
```

The `size_of()` method calculates the size of the data structure specified as `T` and adds `8 bytes` to accommodate the marker information.

* Create a method to pack the data Create a method called `pack()` to convert the data into the storage format and write it to a provided buffer
    

```rust
impl {
  // -- Code snippet --
   pub fn pack(&self, buffer: &mut [u8]) -> AccountStoreResult<usize> {
        // Get the length of the PDA account storage size
        let buffer_length = buffer.len();

        // Check if the size of the `MARKER` is less than the size  of the `buffer_length`
        if buffer_length < MARKER_SIZE {
            // If the size is smaller, return an error indicating this to the user
            return Err(AccountStoreError::BufferTooSmallForMarker);
        }
        // Serialize the user data using `borsh`
        let data = self.data.try_to_vec().unwrap(); //HANDLE THIS BORSH ERROR AS YOU WISH
                                                    // Get the data length
        let data_length = data.len();

        // Check if the sum of the size of the `data_length` and the `MARKER_SIZE` is
        // greater than the `buffer_length`
        if buffer_length < data_length + MARKER_SIZE {
            return Err(AccountStoreError::BufferTooSmallForData);
        }

        // Copy the `data_length` to the buffer as the `MARKER`
        buffer[0..=7].copy_from_slice(&data_length.to_le_bytes());

        // Copy the data into the buffer.
        // If the data is smaller than the buffer then the space filled with
        // zeroes is left intact
        buffer[8..=data_length + 7].copy_from_slice(&data);

        Ok(data_length + 8usize)
    }
}
```

* Create a method `unpack()` to read the storage format
    

```rust
impl {
  // -- Code snippet --


    pub fn unpack(buffer: &[u8]) -> AccountStoreResult<AccountStore<T>> {
        // Get the length of the PDA account storage
        let buffer_length = buffer.len();

        // Check if the size of the `MARKER` is less than the size  of the `buffer_length`
        if buffer_length < MARKER_SIZE {
            return Err(AccountStoreError::BufferTooSmallForMarker);
        }

        // Convert the `MARKER` bytes to and array of `[u8; 8] `
        // since `usize::from_le_bytes` only accepts `[u8; 8]`
        let marker: [u8; 8] = match buffer[0..MARKER_SIZE].try_into() {
            Ok(value) => value,
            Err(_) => return Err(AccountStoreError::CorruptedMarker),
        };
        // Get the last index of the valid data
        let byte_length = usize::from_le_bytes(marker);

        // Check if the last index of the valid buffer is greater than the PDA storage size
        if byte_length > buffer_length {
            return Err(AccountStoreError::CorruptedStorage);
        }

        // Collect the valid data by skipping the `MARKER_SIZE` of `8 bytes`
        // and iterating the rest of the bytes until the index marked by the `byte_length`
        let data = buffer
            .iter()
            .skip(8)
            .take(byte_length)
            .map(|byte| *byte)
            .collect::<Vec<u8>>();

        if byte_length != 0 {
            let data = T::try_from_slice(&data).unwrap(); // Handle error as you see fit
            Ok(AccountStore { data })
        } else {
            // If the `byte_length` is zero it means that no previous
            // data had been written to the PDA account previously
            // so return the `Default` representation of the data structure represented
            // by the generic `T`
            Ok(AccountStore { data: T::default() })
        }
    }
}
```

* Lastly, create a method to add data to the store
    

```rust
impl {
  // -- snippet --
pub fn add_data(&mut self, data: T) -> &mut Self {
        self.data = data;

        self
    }
}
```

##### Utilizing our storage library to serialize and deserialize the `pda_account.data`

* Unpack the Solana PDA data `pda_account.data` using `unpack()` method and `add_data()` to add some data eg. from a Solana `Instruction`
    

```rust
let mut data = AccountStore::MessagingAccount>::unpack(&pda_account.data.as_ref().borrow()).unwrap();

let user_data = MessagingAccount {
    username: "SolanaSeaMonster".into(),
    dh_keys: vec![[1u8; 32], [2u8; 32]],
};

data.add_data(user_data);
```

* Packing the data and writing it back to the Solana PDA account using `pack()` method.
    

```rust
data.pack(&mut &mut pda_account.data.borrow_mut()[..]).unwrap();
```

Since a Solana PDA account storage is a Rust `RefCell<Rc<[u8; T>>`, the `&variable_name.as_ref().borrow()` is used to access the data and convert it into a `&[u8]` as required by the `unpack()` method and the `&mut &mut variable_name.data.borrow_mut()[..]` is used to write data.

Now you know how the PDA storage concept works and how you can create your own lightweight storage format to read and write the data.

That's it.

Keep chewing glass.