The TransactionBatchProcessor handles validation and execution of sanitized batched transactions within the SVM acting as the central interface for the whole transaction execution pipeline.

// TransactionBatchProcessor struct as declared in SVM code
pub struct TransactionBatchProcessor<FG: solana_program_runtime::loaded_programs::ForkGraph> {
    /// Bank slot (i.e. block)
    slot: Slot,

    /// Bank epoch
    epoch: Epoch,

    /// SysvarCache is a collection of system variables that are
    /// accessible from on chain programs. It is passed to SVM from
    /// client code (e.g. Bank) and forwarded to process_message.
    sysvar_cache: RwLock<SysvarCache>,

    /// Programs required for transaction batch processing
    pub program_cache: Arc<RwLock<ProgramCache<FG>>>,

    /// Builtin program ids
    pub builtin_program_ids: RwLock<HashSet<Pubkey>>,

    execution_cost: SVMTransactionExecutionCost,
}

Fields of TransactionBatchProcessor struct

1. slot - a fixed timeframe when a validator can produce a block (can be as low as 150ms for alpenglow consensus). It’s Rust data type is a u64.

2. epoch - the timeframe when a leader schedule is valid divided into slots. The leader schedule is the mapping of validator public keys to each slot within the given epoch. It’s Rust data type is also a u64.

3. sysvar_cache - defined by the Rust type SysvarCache, it contains system variables like the Clock (network time), Epoch Schedule, Epoch Rewards, Rent, Stake History and Last Network Restart Slot.

4. program_cache - defined by the Rust type ProgramCache, it contains loaded, compiled and verified programs required for transaction batch processing, optimizing shared data across validator forks. It tracks program information such as usage stats, verification status, and handles deployments, evictions, and tombstones for unloadable programs. Executable programs that are frequently used are kept in memory to improve execution speed through cooperative batch loading and eager preloading. The program cache isn’t saved to disk, so it must be rebuilt after each SVM restart.

5. builtin_program_ids - It contains builtin program IDs like BPF Loader that own and execute smart contract programs and program configs like the supported SBF versions (all SBF versions are enabled by default).

6. execution_cost - configures the CUs used by various operations like how much computing a SHA256 hash costs, cost of logging (msg!()), CPI, verifying Ed25519 signatures, CUs used for additional heap allocations above the default and more.

Processing batched sanitized transactions

The method on TransactionBatchProcessor for processing sanitized transaction batches is TransactionBatchProcessor:: load_and_execute_sanitized_transactions. It acts as the entry point to the SVM and takes the following parameters:

1. callbacks: This parameter allows the transaction processor to load information about accounts required for transaction execution. It requires the value supplied to this argument to implement the TransactionProcessingCallback trait which is a supertrait of InvokeContextCallback trait (the callback used by solana_program_runtime::invoke_context::InvokeContext in SVM to manage the main pipeline from runtime to program execution). By defining this trait developers can maintain full control of their application while ensuring integration with SVM according to the specification. This trait has three methods:

   • get_account_shared_data() which takes &self, a public key (the address to fetch information for) as parameters and returns an optional AccountSharedData struct which is the in-memory representation of an Account. An Option::None can be returned if the account does not exist.

   • account_matches_owner which takes &self, a public key and a slice of public keys (&[Pubkey]) representing potential owners of an account. The goal of this method is to get the index of the owner of the public key within the slice of owners &[Pubkey], therefore, it returns an Option of type usize. An Option::None can be returned if the account public key does not exist in the Bank or if the lamports balance is zero.

   • add_builtin_account which takes a &self, name of type &str and a program_id public key. This method is used to add a program to the available programs in the execution pipeline, for example adding the native loader to the bank, that would load a smart contract program into the execution pipeline.

2. sanitized_txs: This parameter contains a slice of sanitized Solana transactions (&[impl SVMTransaction]). A sanitized Solana transaction can be any Rust type as long as it implements the SVMTransaction trait which is a supertrait of SVMMessage. These traits allow any type implementing them to provide convenience methods that can fetch information such as the fee payer, the first signature, all the signatures in a transaction, number of transaction signatures, the recent blockhash of the transaction, the number of write locks, the account keys and more. An example of a type implementing this trait is SanitizedTransaction.

3. check_results - this parameter contains a list of the results of checking the validity of a transaction. Some checks performed that produce these results include the validity of a blockhash or nonce, whether an account matches it’s owner and whether the number of sanitized transactions equals the number of transactions that underwent these checks.

4. environment - This parameter is the runtime environment for transaction batch processing instantiated by TransactionProcessingEnvironment struct. It configures the blockhash to use for the current transaction batch, determine rent due for the account and collect rent, the runtime features to use, lamports per signature for nonce accounts and the total stake for the current epoch.

5. config - this parameter configures customization of transaction processing behavior using TransactionProcessingConfig struct. Custom configurations include account overrides, log message byte size limits, configuring of the recording capabilities for transaction execution, whether to limit the number of programs loaded for the transaction batch and whether or not to check a program’s modification slot when replenishing a program cache instance.

The load_and_execute_sanitized_transactions returns a LoadAndExecuteSanitizedTransactionsOutput struct which contains the fields:

1. error_metrics - the error metrics for the processed transactions

2. execute_timings - timings for transaction batch execution

3. processing_results - a vector containing type alias TransactionProcessingResult indicating whether a transaction executed successfully or the transaction execution failed and needs to be rolled back

4. balance_collector - a Option<BalanceCollector> containing balances accumulated for TransactionStatusSender (a channel within the Solana ledger) when transaction balance recording is enabled.

ForkGraph Trait

The struct TransactionBatchProcessor contains a generic FG which requires the passed type to implement the ForkGraph trait. The trait maps the block’s relationship between two slots returning the BlockRelation enum.

// Relationship between two fork IDs
pub enum BlockRelation {
    // The slot is on the same fork and is an ancestor of the other slot
    Ancestor,
    // The two slots are equal and are on the same fork
    Equal,
    // The slot is on the same fork and is a descendant of the other slot
    Descendant,
    // The slots are on two different forks and may have had a common ancestor at some point
    Unrelated,
    // Either one or both of the slots are either older than the latest root, or are in future
    Unknown,
}

The ForkGraph trait requires the relationship(&self, a: Slot, b: Slot) method to be implemented for the generic type. This method requires the two slots arguments a: Slot and b: Slot to be compared in a manner that returns the relationship (BlockRelation enum) between them. The Slot is just a type alias for a Rust u64 representing an entry that estimates wallclock duration (a tick) . An example of implementing this trait:

# Add `solana-clock` dependency used in SVM ecosystem, it contains the `Slot` type
cargo add solana-clock

pub struct SlotRelationships;

impl ForkGraph for SlotRelationships {
    fn relationship(&self, slot_a: Slot, slot_b: Slot) -> BlockRelation {
        // Since a `Slot` is just a Rust `u64`, and a `u64` implement
        // Rust `PartialCmp` and `Cmp`, we can use `.cmp()` method
        // on `slot_a` to compare against `slot_b` and derive a relationship
        // checking if `slot_a` is less than, equal to or greater than `slot_b`
        match slot_a.cmp(&slot_b) {
            std::cmp::Ordering::Less => BlockRelation::Ancestor,
            std::cmp::Ordering::Equal => BlockRelation::Equal,
            std::cmp::Ordering::Greater => BlockRelation::Descendant,
        }
    }
}

The TransactionBatchProcessor has some more methods to configure of get information about the transaction execution pipeline that you can explore.

< Previous Article: SVM Series 03: Compute Budget & Syscalls

Resources

1. Understanding Slots, Blocks, and Epochs on Solana
   https://www.helius.dev/blog/solana-slots-blocks-and-epochs

2. Solana SVM TransactionBatchProcessor module docs
   https://docs.rs/solana-svm/latest/solana_svm/transaction_processor/struct.TransactionBatchProcessor.html

Compute Budget

What is the Compute Budget & Compute Budget Program?

The compute budget defines the maximum compute units (CU) a transaction can consume to prevent resource abuse on the Solana network. The default CUs are 200,000 with a maximum limit of 1.4 million CUs. During program execution, the SVM breaks the transaction logic into opcodes (BPF instructions) where the VM consumes 1 CU per opcode. Examples of operations that are converted to opcodes are program syscalls, signature verification, locking instructions during transaction sanitization, etc. The compute budget program ComputeBudget111111111111111111111111111111 handles requesting more CUs for program execution if the default CUs are not enough.

To use the Compute Budget in the SVM API

cargo add solana-compute-budget

fn main() {
    // Initialize `ComputeBudget` struct with default paramemters.
    // These parameters include:
    //    - compute unit limit (defaults to 1.4 million CUs)
    //    - Maximum nesting of instructions that can happen during a transaction (defaults to 5)
    //    - Maximum cross-program invocation and instructions per transaction (defaults to 64)
    //    - Maximum number of slices hashed per syscall (defaults to 20,000)
    //    - Maximum SBF to BPF call nesting that can happen within a program (defaults to 64 SBF to BPF to BPF calls)
    //    - Size of a stack frame in bytes matching the size specified in the LLVM SBF backend (defaults to 4096)
    //    - Maximum cross-program invocation instruction size (default to IPv6 Min MTU size of 1280)
    //    - Length of the heap memory region used for program heap 
    //      (defaults to solana_program_entrypoint::HEAP_LENGTH of 32768 = 1024 * 32) 
    //    - Loads the default execution costs defined by `solana_program_runtime::execution_budget::SVMTransactionExecutionCost` struct
    //      which is defines cost paramemters for operations like loggin, CPI, signature verification and more. 
    let compute_budget_config = ComputeBudget::default();    
}

The SVMTransactionExecutionCost source code parameters are defined at Solana Program Runtime Modeule #L90.

Syscalls

What are syscalls?

In SVM, a syscall (short for system call) is a low-level function provided by the Solana VM that programs can invoke to interact with the VM environment. Syscalls are predefined entrypoints into the runtime that ensure only specific actions are allowed during program execution. These operations that are not part of the program itself but are offered by the VM as trusted privileged operations. Examples of a syscall is verifying Ed25519 signatures or logging information.

Configuration parameters used in syscalls parameters

Defining syscalls in SVM requires some configuration provided by the solana_sbpf::vm::Config. This configures VM parameters like maximum call depth, stack frame size, copying read-only data, tracing instructions, which SBF versions are allowed and more.

The solana-sbpf Rust crate is used to define syscalls and their configuration parameters. An example of this:

# add `solana-program` to your dependencies
# it contains `solana-sbpf` crate used to configure the VM
cargo add solana-program-runtime 

cargo add solana-bpf-loader-program # Contains the syscalls source code to configure the VM

In the src/main.rs file part of the Compute Budget code above.

use solana_program_runtime::{
    // `InvokeContext` is used to configure the entire exeuction pipeline
    invoke_context::InvokeContext,
    solana_sbpf::{
        program::BuiltinProgram,
        // Rename from `Config` to `SbpfVmConfig` to avoid conflicts with other deps with structs called `Config`
        vm::Config as SbpfVmConfig,
    },
};

    // ... previous code from above

    // Defining syscalls requires some configuration using `solana_sbpf::vm::Config`.
    // This configures maximum call depth, stack frame size, copying read only data, tracing instructions,
    // allowed SBF versions, sanitizing user provided values and many more
    let vm_config = SbpfVmConfig {
        // configure maximum call depth using that of our compute budget `compute_budget_config`
        max_call_depth: compute_budget_config.max_call_depth,
        // configure stack frame size using that of our compute budget `compute_budget_config`
        stack_frame_size: compute_budget_config.stack_frame_size,
        // Enable instruction tracing
        enable_instruction_tracing: true,
        // enable symbol and section labels for BPF (disabled by default)
        enable_symbol_and_section_labels: true,
        // Reject ELF files containing issues that the verifier did not catch before (up to v0.2.21). Disabled by default
        reject_broken_elfs: true,
        // Avoid copying read only sections when possible. Enabled by default
        optimize_rodata: false,
        // Use all other default parameters
        ..Default::default()
    };

We can now specify the syscalls the compiled program can call during execution using solana_sbf::program::BuiltinProgram Rust crate. The SVM configuration defined above as vm_config is used to configure the VM. The BuiltinProgram takes a generic parameter, in our case InvokeContext which defines the main pipeline from runtime to program execution.

    // The SVM configuration `vm_config` defined above is used  to configure the VM.
    // The `BuiltinProgram` takes a generic parameter, in our case the generic parameter will be
    // `solana_program_runtime::invoke_context::InvokeContext`
    let mut loader: BuiltinProgram<InvokeContext<'_>> = BuiltinProgram::new_loader(vm_config);

To register a syscall we use the register_function() method on BuiltinProgram<InvokeContext<'_>>. This function expects the name of the syscall and the syscall itself defined in the solana_bpf_loader_program::syscalls module. For example the msg! macro used in Solana programs to log messages is dependent on the sol_log() function which calls the sol_log_ syscall in the VM.

// How the `msg!()` macro is defined in the source code
#[cfg(feature = "std")]
#[macro_export]
macro_rules! msg {
    ($msg:expr) => {
        $crate::sol_log($msg)
    };

    // The macro calls `sol_log()` function within the same crate and passes a message
    // formatted message using Rust `format!()` macro 
    ($($arg:tt)*) => ($crate::sol_log(&format!($($arg)*))); 
}

// How the `sol_log()` function calls the sycall to show the formatted message
/// Print a string to the log.
#[inline]
pub fn sol_log(message: &str) {
    #[cfg(target_os = "solana")] // Only available on the SVM
    unsafe {
        // syscall is defined as `sol_log_` in the VM
        syscalls::sol_log_(message.as_ptr(), message.len() as u64);
    }

   // ...
}

We can define our own sol_log_ syscall in the VM similar to the above msg!() implementation using the loader variable we defined earlier.

// Import the `SyscallLog` from the `solana_bpf_loader_program::syscalls` module.
use solana_bpf_loader_program::syscalls::SyscallLog;

    loader
        // call this method and pass in the syscall function name in our case `sol_log_` 
        // and `SyscallLog::vm` which provides `vm()` method which is the VM interface that accepts parameters for execution.
        .register_function("sol_log_", SyscallLog::vm)
        .expect("Registration of `sol_log_` syscall failed!"); //Handle the error

These syscalls have a vm() method that provides the VM interface and a rust() method that provides a Rust interface.

We have learnt how to configure the VM and add syscalls. Explore the syscalls of the solana-bpf-loader-program Rust crate to learn other syscalls like SyscallLogPubkey, SyscallGetClockSysvar and many more.

The source code for this article can be found at - https://github.com/448-OG/SVM-Series/tree/master/syscalls

< Previous Article: SVM Series 02: SVM Feature Activation & Management

Next Article: SVM Series 04: TransactionBatchProcessor >

Resources

1. Cost model source code
   https://github.com/anza-xyz/agave/blob/d5a84daebd2a7225684aa3f722b330e9d5381e76/cost-model/src/transaction_cost.rs#L121

2. Compute budget source code
   https://github.com/anza-xyz/agave/blob/d5a84daebd2a7225684aa3f722b330e9d5381e76/compute-budget/src/compute_budget.rs#L22

3. Bytecode / ISA
   https://github.com/anza-xyz/sbpf/blob/main/doc/bytecode.md

4. How does the Solana BPF_VM calculate how many Compute units a program consumes?
   https://solana.stackexchange.com/questions/15119/how-does-the-solana-bpf-vm-calculate-how-many-compute-units-a-program-consumes
   https://github.com/anza-xyz/agave/blob/d5a84daebd2a7225684aa3f722b330e9d5381e76/program-runtime/src/invoke_context.rs#L529
   https://github.com/solana-labs/rbpf/blob/4dc039f4ee7409838c7f230558aebf6869c32db9/src/program.rs#L331
   https://github.com/anza-xyz/agave/blob/d5a84daebd2a7225684aa3f722b330e9d5381e76/programs/bpf_loader/src/lib.rs#L1405
   https://github.com/solana-labs/rbpf/blob/4dc039f4ee7409838c7f230558aebf6869c32db9/src/vm.rs#L422

5. BPF instructions all consume 1 CU in the interpreter (which must match the JIT results)
   https://github.com/solana-labs/rbpf/blob/f3758ecee89198433422f751beee7f0f52dbcd55/src/interpreter.rs#L162

6. Solana sycalls module
   https://docs.rs/solana-bpf-loader-program/latest/solana_bpf_loader_program/syscalls/index.html

Feature activation refers to a community-governed process for enabling new non-controversial network primitives on the Solana network through validator voting based on stake weight. An example of this is SIMD 220 which improves validator performance by using Accounts Lattice Hash for snapshots instead of the Merkle-based account hash.

SPL tokens that represent the total active stake on the network are minted and distributed to all validtors based on their stake. Validators then transfer their vote tokens to a certain predetermined address, in the case of SIMD 220 address LTsNA...3mSQ2 on mainnet-beta. Once vote threshold is met the feature is activated at the beginning of a predetermined epoch. The built-in Feature program present in all validators is responsible for activating new features simultaneously to avoid hard forks that break consensus.

Feature proposal

A feature proposal is added to the code base using a feature id. Feature IDs can be accessed using the agave-feature-set Rust crate. The feature ID is generated in the following steps:

1. Create a Solana keypair for the proposal

    solana-keygen new --outfile <feature-proposal-name>.json --silent --no-passphrase
   

2. Install the (note you need libudev installed) and then run

    # install spl-feature-proposal-cli
    $ cargo install spl-feature-proposal-cli --git https://github.com/solana-program/feature-proposal.git
   
    # Derive the feature ID
    $ spl-feature-proposal address feature-proposal.json
   

   Deriving the feature ID prints the details; example:

    Feature Id: AMJPdBjeCf1694FJcGsYNsGZaXogXuSVRVUVTAX28PEi
    Token Mint Address: 3vYBgFztMaEPByNsk8acEjfjLauswW46hm2ZuMkh3WFy
    Acceptance Token Address: EhfPUgpY4sKPqy8i488cXGxwKwMQ7YcMa5CMMHyzn6w9
   

   where: Feature Id: is the identifier added to the code base for that feature and is also retrieved by solana feature status subcommand of the solana command line tool. Note that only the feature proposal program can activate this feature since there is no private key for the address identified by the Feature id.

3. Initializing the feature - This is only possible after the feature is deployed to a Solana cluster (devnet, testnet or mainnet-beta). Note that features are first deployed to testnet cluster. Running solana feature status should show this feature's ID.

    # Example from Solana docs. Running this mints SPL tokens for the feature proposal and the 
    # `feature proposer` is required to finance creating all the 
    # SPL token accounts (about 0.00204 SOL per account) for all validators
    $ spl-feature-proposal propose feature-proposal.
   

    # The output of the above command example from Solana docs
    Feature Id: HQ3baDfNU7WKCyWvtMYZmi51YPs7vhSiLn1ESYp3jhiA
    Token Mint Address: ALvA7Lv9jbo8JFhxqnRpjWWuR3aD12uCb5KBJst4uc3d 
    Distributor Token Address: GK55hNft4TGc3Hg4KzbjEmju8VfaNuXK8jQNDTZKcsNF 
    Acceptance Token Address: AdqKm3mSJf8AtTWjfpA5ZbJszWQPcwyLA2XkRyLbf3Di 
    Number of validators: 376 
    Tokens to be minted: 134575791.53064314 
    Tokens required for acceptance: 90165780.3255309 (67%) 
    Token distribution file: feature-proposal.csv 
    JSON RPC URL: http://api.mainnet-beta.solana.com
   
    Distribute the proposal tokens to all validators by running: 
        $ solana-tokens distribute-spl-tokens --from GK55hNft4TGc3Hg4KzbjEmju8VfaNuXK8jQNDTZKcsNF --input-csv feature-proposal.csv --db-path db.8CyUVvio --fee-payer ~/.config/solana/id.json --owner <FEATURE_PROPOSAL_KEYPAIR> 
        $ solana-tokens spl-token-balances --mint ALvA7Lv9jbo8JFhxqnRpjWWuR3aD12uCb5KBJst4uc3d --input-csv feature-proposal.csv
   
    Once the distribution is complete, request validators vote for the proposal. To vote, validators should first look up their token account address: 
        $ spl-token --owner ~/validator-keypair.json accounts ALvA7Lv9jbo8JFhxqnRpjWWuR3aD12uCb5KBJst4uc3d 
    and then submit their vote by running: 
        $ spl-token --owner ~/validator-keypair.json transfer <TOKEN_ACCOUNT_ADDRESS> ALL AdqKm3mSJf8AtTWjfpA5ZbJszWQPcwyLA2XkRyLbf3Di
   
    Periodically the votes must be tallied by running: 
        $ spl-feature-proposal tally 8CyUVvio2oYAP28ZkMBPHq88ikhRgWet6i4NYsCW5Cxa
    Tallying is permissionless and may be run by anybody.
    Once this feature proposal is accepted, the HQ3baDfNU7WKCyWvtMYZmi51YPs7vhSiLn1ESYp3jhiA feature will be activated at the next epoch.
   
    Add --confirm flag to initiate the feature proposal
   

4. Tally the votes - anyone can tally the votes to enable the feature to be automatically activated at the start of the next epoch

     $ spl-feature-proposal tally
   

Using features in SVM API

In this series we will use a mock Bank to coordinate the SVM execution pipeline. The agave-feature-set Rust crate will be used to activate features for transaction processing.

FeatureSet struct

Feature set is defined by the struct FeatureSet which contains a list of activated features in the active field and deactivated features in the inactive field.

#[derive(Debug, Clone, Eq, PartialEq)]
pub struct FeatureSet {
    active: AHashMap<Pubkey, u64>,
    inactive: AHashSet<Pubkey>,
}

Default implementation for FeatureSet struct

Instantiating the FeatureSet struct using default method FeatureSet::default() makes all features inactive.

// Default implementation for `FeatureSet`
impl Default for FeatureSet {
    fn default() -> Self {
        Self {
            // All features disabled
            active: AHashMap::new(),
            inactive: AHashSet::from_iter((*FEATURE_NAMES).keys().cloned()),
        }
    }
}

The *FEATURE_NAMES is used to load all features since they are defined by agave_feature_set::FEATURE_NAMES constant which is a can be described as HashMap<Public Key of the feature, the description of the feature>.

Let’s explore this by creating a new Rust project and add the agave-feature-set crate

cargo new explore-feature-set

# Add agave-feature-set to Cargo.toml
cargo add agave-feature-set

Looping through the available features for SVM API

In the src/main.rs folder loop through the feature names

use agave_feature_set::FEATURE_NAMES;

fn main() {
    // Loop through the available features for the SVM version
    for (feature_name: &solana_pubkey::Pubkey, feature_description: &&str) in FEATURE_NAMES.iter() {
        println!("{feature_name} - {feature_description",);
    }
}

Instantiating the defaults for FeatureSet

use agave_feature_set::FeatureSet;

let feature_set_api = FeatureSet::default();

Methods on FeatureSet

FeatureSet methods on can activate or deactivate a feature by providing it’s public key. An example is activating SIMD 256 which raises the block limit to 60M . agave-feature-set crate provides constant modules that have a method id() which contains the Public Key for the feature activation. For SIMD 256, the module is agave_feature_set::raise_block_limits_to_60m .

The activate() method requires the feature public key and the slot that the feature will be activated.

use agave_feature_set::FeatureSet;

fn main() {
    let mut feature_manager = FeatureSet::default();

    // Activate SIMD 256 from slot zero
    feature_manager.activate(&agave_feature_set::raise_block_limits_to_60m::id(), 0);

    // To deactivate a feature, example  SIMD 256
    feature_manager.deactivate(&agave_feature_set::raise_block_limits_to_60m::id());

    // Some more useful methods are
    //

    // To check whether a feature is active, example  SIMD 256
    feature_manager.is_active(&agave_feature_set::raise_block_limits_to_60m::id());

    // To check the slot when a certain feature was activated, example  SIMD 256
    feature_manager.activated_slot(&agave_feature_set::raise_block_limits_to_60m::id());

    // To get all active features
    feature_manager.active();

    // To get all inactive features
    feature_manager.inactive();

    // To get all enabled features that trigger full inflation
    feature_manager.full_inflation_features_enabled();

    // To instantiate a `FeatureSet` struct with all features enabled at once
    let feature_manager = FeatureSet::all_enabled();
}

These methods can be used to instantiate a Bank with the features required in your SVM application.

The source code for this article can be found at - https://github.com/448-OG/SVM-Series/tree/master/svm-featues

< Previous Article - SVM Series 01: Introduction

Next Article - SVM Series 03: Compute Budget & Syscalls >

Resources

1. Feature Proposal Program’s source code - https://github.com/solana-program/feature-proposal

2. solana-feature-gate-interface - crate used within validator client to provide mechanism for features to be simultaneously activated across the network.
   https://crates.io/crates/solana-feature-gate-interface

3. solana feature activate - subcommand part of the solana command line tool used to activate features. Run solana feature --help to show other arguments.

Articles in this series

1. Introduction to SVM, a summary of SVM API blog (this article)

2. SVM Series 02: SVM Feature Activation & Management

3. SVM Series 03: Compute Budget & Syscalls

4. SVM Series 04: TransactionBatchProcessor

Introduction to SVM, a summary of SVM API blog

The Solana Virtual Machine is the entire transaction processing pipeline from validator’s runtime to executing Solana eBPF programs.

The Bank (contained in the Ledger) coordinates the networks state during any given slot by processing transactions, packing them into a block. Validators replaying a block attempt to rebuild a matching instance of the Bank.

• Transactions are processed in batches

• Each transactions contains one or multiple instructions

• Each instruction contains information to load accounts and determine write locks

• Read locks allow parallel data access across Bank instances

• Transactions are sanitized before they are processed

• De-duplication of account keys from a transaction’s instruction is done during sanitization

• After accounts are loaded, SVM loads the executable program for each instruction and provisions an eBPF virtual machine to execute the program, returning that result

• Caching is used to store the machine code from the translated program byte-code until recomputing the program byte-code is needed or when the cache is full

All of these processes are coordinated by the Bank within the SVM.

Agave SVM API

Credit: Anza’s Blog

The solana-svm Rust crate is used to build SVM applications in a protocol compliant manner. The TransactionBatchProcessor struct is the central interface of SVM which processes batches of sanitized transactions using components in the transaction pipeline. During this series we will break down each components.

// From the `solana-svm` crate.
// The method `load_and_execute_sanitized_transactions()` on this struct is used
// to process batched transactions.
pub struct TransactionBatchProcessor<FG: ForkGraph> {
  /// Bank slot (i.e. block)
  slot: Slot,
  /// Bank epoch
  epoch: Epoch,
  /// SysvarCache is a collection of system variables that are
  /// accessible from on chain programs. It is passed to SVM from
  /// client code (e.g. Bank) and forwarded to the MessageProcessor.
  pub sysvar_cache: RwLock<SysvarCache>,
  /// Programs required for transaction batch processing
  pub program_cache: Arc<RwLock<ProgramCache<FG>>>,
  /// Builtin program ids
  pub builtin_program_ids: RwLock<HashSet<Pubkey>>,
}

You can read an overview of this API on Anza’s blog - https://www.anza.xyz/blog/anzas-new-svm-api

Source Code

The source code for each article can be found at https://github.com/448-OG/SVM-Series/

Next Article: SVM Series 02: SVM Feature Activation & Management >

Token accounts are used to store the information about a token. It has the following 8 fields.

1. Mint Public Key - This is the Public Key of the Mint associated with this account

2. Owner Public Key - Represents the Public Key of the Owner of the account

3. Amount - The amount of tokens

4. Optional Delegate Public Key - If there is a Public Key authorized to manage a delegated amount

5. Account State - Whether an account is not initialized, is initialized or if the account is frozen

6. Optional Is Native - Shows the rent-exempt amount if this is a native token if the value is provided

7. Delegated Amount - The amount the Delegate Public Key can manage

8. Optional Close Authority - The public key of authorized to close the account if it is provided.

Let's write some Rust code to implement the structure of the token account. This tutorial assumes you have installed the Rust stable toolchain (version >= 1.78.0 at time of writing this)

• Create a new rust cargo project

    $ cargo new token-account-layout 
  
    # Switch to the project directory
    $ cd token-account-layout
  

• Inside the Cargo.toml file, we then import rand crate that will generate random 32 bytes to simulate a Public Key since onchain there is no random number generation capability.

    [package]
    name = "token-account-layout"
    version = "0.1.0"
    edition = "2021"
  
    [dependencies]
    rand = "0.8.5" #<-- Add this dependency
  

In the main.rs file

Note that Solana uses memory alignment to improve performance since modern CPUs perform better if they are reading or writing to aligned memory. You can do your own research about memory alignment.

In an optional value of the Token Account, due to memory alignment Solana will represent the whether a field is optional using 4 bytes:

where:
    The value exists `Option::Some`  = [1u8, 0, 0, 0]
    The value doesn't exist `Option::None` = [0u8;4]

Add the rust code to represent a padded optional value.

/// If the value is Option::Some(value)
const OPTION_IS_SOME: [u8; 4] = [1u8, 0, 0, 0];

The Account Size is of 165 bytes in length. Define this constant in the code

/// The length of the contiguous chunk of memory representing a token account.
const TOKEN_2022_ACCOUNT_LEN: usize = 165;

An image representing the account structure

Source: Solana Cookbook

Each byte range corresponds to a particular field in a Token account. Let's define these ranges as constants

/// Import `RangeInclusive` from rust core library
use core::ops::RangeInclusive;

/// Mint's Ed25519 Public key of 32 bytes in length
const MINT_ACC_RANGE: RangeInclusive<usize> = 0..=31;

/// Owner's Ed25519 Public key of 32 bytes in length
const MINT_OWNER_RANGE: RangeInclusive<usize> = 32..=63;

/// Eight bytes that hold an amount of type u64
const AMOUNT_RANGE: RangeInclusive<usize> = 64..=71;

/// 4 bytes of a Option + Ed25519 Public key of 32 bytes in length
const DELEGATE_RANGE: RangeInclusive<usize> = 72..=107;

/// One byte representing the Account state
const STATE_RANGE: usize = 108;

/// 4 Bytes Option + 8 bytes amount of type u64
/// representing current amount of rent exempt
// amount of a native token
const IS_NATIVE_RANGE: RangeInclusive<usize> = 109..=120;

/// 8 bytes of an amount of type u64 the delegate
/// can manage
const DELEGATED_AMOUNT_RANGE: RangeInclusive<usize> = 121..=128;

/// 4 bytes of a Option Type representation + Ed25519 Public key of 32 bytes in length
const CLOSE_AUTHORITY_RANGE: RangeInclusive<usize> = 129..=164;

Next, we define a struct to hold the Ed25519 public key (not that Solana public keys can be on the Ed25519 curve or off the curve)

/// Derive `Clone`, `Copy` to allow copying and cloning the Struct.
/// Derive `Default` to allow definition of a zero filled 32byte array.
/// Derive `PartialEq` to allow comparing of two structs
#[derive(Clone, Copy, Debug, Default, PartialEq)]
struct Ed25519PublicKey([u8; 32]);

impl Ed25519PublicKey {
    // generate 32 bytes random numbers
    fn new_random() -> Self {
        use rand::{thread_rng, Rng};

        let mut buffer = [0u8; 32];
        thread_rng().fill(&mut buffer[..]);

        Self(buffer)
    }

    /// Convert to 32 bytes
    fn to_bytes(&self) -> [u8; 32] {
        self.0
    }
}

impl From<&[u8]> for Ed25519PublicKey {
    fn from(bytes: &[u8]) -> Self {
        let valid_bytes: [u8; 32] = bytes
            .try_into()
            .expect("Invalid slice length for Ed25519 public key. 32 bytes are required");

        Self(valid_bytes)
    }
}

In Solana, a token account state can be Uninitialized, Initialized or Frozen.

/// The state of a token account
#[derive(Clone, Copy, Debug, Default, PartialEq)]
enum Token2022AccountState {
    /// not yet initialized
    #[default]
    Uninitialized = 0,
    /// is initialized allowing actions by the owner or delegate
    Initialized = 1,
    /// Account is frozen by the mint freeze authority preventing
    /// Any actions from the account owner or delegate
    Frozen = 2,
}

/// The `Token2022AccountState` is converted to
/// a byte representation by the numbers described
/// in its fields, like `Uninitialized = 0` 
/// using `Token2022AccountState::Uninitialized as u8`
/// converts that field to `0`
impl From<u8> for Token2022AccountState {
    fn from(value: u8) -> Self {
        match value {
            0 => Self::Uninitialized,
            1 => Self::Initialized,
            2 => Self::Frozen,
            _ => panic!("The byte value passed is not supported"),
        }
    }
}

Construct the Token Account struct

#[derive(Clone, Copy, Debug, Default, PartialEq)]
struct Token2022Account {
    /// The mint associated with this account
    mint: Ed25519PublicKey,
    /// The owner of this account.
    owner: Ed25519PublicKey,
    /// The amount of tokens this account holds.
    amount: u64,
    /// If `delegate` is `Some` then `delegated_amount` represents
    /// the amount authorized by the delegate
    delegate: Option<Ed25519PublicKey>,
    /// The account's state
    state: Token2022AccountState,
    /// If is_some, this is a native token, and the value logs the rent-exempt reserve. An Account
    /// is required to be rent-exempt, so the value is used by the Processor to ensure that wrapped
    /// SOL accounts do not drop below this threshold.
    is_native: Option<u64>,
    /// The amount delegated
    delegated_amount: u64,
    /// Optional authority to close the account.
    close_authority: Option<Ed25519PublicKey>,
}

Create helper methods to reuse code

impl Token2022Account {
    /// Converts bytes to an amount
    fn to_amount(bytes: &[u8]) -> u64 {
        // Little-endian is used
        u64::from_le_bytes(
            bytes
                .try_into()
                .expect("Invalid slice length for amount. 8 bytes are required"),
        )
    }

    /// Converts the bytes of an optional public key to Option<Ed25519PublicKey>
    fn to_optional_pubkey(bytes: &[u8]) -> Option<Ed25519PublicKey> {
        // if the first byte is zero then the public key is None
        if bytes[0] == 0 {
            Option::None
        } else {
            // Skip the first 4 bytes.
            // Since we implemented `From<[u8]> for Ed25519PublicKey`
            // using the `into()` method here auto-converts the bytes to
            // Ed25519PublicKey 
            Option::Some(bytes[4..].into())
        }
    }

    /// Converts an optional amount to a `Option<u64>`
    fn to_optional_amount(bytes: &[u8]) -> Option<u64> {
        // if the first byte is 0 then the value is None
        if bytes[0] == 0 {
            Option::None
        } else {
            // skip the first 4 bytes
            Option::Some(Self::to_amount(&bytes[4..]))
        }
    }
}

With the impl we create methods to pack Self and unpack bytes to Self

impl {
    // ... previous code here

    /// An optional public key consists of 36 bytes where
    /// 32 bytes public key and a byte representing whether
    /// it is Option::Some which is represented by `1` or
    /// if it is Option::None which is represented by `0`.
    /// However due to memory alignment, 3 bytes padding is added 
    /// (1 byte optional representation + 3 bytes padding)
    fn pack_optional_pubkey(option_pubkey: Option<&Ed25519PublicKey>) -> [u8; 36] {
        let mut buffer = [0u8; 36];

        if let Some(pubkey) = option_pubkey {
            // The byte range of the Option representation with 3 bytes padding
            buffer[0..=3].copy_from_slice(&OPTION_IS_SOME);
            // Add the 32 bytes for the public key after the padding
            buffer[4..=35].copy_from_slice(&pubkey.to_bytes());
        }

        buffer
    }


    /// Pack the Token2022Account struct (Self)
    fn pack(&self) -> [u8; 165] {
        // Initialize an empty buffer with 165 bytes of zeroes
        let mut buffer = [0u8; TOKEN_2022_ACCOUNT_LEN];

        // The range within the buffer for the `mint public key`
        buffer[MINT_ACC_RANGE].copy_from_slice(&self.mint.to_bytes());

        // The range within the buffer for the `owner public key`
        buffer[MINT_OWNER_RANGE].copy_from_slice(&self.owner.to_bytes());

        // The range within the buffer for the `8 bytes amount`
        buffer[AMOUNT_RANGE].copy_from_slice(&self.amount.to_le_bytes());

        // The range within the buffer for the `optional delegate public key`
        buffer[DELEGATE_RANGE].copy_from_slice(&Self::pack_optional_pubkey(self.delegate.as_ref()));

        // The range within the buffer for the `token account state`
        buffer[STATE_RANGE] = self.state as u8;

        // The range within the buffer for the `is_native optional amount`
        if let Some(amount) = self.is_native.as_ref() {
            // Define a buffer with 1 byte Option representation
            // 3 bytes padding for the Option representation
            // and 8 bytes amount
            let mut inner_buffer = [0u8; 12];
            // Load the optional bytes with the amount
            inner_buffer[0..=3].clone_from_slice(&OPTION_IS_SOME);
            // Add the 8 bytes amount
            inner_buffer[4..].copy_from_slice(&amount.to_le_bytes());
            // Copy this buffer into the buffer for the packed amount
            // within the range
            buffer[IS_NATIVE_RANGE].copy_from_slice(&inner_buffer)
        }

        // The range within the buffer for the `8 bytes delegate amount`
        // which the delegate authority can manage
        buffer[DELEGATED_AMOUNT_RANGE].copy_from_slice(&self.delegated_amount.to_le_bytes());

        // The range within the buffer for the `optional close authority public key`
        buffer[CLOSE_AUTHORITY_RANGE]
            .copy_from_slice(&Self::pack_optional_pubkey(self.close_authority.as_ref()));

        buffer
    }

    /// Code to unpack 165  bytes to `Token2022Account`
    fn unpack_account_data(packed: [u8; TOKEN_2022_ACCOUNT_LEN]) -> Self {
        // Since we derived default for `Token2022Account` we can then
        // we instantiate a `Token2022Account` with default values
        let mut account = Token2022Account::default();

        // Since `From<u8> for Ed25519PublicKey` is implemented,
        // use `.into()` method to auto-convert the byte range to an Ed25519PublicKey
        account.mint = packed[MINT_ACC_RANGE].into();

        // Same as above
        account.owner = packed[MINT_OWNER_RANGE].into();

        // Parse a byte slice to u64
        account.amount = Self::to_amount(&packed[AMOUNT_RANGE]);

        // Parse a byte slice to an optional public key
        account.delegate = Self::to_optional_pubkey(&packed[DELEGATE_RANGE]);

        // Parse a byte into the `AccountState` using `.into()` method
        // since `From<u8>` is implemented for `Token2022AccountState`
        account.state = packed[STATE_RANGE].into();

        // Parse a byte slice to an optional u64 amount
        account.is_native = Self::to_optional_amount(&packed[IS_NATIVE_RANGE]);

        // Parse a byte slice into optional u64 amount
        account.delegated_amount = Self::to_amount(&packed[DELEGATED_AMOUNT_RANGE]);

        // Parse a byte slice inot an optional public key
        account.close_authority = Self::to_optional_pubkey(&packed[CLOSE_AUTHORITY_RANGE]);

        account
    }
}

Lastly test the code by comparing the packed bytes to the original.

fn main() {
    let account = Token2022Account {
        mint: Ed25519PublicKey::new_random(),
        owner: Ed25519PublicKey::new_random(),
        amount: 4,
        delegate: Option::Some(Ed25519PublicKey::new_random()),
        state: Token2022AccountState::Frozen,
        // If is_some, this is a native token, and the value logs the rent-exempt reserve. An Account
        // is required to be rent-exempt, so the value is used by the Processor to ensure that wrapped
        // SOL accounts do not drop below this threshold.
        is_native: Option::Some(2),
        delegated_amount: 1,
        close_authority: Option::Some(Ed25519PublicKey::new_random()),
    };

    let packed = account.pack();
    let unpacked = Token2022Account::unpack_account_data(packed);
    assert_eq!(&account, &unpacked);
}

That's it! A custom implementation of packing and unpacking a Token Account based on Solana's memory layout.

CAUTION: This code is not production grade or optimized, it is meant to illustrate the byte ranges of a region of memory containing a Solana Token Account

The source code — https://github.com/448-OG/Blinded-Diffie-Hellman-merkle/blob/master/src/main.rs

In 1982 David Chaum conceived the idea of anonymous cryptographic electronic cash using cryptographic proofs to mint electronic and melt electronic tokens.

The security proofs were guaranteed using blinded digital signature schemes that provided unlinkability between withdrawal and spending transactions. The concept of blind signatures can be explained by a scenario where a provider wants an entity to commit to a certain proof even though that entity does not know the proof. The provider can then reveal the proof later or send it to another provider who can reveal the proof to the entity.

Ecash has gained popularity in Bitcoin in projects like Cashu and Fedimint due to their anonymity and scalability properties.

In this article we will look at how to construct a simple Ecash token using Blinded Diffie–Hellman–Merkle key exchange.

Diffie–Hellman–Merkle key is a protocol for exchanging secrets between parties over a public channel where at least one of the party’s public keys is known.

Our algorithm for Blinded Diffie-Hellman-Merkle key exchange with comprise of:

1. a secret key and a public key for the mint

2. a secret key for the provider

3. a nonce generated by the provider

4. an algorithm to transform our secret into a point on an elliptic curve (blinding factor)

5. an algorithm to blind the provider secret (blinded message)

6. an algorithm to generate a blinded signature from the blinded message

7. an algorithm to unblind the token from the blinded signature

The steps to generate the token are:

Where we assume:

• Bob is the Mint

• Alice is the Provider

• Eve is another Provider that can redeem the tokens generated by Alice by sending the tokens to Bob

• Generator represented by G is a known point on the elliptic curve, in our case the Ristretto Point on Curve25519 algorithm

• hash_to_curve() is an algorithm that takes a 32 byte secret, hashes it with a cryptographically secure hashing algorithm like SHA256 and computes a point on the elliptic curve using the result of the hashed bytes. The hashing algorithm blinds the secret because one cannot guess the secret from the hash.This is known as preimage resistance.

1. Bob generates a secret key k from some 32 random bytes and then multiplies that by a generator G to generate a public key K and let’s all providers know K

   K = k * G

2. Alice generates a secret key yielding 32 bytes

3. Alice generates a random nonce as the blinding_factor

4. Alice generates a point on the elliptic curve by running the hash to curve algorithm with her the secret key

   hash_to_curve_result = hash_to_curve(sha512_hash(secret))

5. Alice then computes a blinded_message by multiplying the blinding_factor by G and then adding it to the elliptic curve point of the hashed secret hash_to_curve

   blinded_message = hash_to_curve_result + blinding_factor * G

6. Alice then sends the blinded_message to Bob

7. Bob who computes the blinded_signature by multiplying the blinded message with his secret k

   blinded_signature = blinded_message * k

8. Bob then responds to Alice with the blinded_signature

9. Alice computes the unblinded_secret by multiplying the blinding_factor with Bob’s K (public key) and the subtracting this from the blinded_signature

   token = blinded_signature — binding_factor * K

10. Alice can now share her secret s and token secret with Carol

11. Carol can redeem the token by sending the secret s and token to Bob

12. Bob verifies that the token is authentic by running the hash_to_curve() algorithm with the secret key generated by Alice and passed to Carol

    hash_to_curve_result = hash_to_curve(sha512_hash(secret))

13. Bob then multiplies the hash_to_curve with his own secret key k

    hash_to_curve_result * k

14. Bob then compares the result of the previous step with the token and rejects the token if the result of the comparison is false

    assert( (hash_to_curve_result * k) == token )

Let’s implement these steps using the Rust programming language. We assume you have already installed Rust programming language toolchain and its cargo project manager.

Using the commandline create a new Rust project using cargo and switch to that directory

$ cargo new Blinded-Diffie-Hellman-Merkle --name blinded-dhm #Create a new project

$ cd Blinded-Diffie-Hellman-Merkle # Switch to the directory

Inside the Cargo.toml add the dependencies for cryptographically secure random number generation (rand) , elliptic curves) and hashing (sha2)

[dependencies]
curve25519-dalek = { version = "4.1.2", features = [
    "rand_core",
    "digest",
    "precomputed-tables",
] }
rand = "0.8.5"
sha2 = "0.10.8"

In curve25519-dalek - the feature rand_core enables generation of random bytes from the operating system - the feature digest enables the SHA512 hashing algorithm when calling hash to curve -precomputed-tables enables faster multiplication of scalar points with points on the elliptic curve

Inside the main.rs file, lets create the code for our algorithm

1. Import all the data types and traits we will use

    use curve25519_dalek::{ristretto::RistrettoPoint, scalar::Scalar, traits::Identity};
    use rand::rngs::OsRng;
    use sha2::Sha512;
    use std::ops::{Mul, Sub};
   

2. Next, create a struct MintCrypto to perform cryptographic operations for the mint.

        /// Cryptographic operations for the mint (Bob)
        pub struct MintCrypto {
            // the secret key for a particular Ecash denomination
            secret: Scalar,
            // the public key for the Ecash denomination
            // specific to the secret key above
            public: RistrettoPoint,
        }
   
        impl MintCrypto {
            /// Instantiate our struct
        pub fn new() -> Self {
            // generate a secret key and store it in a `Scalar`
            let secret = Scalar::random(&mut OsRng);
            // Generate the public key
            let public = RistrettoPoint::identity().mul(&secret);
   
            Self { secret, public }
        }
   
        /// Generate a blinded token to return to Alice
        pub fn blinded_token(&self, blinded_message: RistrettoPoint) -> RistrettoPoint {
            blinded_message.mul(self.secret)
        }
   
        /// Prove that the unblinded token is valid
        pub fn proof(&self, unblinded_token: UnblindedToken) -> bool {
            // Compute the point on elliptic curve
            let hash_to_curve =
                RistrettoPoint::hash_from_bytes::<Sha512>(unblinded_token.secret.as_bytes());
            // Check if multiplying the elliptic curve point
            // equals the token
            self.secret.mul(hash_to_curve) == unblinded_token.proof
        }
   
        /// Fetch the public key of the mint
        pub fn public_key(&self) -> RistrettoPoint {
            self.public
        }
    }
   

3. Next, create a struct UnblindedToken which contains the secret key of Alice and the token issued by the mint. UnblindedToken is what is shared with Carol.

        /// Contains the unblinded token from the mint and providers secret key
        pub struct UnblindedToken {
            /// Provider's secret key
            pub secret: Scalar,
            /// The unblinded token from the mint
            pub proof: RistrettoPoint,
        }
   

4. Next, we create the cryptographic primitives for the provider

    /// Cryptographic operations for the Provider (Alice)
    pub struct ProviderCrypto {
        // The secret known to the provider
        secret: Scalar,
        // A random nonce that guarantees each mint token is unique (nonce)
        blinding_factor: Scalar,
        // The blinded message
        blinded_message: RistrettoPoint,
        // The public key of the mint which can be for a particular token's denomination
        ecash_token_public: RistrettoPoint,
    }
   
    impl ProviderCrypto {
        /// instantiate the struct by passing in the public key of the mint
        pub fn new(ecash_token_public: RistrettoPoint) -> Self {
            // Generate 32 cryptographically secure bytes
            let secret = Scalar::random(&mut OsRng);
            // Generate 32  cryptographically secure bytes (nonce)
            let blinding_factor = Scalar::random(&mut OsRng);
            // Run the `secret` first through a SHA512 hash and then generate a point on the elliptic curve
            let hash_to_curve_result = RistrettoPoint::hash_from_bytes::<Sha512>(secret.as_bytes());
            // Generate a `blinded_message` that is unique by adding the `hash_to_curve_result`
            // to a point on the elliptic curve generated by the multiplying the `blinding_factor` by a generator `G`
            let blinded_message =
                hash_to_curve_result + (RistrettoPoint::identity().mul(&blinding_factor));
   
            Self {
                secret,
                blinding_factor,
                blinded_message,
                ecash_token_public,
            }
        }
   
        /// Unblind a token from the mint
        pub fn unblind(&self, token: RistrettoPoint) -> UnblindedToken {
            // Performs the unblinding operation `token - blinding_factor * K` where `K` is the public key of the mint
            let proof = token.sub(self.blinding_factor.mul(self.ecash_token_public));
   
            UnblindedToken {
                secret: self.secret,
                proof,
            }
        }
   
        /// Get the unblinded token and provider secret
        pub fn blinded_message(&self) -> RistrettoPoint {
            self.blinded_message
        }
    }
   

5. Lastly, in the main function, we use our data structures to generate and verify a token

        fn main() {
        // Generate primitives for the mint
        let mint = MintCrypto::new();
   
        // Generate primitives for the provider
        let provider = ProviderCrypto::new(mint.public_key());
   
        // The mint generates a blinded token from the provider's `blinded_message`
        let blinded_token = mint.blinded_token(provider.blinded_message());
   
        // The provider unblinds the token
        let unblinded_token = provider.unblind(blinded_token);
   
        // The mint proves whether the token is valid or not
        assert!(mint.proof(unblinded_token));
    }
   

That’s it. We have created an algorithm for generating Ecash tokens using the blinded Diffie-Hellman-Merkle algorithm.

References

1. https://en.wikipedia.org/wiki/Ecash

2. http://www.hit.bme.hu/~buttyan/courses/BMEVIHIM219/2009/Chaum.BlindSigForPayment.1982.PDF

3. https://gist.github.com/callebtc/557e4cc15f9e43d7474c7cb3d31ee8ed

4. https://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exchange

5. https://en.wikipedia.org/wiki/Curve25519

6. https://en.wikipedia.org/wiki/Montgomery_curve

7. https://ristretto.group/

Previous Articles.

1. Parsing Bitcoin Transactions in Rust Programming Language Standard Library — https://448.africa/parsing-bitcoin-transactions-in-rust-programming-language-standard-library-20c06a23a564

In this article we will look at parsing input scriptSig bytes as a Bitcoin script.

Understanding Bitcoin Script

Bitcoin script is a simple, stack-based, Forth-like programming language used in the Bitcoin protocol for transaction processing. It’s not Turing-complete, meaning it doesn’t have loops, which is a design choice to prevent attacks on the network like infinite loops and logic bombs that would be evaluated in a way that causes denial of service attacks.

Each Bitcoin transaction includes a script that validates the conditions under which the transaction can be redeemed. The script is processed from left to right, and if it completes without any errors and the final result is true, the transaction is considered valid.

Most Common Bitcoin Scripts

Bitcoin’s primary focus is sending bitcoins from one address to one or many addresses. This means that most scripts are created to allow one or multiple wallets to spend bitcoins. The Bitcoin core network has 5 standard scripts that are known to all nodes and miners. The standard scripts and their Assembly Protocol (ASM) representation:
 — Note that all the bytes should appear as hex

1. Pay to Public Key (P2PK) — This is rarely used.
   <Public Key> OP_CHECKSIG

2. Pay to Public Key Hash (P2PKH)
   OP_DUP OP_HASH160 <PublicKey Hash> OP_EQUAL OP_CHECKSIG

3. Pay to Multi-signature (P2MS) — limited to 15 keys
   M <Public Key 1> <Public Key 2> … <Public Key N> N OP_CHECKMULTISIG — where M is the threshold and N is the number of signatures

4. Pay to Script Hash (P2SH)
   OP_HASH160 <20 bytes of the hash> OP_EQUAL

5. Pay to Witness Public Key Hash (P2WPKH)
   OP_0 *<20-byte hash>*

6. Pay to Witness Script Hash (P2WSH)
   OP_0 <32-byte hash>

7. Pay to Tap Root (P2TR)
   OP_1 <32-byte hash>

8. Data Output (OP_RETURN)
   OP_RETURN <data>

We will look at decoding our previous script data into these standard scripts in Rust.

Detecting the script type

At the time of writing there are 8 standard scripts which makes it very easy to identify the type of script by parsing the first or first and second OPCODE NOTE that the * in OP_PUSHBYTES_*means the number of bytes:

1. P2PK — OP_PUSHBYTES_65

2. P2PKH — OP_DUP

3. P2MS — OP_1 — OP_16 OP_PUSHBYTES_*

4. P2SH — OP_HASH160

5. P2WPKH — OP_0 OP_PUSHBYTES_20

6. P2WSH — OP_0 OP_PUSHBYTES_32

7. P2TR — OP_1 OP_PUSHBYTES_32

8. Data (OP_RETURN) — OP_RETURN

Now we will create a module called scripts in the scripts.rs file and import it to the main.rs file.

//... main.rs previous code here
mod scripts;
pub use scripts::*;

fn main() {
    // .. previous code from previous tutorial
}

In our scripts.rs file we will import standard library types for addition , I/O and error handling. We will convert all our errors to std::io::Error

use std::{
    io::{self, Cursor, Error, ErrorKind, Read},
    ops::Add,
};

We will create methods to parse the standard scriptSig inside the impl block of StandardScripts struct

/// Handles scriptSig parsing
#[derive(Debug, Clone, Copy)]
pub struct StandardScripts;

impl StandardScripts {}

We will parse based on the following:

If:

• First OPCODE is OP_PUSHBYTES_65 then parse as P2PK

• First OPCODE is OP_DUP then parse as P2PKH

• First OPCODE is OP_RETURN then parse as Data(OP_RETURN)

• First OPCODE is OP_0 and second OPCODE is OP_PUSHBYTES_20 then parse as P2WPKH

• First OPCODE is OP_0 and second OPCODE is OP_PUSHBYTES_32 then parse as P2WSH

• First OP_CODE is between OP_1 and OP_16 and second OPCODE is OP_PUSHBYTES_32 then parse as P2TR

• First OP_CODE is between OP_1 and OP_16 and second OPCODE is OP_PUSHBYTES_* then try parsing as P2MS else return an std::io::Error with ErrorKind::InvalidData with a helpful error message.

Let’s define the OPCODES to parse

/// Supported OPCODEs for standard scripts in scriptSig
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
#[allow(non_camel_case_types)]
pub enum Opcode {
    OP_HASH160,
    OP_CHECKSIG,
    OP_EQUAL,
    OP_EQUALVERIFY,
    OP_CHECKMULTISIG,
    OP_DUP,
    OP_RETURN,
    OP_0,
    OP_1,
    /// Handles OP_2 to OP_16
    Num(u8),
    /// Handles all OP_PUSHBYTES_*
    PushBytes(u8),
    /// Useful in error handling for unsupported opcodes
    UnsupportedOpcode,
}

impl Opcode {
    /// Parse an opcode from a hex decoded byte
    pub fn from_byte(byte: u8) -> Self {
        match byte {
            169 => Self::OP_HASH160,
            /// All OP_PUSHBYTES_*
            1..=75 => Self::PushBytes(byte),
            172 => Self::OP_CHECKSIG,
            135 => Self::OP_EQUAL,
            136 => Self::OP_EQUALVERIFY,
            174 => Self::OP_CHECKMULTISIG,
            118 => Self::OP_DUP,
            106 => Self::OP_RETURN,
            0 => Self::OP_0,
            81 => Self::OP_1,
            /// All OP_2 - OP_16
            82..=96 => {
                let to_num = match byte {
                82 => 2u8,
                83 => 3,
                84 => 4,
                85 => 5,
                86 => 6,
                87 => 7,
                88 => 8,
                89 => 9,
                90 => 10,
                91 => 11,
                92 => 12,
                93 => 13,
                94 => 14,
                95 => 15,
                96 => 16,
                _ => return Self::UnsupportedOpcode,
            };
            Self::Num(to_num)
        }
        _ => Self::UnsupportedOpcode,
        }    
    }

    /// Handles reading OP_PUSHBYTES_*
    pub fn read_bytes(&self, bytes: &mut Cursor<&[u8]>) -> io::Result<Vec<u8>> {
        // Store all parsed bytes for `OP_PUSHBYTES_*`
        let mut buffer = Vec::<u8>::new();

        match self {
            Self::PushBytes(byte_len) => {
            // Gets the current position and adds the length of the
            let new_position = (bytes.position() as usize).add(*byte_len as usize);
            // Read the byte slice from the current cursor position the byte length
            buffer.extend_from_slice(&bytes.get_ref()[bytes.position() as usize..new_position]);
            // Set the cursor position to the previous cursor position + the byte length
            bytes.set_position(new_position as u64);

            Ok(buffer)
        }
        _ => Err(io::Error::new(
            ErrorKind::Unsupported,
            "This operation is not supported",
            )),
        }
    }
}

Next we implement the trait TryFrom for our Opcode enum so that we can convert it easily to a Rust String

impl TryFrom for String {
    // All errors are converted to std::io::Error
    type Error = io::Error;

    fn try_from(value: Opcode) -> Result<Self, Self::Error> {
        let opcode = match value {
            Opcode::OP_HASH160 => "OP_HASH160",
            Opcode::PushBytes(bytes_len) => {
                return Ok(String::from("OP_PUSHBYTES_").add(bytes_len.to_string().as_str()))
            }
            Opcode::OP_CHECKSIG => "OP_CHECKSIG",
            Opcode::OP_EQUAL => "OP_EQUAL",
            Opcode::OP_EQUALVERIFY => "OP_EQUALVERIFY",
            Opcode::OP_CHECKMULTISIG => "OP_CHECKMULTISIG",
            Opcode::OP_DUP => "OP_DUP",
            Opcode::OP_RETURN => "OP_RETURN",    
            Opcode::OP_0 => "OP_0",
            Opcode::OP_1 => "OP_1",
            Opcode::Num(value) => return Ok(String::from("OP_").add(value.to_string().as_str())),
            Opcode::UnsupportedOpcode => {
                return Err(io::Error::new(
                    ErrorKind::InvalidData,
                    "Unsupported Opcode. Opcode not part of Bitcoin Core standard scripts",
                    ))
                }
             };

         Ok(opcode.into())
    }
}

Next we create a struct with methods to build our scriptSig into a String

#[derive(Debug, Default)]
pub struct ScriptBuilder(Vec<String>);

impl ScriptBuilder {
    /// Initialize `Self` with defaults
    pub fn new() -> Self {
        Self::default()
    }

    /// This will receive an `Opcode` and convert it
    /// into a String since we implemented
    /// `TryFrom for String`
    pub fn push_opcode(&mut self, opcode: Opcode) -> io::Result<&mut Self> {
        let opcode_string: String = opcode.try_into()?;
        self.0.push(opcode_string);

        Ok(self)
    }

    /// This will convert our bytes into hex and
    /// push the opcode `OP_PUSHBYTES_*`
    pub fn push_bytes(&mut self, bytes: &[u8]) -> io::Result<&mut Self> {
        self.0.push(hex::encode(bytes));

        Ok(self)
    }

    /// Next we collect the Vector of Strings
    /// into one String
    pub fn build(self) -> String {
        self.0
            .into_iter()
            .map(|mut part| {
                part.push(' ');
                part
            })
            .collect::<String>()
            .trim()
            .into()
    }  
}

Next we implement the parsing methods for our StandardScripts struct

We implement the parse method to detect the scriptSig

impl StandardScripts {
    /// Decides which scriptSig to parse
    pub fn parse(bytes: &mut Cursor<&[u8]>) -> io::Result<String> {
        // Get the first OPCODE
        let mut opcode_buffer = [0u8; 1];
        bytes.read_exact(&mut opcode_buffer)?;
        // Convert our byte into an `Opcode`
        let first_opcode = Opcode::from_byte(opcode_buffer[0]);

        match first_opcode {
            // If `OP_PUSHBYTES_65` then parse as P2PK
            Opcode::PushBytes(65) => Self::parse_p2pk(bytes),
            // If `OP_DUP` then parse as P2PKH
            Opcode::OP_DUP => Self::parse_p2pkh(bytes),
            // If `OP_HASH160` then parse as P2PK
            Opcode::OP_HASH160 => Self::parse_p2sh(bytes),
            // If `OP_RETURN` then parse as Data(OP_RETURN)
            Opcode::OP_RETURN => Self::parse_data(bytes),
            // If `OP_0` as first OPCODE and OP_PUSHBYTES_20 is second OPCODE then parse as P2WPKH
            // Else if `OP_0` as first OPCODE and OP_PUSHBYTES_32 is second OPCODE then parse as P2WSH
            // Else return an an error if `OP_0` is first OPCODE
            Opcode::OP_0 => {
                bytes.read_exact(&mut opcode_buffer)?;
                let second_opcode = Opcode::from_byte(opcode_buffer[0]);
                if second_opcode.eq(&Opcode::PushBytes(20)) {
                    Self::parse_p2wpkh(bytes)
                } else if second_opcode.eq(&Opcode::PushBytes(32)) {
                    Self::parse_p2wsh(bytes)
                } else {
                    return Self::to_io_error(
                    "Invalid Script. Expected OP_PUSHBYTES_20 or OP_PUSHBYTES_32 after OP_0",
                    );
                }
            }
            _ => {
                // If `OP_1` as first OPCODE and OP_PUSHBYTES_32 is second OPCODE then parse as P2TR
                // Else try parsing as P2MS

                bytes.read_exact(&mut opcode_buffer)?;
                let second_opcode = Opcode::from_byte(opcode_buffer[0]);

                if first_opcode.eq(&Opcode::OP_1) && second_opcode.eq(&Opcode::PushBytes(32)) {
                    Self::parse_p2tr(bytes)
                } else {
                    // Reset current position of cursor to the beginning
                    bytes.set_position(bytes.position() - 2);
                    Self::parse_p2ms(bytes)
                }
            }
        }
    }
}

Next, we create a method to handle errors since they are common whereby they return a std::io::Error with ErrorKind and custom message.

impl StandardScripts {
    // ..previous methods here

    // Error Handling returning an `io::Result` to avoid
    // having to add `Err()` whenever we call this method.
    // Our message is unique so we add that as argument
    pub fn to_io_error(message: &str) -> io::Result<String> {
        Err(io::Error::new(ErrorKind::InvalidData, message))
    }
}

1. Method to parse P2PK

impl StandardScripts {
    /// Parse as P2PK
    pub fn parse_p2pk(bytes: &mut Cursor<&[u8]>) -> io::Result<String> {
        // Cursor is already at second byte to we parse
        // 65 bytes from that position to get the
        // Uncompressed Public Key
        let mut public_key_bytes = [0u8; 65];
        bytes.read_exact(&mut public_key_bytes)?;
        // Next we parse OP_CHECKSIG
        let mut op_checksig_byte = [0u8; 1];
        bytes.read_exact(&mut op_checksig_byte)?;
        let op_checksig = Opcode::from_byte(op_checksig_byte[0]);

        if op_checksig.ne(&Opcode::OP_CHECKSIG) {
            return Err(Error::new(
                ErrorKind::InvalidData,
                "Invalid Data. Expected OP_CHECKSIG as last byte of the script.",
            ));
        }

        // Lastly, we build our script
        let mut script_builder = ScriptBuilder::new();
        script_builder
        .push_opcode(Opcode::PushBytes(65))?
        .push_bytes(&public_key_bytes)?
        .push_opcode(Opcode::OP_CHECKSIG)?;

        Ok(script_builder.build())
    }
}

2. Method to parse P2PKH

impl StandardScripts {
    // .. previous methods here
    /// Parse P2PKH
    pub fn parse_p2pkh(bytes: &mut Cursor<&[u8]>) -> io::Result<String> {
        let mut opcode_buffer = [0u8; 1];

        bytes.read_exact(&mut opcode_buffer)?;
        // Parse second OPCODE as OP_HASH160
        let should_be_ophash160 = Opcode::from_byte(opcode_buffer[0]);
        if should_be_ophash160.ne(&Opcode::OP_HASH160) {
            return Err(Error::new(
                ErrorKind::InvalidData,
                "Invalid Data. Expected OP_HASH160 as second byte of the script.",
            ));
        }

        bytes.read_exact(&mut opcode_buffer)?;
            // Parse third OPCODE as `OP_PUSHBYTES_20`
            let should_be_op_pushbytes20 = Opcode::from_byte(opcode_buffer[0]);
            if should_be_op_pushbytes20.ne(&Opcode::PushBytes(20)) {
                return Err(Error::new(
                ErrorKind::InvalidData,
                    "Invalid Data. Expected OP_PUSHBYTES_20 as third byte of the script.",
                ));
             }

            // Get the 20 bytes of the Hash160
            let mut hash160_bytes = [0u8; 20];
            bytes.read_exact(&mut hash160_bytes)?;

            // Parse the next byte as OP_EQUALVERIFY
            bytes.read_exact(&mut opcode_buffer)?;
            let should_be_opequalverify = Opcode::from_byte(opcode_buffer[0]);
            if should_be_opequalverify.ne(&Opcode::OP_EQUALVERIFY) {
                return Err(Error::new(
                    ErrorKind::InvalidData,
                        "Invalid Data. Expected OP_EQUALVERIFY after reading 20 bytes after third byte of the script.",
                ));
            }

            // Parse the next byte as OP_CHECKSIG
            bytes.read_exact(&mut opcode_buffer)?;
            let should_be_opchecksing = Opcode::from_byte(opcode_buffer[0]);
            if should_be_opchecksing.ne(&Opcode::OP_CHECKSIG) {
                return Err(Error::new(
                    ErrorKind::InvalidData,
                    "Invalid Data. Expected OP_CHECKSIG after reading OP_EQUALVERIFY byte in the script.",
                ));
            }

            // Build our script into a Sctring
            let mut script_builder = ScriptBuilder::new();
            script_builder
                .push_opcode(Opcode::OP_DUP)?
                .push_opcode(Opcode::OP_HASH160)?
                .push_opcode(Opcode::PushBytes(20))?
                .push_bytes(&hash160_bytes)?
                .push_opcode(Opcode::OP_EQUALVERIFY)?
                .push_opcode(Opcode::OP_CHECKSIG)?;

                Ok(script_builder.build())
            }
}

3. Method to parse P2SH

impl StandardScripts {
// .. Previous methods here
   /// Parse P2SH
    pub fn parse_p2sh(bytes: &mut Cursor<&[u8]>) -> io::Result<String> {
        let mut script_buffer = [0u8; 1];

        bytes.read_exact(&mut script_buffer)?;

        // Second OPCODE should be OP_PUSHBYTES_20
        let second_opcode = Opcode::from_byte(script_buffer[0]);
        if second_opcode.ne(&Opcode::PushBytes(20)) {
            return Self::to_io_error(
                "Invalid Data. Expected an OP_PUSHBYTES_20 opcode after OP_HASH160",
            );
        }

        // Read the 20 bytes of HASH160
        let mut bytes_20_buffer = [0u8; 20];
        bytes.read_exact(&mut bytes_20_buffer)?;

        bytes.read_exact(&mut script_buffer)?;
        let last_opcode = Opcode::from_byte(script_buffer[0]);
        if last_opcode.ne(&Opcode::OP_EQUAL) {
            return Self::to_io_error(
                "Invalid Data. Expected an OP_EQUAL opcode after reading 20 bytes",
            );
        }

        // Build the script into a String
        let mut script_builder = ScriptBuilder::new();
        script_builder
            .push_opcode(Opcode::OP_HASH160)?
            .push_opcode(Opcode::PushBytes(20))?
            .push_bytes(&bytes_20_buffer)?
            .push_opcode(Opcode::OP_EQUAL)?;

        Ok(script_builder.build())
    }
}

4. Parse Data(OP_RETURN)

impl StandardScripts {
    //.. Previous code here
    // Parse OP_RETURN
    pub fn parse_data(bytes: &mut Cursor<&[u8]>) -> io::Result<String> {
        let mut script_buffer = [0u8; 1];

        bytes.read_exact(&mut script_buffer)?;
        // Get second OPCODE which is `OP_PUSHBYTES_*`
        let second_opcode = Opcode::from_byte(script_buffer[0]);
        /// Read the number of bytes specified by second OPCODE
        let data_bytes = second_opcode.read_bytes(bytes)?;

        let mut script_builder = ScriptBuilder::new();
        script_builder
        .push_opcode(Opcode::OP_RETURN)?
        .push_opcode(second_opcode)?
        .push_bytes(&data_bytes)?;

        Ok(script_builder.build())
    }
}

5. Parse P2WPKH

impl StandardScripts {
    //.. Previous code here
    /// Parse P2WPKH
    pub fn parse_p2wpkh(bytes: &mut Cursor<&[u8]>) -> io::Result<String> {
        /// Read the next 20 bytes
        let mut pubkey_hash_bytes = [0u8; 20];
        bytes.read_exact(&mut pubkey_hash_bytes)?;

        let mut scripts = ScriptBuilder::new();
        scripts
        .push_opcode(Opcode::OP_0)?
        .push_opcode(Opcode::PushBytes(20))?
        .push_bytes(&pubkey_hash_bytes)?;

        Ok(scripts.build())
    }
}

6. Parse P2WSH

impl StandardScripts {
    //.. Previous code here
    /// Parse P2WSH
    pub fn parse_p2wsh(bytes: &mut Cursor<&[u8]>) -> io::Result<String> {
        // Parse next 32 bytes
        let mut hash_bytes = [0u8; 32];
        bytes.read_exact(&mut hash_bytes)?;

        let mut scripts = ScriptBuilder::new();
        scripts
        .push_opcode(Opcode::OP_0)?
        .push_opcode(Opcode::PushBytes(32))?
        .push_bytes(&hash_bytes)?;

        Ok(scripts.build())
    }
}

7. Parse P2TR

impl StandardScripts {
    //.. Previous code here
    /// Parse P2TR
    pub fn parse_p2tr(bytes: &mut Cursor<&[u8]>) -> io::Result<String> {
        // Parse next 32 bytes
        let mut hash_bytes = [0u8; 32];
        bytes.read_exact(&mut hash_bytes)?;

        let mut scripts = ScriptBuilder::new();
        scripts
        .push_opcode(Opcode::Num(1))?
        .push_opcode(Opcode::PushBytes(32))?
        .push_bytes(&hash_bytes)?;

        Ok(scripts.build())
    }
}

8. Parse P2MS

impl StandardScripts {
    // .. previous code here

    /// Parse a P2MS.
    /// Also checks to see if the number of public keys parsed is equal to number of public keys requires
    /// or if the parsed public keys are less than the threshold
    pub fn parse_p2ms(bytes: &mut Cursor<&[u8]>) -> io::Result<String> {
        let mut opcode_buffer = [0u8; 1];
        bytes.read_exact(&mut opcode_buffer)?;
        let threshold_opcode = Opcode::from_byte(opcode_buffer[0]);

        match threshold_opcode {
                Opcode::Num(_) | Opcode::OP_1 => {
                    let mut script_builder = ScriptBuilder::new();
                    script_builder.push_opcode(threshold_opcode)?;
                    // The number of public keys parsed
                    let mut pubkey_count = 0u8;
                    // The number of public keys specified in the scriptSig
                    let parsed_pubkey_count: u8;
                    let mut pushbytes_buffer = Vec::<u8>::new();

                    loop {
                        bytes.read_exact(&mut opcode_buffer)?;
                        let current_opcode = Opcode::from_byte(opcode_buffer[0]);

                        match current_opcode {
                            Opcode::Num(value) => {
                            parsed_pubkey_count = value;
                            script_builder.push_opcode(current_opcode)?;
                            //Break the loop if a `OP_1 to OP_16` is encountered
                            break;
                        }
                }
                Opcode::PushBytes(value) => {
                    let new_position = bytes.position() as usize + value as usize;
                    let read_bytes =
                    &bytes.get_ref()[bytes.position() as usize..new_position];
                    pushbytes_buffer.extend_from_slice(read_bytes);

                    script_builder
                        .push_opcode(current_opcode)?
                        .push_bytes(&pushbytes_buffer)?;

                    pushbytes_buffer.clear();
                    bytes.set_position(new_position as u64);
                    pubkey_count = pubkey_count.add(1);
                }
                _ => {
                    return Self::to_io_error(
                        "Invalid Script. Expected a PUSH_BYTES_* or OP_1..16",
                    )
                }
            }
        }

        if pubkey_count.ne(&parsed_pubkey_count) {
            return Self::to_io_error(
                "Invalid Script. The number of public keys for multisignature is less than or greater than the script requirements.",
            );
        }

        match threshold_opcode {
            Opcode::Num(threshold_inner) => {
                if parsed_pubkey_count.lt(&threshold_inner) {
                    return Self::to_io_error(
                        "Invalid Script. The number of public keys for multisignature is less the threshold.",
                    );
                }
            }
            _ => (),
        }

        // Parse next byte and check if it is OP_CHECKMULTISIG opcode
        bytes.read_exact(&mut opcode_buffer)?;
        let opcheck_multisig = Opcode::from_byte(opcode_buffer[0]);

        if opcheck_multisig.ne(&Opcode::OP_CHECKMULTISIG) {
            return Self::to_io_error(
                "Invalid Script. OP_CHECKMULTISIG opcode should be next",
            );
            }
            script_builder.push_opcode(Opcode::OP_CHECKMULTISIG)?;

            Ok(script_builder.build())
        }
        _ => Self::to_io_error("Invalid Script."),
        }
    }
}

Lastly, we need to test that our code works inside the main function in main.rs file and we assert that each outcome is successful using the .is_ok() method on std::io::Result

fn main() {
    // .. Previous code here

    let p2pk_bytes = hex!("410000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000ac");
    let mut p2pk = Cursor::new(p2pk_bytes.as_ref());
    let outcome = StandardScripts::parse(&mut p2pk);
    assert!(outcome.is_ok());
    dbg!(&outcome.unwrap());

    let p2pkh_bytes = hex!("76a914000000000000000000000000000000000000000088ac");
    let mut p2pkh = Cursor::new(p2pkh_bytes.as_ref());
    let outcome = StandardScripts::parse(&mut p2pkh);
    assert!(outcome.is_ok());
    dbg!(&outcome.unwrap());

    let p2sh_bytes = hex!("a914748284390f9e263a4b766a75d0633c50426eb87587");
    let mut p2sh = Cursor::new(p2sh_bytes.as_ref());
    let outcome = StandardScripts::parse(&mut p2sh);
    assert!(outcome.is_ok());
    dbg!(&outcome.unwrap());

    let op_return_bytes = hex!("6a0b68656c6c6f20776f726c64");
    let mut op_return = Cursor::new(op_return_bytes.as_ref());
    let outcome = StandardScripts::parse(&mut op_return);
    assert!(outcome.is_ok());
    dbg!(&outcome.unwrap());

    let p2wpkh_bytes = hex!("00140000000000000000000000000000000000000000");
    let mut p2wpkh = Cursor::new(p2wpkh_bytes.as_ref());
    let outcome = StandardScripts::parse(&mut p2wpkh);
    assert!(outcome.is_ok());
    dbg!(&outcome.unwrap());

    let p2wsh_bytes = hex!("00200000000000000000000000000000000000000000000000000000000000000000");
    let mut p2wsh = Cursor::new(p2wsh_bytes.as_ref());
    let outcome = StandardScripts::parse(&mut p2wsh);
    assert!(outcome.is_ok());
    dbg!(&outcome.unwrap());

    let p2tr_bytes = hex!("51200000000000000000000000000000000000000000000000000000000000000000");
    let mut p2tr = Cursor::new(p2tr_bytes.as_ref());
    let outcome = StandardScripts::parse(&mut p2tr);
    assert!(outcome.is_ok());
    dbg!(&outcome.unwrap());

    let p2ms_3_bytes = hex!("524104d81fd577272bbe73308c93009eec5dc9fc319fc1ee2e7066e17220a5d47a18314578be2faea34b9f1f8ca078f8621acd4bc22897b03daa422b9bf56646b342a24104ec3afff0b2b66e8152e9018fe3be3fc92b30bf886b3487a525997d00fd9da2d012dce5d5275854adc3106572a5d1e12d4211b228429f5a7b2f7ba92eb0475bb14104b49b496684b02855bc32f5daefa2e2e406db4418f3b86bca5195600951c7d918cdbe5e6d3736ec2abf2dd7610995c3086976b2c0c7b4e459d10b34a316d5a5e753ae");
    let mut p2ms_3 = Cursor::new(p2ms_3_bytes.as_ref());
    let outcome = StandardScripts::parse(&mut p2ms_3);
    assert!(outcome.is_ok());
    dbg!(&outcome.unwrap());

    let p2ms_2_bytes = hex!("51210000000000000000000000000000000000000000000000000000000000000000002100000000000000000000000000000000000000000000000000000000000000000052ae");
    let mut p2ms_2 = Cursor::new(p2ms_2_bytes.as_ref());
    let outcome = StandardScripts::parse(&mut p2ms_2);
    assert!(outcome.is_ok());
    dbg!(&outcome.unwrap());
}

That’s it. We have successfully parsed the input scriptSig part.

References

1. Repository — https://github.com/448-OG/BitcoinTransactions

2. https://wiki.bitcoinsv.io/index.php/Opcodes_used_in_Bitcoin_Script

3. https://learnmeabitcoin.com/technical/transaction/input/scriptsig/

4. Bitcoin scripts — https://github.com/jimmysong/programmingbitcoin/blob/master/ch13.asciidoc

Bitcoin transactions are used to transfer value or inscribe data onchain in an immutable way. In this series of articles we will discuss how to encode and decode a raw hex transaction, contructing a Bitcoin transaction and scripts.

Part 1. Hex Encoded Bitcoin Transactions

Bitcoin transactions can be encoded in raw hexadecimal format. The hexadecimal format is base16 which is represented by characters 0–9 and A-Fwhich create the hex alphabet 0123456789ABCDEF. Each character represents two bytes of data.

Let’s look at how to decode the bitcoin transaction
010000000269adb42422fb021f38da0ebe12a8d2a14c0fe484bcb0b7cb365841871f2d5e24000000006a4730440220199a6aa56306cebcdacd1eba26b55eaf6f92eb46eb90d1b7e7724bacbe1d19140220101c0d46e033361c60536b6989efdd6fa692265fcda164676e2f49885871038a0121039ac8bac8f6d916b8a85b458e087e0cd07e6a76a6bfdde9bb766b17086d9a5c8affffffff69adb42422fb021f38da0ebe12a8d2a14c0fe484bcb0b7cb365841871f2d5e24010000006b48304502210084ec4323ed07da4af6462091b4676250c377527330191a3ff3f559a88beae2e2022077251392ec2f52327cb7296be89cc001516e4039badd2ad7bbc950c4c1b6d7cc012103b9b554e25022c2ae549b0c30c18df0a8e0495223f627ae38df0992efb4779475ffffffff0118730100000000001976a9140ce17649c1306c291ca9e587f8793b5b06563cea88ac00000000

We will be using the Rust standard library without any external crates.

First create a cargo project.

cargo new Bitcoin-Tx-Hex --name btc-tx-hex

This creates a crate called btc-tx-hex in the directory called Bitcoin-Tx-Hex.

In the main.rs file create a variable called raw_tx to represent our hex encoded transaction.

fn main() {
let mut raw_tx = "010000000269adb42422fb021f38da0ebe12a8d2a14c0fe484bcb0b7cb365841871f2d5e24000000006a4730440220199a6aa56306cebcdacd1eba26b55eaf6f92eb46eb90d1b7e7724bacbe1d19140220101c0d46e033361c60536b6989efdd6fa692265fcda164676e2f49885871038a0121039ac8bac8f6d916b8a85b458e087e0cd07e6a76a6bfdde9bb766b17086d9a5c8affffffff69adb42422fb021f38da0ebe12a8d2a14c0fe484bcb0b7cb365841871f2d5e24010000006b48304502210084ec4323ed07da4af6462091b4676250c377527330191a3ff3f559a88beae2e2022077251392ec2f52327cb7296be89cc001516e4039badd2ad7bbc950c4c1b6d7cc012103b9b554e25022c2ae549b0c30c18df0a8e0495223f627ae38df0992efb4779475ffffffff0118730100000000001976a9140ce17649c1306c291ca9e587f8793b5b06563cea88ac00000000";
}

Bitcoin hex encoded transactions have four concatenated parts

version | inputs | outputs | locktime

Version

The Bitcoin version is represented in four bytes which is a u32 in Rust. Create a new module called version.rs and import it into the main.rs file.

mod version;
pub use version::*;

fn main() {
    //... previous code
}

In our version.rs file let’s create a struct to parse our version

/// Bitcoin transactions version one and two are supported
/// by Bitcoin core. A node must pre-configure a transaction
/// version higher than version 2 and this transaction is
/// not guaranteed to be propagated by all Bitcoin core.
#[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Default)]
pub enum TxVersion {
    /// This will be treated as the default version
    /// when calling TxVersion::default()
    #[default]
    One,
    /// The Bitcoin transaction version two which allows
    /// using the OPCODE `OP_CHECKSEQUENCEVERIFY` which allows
    /// setting relative locktime for spending outputs.
    Two,
    /// Custom transaction version which is considered non-standard,
    /// must be set by the Bitcoin node operator and is not guaranteed
    /// to be accepted by other nodes running Bitcoin core software
    Custom(u32),
}

Next we implement our encoder and decoder; to and from bytes for the TxVersion . The version is in little-endian format which is also called reverse byte order .

impl TxVersion {
    /// This converts our version to bytes.
    /// Since version number is four bytes little-endian we use `u32::to_le_bytes()`
    pub fn to_bytes(&self) -> [u8; 4] {
        match self {
        Self::One => 1u32.to_le_bytes(),
        Self::Two => 2u32.to_le_bytes(),
        Self::Custom(version) => version.to_le_bytes(),
        }
    }

    /// This converts from bytes to `Self`
    pub fn from_bytes(bytes: [u8; 4]) -> Self {
        let parsed = u32::from_le_bytes(bytes);

        match parsed {
            1u32 => Self::One,
            2u32 => Self::Two,
            _ => Self::Custom(parsed),
        }
    }
}

Next we write a simple test to check the correctness of our parser.

#[cfg(test)]
mod tx_sanity_checks {
    use crate::TxVersion;

    #[test]
    fn tx_version() {
        assert_eq!([1u8, 0, 0, 0], TxVersion::One.to_bytes());
        assert_eq!([2u8, 0, 0, 0], TxVersion::Two.to_bytes());
        assert_eq!([30u8, 0, 0, 0], TxVersion::Custom(30).to_bytes());

        assert_eq!(TxVersion::One, TxVersion::from_bytes([1u8, 0, 0, 0]));
        assert_eq!(TxVersion::Two, TxVersion::from_bytes([2u8, 0, 0, 0]));
        assert_eq!(
        TxVersion::Custom(30),
        TxVersion::from_bytes([30u8, 0, 0, 0])
        );
    }
}

Let’s run our test with the command

cargo test --verbose --all-features

Our test passes with the following output

# ... Other log data here
# Below is the part we are interested in
running 1 test
test version::tx_sanity_checks::tx_version ... ok

After parsing the version, we need to get the number of outputs in the output section. To do this, we parse the first byte into a Bitcoin VarInt .

VarInt

A VarInt (short for “Variable Integer”) is a crucial format used in Bitcoin to indicate the lengths of fields within transactions, blocks, and peer-to-peer network data. Learn more about VarInt at https://web.archive.org/web/20230331170203/https://learnmeabitcoin.com/technical/varint

NOTE: That a VarInt of 8 bytes is beyond the maximum block size of a Bitcoin block and therefore never used in a Bitcoin transaction.

In our Rust VarInt parser, we will use a std::io::Cursor to read either 0, 2, 4 or 8 bytes from the current position in our byte length field.

Reading:

• 0 bytes will be treated as a u8

• 2 bytes will be treated as a u16

• 4 bytes will be treated as a u32

• 8 bytes will be treated as a u64

Where:

• 0 bytes is represented by a u8 of <= 252

• 2 bytes is represented by a u8 of 253

• 4 bytes is represented by a u8 of 254

• 8 bytes is represented by a u8 of 255

Create a new module in our src directory called varint.rs then in then import our module in the src/main.rs file.

// ... Previous imports here
mod varint;
pub use varint::*;

fn main() {
    // ... Previous code here
}

In our src/varint.rs file.

// We are using a Cursor to have a position
// of up to the index that the bytes have been
// read. This is convinient instead of using
// a counter to keep track of everything
// which can be cumbersome since we need to
// keep track of the length of bytes t
use std::io::{self, Cursor, Read};

/// We create a `VarInt` struct to hold methods for calculating
/// the number of bytes in the `VarInt``
#[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Clone, Copy)]
pub struct VarInt;

impl VarInt {
    /// This converts our VarInt byte into the number of bytes that we need to parse
    pub const fn parse(byte: u8) -> usize {
        match byte {
            // 0 to 252 is treated as a Rust u8 which is 1 byte long
            0..=252 => 1,
            // 253 is treated as a Rust u16 which is 2 bytes long
            253 => 2,
            // 253 is treated as a Rust u32 which is 4 bytes long
            254 => 4,
            // 253 is treated as a Rust u64 which is 8 bytes long
            255 => 8,
        }
    }

    /// Given a Cursor of bytes, we read the current or next number of bytes
    /// then convert them into an integer
    pub fn integer(byte_len: usize, bytes: &mut Cursor<&[u8]>) -> io::Result<usize> {
        let outcome = match byte_len {
                1 => {
                // NOTE - Since we are reading one value and the Cursor always advances
                // by the number of bytes read, we reset the cursor to the last position
                // in order to parse that one byte. First we get the current cursor
                // position using `bytes.position()` and then subtract 1
                bytes.set_position(bytes.position() - 1);

                // A u8 has array length of 1
                let mut buffer = [0u8; 1];
                // Read exactly one byte
                bytes.read_exact(&mut buffer)?;

                buffer[0] as usize
            }
            2 => {
                // A u16 has array length of 2
                let mut buffer = [0u8; 2];
                // Read exactly two bytes
                bytes.read_exact(&mut buffer)?;

                u16::from_le_bytes(buffer) as usize
            }
            4 => {
                // A u32 has array length of 4
                let mut buffer = [0u8; 4];
                // Read exactly four bytes
                bytes.read_exact(&mut buffer)?;

                u32::from_le_bytes(buffer) as usize
            }
            8 => {
                // A u32 has array length of 8
                let mut buffer = [0u8; 8];
                // Read exactly eight bytes
                bytes.read_exact(&mut buffer)?;

                u64::from_le_bytes(buffer) as usize
            }
            _ => {
                    // All other values are not supported and we return an error to
                    // indicate this
                    return Err(std::io::Error::new(
                    std::io::ErrorKind::NotFound,
                    "The byte length specified is not supported",
                    ));
                }
            };

        Ok(outcome)
    }
}

In the same file, we write tests to check if our VarInt is being parsed correctly or does the parse resolve to an error

#[cfg(test)]
mod varint_sanity_checks {
    use crate::VarInt;
    use std::io::{Cursor, Read};

    #[test]
    fn varint_zero_to_252() {
        let bytes = [0u8, 0, 0, 0, 1];
        let mut bytes = Cursor::new(bytes.as_slice());

        // Simulate version bytes by skipping 4 bytes
        bytes.set_position(4);

        let mut varint_byte = [0u8; 1];
        bytes.read_exact(&mut varint_byte).unwrap();
        let varint_byte_len = VarInt::parse(varint_byte[0]);
        let varint_len = VarInt::integer(varint_byte_len, &mut bytes);
        assert!(varint_len.is_ok());
        assert_eq!(1usize, varint_len.unwrap());
    }

    #[test]
    fn varint_253() {
        let mut bytes = vec![0u8, 0, 0, 0, 253];
        let placeholder_bytes = [1u8; 257];
        bytes.extend_from_slice(&placeholder_bytes);
        let mut bytes = Cursor::new(bytes.as_slice());

        // Simulate version bytes by skipping 4 bytes
        bytes.set_position(4);

        let mut varint_byte = [0u8; 1];
        bytes.read_exact(&mut varint_byte).unwrap();
        let varint_byte_len = VarInt::parse(varint_byte[0]);
        let varint_len = VarInt::integer(varint_byte_len, &mut bytes);
        assert!(varint_len.is_ok());
        assert_eq!(257usize, varint_len.unwrap());    
    }

    #[test]
    fn varint_254() {
        let mut bytes = vec![0u8, 0, 0, 0, 254];
        let placeholder_bytes = [1u8; 40];
        bytes.extend_from_slice(&placeholder_bytes);
        let mut bytes = Cursor::new(bytes.as_slice());

        // Simulate version bytes by skipping 4 bytes
        bytes.set_position(4);

        let mut varint_byte = [0u8; 1];
        bytes.read_exact(&mut varint_byte).unwrap();
        let varint_byte_len = VarInt::parse(varint_byte[0]);
        let varint_len = VarInt::integer(varint_byte_len, &mut bytes);
        assert!(varint_len.is_ok());
        assert_eq!(16843009usize, varint_len.unwrap());
    }    

    #[test]
    fn varint_255() {
        let mut bytes = vec![0u8, 0, 0, 0, 255];
        let placeholder_bytes = [1u8; 40];
        bytes.extend_from_slice(&placeholder_bytes);
        let mut bytes = Cursor::new(bytes.as_slice());

        // Simulate version bytes by skipping 4 bytes
        bytes.set_position(4);

        let mut varint_byte = [0u8; 1];
        bytes.read_exact(&mut varint_byte).unwrap();
        let varint_byte_len = VarInt::parse(varint_byte[0]);
        let varint_len = VarInt::integer(varint_byte_len, &mut bytes);
        assert!(varint_len.is_ok());
        assert_eq!(72340172838076673usize, varint_len.unwrap());
    }
}

Next we create our transaction parser by creating a module called tx.rs in the src directory and then registering our module in the src/main.rs file.

// ... Previous imports here

mod tx;
pub use tx::*;

fn main() {
    //... Previous code here
}

In our src/tx.rs file

Bitcoin transaction have inputs and outputs so we create structs to represent them.

/// Our transaction inputs
#[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Clone)]
pub struct TxInput {
    // The SHA256 bytes of the previous transaction ID
    // of the unspent UTXO
    previous_tx_id: [u8; 32],
    // Previous index of the previous transaction output
    previous_output_index: u32,
    // The scriptSig
    signature_script: Vec<u8>,
    // The sequence number
    sequence_number: u32,
}

/// Transaction outputs
#[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Clone)]
pub struct TxOutput {
    // Amount in satoshis
    amount: u64,
    // The locking script which gives conditions for spending the bitcoins
    locking_script: Vec<u8>,
}

Now we combine the TxInput, TxOutput , VarInt and TxVersion as part of the transaction struct

// Import our modules
use crate::{TxVersion, VarInt};
use std::io::{Cursor, Read, self};

/// The structure of the Bitcoin transaction
#[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Default)]
pub struct BtcTx {
    // The version of the Bitcoin transaction
    version: TxVersion,
    // A transaction can have multiple inputs
    inputs: Vec,
    // A transaction can have multiple outputs
    outputs: Vec,
    // The locktime for the transaction parsed
    // from 4 bytes into a u32
    locktime: u32,
}

Next we implement methods to parse our version, inputs, outputs and locktime into our BtcTx

impl BtcTx {
    /// Convert hex bytes into a Transaction struct. This calls all other
    /// methods to parse the version, inputs, outputs and locktime.
    pub fn from_hex_bytes(bytes: impl AsRef<[u8]>) -> io::Result<Self> {
        // Instantiate a new cursor to hold the bytes.
        // The cursor's position advances whenever we read
        // bytes allowing us to simplify the logic
        // instead of using a counter to keep track of bytes read
        let mut bytes = Cursor::new(bytes.as_ref());

        // The version number is always a 4 byte array
        let mut version_bytes = [0u8; 4];
        // Read exactly 4 bytes and advance the cursor to the 4th byte
        bytes.read_exact(&mut version_bytes)?;
        // Get the transaction version from the bytes
        let version = TxVersion::from_bytes(version_bytes);        

        // Get a vector of inputs by calling the `Self::get_inputs()` method
        let inputs = BtcTx::get_inputs(&mut bytes)?;
        // Get a vector of outputs by calling the `Self::get_outputs()` method
        let outputs = BtcTx::get_outputs(&mut bytes)?;
        // Get a vector of inputs by calling the `Self::locktime()` method
        let locktime = BtcTx::locktime(&mut bytes)?;    

        Ok(BtcTx {
            version,
            inputs,
            outputs,
            locktime,
        })
    }

    /// Get all inputs from the current position of the `Cursor`.
    /// This method decodes the number of inputs by first decoding the
    /// `varint` and then looping number of inputs calling
    /// `Self::input_decoder()` on each iteration.
    fn get_inputs(bytes: &mut Cursor<&[u8]>) -> io::Result<Vec> {
        let mut varint_len = [0u8];
        bytes.read_exact(&mut varint_len)?;

        let varint_byte_len = VarInt::parse(varint_len[0]);
        let no_of_inputs = VarInt::integer(varint_byte_len, bytes)?;

        let mut inputs = Vec::::new();

        (0..no_of_inputs).into_iter().for_each(|_| {
        inputs.push(BtcTx::input_decoder(bytes).unwrap());
        });

        Ok(inputs)
    }    

    // Decodes an input from current `Cursor` position.
    fn input_decoder(bytes: &mut Cursor<&[u8]>) -> io::Result {
        // The previous transaction ID is always a SHA256 hash converted to a 32 byte array
        let mut previous_tx_id = [0u8; 32];
        // Read exactly 32 bytes and advance the cursor to the end of the 32 byte array
        bytes.read_exact(&mut previous_tx_id)?;
        // The transaction ID in hex format is in network byte order so we reverse
        // it to little endian
        previous_tx_id.reverse();

        //Previous transaction index is 4 bytes long which is a Rust u32
        let mut previous_tx_index_bytes = [0u8; 4];
        bytes.read_exact(&mut previous_tx_index_bytes)?;
        // Convert the read 4 bytes to a u32
        let previous_output_index = u32::from_le_bytes(previous_tx_index_bytes);

        // Get the length of the scriptSig
        let mut signature_script_size = [0u8];
        bytes.read_exact(&mut signature_script_size)?;
        // Parse the length VarInt
        let varint_byte_len = VarInt::parse(signature_script_size[0]);
        // Get the length by converting VarInt into an integer by calling `integer`
        let integer_from_varint = VarInt::integer(varint_byte_len, bytes)?;

        // Buffer to hold the signature script
        let mut signature_script = Vec::<u8>::new();
        let mut sig_buf = [0u8; 1];
        // Since we are using a cursor, we iterate in order to advance
        // the cursor in each iteration
        (0..integer_from_varint).for_each(|_| {
        bytes.read_exact(&mut sig_buf).unwrap();

        signature_script.extend_from_slice(&sig_buf);
        });

        // The sequence number is a u32 (4 bytes long)
        let mut sequence_num_bytes = [0u8; 4];
        bytes.read_exact(&mut sequence_num_bytes)?;
        // Convert the sequence number to a integer
        let sequence_number = u32::from_le_bytes(sequence_num_bytes);    

        Ok(TxInput {
            previous_tx_id,
            previous_output_index,
            signature_script,
            sequence_number,
        })
    }    

    /// Get the outputs after all inputs have been parsed.
    fn get_outputs(bytes: &mut Cursor<&[u8]>) -> io::Result<Vec> {
        // Get the number of outputs by reading our VarInt
        let mut num_of_output_bytes = [0u8; 1];
        bytes.read_exact(&mut num_of_output_bytes)?;
        let var_int_byte_length = VarInt::parse(num_of_output_bytes[0]);
        // Convert our VarInt to an integer
        let num_of_outputs = VarInt::integer(var_int_byte_length, bytes)?;

        let mut outputs = Vec::::new();

        // Iterate over number of outputs
        (0..num_of_outputs).into_iter().for_each(|_| {
        // The first value of the output is the amount in satoshis
        // which is 8 bytes long (Rust u64)
        let mut satoshis_as_bytes = [0u8; 8];
        bytes.read_exact(&mut satoshis_as_bytes).unwrap();
        // Get the number of satoshis in decimal
        let satoshis = u64::from_le_bytes(satoshis_as_bytes);    

        // Get the exact size of the locking script
        let mut locking_script_len = [0u8; 1];
        bytes.read_exact(&mut locking_script_len).unwrap();
        // Parse the length into a varint
        let script_byte_len = VarInt::parse(locking_script_len[0]);
        // Convert our VarInt to an integer
        let script_len = VarInt::integer(script_byte_len, bytes).unwrap();
        let mut script = Vec::<u8>::new();

        // For the length of the script, read each byte and advance the cursor in each iteration
        (0..script_len).for_each(|_| {
            let mut current_byte = [0u8; 1];

            bytes.read_exact(&mut current_byte).unwrap();
            script.extend_from_slice(¤t_byte);
        });

        // Construct our Transaction Output struct and then push it to the outputs vec
        outputs.push(TxOutput {
                amount: satoshis,
                locking_script: script,
            });
        });

        Ok(outputs)
    }

    // Lastly, after parsing our version, inputs and outputs we parse the locktime
    fn locktime(bytes: &mut Cursor<&[u8]>) -> io::Result<u32> {
        // The locktime is 4 bytes long
        let mut locktime_bytes = [0u8; 4];
        bytes.read_exact(&mut locktime_bytes)?;

        // Convert the locktime into an integer
        Ok(u32::from_le_bytes(locktime_bytes))
    }
}

We can now utilize our code to decode a hex transaction. First add the hex-conservative crate into the Cargo.toml manifest file since creating a library to parse hex into bytes is beyond the scope of this article.

Lastly, we use our parse to parse the transaction hex string we introduced at the beginning of the article.

fn main() {
    let raw_tx = hex!("010000000269adb42422fb021f38da0ebe12a8d2a14c0fe484bcb0b7cb365841871f2d5e24000000006a4730440220199a6aa56306cebcdacd1eba26b55eaf6f92eb46eb90d1b7e7724bacbe1d19140220101c0d46e033361c60536b6989efdd6fa692265fcda164676e2f49885871038a0121039ac8bac8f6d916b8a85b458e087e0cd07e6a76a6bfdde9bb766b17086d9a5c8affffffff69adb42422fb021f38da0ebe12a8d2a14c0fe484bcb0b7cb365841871f2d5e24010000006b48304502210084ec4323ed07da4af6462091b4676250c377527330191a3ff3f559a88beae2e2022077251392ec2f52327cb7296be89cc001516e4039badd2ad7bbc950c4c1b6d7cc012103b9b554e25022c2ae549b0c30c18df0a8e0495223f627ae38df0992efb4779475ffffffff0118730100000000001976a9140ce17649c1306c291ca9e587f8793b5b06563cea88ac00000000");
    let tx_decode = BtcTx::from_hex_bytes(raw_tx);

    dbg!(tx_decode.unwrap());
}

That’s it. We have decoded a raw Bitcoin hex encoded transaction.

In the next article, we will look into converting hash bytes into SHA256 strings and converting our script bytes into a Bitcoin Script.

References

1. VarInt — https://web.archive.org/web/20230331170203/https://learnmeabitcoin.com/technical/varint

2. The code for this article — https://github.com/448-OG/BitcoinTransactions

3. Compact Size — https://learnmeabitcoin.com/technical/general/compact-size/

Base58Check is a binary-to-text encoding algorithm that allows us to convert byte arrays into human-readable strings that are resistant to errors. These byte arrays can be public keys, digests of hash functions and even private keys.

The algorithm used to convert bytes to Base58Check is the Base58 algorithm. We will first look at the Base58 algorithm and later combine it to create the Base58Check algorithm.

Base58Check is primarily used for encoding Bitcoin addresses, but it can be more broadly applied to any data that needs to be represented in a compact, user-friendly format that excludes characters like 0, O, I, and l that can look similar in certain fonts. This prevents confusion and ensures readability.

As illustrated above, the Base58 avoids characters that can create confusion reducing the alphabet of allowed characters to 58 which are 123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz

To convert a decimal value to Base58 we use the modulus function until no remainders are left. Using modulus function makes sure we circle around when we reach the maximum number of characters which is 58. Each remainder corresponds to the character at the index in the Base58 alphabet, for example remainder 10 equals to the character A

Example the decimal number 1024

base10 = 1024

1025 % 58 = 38 <- Ignoring the fractional part
17 % 58 = 17

The numbers are 17 and 38 respectively
So we look for the characters at index 17 and 38 in the Base58 alphabet

123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz
                 |                    |
             index 17 is `J`      index 38 is `f`

Therefore:
Base58 = `Jf`

To convert back to decimal from Base58 string you take each character and lookup it’s index in the Bas58 alphabet and multiply it by how many 58s that position represents and then you add all the values together.

Base58 = `Jf`

J = index 17 in the Base58 alphabet
f = index 38 in the Base58 alphabet

To get the index at character we start with the last one as index 0 moving backwards
J = index 1 in the Base58 string
f = index 0 in the Base58 string

f = 38 * 58^0 = 38
J = 17 * 58^1 = 986
Decimal = 38 + 989 = 1024

In our code, we will use bytes instead of decimals in order to allow conversions from any value that can be converted to and from bytes like public and private keys.

Let’s create a new cargo project.

$ cargo new base58

Switch to the project directory.

cd base58

We need to add some dependencies in the Cargo.toml file

[dependencies]
sha = "0.10.8"
rand_chacha = "0.3.1"
rand_core = { version = "0.6.4", features = ["getrandom"]}
bitcoin =  { version = "0.31.1", features = ["std", "rand-std"] }

Let’s import the data types into our main.rs file

use bitcoin::base58;
use rand_chacha::ChaCha20Rng;
use rand_core::{RngCore, SeedableRng};
use sha2::{Digest, Sha256};
use std::collections::VecDeque;

fn main() {}

We import the base58 type from the bitcoin crate to compare the output of our custom base58 to the output of the mostly use Rust bitcoin crate. This is done to check for correctness in converting to base58 from bytes and vice versa.

rand_chacha and rand_core crates are used to generate random bytes while sha2 will be used to create our checksum for Base58Check algorithm.

Next we create our code for random byte generation. We use a const generic N so that we can pass the number of bytes we want to generate.

/// Our random byte generator
#[derive(Debug, PartialEq, Eq, PartialOrd, Ord)]
pub struct Entropy<const N: usize>([u8; N]);

impl<const N: usize> Entropy<N> {
    /// Generate our random bytes
    pub fn generate() -> Self {
        // Create a chacha based 
        // cryptographically secure psuedo-random
        // number generator.
        let mut rng = ChaCha20Rng::from_entropy();
        // Initialize an empty byte array of `N`
        // number of bytes
        let mut buffer = [0u8; N];
        // Fill our buffer with randomly generated bytes
        rng.fill_bytes(&mut buffer);

        Self(buffer)
    }
}

Next we write code to parse bytes to Base58 encoding.

/// Add our base58 alphabet
const ALPHABET: &str = "123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz";

/// Converts our bytes to base58. 
/// It takes a byte slice
fn to_base58(base58_bytes: &[u8]) -> String {
    // We use a `VecDeque` so that we don't have to
    // call `.reverse()` function on our `Vec`
    // since this can be an expensive operation.
    let mut base58_char = VecDeque::<char>::new();

    // We create a new String to hold our final base58 string
    let mut outcome = String::new();

    // We iterate checking for `0` bytes that appear at the
    // beginning of our array since they are supposed
    // to be prefixed by `1`. We then push a `1` for 
    // every 0 byte we find.
    for _ in base58_bytes.iter().take_while(|&&x| x == 0) {
        outcome.push('1');
    }

    // This will be the result of every modulo operation
    // holding the quotient of our division
    let mut decimal = 0usize;

    // We multiply by 256 since bytes are in
    // base 2 and our division to base58 is in
    // base 10. This is done by using left shift
    // to multiply since 2^8 is equal to 256.
    // we then assign the result to `decimal` variable
    for value in base58_bytes {
        decimal = (decimal << 8) | *value as usize;
    }

    // We know split the alphabet and collect the individual
    // characters so that we can get the index
    let index_alphabet = ALPHABET.chars().collect::<Vec<char>>();

    // We iterate over decimal variable
    // dividing by `58` and getting the remainder.
    // We then lookup the character at the index
    // of that remainder and then we push that character
    // to our vecdeque pushing it to the front.
    // We halt the loop when `decimal` variable equals 0
    while decimal != 0 {
        // This is equivalent to diving and getting the
        // remainder while allowing compilers to
        // optimize for speed.
        let (quotient, remainder) = (decimal / 58, decimal % 58);

        base58_char.push_front(index_alphabet[remainder as usize]);
        decimal = quotient;
    }

    // We know iterate over our characters and add them
    // to the `outcome` variable which might
    // contain leading zeros represented by `1`s
    outcome += base58_char.iter().collect::<String>().as_str();

    outcome
}

Next we create our base58 decoder.

/// This decodes a base58 string into bytes
fn from_base58(base58_str: &str) -> Vec<u8> {
    // Initialize a variable to hold the decimal equivalent of the base58 string
    let mut decimal = 0usize;

    // Create a vector of characters from the base58 alphabet
    let index_alphabet: Vec<_> = ALPHABET.chars().collect();

    // Initialize a variable to count the number of leading zeros in the base58 string
    let mut leading_zeros_total = 0usize;

    // Initialize a vector to hold the outcome of the decoding
    let mut outcome = Vec::<u8>::new();

    // Convert the base58 string into a vector of characters
    let mut split_chars = base58_str.chars().collect::<Vec<char>>();

    // Count the number of leading zeros in the base58 string and add a corresponding number of zeros to the outcome
    for _ in split_chars.iter().take_while(|&x| x == &'1') {
        leading_zeros_total += 1;
        outcome.push(0);
    }

    // Remove the leading zeros from the base58 string
    split_chars.drain(0..leading_zeros_total);

    // Convert each character in the base58 string to its decimal equivalent and add it to the decimal variable
    for current_char in split_chars {
        let value = index_alphabet
            .iter()
            .position(|&in_alphabet| in_alphabet == current_char)
            .unwrap();
        decimal = (decimal * 58) + value;
    }

    // Initialize a vector to hold the bytes of the decimal
    let mut bytes = Vec::<u8>::new();

    // This loop continues as long as the decimal value is greater than 0
    while decimal > 0 {
        // The division operation (decimal / 256) gives the quotient
        // The modulus operation (decimal % 256) gives the remainder
        // These two values are stored in the variables 'quotient' and 'remainder' respectively
        let (quotient, remainder) = (decimal / 256, decimal % 256);

        // The remainder (which is the result of the modulus operation) is converted to an 8-bit unsigned integer (u8)
        // and then pushed onto the 'bytes' vector. This is because each byte represents a value between 0 and 255 (inclusive),
        // which is the range of possible remainders from the modulus operation.
        bytes.push(remainder as u8);

        // The 'decimal' variable is updated to the 'quotient' from the division operation.
        // This effectively reduces the 'decimal' value by a factor of 256 in each iteration of the loop,
        // preparing it for the next iteration where the process repeats.
        decimal = quotient;
    }

    // Reverse the bytes to get them in the correct order
    bytes.reverse();

    // Add the bytes to the outcome
    outcome.extend_from_slice(&bytes);

    // Return the outcome
    outcome
}

Bitcoin uses Base58Check which is an enhanced version of Base58 algorithm. Base58Check has some advantages like a version number and error detection due to the addition of four bytes checksum from the double hashing of a prefix byte and the payload using the SHA256 hash function.

Let add code to create our Base58Check algorithm.

#[derive(Debug, Default)]
pub struct Base58Check {
    // Will hold our prefix
    prefix: Vec<u8>,
    // Our bytes
    payload: Vec<u8>,
    // 4 byte result of double hashing
    checksum: Vec<u8>,
}

Next we implement our initialization function.

impl Base58Check {
  // Initialize Self with defaults since we derive default on our struct
   fn new() -> Self {
      Self::default()
  }
}

We need a way to add our prefix and payload

impl Base58Check {
    /// We call this method to add our prefix
    /// taking a byte slice as argument to allow any prefix
    fn add_prefix(mut self, prefix: &[u8]) -> Self {
        self.prefix.extend_from_slice(prefix);

        self
    }

    /// Method to add byte payload Self
    fn add_payload(mut self, payload: &[u8]) -> Self {
        self.payload.extend_from_slice(payload);

        self
    }
}

To generate our checksum, we first pass our prefix and payload to a SHA256 hashing function and then using our SHA256 hashing function we hash the result of the digest of the hash function (double hashing).

impl Base58Check {
  // This takes a mutable self and assigns the result
  // of the payload to `checksum` field in `Self`
  fn calc_checksum(mut self) -> Self {
        // Initialize our sha256 hashing function
        let mut hasher = Sha256::new();
        // Hash the prefix
        hasher.update(&self.prefix);
        // Hash our payload
        hasher.update(&self.payload);

        // Get the result of the digest
        let first_hash = hasher.finalize();

        let mut hasher = Sha256::new();
        // Hash the hash
        hasher.update(first_hash.as_slice());
        let double_hash = hasher.finalize();

        // Take the first four bytes and add them as the checksum
        self.checksum.extend_from_slice(&double_hash[0..4]);

        self
    }
}

Our last method for our struct with generate a vector of bytes that we can pass to our base58 encoder.

impl Base58Check {
  // This method just combines all fields of `Self`
  // into one vector by draining the other vectors
  fn build(mut self) -> Vec<u8> {
        let mut outcome = Vec::<u8>::new();
        outcome.extend(self.prefix.drain(..));
        outcome.extend(self.payload.drain(..));
        outcome.extend(self.checksum.drain(..));

        outcome
    }
}

Now let’s test our code in the main function.

fn main() {
    // Generate some random bytes
    let private_key = Entropy::<4>::generate().0;
    // Intialize our Base58Check struct
    let mut bytes = Base58Check::new()
        // Add our prefix
        .add_prefix(&[0u8, 0, 0, 0])
        // Add the payload
        .add_payload(&private_key)
        // Calculate the checksum
        .calc_checksum()
        // Combine all our bytes (prefix + payload + checksum)
        .build();
    let custom_conversion = to_base58(&mut bytes);

    // Assert that the base58 string generated
    // equals to the the base58 string 
    // generated by the bitcoin crate
    assert_eq!(base58::encode(&bytes).to_string(), custom_conversion);

    let to_custom_vec = from_base58(&custom_conversion);
    assert_eq!(
        to_custom_vec,
        base58::decode(&base58::encode(&bytes)).unwrap()
    );
}

That’s it! We have implemented a Base58Check encoder and decoder

References

1. https://learnmeabitcoin.com/technical/base58

Seed phrases exist because private keys are big numbers and it’s hard for anyone to memorize them. For example the bytes below as hex.

3AF660000011144004F890FF

The hex characters above have repeating characters like 00000 and 4400 making it easy for someone to miss or add a character. When you’re dealing with digital wallets and cryptocurrencies, it’s really important to get everything right. If you make a mistake with your wallet’s private key, you could end up with an empty wallet. This is because even a small error can create a completely different key, leading to a different wallet.

This is where BIP39 comes in. It replaces complicated keys with a set of simple words, known as a mnemonic phrase. This makes it easier to manage your keys without making mistakes. It’s like turning a hard-to-remember password into a memorable sentence.

So, instead of dealing with complex and error-prone keys, you just need to remember your unique set of words. Several backup and recovery mechanisms have been proposed such as BIP39, Electrum v2, Aezeed, Muun and SLIP39.

In this article we will look at Bitcoin Improvement Proposal 39 (BIP39) and write some Rust code to create and recover a seed using a mnemonic.

BIP39 is a technique that turns complex wallet recovery codes into a sequence of simple, readable words, known as a mnemonic. Here’s how it works:

1. The list of words used to create the mnemonic is designed so that typing just the first four letters of a word is enough to clearly identify it.

2. The list avoids similar word pairs like “build” and “built”, or “quick” and “quickly”. These pairs can make the mnemonic harder to remember and more likely to be guessed incorrectly.

3. The words are sorted in a specific order. This makes it quicker and easier to find a particular word when you need it. It also allows for the use of a ‘trie’ (a type of search tree), which can help to compress the data.

4. The specification also requires that native characters used to be encoded in UTF-8 using Normalization Form Compatibility Decomposition (NFKD)

Here are the steps to create a BIP39 mnemonic:

1. Create random bytes using a Cryptographically Secure Pseudo Random Number Generator. The bytes range between 128 bits (16bytes) to 256 bits (32 bytes) to generate 12–24 words.

2. Create a checksum, which is the first (length of entropy in bits/32) bits of the SHA256 of the entropy.

3. Append this checksum to the end of the initial entropy.

4. Split the result into groups of 11 bits.

5. Convert these bits into their decimal representation.

6. These decimal representations vary from 0–2047 and work as an index to a mnemonic word list. Choose words corresponding to these indices from the word list. For this tutorial we will use the most popular one which is English word list. Download the word list from https://github.com/bitcoin/bips/blob/master/bip-0039/english.txt

Remember, the mnemonic generated from these steps can be used to recover your wallet, so it’s crucial to keep it safe and secure.

This tutorial assumes you have installed Rust Programming Language toolchain which comes bundled with cargo build and dependency management tool.

Let’s create a cargo project called bip39-simple

$ cargo new bip39-simple

Switch to that project directory

$ cd bip39-simple

Add dependencies to Cargo.toml file

[dependencies]
rand_chacha = "*"
rand_core = { version = "*", features = ["getrandom"] }
sha2 = "*"
pbkdf2 = { version = "0.12.2", features = [
    "simple",
] }

We add the getrandom feature to rand_core in order to get random bytes using the operating system cryptographically secure random number generator. sha2 crate will be used to create a checksum from the SHA256 hash of the random bytes generated.

In our main.rs file we import these dependencies and bring some data types into scope

use pbkdf2::pbkdf2_hmac;
use rand_chacha::ChaCha20Rng;
use rand_core::{RngCore, SeedableRng};
use sha2::{Digest, Sha256, Sha512};
use std::{
    fs::File,
    io::{self, prelude::*},
    path::{Path, PathBuf},
};

The PBKDF2 key derivation function requires that an iteration count and a salt. The BIP39 standardized document requires that if a user includes a passphrase in the creation of a seed, that passphrase be appended to the word mnemonic and if there is no seed then we append an empty string "" instead. The specification also requires an iteration count of 2048. So we add two constants to hold these values.

/// Number of iterations to be run by the PBKDF2 for key derivation
pub const ITERATION_COUNT: u32 = 2048;
/// The word used as a prefix for the salt for our key derivation function
pub const SALT_PREFIX: &str = "mnemonic";

We need a way to generate random bytes so we introduce the a way to generate these bytes in a manner that a user can generate varying length of bytes (in our case between 128 to 256 bytes)

// This struct takes a constant `N` as a generic
// enabling one to specify a variable length for the bytes generated
#[derive(Debug, PartialEq, Eq, PartialOrd, Ord)]
pub struct Entropy<const N: usize>([u8; N]);

impl<const N: usize> Entropy<N> {
    // This method generates the bytes 
    pub fn generate() -> Self {
        // Instantiate our cryptographically secure random byte generation algorithm
        let mut rng = ChaCha20Rng::from_entropy();
        // Create a zero filled buffer to hold our bytes
        let mut buffer = [0u8; N];
        // Fill our buffer with random bytes
        rng.fill_bytes(&mut buffer);

        // Return our buffer
        Self(buffer)
    }
}

Next we need to create a struct that can hold methods to generate our mnemonic, create our seed and recover our seed.

#[derive(Debug, Default)]
pub struct Bip39Generator {
    // This holds all our indexes that we will use to fetch
    // our word from the word list 
    // with each index corresponding to an index
    // from our wordlist contained in a Vec<word>
    mnemonic_index: Vec<u16>,
    // This field holds the random bytes with our checksum
    // bytes appended to the end
    appended: Vec<u8>,
    // This contains a path to our wordlist file
    path: PathBuf,
}

Let’s create a method to instantiate our Bip39Generator struct.

impl Bip39Generator {
  // This method takes an argument `path_to_wordlist` which
  // is a path to the wordlist we downloaded
  // where the path is anything that implements the trait
  // AsRef<Path> meaning we pass any data type as convert it 
  // to a path using the `.as_ref()` method as long as that
  // data type implements the `AsRef<Path>` trait.
  pub fn new(path_to_wordlist: impl AsRef<Path>) -> Self {
      Self {
          // Convert `path_to_wordlist` argument to a path
          // using `.as_ref()` method and convert it
          // to a `std::path::PathBuf` using the `.to_path_buf()`
          path: path_to_wordlist.as_ref().to_path_buf(),
           // All other fields can hold default values
          // and we can call this method since 
          // we derived `Default` values using `#[derive(Default)]` 
          // on our struct 
          ..Default::default()
      }
  }
}

Next we create a method to load our word list from the file we download containing our English word list

Note the ... is not part of Rust syntax but to show previous code we implemented.

impl Bip39Generator {
  ...

// This method takes in a mutable `Self`
fn load_wordlist(&mut self) -> io::Result<Vec<String>> {
        // open the file using the path we passed
        // when instantiating our struct 
        // using `Bip39Generator::new()`
        let file = File::open(&self.path)?;
        // Create a buffer so that we can efficiently readd
        // our file
        let reader: io::BufReader<File> = io::BufReader::new(file);

        // Create a Vector to hold our wordlist
        let mut wordlist = Vec::<String>::new();

        // Read each line
        for line in reader.lines() {
          // Push each word to our `wordlist` vector
          // handling any I/O errors using `?`
            wordlist.push(line?);
        }

        // Return our vector of word list
        Ok(wordlist)
    }
}

We need to generate a checksum to append to our randomly generated bytes in order to generate a mnemonic

impl Bip39Generator {
  ...
  // Here we pass our generated random bytes as `entropy` argument
   fn generate_checksum<const N: usize>(&mut self, entropy: [u8; N]) -> &mut Self {
      // BIP39 spec requires a seed to be generated
      // using a SHA256 Psuedo Random Function (PRF)
      // so we instantiate a SHA256 hashing function.
      let mut hasher = Sha256::new();

      // We now pass our random bytes into our SHA256 PRF
      hasher.update(entropy.as_slice());

      // We now get our finalized value. Using
      // SHA256 always ensures that despite being
      // able to use variable length of random bytes
      // we always get back a 256 bit (32 byte) value.
      let entropy_hash = hasher.finalize();

      // Since we get a 32 byte value we multiply by
      // `8` to get number of bits since 1 byte == 8 bits
      let bits_of_entropy = entropy.len() * 8;
      // We get our `n` bits for our checksum from the
      // length of the random bits (entropy) 
      // where `n` is calculated as the 
      // `length of our random bits / 32`
      let bits_of_checksum = bits_of_entropy / 32;
      // We then use bit shifting to get
      // bits of checksum from our
      // 256 bit hash in variable `entropy_hash`
      let significant = entropy_hash[0] >> bits_of_checksum;

      let mut appended = entropy.to_vec();
      // We then append our checksum to our random
      appended.push(significant);

      // We now assign our appended bytes to the `appended`
      // field of our `Bip39Generator` struct which is `Self`
      self.appended = appended;

      self
  }
}

The next step is to get the index of the words to be used in our mnemonic. This step involves splitting our random bytes with the appended checksum into groups of 11 bits.

impl Bip39Generator {
  ...

  // We pass a mutable to self since we want to
  // add the result of this computation to `Self`
  fn compute(&mut self) -> &mut Self {
    // This vector will hold the binary 
    // representation of each byte in the `appended` vector.
      let mut bits = vec![];

      // This line starts a loop that iterates over each byte in the `self.appended` vector.
      for &byte in self.appended.iter() {
          // This line starts a nested loop that 
          // counts backwards from 7 to 0. 
          // The variable `i` represents the position of 
          // the bit we're interested in within 
          // the current byte.
          for i in (0..8).rev() {
          /*
            This line does three things:
             - `byte >> i`: This is a right bitwise shift operation. 
                            It moves the bits in `byte` `i` places to the right. 
                            The bit at position `i` is now at position 0.
             - `(byte >> i) & 1u8`: This is a bitwise AND operation with `1u8` (which is `1` in binary). 
                                    This operation effectively masks all the bits in `byte` except for the one at position 0.
             - `bits.push((byte >> i) & 1u8 == 1);`: This pushes `true` if the bit at position 0 is `1` 
                                                      and `false` otherwise into the `bits` vector.
            */            
              bits.push((byte >> i) & 1u8 == 1);
          }
      }

      // This line starts a loop that iterates over 
      // the `bits` vector in chunks of 11 bits.
      for chunk in bits.chunks(11) {
          // This line checks if the current chunk has 
          // exactly 11 bits. If it does, the code inside 
          // the if statement is executed.
          if chunk.len() == 11 {
              // This line initializes a mutable 
              // variable named `value` and sets it to 0. 
              // This variable will hold the decimal 
              // representation of the current 11-bit chunk.
              let mut value: u16 = 0;

              // This line starts a nested loop that iterates
              // over each bit in the current chunk. 
              // The variable `i` is the index of the current
              //  bit, and `bit` is the value of the current bit.
              for (i, &bit) in chunk.iter().enumerate() {
                  // This line checks if the current bit 
                  // is `1` (true). If it is, it shifts `1` 
                  // to the left by `(10 - i)` places 
                  // (this effectively gives `1` a value of `2^(10 - i)`) 
                  // and then performs a bitwise OR operation with `value`. 
                  // This has the effect of adding `2^(10 - i)` to `value`.
                  if bit {
                      value |= 1u16 << (10 - i);
                  }
              }
              // This line pushes the decimal 
              // representation of the current 11-bit chunk 
              // into the `self.mnemonic_index` vector.
              self.mnemonic_index.push(value);
          }
      }

      self
  }
}

Now that we can get our mnemonic

impl Bip39Generator {
  ...

    pub fn mnemonic<const N: usize>(&mut self) -> io::Result<String> {
        // This generates the number of random bits we need
        let entropy = Entropy::<{ N }>::generate();

        // Next, let's generate our checksum
        self.generate_checksum::<N>(entropy.0);

        // Next we compute the decimal numbers we will use
        // to get our wordlist
        self.compute();

        // Load the wordlist into memory
        let wordlist = self.load_wordlist()?;

        // Iterate through the decimal numbers
        // and for each decimal number get the word
        // in it's index in the wordlist (wordlist[index from decimal number]
        let mnemonic = self
            .mnemonic_index
            .iter()
            // Enumerate to get the current count in our interation
            .enumerate()
            .map(|(index, line_number)| {
                // Convert our decimal index (line_numer) to 
                // a usize since Rust is very strict in that
                // you can only index an array using a usize
                // so we dereference and cast using `as usize`
                let word = (&wordlist[*line_number as usize]).clone() + " ";  // Add a space in each word
                // Since indexes start at zero we add `1`
                // to make them human readable (humans mostly count from 1)
                let index = index + 1;

                // Check if we have our index is less than
                // 10 so we add a padding to make printing
                // to console neat
                let indexed = if index < 10 {
                    String::new() + " " + index.to_string().as_str()
                } else {
                    index.to_string()
                };

                // Print our index and each word. This 
                // will show the user the words in each
                // line but with a number. eg
                //  9. foo
                // 10. bar
                println!("{}. {}", indexed, &word);

                // Return the word
                word
            })
            .collect::<String>(); // Combine all strings into one

        // Trim the last space in the and return the mnemonic
        Ok(mnemonic.trim().to_owned())
    }
}

Now we can generate a seed from our mnemonic

impl Bip39Generator {
...

  // We pass our mnemonic and an optional passphrase
    fn seed(mnemonic: &str, passphrase: Option<&str>) -> io::Result<Vec<u8>> {
        // We check if there is a passphrase provided.
        // if there is one we prefix our salt with the passphrase
        let salt = if let Some(passphrase_required) = passphrase {
            String::new() + SALT_PREFIX + passphrase_required
        } else {
            String::from(SALT_PREFIX)
        };

        // We want to generate a 512bit seed
        // so we create a buffer to hold this.
        let mut wallet_seed = [0u8; 64]; // 512 bits == 64 bytes

        // We generate a key and push all the bytes to the `wallet_seed` buffer
        pbkdf2_hmac::<Sha512>(
            mnemonic.as_bytes(),
            salt.as_bytes(),
            ITERATION_COUNT,
            &mut wallet_seed,
        );

        // We return our seed
        Ok(wallet_seed.to_vec())
    }
}

Create methods to get a seed secured with a passphrase and one without. These methods call seed() which is a private method.

impl Bip39Generator {
...
   /// Generates a seed without a passphrase
    pub fn insecure_seed(mnemonic: &str) -> io::Result<Vec<u8>> {
        Self::seed(mnemonic, None)
    }

    /// Generates a seed with a passphrase
    pub fn secure_seed(mnemonic: &str, passphrase: &str) -> io::Result<Vec<u8>> {
        Self::seed(mnemonic, Some(passphrase))
    }
}

Now in the main function we can instantiate our seed generator to generate and recover our seed

fn main() {
    // Instantiate a mnemonic generator with the location of the wordlist
    let mut generator = Bip39Generator::new("english.txt");
    // Generate the mnemonic
    let mnemonic = generator.mnemonic::<16>().unwrap();

    println!("Your mnemonic is: {}", &mnemonic);

    // Recover a wallet by running the mnemonic through seed generator without a passphrase
    let insecure_seed = Bip39Generator::insecure_seed(&mnemonic);

    let passphrase = "BitCoin_iZ_Awesome";
    // Recover a wallet by running the mnemonic through seed generator without a passphrase
    let secure_seed = Bip39Generator::secure_seed(&mnemonic, passphrase);

    // test that the seed generation succeeds
    assert!(&secure_seed.is_ok(),);
    assert!(&insecure_seed.is_ok(),);
}

That’s it. We have made life easier for humans :) Cheers!

How a Solana PDA initialized

The initialized storage is an array of the maximum memory size the data structure takes up in memory. Storing a data structure like a username as a Rust String would require 24 bytes of memory allocated on the Solana blockchain. The bigger the data structure the higher the rent needed to pay for onchain storage. Calculating the maximum size a data structure would take up can be done using the Rust function core::mem::size_of::<T>() where T is the data structure.

core::mem::size_of::<String>(); //24 bytes

When system_instruction::SystemInstruction::CreateAccountWithSeed is called with the size of the String (24 bytes), will allocate an empty array of 24 bytes [u8; 24].

[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

For large data structures serialize/ deserialize libraries are used to handle the conversion of data structures to and from bytes.

A BIT OF GLASS: Solana uses Rust for programming onchain programs and the most popular serialization framework is serde. Unfortunately, serde uses too many CPU cycles to serialize and deserialize making it expensive to convert data to and from bytes on Solana’s limited BPF smart contract execution stack. borsh crate is the go to serialization library for Solana.

Suppose we need to create an end-to-end encrypted messaging Dapp that stores a user’s Diffie-Hellman public keys onchain in the user’s PDA address. The data structure in Rust would look like this:

use borsh::{BorshSerialize, BorshDeserialize};

#[derive(Debug, BorshSerialize, BorshDeserialize, Default)]
pub struct MessagingAccount {
   username: String,
   dh_keys: Vec<[u8; 32]>,
}

Calculating the size using core::mem::size_of::<MessagingAccount>() will return a maximum size of 48 bytes that can be stored in memory. Creating a PDA on Solana using system_instruction::SystemInstruction::CreateAccountWithSeed will initialize a zeroed array of 48 bytes ([u8; 48]) which can be represented as:

[0u8; core::mem::size_of::<MessagingAccount>()]

Using solana_program::msg to log the PDA would show the initialized account as

AccountInfo {
       key: pda_address,
       owner: program_id,
       is_signer: false,
       is_writable: true,
       executable: false,
       ..,
       data.len: 48,
       data: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000,
   .. }

The data part of the AccountInfo above shows that 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 which is hex for the 48 bytes of the zeroed storage initialized when the PDA account was created [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

We can deserialize the data structure onchain as follows:

// – Code snippet –
entrypoint!(process_instruction);

   pub fn process_instruction(
       program_id: &Pubkey,
       accounts: &[AccountInfo],
       instruction_data: &[u8],
   ) -> ProgramResult {
       let accounts_iter = &mut accounts.iter();

       let pda_account = next_account_info(accounts_iter)?;
       let pda_data = MessagingAccount::try_from_slice(instruction_data)?;
   }

Calling the smart contract code will return a borsh error

Err(
    Custom {
        kind: InvalidData,
        error: "Not all bytes read",
    },
)

The error returned is because borsh tries to deserialize the PDA account based on the data size encoded in the borsh specification - https://borsh.io/, in our case MessagingAccount::default which is [0, 0, 0, 0, 0, 0, 0, 0] which is an array of 8 bytes while the initialized PDA store is an array of 48 bytes.

What To Do?

A simple way to solve this issue is to pack and unpack bytes using a storage format. The storage format will contain a marker to indicate the length of the valid bytes. The marker is of type usize which has a maximum of 8 bytes which can be calculated by core::mem::size_of::<usize>().len() The data storage format can be defined as:

MARKER (8 bytes) | DATA (variable byte length) | Zeros (filled with zeroes)

To Serialize the data into the storage format

1. calculate the size of the valid bytes serialized by borsh: MessagingAccount::try_to_vec()?.len()

2. Get the size of the storage length of the pda account: pda_account.data.len()

3. Add the MARKER length to the serialized data length: 8 + MessagingAccount::try_to_vec()?.len()

4. Check if the length of the MARKER + MessagingAccount::try_to_vec()?.len() is greater than the length of the pda_account data storage.

5. If the size of the MARKER + MessagingAccount::try_to_vec()?.len() os greater, return an error informing the user that the data cannot be written to the Solana PDA storage because it exceeded the capacity of the PDA account.

6. If the data is less than or equal to the capacity of the PDA account storage, concate the MARKER with the bytes of the serialized data and then if the concatenated data is still less than the capacity of the PDA account storage, fill the remaining space with zeroes.

7. Write the data into the PDA storage

To deserialize the data storage format

1. Get the first 8 bytes and convert them to a MARKER : usize::from_le_bytes(bytes_stored[0..8]

2. Skip the first 8 bytes as indicated by the marker and then fetch the rest of the bytes up to the index indicated by the marker: let data = bytes_stored.iter().skip(8).take(MARKER).collect::<Vec<u8>>();

3. Deserialize the data using borsh: MessagingAccount.try_from_slice(&data)?;

Transforming the pseudocode above to code:

The code contains comments that explain each step.

• Create code to handle the errors

/// The result type that encompasses the `AccountStoreError`
pub type AccountStoreResult<T> = Result<T, AccountStoreError>;

#[derive(Debug)]
pub enum AccountStoreError {
    /// The buffer cannot acommodate the size of the `MARKER` which is `8 bytes`
    BufferTooSmallForMarker = 0,
    /// The buffer cannot acommodate the size of the data and the  `MARKER`
    BufferTooSmallForData = 1,
    /// The deserialized bytes do not contain a `MARKER`
    CorruptedMarker = 2,
    /// The data provided does not contain enough data length as specified by the `MARKER`
    CorruptedStorage = 3,
    /// The error provided is invalid
    InvalidError = 4,
}

• Create a data structure to handle these operations:

      pub const MARKER_SIZE: usize = 8;

      #[derive(Debug)]
      pub struct AccountStore<T> {
          pub data: T,
      }

• Create a method to calculate the size needed for a generic data structure to be stored onchain with our storage format. The impl {} block to take a generic parameter T that must implement BorshSerialize, BorshDeserialize and core::fmt::Debug.

impl<T> AccountStore<T>
where
    T: BorshDeserialize + BorshSerialize + Default + Sized,
{
    pub fn size_of() -> usize {
        core::mem::size_of::<T>() + MARKER_SIZE
    }
}

The size_of() method calculates the size of the data structure specified as T and adds 8 bytes to accommodate the marker information.

• Create a method to pack the data Create a method called pack() to convert the data into the storage format and write it to a provided buffer

impl {
  // -- Code snippet --
   pub fn pack(&self, buffer: &mut [u8]) -> AccountStoreResult<usize> {
        // Get the length of the PDA account storage size
        let buffer_length = buffer.len();

        // Check if the size of the `MARKER` is less than the size  of the `buffer_length`
        if buffer_length < MARKER_SIZE {
            // If the size is smaller, return an error indicating this to the user
            return Err(AccountStoreError::BufferTooSmallForMarker);
        }
        // Serialize the user data using `borsh`
        let data = self.data.try_to_vec().unwrap(); //HANDLE THIS BORSH ERROR AS YOU WISH
                                                    // Get the data length
        let data_length = data.len();

        // Check if the sum of the size of the `data_length` and the `MARKER_SIZE` is
        // greater than the `buffer_length`
        if buffer_length < data_length + MARKER_SIZE {
            return Err(AccountStoreError::BufferTooSmallForData);
        }

        // Copy the `data_length` to the buffer as the `MARKER`
        buffer[0..=7].copy_from_slice(&data_length.to_le_bytes());

        // Copy the data into the buffer.
        // If the data is smaller than the buffer then the space filled with
        // zeroes is left intact
        buffer[8..=data_length + 7].copy_from_slice(&data);

        Ok(data_length + 8usize)
    }
}

• Create a method unpack() to read the storage format

impl {
  // -- Code snippet --


    pub fn unpack(buffer: &[u8]) -> AccountStoreResult<AccountStore<T>> {
        // Get the length of the PDA account storage
        let buffer_length = buffer.len();

        // Check if the size of the `MARKER` is less than the size  of the `buffer_length`
        if buffer_length < MARKER_SIZE {
            return Err(AccountStoreError::BufferTooSmallForMarker);
        }

        // Convert the `MARKER` bytes to and array of `[u8; 8] `
        // since `usize::from_le_bytes` only accepts `[u8; 8]`
        let marker: [u8; 8] = match buffer[0..MARKER_SIZE].try_into() {
            Ok(value) => value,
            Err(_) => return Err(AccountStoreError::CorruptedMarker),
        };
        // Get the last index of the valid data
        let byte_length = usize::from_le_bytes(marker);

        // Check if the last index of the valid buffer is greater than the PDA storage size
        if byte_length > buffer_length {
            return Err(AccountStoreError::CorruptedStorage);
        }

        // Collect the valid data by skipping the `MARKER_SIZE` of `8 bytes`
        // and iterating the rest of the bytes until the index marked by the `byte_length`
        let data = buffer
            .iter()
            .skip(8)
            .take(byte_length)
            .map(|byte| *byte)
            .collect::<Vec<u8>>();

        if byte_length != 0 {
            let data = T::try_from_slice(&data).unwrap(); // Handle error as you see fit
            Ok(AccountStore { data })
        } else {
            // If the `byte_length` is zero it means that no previous
            // data had been written to the PDA account previously
            // so return the `Default` representation of the data structure represented
            // by the generic `T`
            Ok(AccountStore { data: T::default() })
        }
    }
}

• Lastly, create a method to add data to the store

impl {
  // -- snippet --
pub fn add_data(&mut self, data: T) -> &mut Self {
        self.data = data;

        self
    }
}

Utilizing our storage library to serialize and deserialize the pda_account.data

• Unpack the Solana PDA data pda_account.data using unpack() method and add_data() to add some data eg. from a Solana Instruction

let mut data = AccountStore::MessagingAccount>::unpack(&pda_account.data.as_ref().borrow()).unwrap();

let user_data = MessagingAccount {
    username: "SolanaSeaMonster".into(),
    dh_keys: vec![[1u8; 32], [2u8; 32]],
};

data.add_data(user_data);

• Packing the data and writing it back to the Solana PDA account using pack() method.

data.pack(&mut &mut pda_account.data.borrow_mut()[..]).unwrap();

Since a Solana PDA account storage is a Rust RefCell<Rc<[u8; T>>, the &variable_name.as_ref().borrow() is used to access the data and convert it into a &[u8] as required by the unpack() method and the &mut &mut variable_name.data.borrow_mut()[..] is used to write data.

Now you know how the PDA storage concept works and how you can create your own lightweight storage format to read and write the data.

That's it.

Keep chewing glass.