Benchmarks

Benchmark tests on Acurast phones are vital to determine the compute power of the participating devices. The compute metrics determined by these benchmark tests are relevant for matching phones to the right tasks based on their capabilities, calculating benchmark rewards distributed for the provision of phones, and determining staked compute rewards. Better hardware usually leads to a higher benchmark score.

Overview

The Acurast Benchmark suite is designed to evaluate the performance of Processors running in the Acurast network. It provides a comprehensive set of tests to measure computational throughput, memory allocation and access efficiency, and storage read/write performance. The benchmarks include support for multithreaded execution to leverage modern multi-core architectures.

The benchmark suite evaluates three main components: CPU, RAM, and Storage. Each component contributes to the overall performance score of the device.

CPU Benchmarks

The CPU benchmarks evaluate the computational capabilities of the device through three test categories:

Test Categories

• Crypto: Measures encryption and decryption throughput using AES-256 and hashing throughput using SHA-256.
• Math: Tests matrix multiplication performance, with support for three algorithms:
  • Divide and Conquer: A recursive algorithm used for single-threaded execution when SIMD is disabled or for multithreaded benchmarks in general.
  • Iterative: A straightforward implementation used when SIMD optimization is enabled but the device lacks the required hardware capabilities.
  • SIMD-Optimized: An optimized version of the iterative algorithm leveraging SIMD instructions, used when the device supports the required hardware capabilities.
• Sort: Benchmarks sorting algorithms, including single-threaded and multithreaded merge sort.

Execution Modes

The CPU benchmark suite is available in two variations:

• Single-Core: Executes all tests using a single thread to evaluate the performance of a single core.
• Multi-Core: Executes all tests using multiple threads to leverage the full computational power of the device.

Configuration

• Crypto: Configure the duration of the test and the size of the data to encrypt and hash.
• Math: Specify the matrix size, whether to enable SIMD optimizations, and the duration of the test.
• Sort: Define the size of the dataset and the duration of the test.

Score Calculation

The CPU score is calculated as the average of the throughput ($TPS$) values from the Crypto, Math, and Sort benchmarks. Higher throughput results in a higher score.

$$score_{CPU} = \frac{TPS_{crypto} + TPS_{math} + TPS_{sort}}{3}$$

RAM Benchmarks

The RAM benchmarks assess memory allocation and access patterns through two test categories:

Test Categories

• Allocation: Measures the time taken to allocate and initialize memory.
• Access: Evaluates sequential, random, and concurrent memory access patterns.

Configuration

• Allocation: Specify the number of iterations and the size of memory to allocate in each iteration.
• Access: Configure the number of iterations and the size of data for sequential, random, and concurrent access patterns.

Score Calculation

The RAM score is calculated based on the inverse of the average times ($T$) for Allocation and Access benchmarks. Higher efficiency results in a higher score. The total available RAM memory is also recorded.

$$score_{RAM} = \frac{T_{alloc}^{-1} + T_{access\_seq}^{-1} + T_{access\_rand}^{-1} + T_{access\_concurr}^{-1}}{4}$$

Storage Benchmarks

The storage benchmarks focus on file I/O performance through read and write operations.

Test Categories

• Access: Measures sequential and random read/write throughput for files on the storage medium.

Configuration

• Access: Define the number of iterations and the size of data for sequential and random read/write operations.

Score Calculation

The Storage score is calculated as the average of the inversed average times ($T$) for sequential and random read/write operations. Lower average times result in a higher score. The available storage capacity is also recorded.

$$score_{storage} = \frac{T_{access\_seq}^{-1} + T_{access\_rand}^{-1}}{2}$$

Overall Scoring

The compute overall score is calculated as a weighted average that combines performance metrics from CPU, RAM, and Storage benchmarks to provide a comprehensive evaluation of a device's computational capabilities.

$$score_{all} = w(score_{CPU\ SC}) + w(score_{CPU\ MC}) + w(mem_{RAM}) + w(score_{RAM}) + w(mem_{storage}) + w(score_{storage})$$

Where:

$$w(m) = \frac{m}{N_{m}} \times W_{m}$$

And:

• $W_{m}$: Weight assigned to each metric pool ($0 \leqslant W_{m} \leqslant 1$ and $\sum_{m} W_{m} = 1$)
• $N_{m}$: Normalization factor representing the sum of all scores submitted to the metric pool. Uses the previous epoch's total if available, otherwise the current epoch's total, or defaults to $m$ if no data exists on the chain.
• $score_{CPU\ SC}$: CPU performance score (Single-Core)
• $score_{CPU\ MC}$: CPU performance score (Multi-Core)
• $mem_{RAM}$: Total available RAM memory
• $score_{RAM}$: RAM performance score
• $mem_{storage}$: Available storage capacity
• $score_{storage}$: Storage performance score

To make the final score human-readable, a scaling factor of $10^9$ is applied.

$$score_{display} = score_{all} \times 10^9$$

Component Weights

The components are weighted as follows:

• CPU Single-Core: $W_{CPU\ SC} = 0.2307$
• CPU Multi-Core: $W_{CPU\ MC} = 0.2307$
• RAM Total: $W_{RAM\ mem} = 0.4615$
• RAM Speed: $W_{RAM\ score} = 0$
• Storage Total: $W_{storage\ mem} = 0.0769$
• Storage Speed: $W_{storage\ score} = 0$

If all weights equal $0$, the components are weighted equally:

$$W_{m} = \frac{1}{n} \text{ if active, or } W_{m} = 0 \text{ otherwise}$$

Where:

• $n$: The number of active metric pools