Parallel TIP5 hashing on AMD Virtex UltraScale+ FPGAs — designing a Neptune Cash miner on AWS EC2 F2

This is the first post in a three-part build log about mining Neptune Cash — a privacy coin whose proof-of-work is built on the TIP5 algebraic hash, the same hash that sits inside its recursive STARK proofs. I started where a hardware person instinctively starts when the task is “compute one fixed function as many times per second as physically possible”: an FPGA.

The full code for all three posts lives in ai-hpc/tip5-neptune. This post covers the AWS EC2 F2 design — the ambitious, cloud-scale starting point, where the real work turned out to be the PCIe interface, not the hash. The next post moves it onto three VU35P UltraScale+ on my desk, and the third explains why I eventually abandoned FPGAs for a single GPU.

Table of contents

Why FPGA, and why TIP5 is the right target

Neptune’s guesser proof-of-work is, at its core, a loop: take a block-proposal digest, append a nonce, hash it with TIP5, and check whether the result is below a threshold. Miss, increment the nonce, repeat. The miner that computes the most TIP5 hashes per second wins the block. There is no memory-hardness, no large working set — it is a pure compute firehose pointed at one fixed function. That is the textbook case for an FPGA: build the function once in silicon, replicate it, pipeline it, and run it at a deterministic rate with no instruction-fetch overhead.

TIP5 itself is an arithmetization-oriented hash — designed to be cheap inside a STARK, which means it is defined over a finite field rather than over bits. Concretely (these constants are straight out of the implementation):

constexpr int STATE_SIZE = 16;   // 16 field elements
constexpr int NUM_ROUNDS  = 5;
constexpr int RATE        = 10;  // sponge rate
constexpr int CAPACITY    = 6;   // sponge capacity (10 + 6 = 16)
constexpr int DIGEST_LEN  = 5;
constexpr uint64_t GOLDILOCKS_MODULUS = 0xFFFFFFFF00000001ULL; // 2^64 - 2^32 + 1

The state is 16 elements of the Goldilocks field, p = 2^64 − 2^32 + 1. One TIP5 permutation is 5 rounds, and each round is three layers:

1. S-box layer. The first 4 elements go through a split-and-lookup S-box (split the 64-bit element into bytes, run each through a fixed 256-entry table, recombine). The remaining 12 elements go through the algebraic power map x → x⁷ mod p.
2. MDS layer. Multiply the 16-element state by a fixed 16×16 MDS matrix — this is the diffusion step, and it is the most arithmetic-heavy part.
3. Round constants. Add 16 round-specific field constants.

Everything is modular arithmetic over a 64-bit prime. On a CPU that means a lot of 64-bit multiplies and conditional subtractions; on an FPGA it means DSP slices and a carefully pipelined datapath. The appeal is obvious: every one of those 16 lanes can run in parallel, every round can be a pipeline stage, and there is no reason a hash should take more than a handful of cycles once the pipeline is full.

AWS EC2 F2: eight UltraScale+ FPGAs you can rent by the hour

The reason this started in the cloud rather than on a desk is AWS EC2 F2 instances — the second generation of Amazon’s FPGA instances, which launched with a meaningful price/performance jump over the old F1 family. Each F2 FPGA is an AMD Virtex UltraScale+ HBM VU47P with 16 GB of HBM on-package, and you can rent up to eight of them in a single instance:

The host is a 3rd-gen AMD EPYC, the FPGA talks to it over PCIe, and AWS gives you a shell — a pre-built design that owns the PCIe endpoint, the DMA engines, and the HBM controllers, and exposes an AXI interface where your custom logic (the “CL”, custom logic) plugs in. You write the accelerator; the shell handles getting data in and out. Build the design, register it as an Amazon FPGA Image (AFI), and any F2 instance can load it.

The attraction was that I could prototype a single-FPGA design on an f2.6xlarge and, if it worked, fan it out to eight VU47Ps on an f2.48xlarge without touching the accelerator logic.

The main challenge: customizing the Shell/CL PCIe interface

Here is the thing nobody tells you about FPGA development on AWS: the hash was the easy part. The TIP5 datapath is a self-contained math problem I could simulate on my laptop. The hard, time-consuming, genuinely-new-to-me part was the interface — getting my custom logic to appear as a device the EC2 host can talk to, across the PCIe and AXI fabric that AWS’s Shell provides. That is where most of the engineering actually went, and it’s almost entirely written in SystemVerilog against the aws/aws-fpga HDK — I started, as everyone does, from its reference CLs (CL_ADD_TWO, cl_sde) and the sh_cl_interface definition.

The Shell/CL split. An F2 FPGA is divided into two halves. The Shell (SH) is AWS’s — it owns the PCIe endpoint, the DMA engines, the HBM/DDR controllers, and everything that must not go wrong for a multi-tenant cloud FPGA to be safe. The Custom Logic (CL) is yours. You never instantiate a PCIe core; you connect to the Shell across a fixed bundle of AXI interfaces (the sh_cl_* / cl_sh_* signals) and the Shell does the rest.

Two PCIe physical functions. The Shell presents the FPGA to the host as two PCIe Physical Functions:

• Application PF (AppPF) — yours. It exposes the BARs (Base Address Registers) that map the FPGA’s address space into the EC2 instance’s memory, so the host can read and write your logic.
• Management PF (MgmtPF) — reserved exclusively for AWS’s FPGA management tools (loading an AFI, reading metrics, clearing the image). You never touch it; it’s how AWS keeps the platform safe across tenants.

Two kinds of traffic. Within the AppPF, data moves over two AXI flavors, and picking the right one for each job is the core design decision:

• AXI4-Lite — for registers. Small, single-beat reads and writes to a memory-mapped control/status register file. This is the OCL interface (AppPF BAR0).
• AXI4 — for memory-mapped I/O. Wide (512-bit), bursting, high-throughput bulk transfers; this is the PCIS interface, fed by the Shell’s XDMA engine.

For the TIP5 miner I built the entire control plane on the OCL AXI4-Lite path and hand-wrote the register map. A mining job is just a set of registers the host pokes:

// OCL AXI-L: AppPF BAR0 (64 MiB window, 12-bit register space)
localparam DIGEST_WIDTH = 320;                 // 5 BFieldElements × 64b, laid out as 10× 32b regs
localparam ADDR_KERNEL_AUTH_PATH_0 = 12'h000;  // job input  ┐
localparam ADDR_HEADER_AUTH_PATH_0 = 12'h050;  //            │ the puzzle from neptune-core
localparam ADDR_THRESHOLD          = 12'h0C8;  //            │
localparam ADDR_NONCE_START        = 12'h0F0;  // job input  ┘
localparam ADDR_CONTROL            = 12'h200;  // write: start the kernels
localparam ADDR_STATUS             = 12'h204;  // read:  busy / found
localparam ADDR_VALID_NONCE        = 12'h210;  // result: the winning nonce
localparam ADDR_FINAL_HASH         = 12'h238;  // result: the digest that cleared the threshold

The host writes the puzzle (the kernel and header authentication paths, the threshold, the nonce base), writes CONTROL to start, polls STATUS until a nonce is found, then reads VALID_NONCE and FINAL_HASH back out. Behind those addresses is a hand-coded AXI4-Lite slave: the write and read state machines that drive the awready / wready / bvalid / arready / rvalid handshakes the protocol demands. (Unmapped reads return 0xDEADBEEF — a small thing that saved a lot of debugging.)

Two things make this harder than it sounds:

1. You must correctly tie off what you don’t use. A CL that leaves the PCIM, DMA_PCIS, or CL_SDA interfaces floating won’t pass the Shell’s checks. AWS ships stubs for exactly this, and the design includes them — unused_pcim_template.inc, unused_dma_pcis_template.inc, unused_cl_sda_template.inc — so every unused port is driven to a safe default.
2. Your FPGA has a PCIe identity you have to declare. cl_id_defines.vh carries the Vendor/Device and Subsystem IDs baked into the AFI manifest: CL_SH_ID0 = 0xF002_1D0F — Amazon’s 0x1D0F vendor ID with a device ID in the allowed 0xF000–0xF0FF range. Get this wrong and the AFI won’t register.

And the feedback loop is unforgiving. A mistake on the interface won’t surface in simulation if the testbench is wrong, and confirming it on real hardware means a full AFI build — synthesis, place-and-route, timing closure, image registration — measured in hours. So before any AFI, the CL is simulated against the Shell’s bus-functional model in verif/sim/test_cl_tip5.sv, with a C host-side counterpart (software/runtime/test_tip5_if.c) exercising the same register map. Getting the interface right in simulation is the difference between a one-hour iteration and a one-day one — and it’s exactly the pain that, six months later, pushed me off FPGAs entirely.

The accelerator: pipelined TIP5 cores behind an AXI/DMA front end

The design splits cleanly into a host side and an FPGA side.

FPGA side (cl_tip5). The custom logic is a tree of SystemVerilog modules, each one a layer of the hash:

• tip5_b_field_element.sv — Goldilocks field arithmetic (P = 64'hFFFFFFFF00000001), the primitive every other module is built on. Modular add, sub, and the Montgomery multiply that all the higher layers call.
• split_lookup.sv / sbox_layer.sv — the split-and-lookup S-box and the x⁷ power map.
• mds_matrix.sv / mds_layer.sv — the 16×16 MDS multiply.
• round_constants.sv — the per-round constant injection.
• tip5_permutation.sv — the 5-round permutation, wiring the layers into a pipeline.
• tip5_hash_pair.sv / tip5_hash_varlen.sv — sponge wrappers (hash a pair of digests; hash a variable-length input).
• fast_kernel_mast_hash.sv — the mining kernel: feed nonces, run the permutation, compare against the threshold.
• cl_tip5_wrapper.sv / cl_tip5.sv — the AWS Shell interface and the OCL AXI4-Lite register map (covered in detail above).

The whole thing synthesizes for xcvu35p-fsvh2104-2-e (the development part I used before the VU47P AFI build). fast_kernel_mast_hash is the bridge between the register map and the hash pipeline: it reads the job out of the OCL registers, runs the permutation across its slice of nonce space, and raises the STATUS flag with the winning nonce.

Host side (neptune-fpga-miner). Neptune Core is Rust. The AWS FPGA SDK is C. So the host miner is a standalone Rust binary that:

1. Talks to a neptune-core node over RPC to fetch a proof-of-work puzzle (the kernel auth path, the threshold, the guesser reward).
2. Uses bindgen-generated FFI bindings over the AWS FPGA SDK — fpga_pci peek/poke for the AXI4-Lite registers, fpga_dma for the bulk path — to open the device, write the job, start the kernels, and read the winning nonce back.
3. Wraps all of that in a safe Rust API — open_fpga_session(), transfer_data_to_fpga(), trigger_tip5_computation(), retrieve_fpga_results() — so the mining loop reads like ordinary Rust instead of raw FFI.

┌─────────────────┐    RPC/HTTP    ┌──────────────────┐    PCIe/DMA   ┌──────────┐
│  Neptune Node   │◄──────────────►│ Standalone Miner │◄─────────────►│  VU47P   │
│ • RPC server    │                │ • puzzle solver  │   AWS shell   │  TIP5 CL │
│ • block creation│                │ • FFI + DMA mgmt │               │  cores   │
└─────────────────┘                └──────────────────┘               └──────────┘

The register map is enough to drive a single miner, but it isn’t how you’d feed eight FPGAs at full tilt — that’s what the AXI4 PCIS/XDMA path is for, and there batching is the whole game. A single hash is far too small to justify a PCIe round-trip, so the host packs thousands of nonces per DMA transfer and overlaps the next transfer with the current computation — while the FPGA is grinding batch N, batch N+1 is already in flight across PCIe.

The optimization that mattered: parallelizing the S-box layer

The first working S-box layer was embarrassingly slow: 3,935 ns of latency per round, because it processed the 16 field elements sequentially. Each element waited for the one before it, and each x⁷ was itself a sequence of dependent multiplies (x² → x⁴ → x⁶ → x⁷) that stalled on a single shared field multiplier.

The fix was to stop being stingy with DSP blocks and lay the whole layer out in space instead of time:

• 16 lanes in parallel. All 16 field elements processed simultaneously — 4 split-and-lookup units and 12 power-map units, not one shared datapath.
• A 3-stage x⁷ pipeline. Stage 1 computes all the x² in parallel, stage 2 all the x⁴, stage 3 all the x⁷ — using 36 DSP48 multipliers (12 elements × 3 stages) instead of one.
• Parallel byte lookups. The split-and-lookup S-box does all 8 byte-table lookups at once out of 4 × 256×8 distributed-RAM tables, so a lookup is a single cycle instead of a serialized walk.

genvar i;
generate
    for (i = 0; i < NUM_SPLIT_AND_LOOKUP; i++) begin : gen_split_lookup_parallel
        split_lookup_optimized u (/* one instance per lane */);
    end
    for (i = 0; i < NUM_POWER_ELEMENTS; i++) begin : gen_mult_parallel
        // 3 pipeline stages × 12 elements = 36 parallel field multipliers
    end
endgenerate

At 100 MHz the pipelined layer settles in 4 cycles = 40 ns, down from 3,935 ns — a 98.4× latency reduction, and after the pipeline fills it retires one full 16-element state every cycle. That single change is the difference between a design that is theoretically an FPGA miner and one that actually saturates the fabric. The same idea then propagates up: instantiate many fast_kernel_mast_hash cores, give each its own slice of the nonce space, and let HBM bandwidth — not the hash datapath — become the limit.

What this design got right, and where it was heading

The F2 design was the right first instinct and it taught me the shape of the problem: TIP5 is wildly parallel, the field arithmetic is the cost center, and pipelining the permutation is where the wins are. The 98.4× S-box result was real and satisfying.

But two things were already nagging. First, an f2.48xlarge is expensive to leave running, and the AFI build loop — synthesis, place-and-route, timing closure, AFI registration — is measured in hours, not minutes. Second, renting eight FPGAs in the cloud is a strange way to learn the hardware; I wanted the bitstream on a board I could put a scope on.

So the next step was to bring it onto hardware I owned: an Osprey E300 carrying three VU35P UltraScale+ FPGAs, with a Zynq 7010 acting purely as the controller that programs them and relays UART. Same UltraScale+ silicon, now on my desk — and because the VU35P is a Virtex part with no processor, the miner’s whole controller had to move into the programmable logic, with no soft processor at all. That’s the next post.

References

• tip5-neptune — the full source for this build log
• Neptune Cash
• The Tip5 hash function (ePrint 2023/107)
• AWS EC2 F2 instances
• AWS FPGA development kit