Implement bitcoind-compatible JSON-RPC and REST blockchain methods by dsbaars · Pull Request #795 · libbitcoin/libbitcoin-server

dsbaars · 2026-06-08T17:24:34Z

Summary

The bitcoind compatibility protocol (src/protocols/bitcoind/) was largely stubbed. This PR implements the read-only blockchain surface plus raw-transaction retrieval and broadcast, reusing the existing query layer and the libbitcoin-system bitcoind* JSON serializers. Chain-context fields that the serializers intentionally omit (height, confirmations, next/prev hash, mediantime, blocktime) are injected at the protocol layer, mirroring protocol_native.

Motivation: enabling Bitcoin Core-compatible clients to read chain state and broadcast transactions against a libbitcoin node.

JSON-RPC methods implemented (return live data)

getbestblockhash, getblock (verbosity 0/1/2), getblockchaininfo, getblockcount, getblockhash, getblockheader, getblockfilter, gettxout, getnetworkinfo, getrawtransaction, sendrawtransaction.

getrawtransaction serves any archived (confirmed) transaction by txid. libbitcoin archives all confirmed transactions hash-addressably, i.e. an implicit txindex. Returns raw hex (verbosity 0) or Core-format JSON (txid/hash/size/vsize/weight/vin/vout/hex plus injected in_active_chain/blockhash/confirmations/blocktime/time).
sendrawtransaction deserializes the tx, runs context-free check(), archives it via query.set_code() so the existing protocol_transaction_out_106 can serve it on getdata, broadcasts it onto the shared peer-message bus to announce to peers, and returns the txid. No mempool subsystem is required for this path.

REST endpoints implemented

/rest/chaininfo.json, /rest/block/<hash>, /rest/block/notxdetails/<hash>, /rest/block/spent/<hash>, /rest/blockhashbyheight/<height>, /rest/headers/<count>/<hash>, /rest/blockfilter/basic/<hash>, /rest/blockfilterheaders/basic/<hash>, /rest/blockpart/<hash>/<offset>/<size>. Per-endpoint media type (binary / hex / json) is derived from the URL extension. bitcoind_target parses the Core REST URL scheme into a json-rpc request model that the REST dispatcher routes.

Deliberately left as clean `not_implemented`

getblockstats, getchaintxstats, getchainwork (need aggregate/cumulative indexes), gettxoutsetinfo, verifytxoutset, scantxoutset (need a UTXO-set scan/snapshot), verifychain (full revalidation), pruneblockchain (pruning), savemempool (needs a mempool). These dispatch and return a structured not_implemented error rather than failing.

Notes / design

Shared helpers (median_time_past, inject_block_context, inject_tx_context, header_to_bitcoind, chain_name) are hoisted into include/bitcoin/server/protocols/bitcoind_json.hpp and used by both the RPC and REST units.
getnetworkinfo string fields (subversion, warnings) were being selected as value_t(bool) from bare string literals; fixed.
Signatures were cross-checked against Bitcoin Core's RPCMethod definitions (bitcoin/bitcoin src/rpc). getblockstats hash_or_height now accepts a height or a hash (Core's skip_type_check behavior, via value_t), and getrawtransaction's verbosity param uses Core's canonical name verbosity.

Testing

Built and runtime-tested against an isolated regtest node: all implemented RPC methods and REST endpoints return correct genesis-block data; getrawtransaction (raw + verbose), sendrawtransaction (submit / invalid / unknown paths), and getblock v1/v2 context injection verified.
No new build-system entries required (the bitcoind translation units already existed; the one new file is header-only).

Open questions for maintainers

sendrawtransaction currently archives the tx before broadcasting and performs only context-free check(). Without chaser_transaction there is no contextual (UTXO/policy) validation or eviction of never-mined transactions. This is acceptable for broadcast, but I would welcome guidance on whether to gate it behind a flag or add minimal validation. A TODO marks the spot.
The interface declares getchainwork and verifytxoutset, which are not standard Bitcoin Core RPCs (Core exposes chainwork as a field, and has loadtxoutset/dumptxoutset).
Not yet covered (no current subsystem): getrawmempool/getmempoolinfo/testmempoolaccept and the REST getutxos/mempool/fork endpoints. These require a queryable mempool, which is currently stubbed in chaser_transaction / protocol_transaction_in.

getblock (verbosity 1/2), getblockcount, getblockhash, getblockheader, getblockchaininfo, getblockfilter, gettxout and getnetworkinfo, reusing the libbitcoin-system bitcoind json serializers with a protocol-level context injector (height, confirmations, mediantime, prev/next hash). Aggregate-heavy methods (getchainwork, getchaintxstats, getblockstats, gettxoutsetinfo, scantxoutset, verifychain, verifytxoutset, pruneblockchain, savemempool) remain explicit not_implemented.

Activate the REST path: bitcoind_target parses /rest/block/<hash>.<bin|hex|json> into a json-rpc model, handle_receive_get dispatches via rest_dispatcher_, and handle_get_block serves all three media types through new raw-http senders (send_data/send_hex/send_dom), which the bitcoind base lacked.

Extend the REST interface beyond block: block_hash (blockhashbyheight), block_txs (notxdetails), block_headers, block_part, block_spent_tx_outputs, block_filter, block_filter_headers and chain_information, with their Core REST url patterns parsed in bitcoind_target. Remaining endpoints (get_utxos[_confirmed], mempool[_information], fork_information) need mempool enumeration / deployment status / utxo semantics not yet exposed, so are left unimplemented.

Move median_time_past, inject_block_context, header_to_bitcoind and chain_name out of the duplicated anonymous namespaces in the rpc and rest protocol units into bitcoind_json.hpp, included by both.

A bare string literal selects value_t(boolean_t) over value_t(const string_t&) in the rpc::object_t initializer (const char* -> bool beats the user-defined string conversion), so subversion and warnings serialized as 'true'. Wrap them in std::string. Caught runtime-testing against a regtest node.

getrawtransaction serves any archived (confirmed) tx by txid: raw hex (verbose 0) or verbose JSON via the existing bitcoind_verbose serializer plus a new inject_tx_context helper (in_active_chain/blockhash/ confirmations/blocktime/time). libbitcoin archives all confirmed tx hash-addressable, so this is a built-in txindex. sendrawtransaction deserializes the hex tx, runs context-free check(), archives it via query.set_code() (so the existing protocol_transaction_ out_106 can serve it on getdata), broadcasts it onto the shared peer message bus to announce to peers, and returns the txid. No mempool subsystem required. TODO: contextual (connect) validation before archiving for policy/DoS hardening.

Two issues caught by runtime-testing against a regtest node: - The verbose/maxfeerate params were declared optional<0_u32> but the handlers take double; the rpc dispatcher threw bad_variant_access ("unexpected type") on any numeric arg. Declare as optional<0.0> to match, consistent with getblock's verbosity. - getrawtransaction verbose used bitcoind_verbose(tx), which on a standalone transaction falls back to libbitcoin's plain inputs/outputs form (no txid). Use bitcoind(tx) for Core's txid/hash/size/vsize/ weight/vin/vout/hex fields (same encoding getblock verbosity 2 embeds).

Cross-checked all declared bitcoind signatures against Core's RPCMethod definitions (bitcoin/bitcoin src/rpc). Two fixes: - getblockstats: hash_or_height was declared string_t, but Core accepts a height number OR a block hash (RPCArg::Type::NUM with skip_type_check). A numeric height threw 'unexpected type' at dispatch. Declare as value_t (the dispatcher passes it through untyped), matching Core; verified both a numeric height and a hash now reach the handler. - getrawtransaction: param named 'verbose'; Core's canonical name is 'verbosity' (with 'verbose' as an alias), and getblock already uses 'verbosity'. Rename for named-parameter compatibility; positional dispatch (used by LND/btcwallet) is unaffected.

evoskuil · 2026-06-08T22:29:33Z

Thanks! Just giving this a quick visual scan, and it looks good. A full set of passing functional tests (see electrum and native/block) would help get this merged much more quickly. We should also have acceptance tests (@eynhaender @echennells ?) to verify interface compliance.

Return target (expanded from bits), verificationprogress (confirmed/candidate height), initialblockdownload, and warnings, alongside the existing fields. chainwork and size_on_disk remain omitted, as they require a cumulative-work index and store-size accounting respectively.

Covers getrawtransaction (raw, verbose, coinbase, segwit, unknown txid) and sendrawtransaction (invalid and malformed input; the broadcast path is gated behind BITCOIND_ALLOW_BROADCAST to avoid relaying on mainnet), plus the REST endpoints (block json/hex/bin, notxdetails, spent, blockhashbyheight, headers, blockpart, and the basic filters). Verified against a synced mainnet node. getblockchaininfo's chainwork and size_on_disk assertions are relaxed to match the implementation.

Build an in-process HTTP harness on the bitcoind test fixture (beast POST for json-rpc, GET for REST, replacing the raw-socket placeholder) and add deterministic acceptance tests against the ten-block mock store: getblockcount/getbestblockhash/getblockhash/getblockheader/getblock/getblockchaininfo/gettxout/getrawtransaction/sendrawtransaction/getnetworkinfo, the not_implemented set, and the REST endpoints (chaininfo, block json/hex/bin, notxdetails, spent, blockhashbyheight, headers, blockpart, blockfilter). A witness-store fixture covers segwit getrawtransaction (wtxid != txid, vsize == ceil(weight/4)). These run in CI without a synced node.

dsbaars · 2026-06-09T11:53:50Z

Thanks for the suggestion. I have added both.

Functional tests (endpoints/, against a running node, in the style of the native and electrum suites): test_bitcoind_rpc.py now covers getrawtransaction and sendrawtransaction, and test_bitcoind_rest.py (new) covers the REST surface (chaininfo, block json/hex/bin, notxdetails, spent, blockhashbyheight, headers, blockpart, basic filters). Verified against a node synced past segwit activation: 28 passed, 1 skipped (a broadcast test gated behind an env flag), the rest xfail for the not_implemented methods and for filters when bip158 is off.

Acceptance tests (test/protocols/bitcoind/, in-process, run in CI without a synced node, in the style of native/electrum): the fixture only had a placeholder request helper, so I built an HTTP harness on it (beast POST for json-rpc, GET for REST) and filled in bitcoind_rpc.cpp and bitcoind_rest.cpp. 30 cases assert exact responses for every wired RPC method and REST endpoint against the ten-block mock store, plus a witness-store case for segwit getrawtransaction (reusing the existing block1a/block2a witness mocks).

I also extended getblockchaininfo to return target, verificationprogress, initialblockdownload, and warnings (chainwork and size_on_disk still need a cumulative-work index and store-size accounting). One compatibility note: the system bitcoind(tx) serializer reports size as the stripped size, whereas Core reports the total size for segwit transactions.

evoskuil · 2026-06-09T17:51:41Z

Thanks! Just a clarification on test terminology. We use these terms (sort of informally):

Unit Test - smallest unit possible/practical (e.g. lowest level method/function).
Component Test - aggregate of units (e.g. class or aggregating function/method).
Functional Test - system function (e.g. endpoint communication over a socket, public API).
Acceptance Test - performed over the compiled executable (e.g. customer acceptance).

We target full Unit Test coverage and Component and/or Functional as necessary to ensure expected aggregate behavior. No external dependencies (e.g. Python or other tools), just Boost Test in our pattern. Can be data-driven (which is when we allow loops in a test).

Acceptance is outside of the build (against the executable or compiled lib). Currently we have Python client-server tests in libbitcoin-server but I'm not sure what the status of that is. @eynhaender ?

Boost.Test, in-build, no external dependencies, per the libbitcoin test taxonomy (unit = lowest-level function, component = aggregate over a class). Unit (test/parsers/bitcoind_target.cpp, replacing the stub): cover the pure bitcoind_target() REST path parser across every route and error path (media mapping, missing/invalid target/hash/number, leading-zero, non-basic filter); error and media cases are data-driven. Unit + component (test/protocols/bitcoind/bitcoind_json.cpp, new): pure header_to_bitcoind field mapping, plus chain_name, median_time_past (BIP113), inject_block_context (genesis/middle/tip) and inject_tx_context (confirmed/unknown) against the ten-block mock store via a minimal store+query fixture (no server/socket). Register the new file in Makefile.am and the vs2022/vs2026 vcxproj/filters.

evoskuil

Very nice, minor comments. Suggest @eynhaender look at the python acceptance tests.

evoskuil · 2026-06-09T17:59:08Z

Only protocol class files in protocols. As bitcoind-specific utils these can be organized into the protocol_bitcoind_rpc base class (to support protocol_bitcoind_rest as well) or otherwise, as protected static methods. As desired these can be factored an independent implementation file for organizational purposes (e.g. protocols/bitcoind/protocol_bitcoind_rpc_json.cpp).

Also by placing into a class it can be removed from the server namespace into a more targeted space (the class name), so only where it is relevant. Class static methods use the class as a namespac.

evoskuil · 2026-06-09T18:01:03Z

+/// context (height, confirmations, etc.); these add it at the protocol layer.
+
+/// BIP113 median of up to 11 block timestamps ending at the given height.
+inline uint32_t median_time_past(const auto& query, size_t height) NOEXCEPT


We should avoid inline (header implementation) when not a necessary template or for performance. These aren't either so should go into .cpp. This helps keep build times down (e.g. test changes will rebuild all of these, as would dependent libs (this is a developer lib).

evoskuil · 2026-06-09T18:08:20Z

+    const auto genesis = system::encode_hash(
+        query.get_header_key(query.to_confirmed(0)));
+
+    if (genesis == "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f")


These hashes should ideally be derived from a on-off system::settings constructing, using the applicable enumeration. Currently the full genesis block is stored, and the hash can be obtained from that directly. Signet is not implemented there yet but can be stubbed in for this.

evoskuil · 2026-06-09T18:14:43Z

+    bool handle_get_block_hash(const code& ec, rest_interface::block_hash,
+        uint8_t media, uint32_t height) NOEXCEPT;
+    bool handle_get_block_txs(const code& ec, rest_interface::block_txs,
+        uint8_t media, system::hash_cptr hash) NOEXCEPT;


Use the convention of passing small objects as const& (see native and electrum) to avoid copy construction. While this may be optimized out by the compiler it's good to be clear about intent. A std::shared_ptr<T> copy increments the reference count (for no reason here, and hits a shared lock), and copies both members (pointer and counter). A const& just passes the address of the shared_ptr object.

evoskuil · 2026-06-09T18:18:43Z

-    return {};
+    hash_digest out{};
+    return decode_hash(out, token) ?
+        emplace_shared<const hash_digest>(std::move(out)) : hash_cptr{};


Can use to_shared(std::move(out)) here, which accepts a move (&&) arg, infers type, sets const and is NOEXCEPT (like emplace_shared). Just a little more compact syntactically. Also in this case it's not an actual emplace (in-place construction), it's move construction, so technically this implies an in-place construction with an intermediate move into the new object, vs just the move.

evoskuil · 2026-06-09T20:22:34Z

+BOOST_AUTO_TEST_CASE(bitcoind_rest__block_hex__hashes_to_block9)
+{
+    auto hex = rest_text("/rest/block/" + encode_hash(test::block9_hash) + ".hex");
+    while (!hex.empty() && (hex.back() == '\n' || hex.back() == '\r'))


Avoid conditions (including loops) in test cases that are not strictly looping over external text vectors (data-driven).

evoskuil · 2026-06-09T20:26:17Z

+
+    data_chunk wire{};
+    BOOST_REQUIRE(decode_base16(wire, hex));
+    BOOST_REQUIRE_EQUAL(block_hash_hex(wire), encode_hash(test::block9_hash));


Avoid calling encode_hash(test::block9_hash) twice in same method. This can be declared at file scope as constexpr and computed at compile for the full set of tests.

evoskuil · 2026-06-09T20:29:54Z

+{
+    auto hex = rest_text("/rest/block/" + encode_hash(test::block9_hash) + ".hex");
+    while (!hex.empty() && (hex.back() == '\n' || hex.back() == '\r'))
+        hex.pop_back();


This terminators stripping should be implemented in rest_text().

evoskuil · 2026-06-09T20:31:54Z

+
+static std::string as_text(const boost::json::value& value) NOEXCEPT
+{
+    return std::string{ value.as_string().c_str() };


Style: avoid unnecessary class name on return.

return { value.as_string().c_str() };

evoskuil · 2026-06-09T20:36:02Z

+// Reconstruct a block from wire bytes and return its hash as display hex.
+static std::string block_hash_hex(const data_chunk& wire) NOEXCEPT
+{
+    return encode_hash(chain::block{ wire, true }.hash());


This is unnecessary if you have the wire, can just hash the header bytes directly using:

return system::bitcoin_hash(system::chain::header::serialized_size(), wire.begin());

Relocate the bitcoind json context helpers out of the inline header (bitcoind_json.hpp) into protocol_bitcoind_rpc as protected static methods, implemented in protocol_bitcoind_rpc_json.cpp and inherited by the rest protocol. Removes them from the broad server namespace and keeps the implementation out of headers. chain_name now derives each network's genesis from system::settings (per the chain::selection enumeration) rather than hardcoded hashes; signet remains a documented stub pending its selection. Source idiom: pass hash_cptr by const& in the rest handlers (native/electrum convention), use to_shared(std::move()) and std::from_chars() in the target parser. Tests: bitcoind_target gains real coverage, bitcoind_json exercises the helpers via a test seam; rest_text() strips line terminators, display hashes are computed once at file scope, header bytes are hashed directly, and the verbose comments are dropped. Register protocol_bitcoind_rpc_json.cpp in Makefile.am and the vs2022/vs2026 project files.

dsbaars · 2026-06-09T22:25:18Z

Thanks for the review, all addressed in 0ef2d05d.

Json helpers moved out of the inline header into protocol_bitcoind_rpc as protected static methods (in protocol_bitcoind_rpc_json.cpp), inherited by rest.
chain_name derives genesis from system::settings per chain::selection; signet stays a documented stub.
Rest handlers take hash_cptr by const&; target parser uses to_shared and std::from_chars.
Tests: rest_text() strips terminators, hashes computed once at file scope, header hashed directly, no wrapping or non data-driven loops.

One deviation: I kept a leading-zero guard before from_chars, since bare from_chars accepts "01" which these fields should reject as non-canonical (matching Core).

Full Boost suite passes. @evoskuil would you mind taking another look when you have a moment?

dsbaars added 8 commits June 8, 2026 16:28

Hoist shared bitcoind json helpers to a header.

51b7337

Move median_time_past, inject_block_context, header_to_bitcoind and chain_name out of the duplicated anonymous namespaces in the rpc and rest protocol units into bitcoind_json.hpp, included by both.

evoskuil self-requested a review June 8, 2026 22:32

dsbaars added 3 commits June 9, 2026 13:19

evoskuil requested a review from eynhaender June 9, 2026 17:53

evoskuil assigned eynhaender Jun 9, 2026

evoskuil added the enhancement label Jun 9, 2026

evoskuil requested changes Jun 9, 2026

View reviewed changes

Conversation

dsbaars commented Jun 8, 2026

Summary

JSON-RPC methods implemented (return live data)

REST endpoints implemented

Deliberately left as clean not_implemented

Notes / design

Testing

Open questions for maintainers

Uh oh!

evoskuil commented Jun 8, 2026

Uh oh!

dsbaars commented Jun 9, 2026

Uh oh!

evoskuil commented Jun 9, 2026

Uh oh!

evoskuil left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dsbaars commented Jun 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Deliberately left as clean `not_implemented`