This week’s newsletter announces the disclosure of two vulnerabilities affecting older versions of Bitcoin Core and summarizes a proposed approach to optimizing miner transaction selection when cluster mempool is in use. Also included are our regular sections announcing new releases and release candidates and describing notable changes to popular Bitcoin infrastructure software.

News

  • ● Disclosure of vulnerabilities affecting Bitcoin Core versions before 22.0: Niklas Gögge posted to the Bitcoin-Dev mailing list a link to announcements of two vulnerabilities affecting versions of Bitcoin Core that have been past their end of life since at least October 2022. This follows a previous disclosure last month of older vulnerabilities (see Newsletter #310). We summarize the disclosures below:

    • ● Remote crash by sending excessive addr messages: before Bitcoin Core 22.0 (released September 2021), a node that was told about more than 232 other possible nodes would crash due to exhaustion of a 32-bit counter. This could be accomplished by an attacker sending a large number of P2P addr messages (at least 4 million messages). Eugene Siegel responsibly disclosed the vulnerability and a fix was included in Bitcoin Core 22.0. See Newsletter #159 for our summary of the fix, which was written without us knowing that it patched a vulnerability.

    • ● Remote crash on local network when UPnP enabled: before Bitcoin Core 22.0, nodes that enabled UPnP for automatically configuring NAT traversal (disabled by default due to previous vulnerabilities, see Newsletter #310) were vulnerable to a malicious device on the local network repeatedly sending variants of a UPnP message. Each message could result in the allocation of additional memory until the node crashed or was terminated by the operating system. An infinite loop bug in Bitcoin Core’s dependency miniupnpc was reported to the miniupnpc project by Ronald Huveneers, with Michael Ford discovering and responsibly disclosing how it could be used to crash Bitcoin Core. A fix was included in Bitcoin Core 22.0.

    Additional vulnerabilities affecting later versions of Bitcoin Core are expected to be disclosed in a few weeks.

  • ● Optimizing block building with cluster mempool: Pieter Wuille posted to Delving Bitcoin about ensuring that miner block templates can include the best set of transactions when using cluster mempool. In the design for cluster mempool, clusters of related transactions are divided into an ordered list of chunks, with each chunk obeying two constraints:

    1. If any transactions within the chunk depend on other unconfirmed transactions, those other transactions must either be a part of that chunk or appear in a chunk earlier in the ordered list of chunks.

    2. Each chunk must have an equal or higher feerate than the chunks that come after it in the ordered list.

    This allows every chunk from every cluster in the mempool to be placed into a single list in feerate order—highest feerate to lowest feerate. Given a chunked mempool in feerate order, a miner can construct a block template by simply iterating over each chunk and including it in their template until they reach a chunk that will not fit their desired maximum block weight (which is usually a bit below the 1 million vbyte limit to leave room for the miner’s coinbase transaction).

    However, clusters and chunks vary in size, with the default upper limit for a cluster in Bitcoin Core expected to be about 100,000 vbytes. That means a miner constructing a block template that is targeting 998,000 vbytes, and which already has 899,001 vbytes filled, may encounter a 99,000 vbyte chunk that doesn’t fit, leaving roughly 10% of their block space unused. That miner can’t simply skip that 99,000-vbyte chunk and try to include the next chunk because the next chunk might include a transaction that depends on the 99,000-vbyte chunk. If a miner fails to include a dependent transaction in their block template, any block they produce from that template will be invalid.

    To work around this edge case problem, Wuille describes how large chunks can be broken down into smaller sub-chunks that can considered for inclusion in the remaining block space based on their feerates. A sub-chunk can be created by simply removing the last transaction in any existing chunk or sub-chunk that has two or more transactions. This will always produce at least one sub-chunk that is smaller than its original chunk and it may sometimes result in several sub-chunks. Wuille demonstrates that the number of chunks and sub-chunks equals the number of transactions, with each transaction belonging to a unique chunk or sub-chunk. That makes it possible to precompute each transaction’s chunk or sub-chunk, called its absorption set, and associate that with the transaction. Wuille shows how the existing chunking algorithm already calculates each transaction’s absorption set.

    When a miner has filled a template with all of the full chunks possible, it can take the precomputed absorption sets for all transactions not yet included in the block and consider them in feerate order. This only requires a single sort operation on a list with the same number of elements as there are transactions in the mempool (almost always less than a million with current defaults). The best feerate absorption sets (chunks and sub-chunks) can then be used to fill the remaining block space. This requires tracking the number of transactions from a cluster that have been included so far and skipping any sub-chunks that don’t fit or which have already had some of their transactions included.

    However, although chunks can be compared with each other to provide the best order for block inclusion, the individual transactions within a chunk or sub-chunk are not guaranteed to be in the best order for only including some of those transactions. That can lead to non-optimal selection when a block is nearly full. For example, when only 300 vbytes remain, the algorithm might select a 200-vbyte transaction at 5 sats/vbyte (1,000 sats total) instead of two 150-vbyte transactions at 4 sats/vbyte (1,200 sats total).

    Wuille describes how precomputed absorption sets are especially useful in this case: because they only require tracking the number of transactions from each cluster that have been included so far, they make it easy to restore to an earlier state in the template-filling algorithm and replace the previously made choice with an alternative to see if it results in collecting more total fees. This allows implementing a branch-and-bound search that can try many combinations of filling the last bit of block space in the hopes of finding a better result than the simple algorithm.

  • ● Hyperion network event simulator for the Bitcoin P2P network: Sergi Delgado posted to Delving Bitcoin about Hyperion, a network simulator he’s written that tracks how data propagates through a simulated Bitcoin network. The work is initially motivated by a desire to compare Bitcoin’s current method for relaying transaction announcements (inv inventory messages) to the proposed Erlay method.

Releases and release candidates

New releases and release candidates for popular Bitcoin infrastructure projects. Please consider upgrading to new releases or helping to test release candidates.

  • ● BDK 1.0.0-beta.1 is a release candidate for “the first beta version of bdk_wallet with a stable 1.0.0 API”.

Notable code and documentation changes

Notable recent changes in Bitcoin Core, Core Lightning, Eclair, LDK, LND, libsecp256k1, Hardware Wallet Interface (HWI), Rust Bitcoin, BTCPay Server, BDK, Bitcoin Improvement Proposals (BIPs), Lightning BOLTs, Lightning BLIPs, Bitcoin Inquisition, and BINANAs.

  • ● Bitcoin Core #30515 adds a UTXO’s block hash and confirmation count as additional fields to the scantxoutset RPC command response. This provides a more reliable identifier for the UTXO’s block than just the block height, especially since chain reorganizations can occur.

  • ● Bitcoin Core #30126 introduces a cluster linearization function Linearize that operates on clusters of related transactions to create or improve linearizations, as part of the cluster mempool project. Cluster linearizations suggest a fee-maximizing order in which a cluster’s transactions could be added to block templates (or a minimal-fee-loss order in which they can be evicted from a full mempool). These functions are not yet integrated into the mempool, so there’s no behavior change in this PR.

  • ● Bitcoin Core #30482 improves parameter validation for REST endpoint getutxos by rejecting truncated or overlarge txids and throwing an HTTP_BAD_REQUEST parse error. Previously this would also fail, but would be handled silently.

  • ● Bitcoin Core #30275 changes the default mode of the estimatesmartfee RPC command from conservative to economical. This change is based on user and developer observations that the conservative mode often leads to overpayment of transaction fees because it is less responsive to short-term fee market drops than the economical mode when estimating fees.

  • ● Bitcoin Core #30408 replaces the use of the wording “public key script” to “output script” to refer to a scriptPubKey in the help text for the following RPC commands decodepsbt, decoderawtransaction, decodescript, getblock (if verbosity=3), getrawtransaction (if verbosity=2,3), and gettxout. This is the same wording used in the proposed BIP for transaction terminology (See Newsletter #246).

  • ● Core Lightning #7474 updates the offers plugin to allow for the newly defined experimental ranges for Type-Length-Value (TLV) types used in offers, invoice requests, and invoices. This was recently added to the unmerged BOLT12 pull request in the BOLTs repository.

  • ● LND #8891 adds a new min_relay_fee_rate field to the expected response from an external fee estimation API source, allowing the service to specify the minimum relay fee rate. If not specified, the default FeePerKwFloor of 1012 sats/kvB (1.012 sats/vbyte) will be used. The PR also improves startup reliability by returning an error from EstimateFeePerKW if called before the fee estimator has fully initialized.

  • ● LDK #3139 improves the security of BOLT12 offers by authenticating the use of blinded paths. Without this authentication, attacker Mallory can take Bob’s offer and request an invoice from each node on the network to determine which one of them belongs to Bob, negating the privacy benefit of using a blinded path. To fix this, a 128-bit nonce is now included in each offer’s encrypted blinded path, rather than in the offer’s unencrypted metadata. This change invalidates outbound payments and refunds with non-empty blinded paths created in prior versions. On the other hand, offers created in prior versions are still valid but are vulnerable to de-anonymization attacks, so users may want to regenerate them after they update to a version of LDK that includes this patch.

  • ● Rust Bitcoin #3010 introduces a length field to sha256::Midstate, allowing for more flexible and accurate tracking of the hash state when incrementally generating a SHA256 digest. This change may affect existing implementations that rely on the previous Midstate structure.