This week’s newsletter describes a proposal to begin testing HTLC endorsement, relays a request for feedback about proposed specifications for Lightning Service Providers (LSPs), discusses challenges with opening zero-conf channels when using dual funding, looks at a suggestion for advanced applications of payjoin transactions, and links to summaries of a recent in-person meeting of Bitcoin Core developers. Also included in this week’s newsletter is the first part of a new series about policies for transaction relay and mempool inclusion, plus our regular sections announcing new releases and release candidates (including a security release of libsecp256k1) and describing notable changes to popular Bitcoin infrastructure software.

News

• ● Testing HTLC endorsement: several weeks ago, Carla Kirk-Cohen and Clara Shikhelman posted to the Lightning-Dev mailing list about the next steps they and others planned to take to test the idea of HTLC endorsement (see Newsletter #239) as part of a mitigation for channel jamming attacks. Most notably, they provided a short proposed specification that could be deployed using an experimental flag, preventing deployments of it from having any effect on interactions with non-participating nodes.

  Once deployed by experimenters, it should become easier to answer one of the constructive criticisms of this idea, which is how many forwarded payments would actually receive a boost from the scheme. If the core users of LN are frequently sending payments to each other over many of the same routes, and if the reputation system works as planned, then that core network will be more likely to keep functioning during a channel jamming attack. But if most spenders only send payments rarely (or only send their most critical types of payments rarely, such as high-value payments), then they won’t have enough interactions to build a reputation, or the reputation data will lag far behind the current state of the network (making it less useful or even allowing reputation to be abused).

• ● Request for feedback on proposed specifications for LSPs: Severin Bühler posted to the Lightning-Dev mailing list a request for feedback on two specifications for interoperability between Lightning Service Providers (LSPs) and their clients (usually non-forwarding LN nodes). The first specification describes an API for allowing a client to purchase a channel from an LSP. The second describes an API for setting up and managing Just-In-Time (JIT) channels, which are channels that start their lives as virtual payment channels; when the first payment to the virtual channel is received, the LSP broadcasts a transaction that will anchor the channel onchain when it is confirmed (making it into a regular channel).

  In a reply, developer ZmnSCPxj wrote in favor of open specifications for LSPs. He noted that they make it easy for a client to connect to multiple LSPs, which will prevent vendor lock-in and improve privacy.

• ● Challenges with zero-conf channels when dual funding: Bastien Teinturier posted to the Lightning-Dev mailing list about the challenges of allowing zero-conf channels when using the dual-funding protocol. Zero-conf channels can be used even before the channel open transaction is confirmed; this is trustless in some cases. Dual-funded channels are channels that were created using the dual-funding protocol, which may include channels where the open transaction contains inputs from both parties in the channel.

  Zero-conf is only trustless when one party controls all of the inputs to the open transaction. For example, Alice creates the open transaction, gives Bob some funds in the channel, and Bob tries spending those funds through Alice to Carol. Alice can safely forward the payment to Carol because Alice knows she’s in control of the open transaction eventually becoming confirmed. But if Bob also has an input in the open transaction, he can get a conflicting transaction confirmed that will prevent the open transaction from confirming—preventing Alice from being compensated for any money she forwarded to Carol.

  Several ideas for allowing zero-conf channel opens with dual funding were discussed, although none seemed satisfying to participants as of this writing.

• ● Advanced payjoin applications: Dan Gould posted to the Bitcoin-Dev mailing list several suggestions for using the payjoin protocol to do more than just send or receive a simple payment. Two of the suggestions we found most interesting were versions of transaction cut-through, an old idea for improving privacy, improving scalability, and reducing fee costs:

  • ● Payment forwarding: rather than Alice paying Bob, Alice instead pays Bob’s vendor (Carol), reducing a debt he owes her (or pre-paying for an expected future bill).

  • ● Batched payment forwarding: rather than Alice paying Bob, Alice instead pays several people Bob owes money (or wants to establish a credit with). Gould’s example considers an exchange that has a steady stream of deposits and withdrawals; payjoin allows withdrawals to be paid for by new deposits when possible.

  Both of these techniques allow reducing what would be at least two transactions into a single transaction, saving a considerable amount of block space. When batching is used, the space savings may be even larger. Even better from the perspective of the original receiver (e.g. Bob), the original spender (e.g. Alice) may pay all or some of the fees. Beyond the space and fee savings, removing transactions from the block chain and combining operations like receiving and spending makes it significantly more difficult for block chain surveillance organizations to reliably trace the flow of funds.

  As of this writing, the post had not received any discussion on the mailing list.

• ● Summaries of Bitcoin Core developers in-person meeting: several developers working on Bitcoin Core recently met to discuss aspects of the project. Notes from several discussions during that meeting have been published. Topics discussed included fuzz testing, assumeUTXO, ASMap, silent payments, libbitcoinkernel, refactoring (or not), and package relay. Also discussed were two other topics we think deserve special attention:

  • ● Mempool clustering summarizes a suggestion for a significant redesign of how transactions and their metadata are stored in Bitcoin Core’s mempool. The notes describe a number of problems with the current design, provide an overview of the new design, and suggest some of the challenges and tradeoffs involved. A description of the design and a copy of the slides from the presentation were later published.

  • ● Project meta discussion summarizes a varied discussion about the project’s goals and how to achieve them despite many challenges, both internal and external. Some of the discussion has already led to experimental changes in the project’s management, such as a more project-focused approach for the next major release after version 25.

Waiting for confirmation #1: why do we have a mempool?

The first segment in a limited weekly series about transaction relay, mempool inclusion, and mining transaction selection—including why Bitcoin Core has a more restrictive policy than allowed by consensus and how wallets can use that policy most effectively.

Many nodes on the Bitcoin network store unconfirmed transactions in an in-memory pool, or mempool. This cache is an important resource for each node and enables the peer-to-peer transaction relay network.

Nodes that participate in transaction relay download and validate blocks gradually rather than in spikes. Every ~10 minutes when a block is found, nodes without a mempool experience a bandwidth spike, followed by a computation-intensive period validating each transaction. On the other hand, nodes with a mempool have typically already seen all of the block’s transactions and store them in their mempools. With compact block relay, these nodes just download a block header along with shortids, and then reconstruct the block using transactions in their mempools. The amount of data used to relay compact blocks is tiny compared to the size of the block. Validating the transactions is also much faster: the node has already verified (and cached) signatures and scripts, calculated the timelock requirements, and loaded relevant UTXOs from disk if necessary. The node can also forward the block onto its other peers promptly, dramatically increasing network-wide block propagation speed and thus reducing the risk of stale blocks.

Mempools can also be used to build an independent fee estimator. The market for block space is a fee-based auction, and keeping a mempool allows users to have a better sense of what others are bidding and what bids have been successful in the past.

However, there is no such thing as “the mempool”—each node may receive different transactions. Submitting a transaction to one node does not necessarily mean that it has made its way to miners. Some users find this uncertainty frustrating, and wonder, “why don’t we just submit transactions directly to miners?”

Consider a Bitcoin network in which all transactions are sent directly from users to miners. One could censor and surveil financial activity by requiring the small number of entities to log the IP addresses corresponding to each transaction, and refuse to accept any transactions matching a particular pattern. This type of Bitcoin may be more convenient at times, but would be missing a few of Bitcoin’s most valued properties.

Bitcoin’s censorship-resistance and privacy come from its peer-to-peer network. In order to relay a transaction, each node may connect to some anonymous set of peers, each of which could be a miner or somebody connected to a miner. This method helps obfuscate which node a transaction originates from as well as which node may be responsible for confirming it. Someone wishing to censor particular entities may target miners, popular exchanges, or other centralized submission services, but it would be difficult to block anything completely.

The general availability of unconfirmed transactions also helps minimize the entrance cost of becoming a block producer—someone who is dissatisfied with the transactions being selected (or excluded) may start mining immediately. Treating each node as an equal candidate for transaction broadcast avoids giving any miner privileged access to transactions and their fees.

In summary, a mempool is an extremely useful cache that allows nodes to distribute the costs of block download and validation over time, and gives users access to better fee estimation. At a network level, mempools support a distributed transaction and block relay network. All of these benefits are most pronounced when everybody sees all transactions before miners include them in blocks - just like any cache, a mempool is most useful when it is “hot” and must be limited in size to fit in memory. Next week’s section will explore the use of incentive compatibility as a metric for keeping the most useful transactions in mempools.

Releases and release candidates

New releases and release candidates for popular Bitcoin infrastructure projects. Please consider upgrading to new releases or helping to test release candidates.

• ● Libsecp256k1 0.3.2 is a security release for applications that use libsecp for ECDH and that may be compiled with GCC version 13 or higher. As described by the authors, new versions of GCC attempt to optimize libsecp code that was designed to run in a fixed amount of time, making it possible to execute a side-channel attack under certain circumstances. It’s worth noting that Bitcoin Core does not use ECDH and is not affected. There is ongoing work to try to detect when future changes to compilers might cause similar problems, allowing changes to be made in advance.

• ● Core Lightning 23.05rc2 is a release candidate for the next version of this LN implementation.

• ● Bitcoin Core 23.2rc1 is a release candidate for a maintenance release of the previous major version of Bitcoin Core.

• ● Bitcoin Core 24.1rc3 is a release candidate for a maintenance release of the current version of Bitcoin Core.

• ● Bitcoin Core 25.0rc2 is a release candidate for the next major version of Bitcoin Core.

Notable code and documentation changes

Notable changes this week in Bitcoin Core, Core Lightning, Eclair, LDK, LND, libsecp256k1, Hardware Wallet Interface (HWI), Rust Bitcoin, BTCPay Server, BDK, Bitcoin Improvement Proposals (BIPs), Lightning BOLTs, and Bitcoin Inquisition.

• ● Bitcoin Core #26076 updates RPC methods that show derivation paths for public keys now use h instead of a single-quote ' to indicate a hardened derivation step. Note that this changes the descriptor checksum. When handling descriptors with private keys, the same symbol is used as when the descriptor was generated or imported. For legacy wallets the hdkeypath field in getaddressinfo and the serialization format of wallet dumps remain unchanged.

• ● Bitcoin Core #27608 will continue trying to download a block from a peer even if another peer provided the block. Bitcoin Core will continue trying to download the block from peers that claim to have it until one of the received blocks has been written to disk.

• ● LDK #2286 allows creating and signing PSBTs for outputs controlled by the local wallet.

• ● LDK #1794 begins adding support for dual funding, starting with methods needed for the interactive funding protocol that is used for dual funding.

• ● Rust Bitcoin #1844 makes the schema in a BIP21 URI lowercase, i.e. bitcoin:. Although the specification for URI schema (RFC3986) says the schema is case insensitive, testing shows that some platforms are not opening the application assigned to handle bitcoin: URIs when an uppercase BITCOIN: is passed. It would be preferable if uppercase was handled correctly, as it allows the creation of more efficient QR codes (see Newsletter #46).

• ● Rust Bitcoin #1837 adds a function for generating a new private key, simplifying what previously required more code to accomplish.

• ● BOLTs #1075 updates the specification so that nodes should no longer disconnect from a peer after receiving a warning message from it.