This week’s newsletter describes the proposed Erlay protocol that could
significantly reduce the overhead of relaying unconfirmed transactions between
nodes, summarizes an executive briefing by Bitrefill CEO Sergej Kotliar about
LN, and lists some recent changes to the COSHV proposal described last week.
Also included are our regular sections on bech32 sending support and notable
code changes in popular Bitcoin infrastructure projects.
None this week.
Erlay proposed: a new paper by Gleb Naumenko, Pieter Wuille, Gregory
Maxwell, Sasha Fedorova, and Ivan Beschastnikh describes an alternative
transaction relay protocol, Erlay. Currently, when a node learns about a
new transaction that passes its relay policy, it announces the txid of that
transaction to all of its peers (except any peers that announced it
previously). This is a very simple and effective policy, but it’s not
efficient—most of the announcements a node receives are announcements for
transactions it has already learned about from a different peer a few moments
before, a redundancy that wastes about 44% of all node bandwidth according to
The paper attempts to reduce that redundancy using the authors’ Erlay
protocol which separates relaying into two phases, a fanout phase and a
reconciliation phase. In the fanout phase, a node will only directly announce
new transactions it has learned about to a maximum of eight peers selected from
the node’s outbound peers.
In the reconciliation phase, a node will periodically request from each of
its outbound peers a sketch of short transaction identifiers (short txids)
for all the new transactions that peer would normally announce to the node.
Sketches are created using the libminisketch library described in
Newsletter #26 which implements a highly bandwidth efficient set
reconciliation technique based on error correction codes. Upon receipt of a
requested sketch, the node itself also generates a sketch of all the new
transactions that it would normally announced to its peer. The node
combines its sketch with the peer’s sketch to produce a list of any differences
between the sketches. That diff contains the short txids of any transactions
that are in one set of new transactions but not the other. The node can then
request any transactions it doesn’t have from that peer and proceed to querying
its next peer for that peer’s sketch. This is repeated for a new peer once
each second, allowing the node to quickly receive any transactions it didn’t
receive via a fanout announcement. This simple two-step protocol of fanout and
reconciliation describes the primary operation of Erlay; the remaining parts of
the protocol described and analyzed in the paper cover the optimal parameters
to use for the sketches and a series of fallback steps to take if a particular
attempt at set reconciliation fails.
After describing the Erlay protocol, the paper analyzes its performance
using both a simulated network of 60,000 nodes (similar in number and use to
the current Bitcoin network) and a live set of 100 nodes spread internationally
across several datacenters. The most notable result is that Erlay reduces the
amount of bandwidth used to announce the existence of new transactions by 84%.
However, it does take about 80% (2.6
seconds) longer for transactions to propagate to all of the nodes in the
network. It’s still the case that Bitcoin transactions can only
be confirmed once every ten minutes on average, so a three second slowdown
seems like a reasonable price to pay for a major reduction in bandwidth.
Having established that the protocol is a worthwhile efficiency
improvement, the paper considers the most important of its secondary aspects:
its effect on privacy. Currently, each Bitcoin Core node slightly delays
the relaying of transactions to
its peers; this makes it more difficult for spy nodes to use timing
correlations to guess that the first peer they receive a transaction from is
the peer that created the transaction. Multiple simulations were run testing
the effectiveness of the current relay protocol against the effectiveness of
Erlay for various amounts of spy nodes among network peers, from 5% spy nodes
to 60% spy nodes. In cases where the spy nodes were public nodes accepting
incoming connections, Erlay always performed as well or better than the current
protocol. In cases where the spy nodes were private nodes making connections
to honest nodes, Erlay sometimes performed better and sometimes performed
worse—but never more than 10% worse (and this worst case is an unlikely situation). The paper also notes that Erlay is compatible with the
proposed BIP156 Dandelion protocol for further improving the network’s
resistance to spy nodes.
Considering future possible changes to Bitcoin relay policy, the paper
notes that an increase in the number of outbound peers from 8 to 32 would only
increase the bandwidth used by a node to announce the existence of new
transactions by 32% with Erlay compared to 300% using the current protocol.
As described in the paragraph above about Erlay’s two
phases, new transactions would still only be fanned out via direct announcements
to 8 peers, but nodes would perform set reconciliation with all 32 peers.
A four-fold improvement in relay connectivity improves the chance that
time-sensitive transactions for contract protocols like LN will make it
to miners quickly.
In addition to libminisketch, the code changes necessary to implement Erlay
in Bitcoin Core amount to only 584 lines of code, and the CPU intensive part of
set reconciliation has been benchmarked to take less than a millisecond under
conditions that are worse than expected in practice. If no objections are
raised over the paper, Naumenko has announced his intention to write a BIP and
work to get Erlay included in a subsequent version of Bitcoin Core. The method
used for transaction relay is part of the P2P network protocol (not a consensus
rule), so the change could begin operation as soon as multiple users upgrade to
it, although we expect nodes will also include a backwards-compatible mode to
support peers who haven’t upgraded.
Presentation: Operating on Lightning: Bitrefill CEO Sergej Kotliar gave a
presentation about LN for the Optech executive briefing last month. The
video is now available. Kotliar begins by explaining that high
transaction fees during previous years had a significant effect on Bitrefill’s
business, so they made a special effort to get really good at minimizing
fee-related expenses. The ability to receive LN payments supported that goal,
and he believes they were the first service on mainnet to sell real items for
LN payments. Today, LN payments represent about 5% of their sales, similar to
the amount of business they do using Ethereum.
He believes it’s important for businesses to start working on LN now. “In
Bitcoin we’ve gotten used to waiting for things […] but making customers wait
an unknown amount of time creates a risk that the customer will go away.” For
example, by the time a deposit clears at an exchange, the customer may no
longer be interested in making the trade that would’ve earned the exchange a
commission. Additionally, Bitrefill’s experience with LN is that LN’s improved
invoicing eliminates a number of different payment errors seen with onchain
bitcoin payments, including overpayments, underpayments, stuck transactions,
copy/paste errors, and other problems. Receiving payments over LN also
eliminates the need to consolidate UTXOs and reduces the need to rotate money
between hot and cold wallets. Eliminating all of these problems has the potential to significantly reduce customer support and backend expenses.
In a particularly interesting section of his talk, Kotliar shows how perhaps as
much as 70% of current onchain payments are users moving money
from one exchange to another exchange (or even between different users of the
same exchange). If all that activity could be moved offchain using LN
payments, exchanges and their users could save a considerable amount of money and everyone in Bitcoin would benefit from the increase in available block space.
Kotliar concluded his talk with a few short segments. He described what
software and services Bitrefill sees LN users using today and what he
expects them to be using in the near future. He then explained two of
Bitrefill’s services for LN users (including businesses), Thor and Thor
Turbo. Finally, he briefly described several planned improvements to LN:
reusable addresses (see Newsletter #30), splicing in and out (see
#22), and larger channels (also #22).
Overall, Kotliar made a compelling case that LN’s faster speed, lower fees,
and improved invoicing means businesses that expect to remain competitive
serving Bitcoin customers in the near future should start working on
implementing LN support today.
COSHV proposal replaced: the author of the COSHV proposal we described
last week has replaced it with a similar
proposal under a different name. The new proposal checks more
than just the hash of a transaction’s outputs—now the hash also includes the transaction’s
version number, number of inputs, sequence numbers, and locktime. This
change eliminates concerns related to transaction malleability that would’ve
affected using the opcode with some types of contract protocols (such as
LN). Additionally, by hashing the
number of inputs allowed in the transaction, the original proposal’s
restriction on just one input is lifted—however, the proposal warns that only
allowing a single input is still recommended to avoid unwanted interactions.
(Note: except for the new name, the changes don’t affect the summary
of COSHV we wrote last week.)
As of this writing, the Bitcoin Core project currently has over 300 open Pull
Requests (PRs). Some of these are works in progress, but most of
the rest are waiting for developers to review them and either identify any
problems that need to be fixed or acknowledge that they’re ready to be
If you want to help Bitcoin Core improve by reviewing PRs but aren’t
sure how to get started, we recommend you check out the Bitcoin Core PR
Review Club. In a weekly IRC meeting, an experienced Bitcoin Core
contributor provides background information
about a selected PR and then answers live questions from new
contributors. Often they are assisted by other established Bitcoin Core
contributors, including sometimes the authors of the PRs being
Optech recommends attendance to any engineer who wants to get more
involved in the Bitcoin Core process. IRC meetings are held at 17:00
UTC on Wednesdays.
Bech32 sending support
Week 12 of 24 in a series about allowing the people
you pay to access all of segwit’s benefits.
This segment marks half-way through our series about bech32, so we decided to
have some fun this week by describing some bech32-related trivia that’s
interesting but not important enough for its own segment.
How is bech32 pronounced? Pieter Wuille, co-author of the proposal, uses a
soft “ch” so that the word sounds like “besh thirty two”.
The name is a portmanteau that mixes the letters of the address’s error
correction coding (BCH) into the name of its numeric base (base32).
Pronouncing it with a soft “ch” allows the first syllable of bech32 to be
similar to Bitcoin’s legacy address format, base58. We admit this extended
explanation ruins the joke, but it’s a clever and amusing bit of wordplay.
BCH has nothing to do with Bitcoin Cash’s ticker code: the name of the
BCH codes bech32 is based upon are an abbreviation for
Bose-Chaudhuri-Hocquenghem, with Hocquenghem inventing this
type of cyclic codes in 1959 followed by Bose and Ray-Chaudhuri independently
rediscovering them in 1960. Moreover, the bech32 address format was announced
in March 2017, three months before the first plans for what would later be
labeled Bitcoin Cash (which initially planned to use the ticker code BCC).
Over ten CPU years consumed: using existing information about
BCH codes, the authors of bech32 were able to find the set of codes that
provided the minimum amount of error detection they desired for Bitcoin
addresses. However, there were almost 160 thousand eligible codes in this
set, and the authors expected some of them to be better than others. To find the best
code among them, over 200 CPU cores and “more than 10 years of computation time” was used.
Bitcoin Core #15741 speeds up importing keys, addresses, and other
information into the wallet using the importmulti RPC by batching the
database updates rather than making them sequentially. In a test performed by
the PR author, this reduced the time to import 10,000 addresses from 465
seconds to 4 seconds.
LND #2985 waits to relay gossip announcements until there are there are
at least ten of them to send and five seconds have elapsed since the previous
batch, reducing bandwidth overhead. This continues the work done by
LND and other implementations towards reducing the overhead of the
gossip protocol given the significant growth in the size of the LN
network over the past year.
LND #2761 switches to using a persistent state machine for routed
payments and to storing some additional state data for the machine, improving
the program’s ability to correctly handle HTLCs that were unresolved when the
daemon was restarted.