(An excerpt from an article originally published on the BU blog)
Examining the history behind the mempool chaining limit is instructive. When Satoshi first released the bitcoin client, there was no such limit. In a certain sense, there was no mempool either [1] — miners simply collected valid transactions and confirmed them in the next block. And nothing broke. For years. In fact, it wasn’t until blocks became congested, in the summer of 2015, that the “mempool” really became a distinct concept unto itself [2]. The Core devs decided at that time that it needed new rules to ensure that it was more carefully managed [3].
The problem was that if transactions were admitted into the node faster than they were being written out to blocks, then a queue of unconfirmed transactions would form. This queue is what we call the mempool. It takes up space in RAM, and since a computer only has so much RAM, the queue could only grow so long before the computer would catch on fire and cats and dogs would become friends (ok the part about the fire was a stretch but some bitcoin nodes crashed and had to be rebooted that summer).
The obvious solution was to increase the block size limit to allow mempool to clear each block, like it had for the previous 6 years. But who wants a simple solution when a complex one will do? Certainly not the Core developers.
And besides, the Core devs of the time subscribed to the Maxwellian theory of block space economics. The fundamental premise of the theory was that the natural state of a block is full. The Maxwellians believe that if the block size limit were maintained above demand, then blocks would soon grow to infinite sizes clogging the Internet and consuming every bit of flash memory on Earth (this might be a slight exaggeration). It takes a group of very smart people to act this dumb. But if you accepted the Maxwellian premise, then the change in direction that bitcoin development took might have appeared rational.
The Core devs carefully considered what to do about the mempool problem [4]. They rejected simple solutions like ejecting transactions at random when low on RAM, as proposed by Mike Hearn [5]. The mempool couldn’t be just a simple queue where transactions waited their turn. The Core devs demanded an optimal solution to the problem they had invented.
They wanted new transactions to be able to cut in line by paying a higher fee, such that the queue was always sorted by fee rate. If the queue got too long, then the lowest-fee-rate transactions would be dropped. Miners would build blocks by selecting the set of transactions that paid the highest total fees while consuming no more than 1 MB of block space. By the power of Central Planning, the Maxwellian Fee Market was born.
(Aside: You can begin to see how something that barely even existed was transformed by the Core devs into a complex mempool management machine that Satoshi had never intended. The irony is that all of this complexity goes away if the block size limit is maintained above the size of blocks demanded by users. It’s replaced by the natural fact that a miner will increase his profit by including any transaction that pays more than some minimum fee rate (that depends on block propagation and other factors) [6]. Personally, I think we should leave miners alone to build blocks how they please.)
Where were we. Oh yes….complexity bred further complexity. The Core devs’ solution to overflowing mempools (the Maxwellian Fee Market) caused additional problems that needed their own clever solutions. For example, a bitcoin user might submit a transaction with a fee rate of 2 sat/byte, that would normally be confirmed quickly, but because of a spike in demand his transaction might be pushed to the back of the queue over and over, never confirming. This was a real problem that many bitcoin users were complaining about [7].
There were two solutions proposed. The first was called “replace-by-fee” (RBF) and it allowed the sender to replace the stuck transaction with a higher paying version (which caused further problems in BTC because this higher paying version could be made payable to someone other than the original payee (are you seeing a pattern yet?)). Thankfully, RBF was purged from all implementations of BCH.
The second solution was called “child-pays-for-parent” (CPFP). Child-pays-for-parent allowed the effective fee rate of the stuck transaction to be bumped so that it (the parent transaction) would be treated together with its child as a “family.” The family would be placed within the queue based on the family’s average fee rate. The recipient of a stuck transaction could bump it by spending the unconfirmed transaction again back to himself with a high fee. Or the sender of the stuck transaction could bump it by spending the change output (assuming there was one) back to himself with a high fee.
Clever, right? Now the “mempool manager” not only needed to sort incoming transactions by fee rate and stick them into the correct location in the queue and eject low fee transactions before running out of RAM, but it also needed to check to see if any of these new transactions were respending a transaction that was already waiting in the queue. That is, it had to check if a child of one of the parents waiting in line had arrived, and if so, it had to relocate both the child and parent together as a family within the queue.
But Bitcoin transactions breed much faster than humans, and next thing the Core devs knew the kids were having kids that were having kids, there’s some weird cross-breeding action going on in the corner, and that funny-looking guy waiting in the movie theatre line 50 feet back is his own grandfather [8]. Of course, what I’m describing here are long chains of unconfirmed transactions and the relations between them can be as confusing as that between your third-cousin twice removed and the mother of your grandfather’s daughter. The point I’m trying to communicate is that as these chains become longer, it becomes more difficult for the “mempool manager” running its CPFP algorithm to figure out where each family belongs in the queue. And so the Core devs just added a stop-gap rule forbidding parents from watching the show with more than 25 of their descendants. Not only were additional relatives forbidden from the theatre regardless of the amount they were willing to pay, they weren’t even allowed to wait at the end of the line outside in the rain.
This brings us to where we are today. The 25-descendent limit and the CPFP code that ABC inherited from Core is now causing problems for Satoshi Dice, Badger Wallet, Memo and others. To summarize, the current problem was caused by a solution (CPFP) to a different problem (stuck transactions), which in turn was caused by another solution (the Maxwellian Fee Market) to a third problem (overflowing mempools) that would never have been a problem in the first place had the block size limit been maintained above demand.
Notes
[1] To all the pedants out there, yeah yeah yeah strictly speaking there was still a mempool, but it as its own cog in the system hadn’t yet entered the gestalt of bitcoin. The mempool concept was not well defined from the collection of transactions a node was trying to confirm.
[2] See [1]
[3] Early discussion of the mempool overflowing problem:
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-October/011368.html
[4] There were several proposals for addressing the mempool problem, for example:
https://github.com/bitcoin/bitcoin/pull/6557
https://github.com/bitcoin/bitcoin/pull/6673
https://github.com/bitcoin/bitcoin/pull/6722
[5] Mike Hearn on why ejecting transactions at random is better than sorting by fee rate:
https://medium.com/@octskyward/mempool-size-limiting-a3f604b72a4a
[6] I wrote a paper and gave a talk on the topic of how miners actually lose money confirming transactions with too small of fees:
https://www.bitcoinunlimited.info/resources/feemarket.pdf
https://www.youtube.com/watch?v=ad0Pjj_ms2k
[7] One of the many discussions on stuck transactions:
https://www.reddit.com/r/Bitcoin/comments/4o9tb1/mempool_megathread/
[8] “I’m my own Grandpa”:
Love it can you sponsored me??