read.cash is a platform where you could earn money (total earned by users so far: $ 828,514.27).
You could get tips for writing articles and comments, which are paid in Bitcoin Cash (BCH) cryptocurrency,
which can be spent on the Internet or converted to your local money.
I'm sure many of you have noticed on November 13th a red banner that said that we were having a really-really bad day with the Bitcoin Cash REST endpoint at rest.bitcoin.com.
Don't get us wrong - we're greatly thankful to the bitcoin.com and Roger Ver for providing this node for free - it's a great service to Bitcoin Cash community! We understand that you can't demand 99.999% reliability from something you use for free.
But the prolonged downtime happened and the previous 2 days we were erratically exploring options on having a backup in case this happens again, because the payments were failing.
If you had an account - that's great - that means that in the worst case your payment didn't go through and stayed in your wallet (bad news for the author, though).
The situation was much worse for the "One time QR code" payments. Some people lost some money forever. As far as we know it's about $2 in total, but still it's their money.
In one case that was reported to us - we sent the author our own BCH instead of the forever lost $1.50 that the upvoter sent.
Here's the problem.
When you upvote something on read.cash - this needs to happen:
Often it's simpler:
But still it's not something that we can do with a single QR code. We could do it with BitPay invoices, I think, but not all devices and software support those.
In order to achieve this, we create an ephemeral (one-time) Bitcoin Cash wallet right in your browser.
Your browser controls it.
We don't know its private key or mnemonic, or anything (that's why there's now at least $1.50 in BCH that are "lost forever" on the blockchain).
So, usually this happens:
You send a transaction to your browser. Your browser crafts a really cool "fan out" transaction to pay all the involved (mostly the author, though).
Now, imagine the situation like yesterday. The Upvoter has a connection to a working BCH node. But their browser doesn't have, because it tries to access the node that is failing.
The problem is that this first arrow is really one-way.
Your browser can't use the same node that you used to send the transaction to it.
Your browser can't find out where did the money came from to send it back.
Even if it could - it would still not be able to send it back.
The first and simplest thing that we did was to ensure that the node is working before showing the QR code.
That didn't help much, since the node was "flashing" - it would seem like it was working, then it was down again the next second, when it needed to send "Transaction 2".
Our guess is that there is a load balancer and some of the nodes behind it were down, so the connectivity appeared randomly affected.
Since the private key was generated in the browser for the one-time transaction - after the failed attempt - the browser was closed and the private key was gone forever with the browser's garbage collector (a real thing).
First of all, we know that running a node is not easy. It's not like dusting off a Raspberry Pi and running a few commands.
It requires constant attention, especially during upgrades (and one is coming today!) But, it seemed like the only way. We needed a working copy of rest.bitcoin.com... and fast...
Regular full nodes would take weeks (days?) to synchronize all the blockchain. We didn't have weeks or days, so we decided to install Flowee The Hub - the fastest Bitcoin Cash node alive.
Let's just say - the documentation greatly oversimplifies what you need to do. It took us about 24 hours of stumbling across the docs and source code to get everything working ("The Hub" + "Indexer" + PostgreSQL + "Bitcore-Proxy" + "rest.bitcoin.com" copy).
Here comes the bad news. It didn't work in the end.
It turns out that the Bitcore Proxy doesn't implement all the functions that rest.bitcoin.com needs and that we need for read.cash.
Next on the list was installing Bitcoin ABC (which runs like close to 100% of Bitcoin Cash network) and then Bitcore, Insight API and then rest.bitcoin.com code. There should be a MongoDB node somewhere in between (which is already scary).
Turns out it's not that easy. You can't even use regular Bitcoin ABC, you need to use the modified one from BitPay and there's only version 0.18, when Bitcoin ABC is at 0.20 now, which is also scary to try. We would have been probably be out-of-sync after the upgrade on November 15th. But besides, the documentation for all of this is really non-existent. Do we need Bitcore or just Insight? Which of the commands do we need to run? There's a lot of info and we haven't found a clear description for how to do it. And we needed something quickly.
Seriously, try to read this through. It's one year old description and it's still "Open".
After that, we took a look at our Electron Cash wallet. It was running pretty well. How?
Next was a prototype in Python. We've had to use the new Async Python stuff and stumbled mostly into the same problems. When we turned Wi-Fi off and on again - occasionally async Python would just sit there and do nothing at all, blocking everything. Without a hint of an error or timeout.
Ok, again, not Pythons fault, it's we, the shitty developers, who couldn't do it.
Finally, the last prototype was in Go. Using a ton of hacks (it is very long 48 hours and we've had very little sleep during those), but it kind of... sort of... works. Pretty resiliently.
We've even added the famous "Chaos Monkey" to the code that randomly disconnects nodes to see how would it behave. So far, quite good..
But we're the realists. Our code is only good enough as a "backup" node, when this happens to rest.bitcoin.com:
So, we'll still keep using rest.bitcoin.com and in the meantime keep exploring the options to run our own node.
The proxy is up and ready to take care of the users if rest.bitcoin.com starts having problems again.
It doesn't implement everything that rest.bitcoin.com does, but just the bare minimum required to make sure that read.cash works.
To eat our own dog food - all the development is now happening using this node.
Hopefully you'll never see this:
...but we'll sleep better knowing that this exists.
Every time something like this happens I never stop to wonder - how amazing it is that Bitcoin Cash "just works". I mean, sure, you have local failures like ours, but "the network" overall "just works".
Huge thanks goes to everyone making this possible!
Hopefully tomorrow we'll be developing something more interesting for read.cash, rather that dealing with the infrastructure again.