Toms Blog

Where I talk about Bitcoin and Technology

Bitcoin Classic and the User-Activated-Hardfork (UAHF)

中文 (Chinese translation)

Over the last month we have seen an accelerated amount of effort being put into finally fixing the problems that have been bothering real users for months now. As the dust is not nearly ready to settle, we can at least make out the terrain and the big players.

What many have been expecting is playing out before our eyes. We are going to see a divorce and two chains will continue their own ways. It does not help anyone to go over the history of how we got here, it just matters that there is too much fighting and sleepless nights on all sides. The end result has been becoming more and more clear to everyone close enough, Bitcoin as a whole will lose everything if we can't move forward.

When the developers and self-appointed experts can't decide on which product to put in the market, we should not stall all progress but instead bring both of them on the market and let the people vote with their money. Let the market decide. In the long term the open market always is the wisest of us all.

For me the math is simple, if we get 5x the amount of paying customers in a year, then the SegWit chain can't support them, which will be a horrible experience due to long waiting times again. So the greatest growth, and the highest prices, are going to be on the chain that has its ducks in a row and allows for scaling in the short term.

Based on this, the most interesting solution we observed is the UAHF. Which is short for User-Activated Hardfork. I want to share my findings on how this one works here;

The chain-fork activates on a specified date. Set to 2017-08-01 12:20:00 GMT

Miners that switched to mining this will follow the current chain up until that point, but diverge after this time.

Major features are;

  • First block (one block) has to be larger than 1 MB.
    This definitively marks the chain-split and old clients won't follow it so we cause minimal disruption. Similarly, we avoid many problems with regards to attacks by other hashpower. Once this block is mined, we are following the updated rules, nothing can be done to change that.

  • Block-size limit is enlarged.
    We get a large breath of fresh air. The limit of 1 MB is removed and miners will mine up to 2 MB blocks. We allow for future growth without additional protocol upgrades until 8 MB.

  • Optional Bi-directional Replay protection.
    When the chain-split happens normal transactions created on one chain will be valid on "the other chain". This is an issue for people that want to trade between those chains or simply want to avoid a transaction on one chain to be valid on another.
    Software will need to be updated to support sending and receiving those special transactions. Normal transactions work fine in all existing chains.

  • Transactions can opt-in to use a more secure way to sign transactions. Without making transactions larger, more information is being used in the creation of the signature which will allow a number of new features immediately. These include malleability fixes, hardware wallet security and replay-protection.

Bitcoin Classic will support the UAHF chain with a special release soon.

Read full...
Posted


Quadratic Hashing

Quadratic scaling of hashing time

Theory

In Bitcoin a transaction moves money from one address to another and in order to only allow the actual owner of money the ability to move this, we use cryptographic signatures.

An owner of a Bitcoin address can only spend money by adding a cryptographic signature to a transaction that spends it. The act of adding a cryptographic signature includes the hashing of the entire transaction. As such the size of the transaction matters to the speed of this signing, as well as the validation of said signature.

There is a design flaw in Transactions v1, as Satoshi created it, which can lead to the validation of signatures taking longer than expected.

The problem we see is the basic choice Satoshi made about what it means to sign a transaction. The problem is that each signature signs a slightly different version of the entire transaction. And this means that should you have 10 signatures, this means you need to hash the entire transaction 10 times to sign it. People validating those transactions similarly need to hash the entire transaction 10 times, with minor differences between each run.

A natural consequence of this is that adding another input and associated signatures will cause another read and hashing of the transaction. But at the same time it will make the transaction larger and as such make all the existing signature checks take longer.
This makes the amount of work validation takes grow quadratic.

Practice

The actual checking of signatures is incredibly fast. The largest transaction ever mined on mainnet was in block 364292 as reported on reddit.

This will most likely stay the largest transaction based on the protocol limit of 1MB for transactions. Even if we change the block size limits, the transaction size limits (at least for old style transactions) stays.

As such we can conclude this flaw in the design is not really an issue that can be exploited to cripple or somehow cause problems with the network. This is illustrated by the fact that this huge transaction on today's hardware takes 5.5 seconds to validate. On future hardware this will be even faster, and software will likely speed up as well.

Worst case scenario

Some years ago Bitcoin researcher Sergio Demian Lerner posted about the worst case scenario. This setup is 100% meant to exploit the flaw. In a normal usage of Bitcoin this construction has no meaning. The post describes that with the maximum of a 1MB transaction we can end up hashing that one megabyte 19,100 times.

Larger transactions are not allowed, even with larger blocks. This is about the maximum validation time we can reach under attack circumstances.

A simple solution to this worst case scenario is to cache the output of the hashing function. With that the amount of times we actually hash the transaction goes down to less than 100 times.

How does Flexible Transactions solve this?

To be clear, the issue is something we may call solved for all regular purposes. Specifically, there is no danger of people abusing the system to cause a slow down. And this is what most people care about.

It is however still important to address the underlying cause and fix this. We currently have a 1MB transaction limit, and we may want to lift that limit in the future without having to worry about the scaling issue again.

The solution in Flexible Transactions is rather simple and elegant, we replaced repeated hashing of the entire transaction with using the transaction-ID as the basis of signing.
This means that the size of the transaction no longer is relevant and we only need to calculate the transaction-ID (txid) once, regardless of the amount of inputs.

Using the method of hashing against a fixed TXID permanently solves both issues of quadratic hashing attacks and transaction malleability.

Read full...
Posted


Who Wants SegWit?

In Bitcoin the one thing that is still being discussed quite a lot it is Segregated Witness, the hot technology that the people from Bitcoin Core announced to the world in December 2015.

As the months go by and the realisation starts to set in that there is no way that the 25% miner support it currently has will get turned into the required 95%, we have to take a step back and think. Who wants SegWit?

Talking to companies and people there are a couple of reasons that people want to see SegWit activated, the most important ones are these;

  • SegWit is seen as the "first" step, its activation is required to proceed with Bitcoin innovation.

The question we should ask is; Do we need SegWit for innovative technologies like Schnorr, Lightning and Parallel validation and many others?
And the answer is, no, SegWit is not required for any of those.

  • SegWit fixes various problems in the protocol and those fixes help to scale Bitcoin.

SegWit has a lot of good ideas, but its implementation approach has made it tainted and over-complex. This together with the fact that Flexible Transactions is finished and does all that SegWit does, but much much simpler is the main reason that SegWit will never activate.

This is is not bad for Bitcoin as it will still get its malleability fix, this is only sad for the team that has worked on SegWit project for so long and didn't get much return on investment.

In Bitcoin, like any mature project, we have to prioritise projects one by one and roll them out safely. This means these important issues will get fixed, the question is purely one of priorities.
To be clear, nobody is saying we should not get the malleability issue fixed, it is just not the priority for this year.

And as I have been saying more often recently, the main way to get malleability fixed is to help get the block size issue resolved so we can get to the next item on the priority list.

Read full...
Posted


Layout of my blog

Yesterday I finally sat down and gave the blog a small make-over. It is loosely based on a theme made by mattgraham, and all bugs that occurred due to my changes are certainly introduced by me.

This layout has several advantages over the previous one, the main advantage is that the content now is readable on mobile devices as the layout auto-adjusts for different form factors.

Some more Javascript based features were added, which will help usability. For instance the chapters appear to the left of the main column for longer stories. I try to keep the balance that Javascript is never required to view a page as there are still many security conscious people in Bitcoin that browse without it.

Hope you enjoy!

Read full...
Posted


Statement-03-14

Bitcoin Classic Statement.

Bitcoin Classic is not affected by the remote-crash bug publicly displayed in Bitcoin Unlimited. This clear message is made in response to various people making statements about Bitcoin Classic. Bitcoin Classic is NOT affected by this issue, and has very strict quality procedures. . While I won't say this will never happen, we do as much as we can to maintain our high standards.

Read full...
Posted


BitClub-attack

The battle for the direction that Bitcoin will take is heating up when SegWit supporting pool BitClub chose to go rogue and instead of protecting Bitcoin with hash power, it has now made moves to attack it.

more on 8btc, translation here.

SegWit is too many changes in one go, one of them is that it tries to fix transaction-malleability. A technique that changes a transaction as they were initially created to still be valid, but the transaction-ID ends up being changed.
The direct effect of this is that some wallets or businesses can get confused for some days. Also financial transactions can get cancelled and need to be re-send.

The effect is not dramatic, the architectural problem has been in Bitcoin from the beginning and has been known for many years.

But today we learned that the BitClub mining pool is actively malleating transactions they mine. changing their transaction-id so the original sender may have problems. This is no accident, you need to write special software to alter those transactions so they don't break. And while the network as a whole will really not be bothered to much by this minor attack, we have to understand what this means.

On one side we have Bitcoin Classic and Bitcoin Unlimited arguing for reasonably sized blocks , and on the other side we have some people argue strongly for SegWit. Up until now the argument was always that the 1.7MB that SegWit offers should be good enough, but the market isn't buying it. The support for bigger blocks is now much larger than the support for SegWit. The SegWit support has been around 23% for months, while the BU blocks get about 32% support, which is just in the middle of a rather larger increase.

Now we have a change in strategy, BitClub is actually trying to cause a problem by actively changing transactions while openly supporting SegWit, which no doubt they will argue is the solution to the problem they just created.

The truth of the matter is that SegWit doesn't solve malleability, and what it does solve comes at a great price.
What SegWit does instead is it introduces a new transaction type, with many problems at the same time, and only should users choose to upgrade their wallets and then actively use those new type of transactions will their transactions no longer be malleable.

Agreed, it is rather annoying that anyone can change your transactions and I agree we should solve that. For this reason I wrote Flexible Transactions (more here). It is new transaction type which doesn't suffer from the malleability issue.

The beauty of Flexible Transactions is that it doesn't pretend to be a block size increase, but it does solve 100% of all the issues that SegWit solves. And more! Additionally, Flexible Transaction is much safer to use and more future-ready than SegWit. Flexible Transactions has been running on its own testnet for months now and various alt-coins are integrating it.

In short, Flexible Transactions is a next-generation solution for encoding transactions, using a tagged system which is future-proof and allows unlimited future growth of Bitcoin. It generates smaller transactions, it is safe from malleability attacks and quadratic scaling attacks.

I would like to suggest to mining pools like BitClub that they realise they are actually hurting Bitcoin users and their actions are indeed annoying actual users that create Bitcoin transactions. Going forward, we do have a good solution for malleability and it is in the form of Flexible Transactions. Please don't be fooled by aggressive marketing techniques which will want to make you believe SegWit is the only solution. It is not the only one, it is certainly not the best solution.

I am saddened that there are those that hold the misguided belief that attacking your users is a good way to keep control over Bitcoin.

In Bitcoin Classic and Unlimited we support the users, we fight for normal level transaction fees, and we will never attack our user in order to try to prove some solution is needed.

Read full...
Posted


What is new in Bitcoin Classic (2017/02/20)

This post will answer what happened in Bitcoin Classic since the last report

New stuff

In an continuing push to make the Bitcoin Classic client the best one to use on Linux we have seen some support added some time ago for the XDG Base Directory Spec for config files. Continuing in that direction Contributor Stephen McCarthy has been hard at work with the command-line parser code.

The code we inherited when we started Classic is shared between various projects, in all of those a set of problems are present which make sysadmins and operators life a lot more difficult.

  • Passing in a not-existing config option or command line argument was silently ignored. So typos would go undetected. The most common problem was that when you added a - to a config file option you would have your option silently ignored.

  • Flags usage was very inconsistent and not very predictable. The only solution that consistently worked was '-flag=1', but sometimes -flag works too and the most fun problem was that '-flag=true' would more often than not end up turning off the flag.

  • Inputs were not validated at all. If you passed something that wasn't numbers to a option that expected numbers, the effect typically was that the option would be set to zero. At best you might get a warning in the logs.

In the last weeks, Bitcoin Classic contributor Stephen McCarthy fixed all those issues and many small ones as well. If any command-line option is passed in that doesn't exist, or is expecting a number and you passed something else, you will get an error and your node won't start. Additionally, yes/no flags become more intuitive and will work properly with any of 0/1, false/true and also with yes/no.

Pull requests; 225 226 227 228

Tom Zander committed a small change to prepare for a world where block-size is no longer a centralized rule, allowing for a healthier network; Punish nodes relatively for oversize blocks commit.

Tom Zander added a new feature for Flexible Transactions called -ft-strict. The commit message reads:

In Flexible Transactions we have undefined tokens which are ignored by current software if present. This allows nodes to stay compatible when new tokens are invented. For miners this policy rule should likely be turned on so transactions that use tokens that are not yet defined will be rejected.

Tom Zander imported the "Expedited forwarding" technology to Classic's master branch commit.

Code cleanups / improvements;

Tom Zander created a new class which now becomes responsible for all orphan transaction handling. This is one step towards solving the problem of having no code structure and everything in one big 5000 line sourcefile. This a non-functional change, aimed to making the code better to maintain and easier to understand. commit.

Tom Zander noticed a potential bug, or at least a bad design in the usage of the assert command. This is better explained in the resulting 'Junior Job' issue. The fix is in commit.

Tom Zander did a refactor of the blocks database handling code. Similar to the orphan code above this is a non-functional change that moves code to be easier to understand and maintain. commit, commit, commit.

Amaury Sechet created a new PR which updates the 3rd party library secp256k1 to a newer version. We also realised that it is confusing for many that imported libraries are placed in the src directory as that makes the fact that they are snapshots and developed elsewhere not very obvious. But no solution has appeared at this time.

GUI (Qt)

A PR by Stephen McCarthy was merged and had the purpose to Fix assertion error on -disablewallet, fixing a regression introduced by another dev just a couple of days before.

Another PR by Stephen McCarthy was merged and introduced a change that made 'Abort' the default button on the dialog that is very destructive, and a user input mistake there could cause many hours of work to recover from.

Tom Zander created a commit with the subject Make splashscreen work on hi-DPI displays commit.

Misc

New contributor Benjamin Arntzen created a PR to fix some text which ended up being applied. also to the website as well.

Tom Zander imported the historical release notes on the website; https://bitcoinclassic.com/news/


If you are interested in the Bitcoin Classic project, please look at our community page for how to get involved.

Read full...
Posted


UTXO-Growth

Over the last year we have seen quite a lot of people that get worried about the UTXO growth. Lets find out why.

For those catching up; the UTXO is the Unspent Transaction Output database. The essential database that is needed to find out if a new incoming transaction is actually spending money that exists.

The growth is explained well in this blog from Gavin Andresen, the blog is already older, so the growth has continued.

There are some reasons why people claim this is an issue. Lets address each of them in turn.

The UTXO needs to be in memory completely. It will grow faster than internal memory can.

Answer: The implementation in all Bitcoin full nodes is that the entire database is stored on disk. With a smart memory-cache to speed up lookups. The memory-cache is configurable and the default is 300MB. (it's the dbcache config option) The entire database is significantly larger than 300MB, as the above blog states.

The claim that it has to be in memory is weird because none of the full node implementations actually do that. Sure, miners can adjust their settings to allow a bigger cache, and be different from everyone else. But this doesn't mean the UTXO should stop growing.

Bigger blocks make the UTXO grow faster!

Answer: The UTXO is a database of where money is stored. With more people using and holding Bitcoin, this will grow. The UTXO will not just grow based on more transactions, the UTXO will grow because we get more customers. Which can only be done when we have bigger blocks.

So the argument isn't directly false, it is misleading. It is missing the fact that UTXO growth is based on more people using Bitcoin. Maybe the people making this accusation are aware that saying they do not want more customers will make them look stupid.

UTXO on disk will make miners lose money because of slow block validation.

Answer: This may have been true in the past, but is no longer true since we introduced xthin blocks. This technology allows any node to communicate a new block with only the header and some meta-data. The receiving node will then use the transactions that are in his memory pool to reconstruct the block.

The direct result of this is that when a new block is mined miners will not validate the vast majority of transactions again because the transactions in their mempool were already validated. This means that there is no bottleneck any more for miners using xthin (available in BU and Classic).

Bottom line is; it's a database. MySql/PostgreSQL/MongoDB etc. have had decades to perfect database technology. Being scared of a database growing on today's hardware is unfounded.

Read full...
Posted


What is new in Bitcoin Classic (2017/01/30)

Bitcoin Classic is a typical open source project where individuals come together in order to join forces and create the best solution they can. It can be a bit chaotic to keep track of what is going on and for this reason I want to write a little update on what has been going on in Bitcoin Classic.

There have been two similar posts in another forum before: 10-jan & 18-Jan.

A very brief recap for those that have not followed along. Bitcoin Classic got a restart that I introduced with Classic is Back here 2 months ago. We released version 1.2 some weeks ago that introduced various new technologies. The CMF bindings project got its release shorty after.

Last week I spend most of the time in a plane or in the airport going to from the EU to Mexico where the Satoshi Roundtable III was. I very much enjoyed meeting many Bitcoin friendly faces and having discussions about how we can further Bitcoin and Bitcoin Classic. At the same time this is my apology for having much less coding based progress.

  • We have seen bugfixes in the 1.2 stable branch (to become the 1.2.1 release).

    • [QT] Bugfix: ensure tray icon menu is not empty. By Stephen McCarthy.
    • Fix failing maxuploadtarget.py QA test from the -extended set. By Stephen McCarthy.
    • Make configure fail on old Boost. (1.55 has been the minimum for some time)
  • On the 'master' line of coding (to become the 1.3 version) we have seen various improvements as well.

    • We merged a PR written by Pat O'Brien that sanitises the default permissions of files written by the node on Unix. Adding code to make sure that the wallet.dat is only ever visible to the main user.
    • Merge a PR from Søren B.C. to remove some dead code (semicolons) from many files.
    • Merge various PRs from Amaury Sechet for backports and to add sublime text project file to gitignore
    • Merge a PR from Stephen McCarthy that makes sure that any unrecognised command line arguments or config-file lines will cause a warning on startup. This should make it much easier to configure the application.
    • Cleanup the NetworkStyle usage. This makes code less fragile and more coherent.
    • Remove duplicate (copy/pasted) code originally imported from the xthin project.

If you are interested in the Bitcoin Classic project, please look at our community page for how to get involved.

Read full...
Posted


Governance Starts Somewhere

Last week I've been at the Satoshi Roundtable III, which was very effective in bringing about 100 Bitcoin interesting people together in a way that makes it very easy to productive to just sit down on one of the many sofas (or even on the beach) and talk with a number of people that share your interests.

To link to someone that is a much better writer than I am, Rick Falkvinge wrote "Overall, my assessment is that bitcoin is lacking project management".

I fully agree on that, in Bitcoin we now have different groups creating their own solutions and then trying to push them to the rest of the world, typically without actually listening to criticism, and then being surprised when the world rejects it.

I re-read a novel last month (Moving Mars) that was a science fiction where some 20 large families live on Mars as separate settlements. They started to create a new, global, government. To my surprise the process to get there was as step 1 to decide on a way to make decisions. And only after that create a constitution.

If we want to have any hope of so many smart people making progress again, we need to stop trying to skip step 1. We need to allow people to talk to each other and start the conversation on how to address the concern Rick formulated on his blog.

In Bitcoin we used to have a solution for that, the bitcoin-dev mailinglist. This has been a failed experiment with many examples of lost opportunities and even a group like Bitcoin Unlimited just refusing to use it because it is not neutral. The secondary effect is that this same group has refused to make BIPs (Bitcoin Improvement Proposal) because of the place where they are proposed and discussed is that same list they won't join.

This week I reached out to the Bitcoin Unlimited team on their slack in order to get their input on starting a new mailinglist under new management that can accomplish the goal of cross-team communication. We had some good ideas, one of them was to call it the 'post-1MB planning' list.

Specifically, what am I thinking about?
If anyone has any changes they want to make to the protocol as that affects all implementations this should have a neutral (not affiliated to any specific implementation) place where this can be discussed.

  • In Bitcoin we currently are having problems with the blocksize-limit of 1MB. This is a difficult topic because the parameter is part of the protocol. A "Consensus rule". What is interesting is that there currently is no bitcoin-wide document how the consensus rules are actually supposed to be changed to support Bitcoin Classic or Unlimited. This needs to change.

  • There is no place where people that develop a wallet or other bitcoin infrastructure can talk about consensus changes, and get familiar with what others propose.

  • Classic recently published its long term (5 year) plan. It would be great to be able to talk about this on a place where multiple implementations can work together so we can all makes sure we work in the same direction and avoid nasty surprises when other groups went a different (maybe better) direction!

With the modern web I've seen people suggest websites like forums to discuss this. The problem with a forum or site is that it excludes people that are not able to reach it, and it opens the risk that the operator of the site may go offline with all the history and the content.

Traditionally the best way to do this has been email lists. Thousands of projects use it successfully. I would like to suggest the best way forward is doing that again. But I'm open to better suggestions if they don't introduce any problems.

This is a question to all the software developers and organisations that want to be involved in the way that Bitcoin evolves over time, what are your requirements and what can you offer for such a platform of coordination to move Bitcoin forward?

Read full...
Posted