When I first started working with Bitcoin, a couple of years ago, I tried to find out how to contribute code and to understand the Quality Assurance policies they held.
At this time I naturally was talking to people from the Bitcoin Core group, and the answers I got were quite confusing to me.
After having worked in the software industry for 20 years I had become very reliant on good quality assurance policies. As well as good usage of distributed source management control. Turns out that Core didn't really have any of that in place.
When I was asked to work with Gavin on Bitcoin Classic I first started to work on quality assurance policies and practices. And I'm happy to say that we have reached at least industry standard, if not a little above. But I haste to say that there is still room for improvement.
The first thing that I changed is that we now have a proper git workflow. This means that developers fix bugs on the stable branch and we use git merges to make those bugfixes go up to the master branch. You will find this process in any git book, so I wont' spent too much time on this. The important part is that merges between branches is done often and that way we can't lose work or introduce bugs by doing it manually. See also our contributing document.
The second part that changed is the release process itself. Based on the fact that we now have a stable branch that is always releasable the release process will become easier. The aim is to follow the good old open source concept of "Release early, release often".
One of the biggest changes is that we now follow the Semantic Versioning concept which is almost universal for open source software as well as for many closed source products.
Let me quote the core concept;
Given a version number MAJOR.MINOR.PATCH, increment the:
- MAJOR version when you make incompatible API changes,
- MINOR version when you add functionality in a backwards-compatible manner, and
- PATCH version when you make backwards-compatible bug fixes.
Additionally I wrote this in the release tag of v1.1.0
At first we wanted to follow the Core releases, but they have different
core-values and push out hundreds of lines of new code in a bugfix release
which feels dirty to have to inherit. The result of that is that the CSV
soft fork code feature release has received a bigger version bump. And
when that disconnect between Core and Classic versions it also stopped
making sense to stay close to versioning of Core.
It was time to review the numbering scheme altogether.
When I talked about the "less than 1.0" versioning over beers with
Eric S. Raymond (author of The Cathedral and the Bazaar) he explained it like this:
The idea was that when the app reaches the set of features required for the user with the lowest demands to use it for its intended usage, that's when you call it 1.0.
Bottom line, Bitcoin is certainly far past its 1.0 release. We will follow the Semantic Versioning from now on.
Continues Integration building
What I was really missing was a proper Continuous integration (CI) system. Naturally, there is the freely available Travis. But thats been a frustrating experience and really not useful as a build server. Apart from the fact that Travis is really slow (taking an hour or more), some and 60% of the time when it says there is a failure, it actually isn't a problem in our code, its really not useful to rely on.
I got a dedicated server in a secure environment and installed Teamcity on it. Teamcity is one of the most used CI software systems out there and quite useful for our purposes.
It does a couple of things;
- every single time a change is made in Git (in all branches) it builds this for all platforms and runs unit tests at least for the Linux ones. Notice that it does the building in a fraction of the time that Travis took. Typically a build completes in less than 15 min.
- It builds both using the 'gitian' concept of reproducible (or static) build environment for all platforms as well as using the native Linux way of using the platform dynamic-libraries (dlls). This makes sure we know it works on more than one version of boost and all the other supporting libraries.
- It builds nightly builds of the master branch allowing people to test these that don't want to compile.
- It pushes the build to remote builders (like Ubuntu's launchpad) to have an easy way to distribute nightly builds or maybe even future releases with minimum effort.
- It tests the Debian contrib to actually build properly.
As I mentioned, the server is in a secure location. What that means is that it is not connected to the internet and physically protected to make sure we can trust the output of that server. When we ship sources anyone can check quickly if they are unaltered from the tags in git. But with binaries (evil) changes can be much harder to detect. So I refuse to gpg-sign any releases unless I know they actually came from the exact sources that are in git, visible for all.
There is plenty of issues that could use improvements. It turns out that the original developers don't really like writing unit tests and the code lacks any organization to make unit testing practical or easy. With some small changes such efforts could be made much easier.
We also have exactly one executable that runs all the test. Which is against the unit testing practices of having small units because one failure in one test can cause a lot of tests afterwards to start failing too, mostly because the memory states were not cleaned up properly after a failure.
Ideally we have a lot of small test executables which can be run by a simple runner. With the added benefit that you can run various in parallel (one per cpu-core) which makes the whole complete much faster as well.
We currently have a benchmark framework, but nothing in there. The idea of
a benchmark is that code which is time critical can be tested in there and
the output can be printed and plotted over a long time (months) to see that
such time critical code doesn't get slower by unintended changes from
release to release.
I find it hard to believe that Bitcoin has no time critical code, as such we should be able to find value to locate, test and keep code from getting slower.
Better integration Linux distro's. Bitcoin was made to be shipped as a big
executable with all the libraries inside. The thinking was that this saves
the application from misbehaving when another library changes behavior.
While the idea seems logical, it shows a lack of experience because the
open source community has been solving this problem for 30 years and it
does so quite successfully. Going against that only introduces a different
set of problems. For instance when the libc DNS exploit was made public
Bitcoin dodged a bullet because libc is one of the very few libraries we
don't actually compile in. But imagine any of the libraries we include has
such a similar vulnerability and the only way to solve that is to re-build
Bitcoin and get everyone to manually update. The Linux alternative of
just running an
apt-get upgrade or similar is vastly more secure and
As mentioned in the CI section, I already added support for Classic in launchpad. This may be something to expand upon and support more Linux distros in a similar fashion.
Bitcoin Classic has a strong Quality Assurance policy, adopts long time industry standard practices and sheds unsafe self-invented ones from upstream.
We also take all the best practices from Open Source and Free Software to make releases often for those that want to test the latest and contribute by reporting issues, or simply by being part of the network.
The future is bright!