Aymeric on Software

Because we needed another of these blogs...

Migrated to Kimsufi

Originally this website was powered by a shared hosting plan on Dreamhost, but I migrated to a fully hosted solution over a year ago.

After considering many options, I chose a Dedibox SC gen 2, a dedicated server solution from Online.net a French server manged hosting provider. The machine features a fairly low spec CPU, a 500GB hybrid SSD-HDD disk, 2GB of RAM and unlimited bandwidth with guaranteed throughput.

All that for €12 a month. It is much cheaper that all other hardware of VPS solutions I could find, and in fact about the same price as the Dreamhost hosting plan it replaced.

France seems to have two companies (Online with Dedibox and OVH with Kimsufi) dedicated to offer very low cost hosting solutions. The prices can be low thanks to custom dedicated hardware that is both small and power efficient. As you can see below, a Dedibox SC is barely larger than a 3.5 inch HDD. My gut feeling is that the emergence of these offers in France is linked to the three strike piracy law and the disappearance of Megaupload. A lot of these servers were used as seed boxes, at least in the beginning…

The hosting on Dedibox turned out to be very good and sufficient for my needs. A properly configured nginx server serving mostly static pages has no problem serving the 20,000 hits per month my sites require. I initially hesitated using Amazon S3 for some of the most popular pages, but dropped the plan entirely. I also have a few dynamic pages, but these are mostly for personal use. The PHP code of Exposition, my unfinished photo gallery software, works a lot faster than on Dreamhost.

The support on Dedibox is not great though. I had some billing issue at the beginning and it took forever for it to be resolved. I’m not sure I would trust them for something critical, like a business.

In the past few days, I made the switch to their competitor Kimsufi. I was running out of space on the 500GB disk of the Dedibox, because of my large photo collection. I opted for a KS-2, which is the exact same price price as the Dedibox SC Gen 2. It is normally fitted with a 1TB disk, but I got 2TB disk instead (surprise!). The KS-2 is also fitted with 4GB of RAM instead of the 2GB of the Dedibox.

The rest is worse though. On Dedibox the bandwidth is guaranteed, whereas on Kimsufi it’s best effort. In practice, I have not noticed any differences. The disk on the Dedibox is a hybrid SDD; Kimsufi uses a traditional slower spinning disk. Dedibox offers free monitoring services, like email notifications when the machine is turned on or off. Kimsufi offers nothing. OVH has a full featured iOS app that used to support Kimsufi users, but is now restricted to the pro tier of OVH hosting. Shame on you OVH! So I ended up configuring a free pingdom.com account for basic monitoring.

In the end, it’s a pretty good deal if you need the storage or the RAM. 2TB of “Cloud Storage” (it should be 1TB) for €12 month and similar fees with no limits in bandwidth is pretty good. It’s more than what you would get from Amazon S3 or Dropbox for the same price. But if you’re fine with 500GB or only 2GB of RAM, I would recommend the Dedibox offer instead.

Let’s hope the new hosting is as reliable as the previous one…

Compute the Balance of a Bitcoin Wallet With node.js and blockchain.info

The Problem

As I have explained in great details in a previous post, the balance of a Bitcoin wallet differs from the balance that can be computed by looking at the blockchain. The reason is that practically every time you make a transaction, the Bitcoin software sends a fraction of the funds back to you, to a new address. Such addresses are invisible in the user interface, and for all practical purposes can be ignored by end users. These addresses are pre-allocated and constitute the key pool.

In this article I explain how you can compute the balance of a BitcoinQt wallet.dat file using node.js and the JSON API of blockchain.info. I will show how you can extract the public addresses contained in the wallet, whether it is encrypted or not, and demonstrate how to use the blockchain.info API efficiently.

Extracting the Keys

Bitcoin relies on public/private key cryptography. The public key is what is turned into a public address through hashing (detailed below) and the private key is the one used to sign the transactions. In order to compute the balance of a wallet, we need to extract all the public keys from the key pool.

The wallet.dat is a Berkley DB database. This is an old NoSQL embedded database from long before the term was coined. Unlike relational databases such as SQLite, it is not possible to make complicated queries in SQL. The wallet uses a single key/value dictionary to store all public and private keys, the last transactions, and all other account informations.

The keys from the key pool are stored directly into the database as individual records. The database keys are split into two parts. The first part of the DB key is some kind of identifier, that describes the type of record. Non encrypted Bitcoin keys are identified with key while encrypted keys are identified with ckey. The second part of the DB key is the Bitcoin public key itself, preceded by a single byte that indicates the length in bytes of the public key. The database record associated to the DB key is the private Bitcoin key, which may or may not be encrypted. This splitting of the DB key into two parts may sound odd, you have to remember that the DB keys must be unique to identify a record.

Here’s a drawing, in case you’re starting to be confused between the DB keys and the Bitcoin public and private keys.

You can see details about the key loarding algorithm in walletdb.cpp from the source code of Bitcoin. Look at the CWalletDB::ReadKeyValue() method.

I had no luck using node-bdb for reading the wallet with node.js, because cursors are not supported. Instead, I used a crude algorithm looking for the ckey and key strings within the wallet. This works because BerkeleyDB does not use any compression, but that’s not ideal.

Computing Bitcoin public addresses

Once you have obtained the Bitcoin public keys from the pool, you can turn them into a Bitcoin addresses through a complicated process of hashing, which is well documented.

However, I found a discrepancy with the documentation. The length of the keys I read from my wallet were all 33 bytes instead of the expected 65 bytes. Bitcoin uses ECDSA, a variant of elliptic curve cryptography. Both public and private keys are points on a specific elliptic curve (Secp256k1) whose equation is y^2 = x^3 + 7. The keys are the x and y coordinates stored as 32 bit integers preceded by one extra header byte that OpenSSL uses to store format information. Since x and y satisfy the known equation of the curve, you can store a single coordinate and a sign to compute the other coordinate with y = +/- sqrt(x^3 + 7). This is what is called a compressed key, and it is 33 bytes long.

I initially thought that I would have to decompress the key, and convert it to the 65 bytes format in order to compute the address. But it turns out that Bitcoin hashes the OpenSSL ECDSA public key as it is. I suspect that means you could have multiple public addresses associated to the same public key: one public address for uncompressed 65 bytes key, and one public address for compressed 33 bytes keys…

You can simply hash the public keys with with the ripemd160 and sha256 algorithms as documented. The details of the hashing are strangely complicated, but easy to follow, and you end up with a 25 bytes binary address. This binary address is then converted into an ASCII string using the Base58 Check encoding. As explained in the source code and the docs, this encoding is used instead of the easier to calculate and more base64 encoding, to avoid confusion between similar characters like 0OIl.

Implementation with node.js is relatively straightforward using the crypto and bs58 modules

Computing the balance with Blockchain.info

You can use the JSON API from blockchain.info to compute the balance of each address in the key pool, and sum them up to obtain the final balance. These APIs are public and can be used without authentication, but they are rate limited. This can be a problem if you use naively the single address API. Instead you should use the multi address API or alternatively the unspent outputs API.

Source code

I wrote a quick and dirty sample using node.js and published it on Github.