This blog had moved to Cloudflare Pages in 2022, and previously it had been hosted on a dedicated server provided by Kimsufi and, before that, Dedibox.

However, half a dozen sites and applications remained on that server. It ultimately took about four years to complete the transition. 😱

Why Not a VPS?

Renting a dedicated server simply stopped making any economic sense. The underpowered machine that previously hosted this site cost €15 a month; a price point that had long ceased to be competitive.

You can host a collection of static HTML/CSS files at zero or near zero cost on either Cloudflare Pages or AWS Cloudfront/S3.

While SQL databases, PHP, docker images have their use, they always felt like an overkill for a simple blog or the kind of simple sites I was hosting. This blog, for instance, is just a collection of static files generated by the Hugo template generator.

Where Are Things Hosted Now?

It depends:

• Simple sites like this one are hosted on either Cloudflare Pages or AWS Cloudfront.

• For public-facing sites, I converted some simple PHP or Ruby scripts to run on Cloudflare Workers/Pages and store their data in a D1 database instead of self-hosting PostgreSQL. My usage is small enough to fit comfortably within the free tier.

• Finally, I have run a few servers at home for over a decade. Some sites that were previously hosted on the VPS are now are served from home and put behind a Cloudflare Tunnel. A few are additionally gated behind Cloudflare Access. Faster internet connections with Gigabit upload now makes this a viable option, at a fraction of the cost of a VPS.

• The home sites all run in Docker containers, so moving or migrating them in the future should be easier. This also helps with the security aspect, as you would need to break out of containment to compromise the host.

Why Did It Take So Long?

I made an historical mistake when I set up my first Linux servers. Initially, I set them up with little to no documentation… making migrating and rebuilding them a challenge.

Having learned from my mistakes, everything I host at home now follows the tenets of infrastructure as code religiously. The base system is installed automatically via Ansible, and every web application or site runs on top as low-privilege Docker containers. Installation, deployment, and updates are largely automated, and all changes are tracked in Git, which serves as both documentation and historical record.

In addition, recent advances in AI have made this setup even more useful: agentic LLMs do a reasonable job of maintaining the repository. They can also explain how things work when you invariably forget a decade later.

In fact, with Claude’s help I was able to move the remaining sites that had been hosted on the Kimsufi VPS. For some of these, Claude also assisted with the conversion from Ruby/PHP-FPM/PostgreSQL to Cloudflare Workers and the D1 Database. All in all, four years of procrastinated drudge work and technical debt disappeared over the course of a couple of long evenings.

I may use a VPS again someday if the free or near-free services I rely on disappear, but I will make sure to have a clear exit plan in place.

However, this blog post is not about COVID-19. In fact, I have been thinking of doing some kind of the yearly retrospective for the last couple of years. So here I am, starting one now, and trying to leave the COVID topic aside.

I do read a lot of development blogs. My interests in programming languages and technologies are varied and somewhat eclectic. The blogs I read range from the serious and in-depth analyses to more light-weight off the cough testimonies. Oftentimes, this is how I stumble on some interesting tool or a piece of technology which I may end up using months or years later. I am writing this blog post with the hope that it may be useful to someone else in the same manner.

TL;DR

So here are three things I have learned in 2020: Terraform, Notion and I somewhat improved my writing skills.

Terraform

I had been aware of Terraform for some time. It is a tool born of the infrastructure as code methodology from DevOps. Unlike older solutions, which were historically rooted in imperative scripts such as Chef or Puppet, Terraform only supports the declarative approach. In other words, you describe the end state of the cloud infrastructure you want, and Terraform figures out which steps are necessary, if any, to bring the system in line with expectations.

Terraform is a cloud agnostic tool. I did not need a multi-cloud tool, so I had reservations in adopting Terraform over a first party solution like CloudFormation for AWS. Both tools play the same role and are declarative, but CloudFormation being a first party product, it has the advantage that it integrates better with the AWS stack. The reasoning is correct, but what I failed to appreciate is how poor and painful it was to use CloudFormation in practice. Terraform provides a much better experience:

• The description language of Terraform is less verbose and better designed than CloudFormation’s. It feels like Terraform was designed to be read and written by humans, whereas CloudFormation looks like a memory dump transcribed into YAML or JSON.
• The documentation of Terraform is excellent despite being a third party provider. Not that the AWS documentation is bad either.
• The killer feature of Terraform is the ability to refactor re-usable code into modules. This is a very straightforward and well-documented process. It compares infinitely better to the kind of hacks one would have to do with CloudFormation to achieve similar results.

One of the drawbacks of Terraform is that the current state of the infrastructure is stored on a file locally, whereas a first party solution like CloudFormation can store the state in the cloud. Personally, I like having the state in a GIT repository and I find it somewhat reassuring.

I currently use Terraform to use all my AWS resources: Route 53 DNS entries, EKR Docker Registries and a handful of Amazon S3 buckets.

Notion

I have been looking for a tool to take and organize notes. One of the many requirements was that the tool needed to be truly cross platform with support for macOS, Windows and iOS and features cloud syncing. In addition, I wanted a tool which can integrate very well with Markdown.

I essentially use Notion to collect links and information when I research a topic and to write design documents. I also adopted the practice of scientists who keep logs about their work. The draft of this blog post was also written in Notion first, before being exported as Markdown and published via Hugo.

I have been looking for a good note taking tool since 2019. I have looked at many and tried a handful. Even though I was aware of Notion for a long time, I was somewhat put off by it. I understood it could be used to publish things on the Internet and it had some kind of database features. Those features are not exactly something I cared for, so they acted as repulsive force, because I was worried about feature creep and unnecessary complexity. These thoughts were echoed by CGP Grey from Cortex; so Notion was put to the back of the pile. How wrong was I!

The truth is that Notion is still a pretty good way to take and organize textual notes. All the normal features from a Markdown editor are well supported, including syntax highlighting for code snippets, even though it uses a WYSIWYG editor. There are some useful additional features like the ability to use Callouts, Emojis or insert LaTeX formulas. The import and export of Markdown is not perfect but it is usable. I have used some of the advanced features like tables, databases and calendars and they are fine. The ability to publish a note as a public webpage on the Internet in one click has turned out to be surprisingly useful, especially since that page automatically updates when the source document in Notion is updated. And to top it all, it looks like my usage of the tool would fit in the free tier forever.

Although I am overall satisfied, Notion has some usability annoyances. Non standard key bindings and mouse control for text selection drive me mad. Occasionally, the WYSIWYG editor will require some non obvious combination of key strokes. But the main downside, by far, is that I am not in control of the data. The way information is stored on the file system or on Notion’s servers is completely undocumented. The Notion API was successfully reversed engineered, but an official Content API should be launching in 2021. Until then, it seems like the official way to backup data is to Export the Workspace as HTML or Markdown, but it is a lossy export. I am currently relying on exports but I hope to start using the upcoming Content API when it becomes available.

Notion’s Alternatives

While researching a note taking tool, I stumbled on Bear. It provides a native editing experience that I like more, but fewer features than Notion overall. The downside is that it is only supported on iOS and macOS (no Windows / no Web). But if that’s not a deal breaker, it is an excellent tool. I also stumbled on Typora which is a very good Markdown Editor, that I use regularly with the Ursine theme. I don’t use Notion’s project management features very much. For very simple lists of things to do, I tend to rely on the much simpler Microsoft To Do.

Improved Written Communication

I have worked from home for most of the year. Even though there are some obvious advantages like avoiding the daily commute grind, the lack of face to face communication has many drawbacks that cannot be fixed by video conferencing.

One of the obvious consequences is that all but trivial communications (which can be done via instant messaging with tools like Slack) must be formalized into a meeting or an exchange of e-mails. Consequently, I have found myself reading and writing more prose than ever before.

The trick with written communication is that you need to be as clear and non-ambiguous as possible while remaining concise. Clarity is very important. If one of your colleagues were to misinterpret one of your ideas, you an easily end up exchanging a few e-mails just to clear things out. This translates into wasted time, and some frustration, for everybody involved. If you had been in the same room, you would have been able to intervene sooner and save that time. Making things clear can involve some amount of repetition or rephrasing, or showing examples. Whenever possible I try to show a diagram or a table instead of prose.

The goal of clarity needs to counter-balanced by the necessity to be short and concise. Long and winding emails are boring to read, so people will just skim them or put them off until the last minute. I know, I am guilty of the same thing. For communication to have an impact, it needs to be read, therefore it needs to be short.

Through repetition, I would like to believe that I have somewhat achieved a basic level of competency at written communication. I am better at it than last year, and definitely better than ten years ago, as plainly demonstrated by the earliest – cringeworthy – posts on this very blog.

As an aside, I would like to recommend “Have You Eaten Grandma?”, a book which is very humorous way to review the basic rules of grammar and punctuation in British English. (Yes this blog is written in American English except for rule about punctuation within quotes or brackets.)

I have used Firefox since the very beginning on Windows. I stuck with it even in the early 2010s, when it clearly lagged behind Chrome, because of a profound distaste for Chrome’s GUI and Google’s business practices. In my opinion, Firefox Quantum released in 2017, has addressed most of the performance gap between Chrome and Firefox. In many cases, Firefox feels even snappier than Chrome. I still use Chrome every day, but mostly as my “debugging” browser for web development, my “translator” and, very rarely, my “Flash player”.

On macOS, I have used Safari since the beginning. However, as of macOS 10.15, I was forced to switch to Firefox when Apple decided to disable the legacy JavaScript API for extensions. The new extension model involves installing an App from the App Store. The “container” App provides an XPC extension which runs inside the Safari browser. This is arguably more efficient than JavaScript extensions, but this choice makes it very difficult for developers to port existing extensions to Safari. The “container” app also provides a much greater attack surface than a mere JavaScript extension.

Content Blocking Extensions

The move to the new extension API in Safari killed support for my favorite extension: uBlock Origin.

For performance and security reasons, Apple decided to no longer let extensions intercept HTTP requests in Safari. Instead, extensions had to migrate to the newer content blocking API introduced in 2015, as part of iOS 9. This API requires extensions to provide a fixed list of blocking rules, identical for all sites. This is nowhere near as effective as uBlock Origin’s dynamic rules. However, the privacy advantage is undeniable, since extension no longer have access to the sensitive information that can be inferred from intercepting all HTTP requests.

This debate about content blocking extensions was recently revived by the upcoming transition to Manifest v3 for Google Chrome extensions. Google is pushing for a similar approach to Apple’s with their new declarativeWebRequest API. They also argue that their new API would be more efficient and safer. However, I am somewhat skeptical, since the old webRequest API will be kept as part of manifest v3 minus the ability to block requests. In other words, Google plans to cripple content blocking extensions like uBLock Origin but other extensions would still be able to spy on users by sniffing requests. The old webRequest blocking API will also be kept for their “Enterprise customers”, which I interpret as Google’s partial acknowledgement that their new API does… indeed… suck.

My Firefox Extensions

This is the list of my Firefox Extensions:

• uBlock Origin. As previously mentioned, I ditched Safari in favor of Firefox because of this extension. It is the best and fastest ad blocker. It can also be configured to block any kind of content on the spot. Content blocking extensions not only make web pages render better and faster, but also more securely since legitimate ad networks have been tricked many times in the past into distributing malware.

• HTTPS Everywhere. This extension ensures that you always use HTTPS for a site that support it, even though you may be following an HTTP link. It is becoming less of an issue, but still worth having.

• Cookie AutoDelete. This extension can be configured to delete cookies and local storage data for all visited sites, except for a short list of exceptions. This is more aggressive than the built-in delete cookies when Firefox is closed option since the extension will activity attempt to delete cookies while Firefox is running. I would recommend turning off notifications, since this extension is a bit too proud of itself for deleting cookies.

• I don’t care about cookies. This extension automatically discards the annoying EU Cookie Banners that seem to spread around the Internet like the plague. I really do not need know about your cookies, nor care, since they will soon be deleted (see above). The only downside of this extension is that it does occasionally break sites. When this occurs, disable the extension for the site, click on the banner, and move on.

• Dark Reader. I use macOS built-in feature to automatically switch to dark mode at night. The problem is that many web pages do not implement dark mode and carry on using very light backgrounds. Dark Reader attempts to automatically switch the pages to have light text on dark background. It usually does a decent job and it can be tweaked for more challenging sites. The only downside is that it seems unable to detect sites that natively support dark mode via the prefers-color-scheme media query.

• Firefox Multi-Account Containers. This extension creates boundaries for cookies and local storage data. It is conceptually similar to running different tabs in the browser under different user profiles. I use this extension to segment sites where I need to remember cookies. For personal use the obvious example is social media. I also use the extension to keep multiple Google Classroom accounts simultaneously opened, to track the progress of my children’s online learning. I also use dedicated containers for work.

• Tampermonkey. This extension allows you to run a custom scripts to modify the appearance and functionality of any sites. I use it to improve some internal web applications we use at work. I also use it to make Google search results look the same as before. It is also trivial to remove ads from Google search results.

• Instapaper. I use Instaper to quickly save interesting web pages I do not have the time to read right away. I can go back to them later in the evening and read them on my phone.

• Lastpass is my password manager. It works and it is cross-platform. I am very convinced one needs to use a password manager, but there are many things in Lastpass that annoy me. I may change in the future.

Configuration Tweaks

I use the following configuration tweaks to improve privacy without resorting to a VPN:

• Enable DNS over HTTPS (aka DoH). This feature is normally turned on by default, except if you live in the UK as it frustrates the government surveillance efforts (see Snoopers’ Charter) and the “voluntary” porn filters run by ISPs. The UK Internet Service Providers Association has named Firefox an Internet Villain over the issue.

• Turn on Encrypted SNI support (aka eSNI). This ensures that Firefox does not leak the name of the visited sites in the clear, when doing a TLS handshake. By default, over plain old SNI, the domain name is sent in plain text before the encrypted connection is established.

You can verify the state of DoH and eSNI using this convenient Test Page from Cloudflare.

Alternatively, if you live in the UK, you could also try to visit the top banned site which cannot effectively be blocked by ISPs when using DoH and eSNI. Since the site is hosted by CLoudFlare, IP blocking would result in massive over blocking of thousands of legitimate sites.

The Future is Uncertain

Unfortunately, it looks like my current set up is living on borrowed times. Mozilla laid off the majority of the Servo team last August, which begs the question about the future of the Firefox, since Servo backported a lot of their technologies to make Firefox Quantum possible. The Servo project on Github is now essentially dead. I am somewhat less worried about the Rust language itself, since it is now large and useful enough to survive on its on merits.

Instead, Mozilla seems to be focusing their energy on their VPN offer and their clone of Have I been pwned?. Obviously these services have a clear monetization strategy compared to a free web browser, but it’s hard to imagine how Mozilla intends to fulfill their self professed goal of providing a better Internet for People with these niche and unambitious projects. If the Firefox browser is unable to keep pace with its commercial competitors, it will fail to attract new users.

In any case, I may transition back to Safari on the Mac. The upcoming version of Safari will support manifest v2 JavaScript extensions as announced at WWDC. This appears to support webRequest, so a port of uBlock Origin may be theoretically possible.

Another alternative would be to consider using Brave, which is yet another Chromium based browser with a focus on privacy. However, the browser has made some controversial moves in the past.

Old Setup

When I started the blog in 2009, it was powered by a self-hosted Wordpress instance. Since I was getting frustrated with maintaining PHP, MySQL and Wordpress up to date, I decided to switch to Octopress, a statically generated site, in 2013.

Overall, I have been satisfied with Octopress, a somewhat abandoned setup for Jekyll, the static site generator written in Ruby that powers Github Pages.

The setup enabled me to write blog posts in Markdown rather than plain HTML. The beginning of each markdown file contained a small front matter header to describe the post’s meta data such as titles and publication dates. These files were then fed into the Jekyll machinery that generated this complex web site, including an Atom feed and a sitemap.

This setup served me well for many years but I had two growing sources of annoyances. First, the theme I was using, looked incredibly dated and was overdue for a refresh. Second, the Ruby stack that powers Jekyll can be very finicky: I had to install a very specific version of Ruby (via rvm or rbenv) to make things work.

New Setup

For the new setup, I have decided to use Hugo, another static site generator, written in Go. Hugo is very similar to Jekyll, as it also uses Markdown and Front Matter. This made the transition easier. For the most part, I was able to re-use my existing markdown source files unchanged.

The motivation to choose Hugo over Jekyll is that Hugo is a batteries included solution, which covers all the needs of this blog. I did not have to worry about Ruby and Gem dependencies. My Hugo setup is as simple as sudo port install hugo. Moreover, I have found that Hugo’s documentation is really excellent: every time I wondered how to do something, I was able to figure it out rapidly.

This blog now features a modern theme, with a refined minimalist layout, that let readers concentrate on the content. The garish gradients and drop shadows of the previous theme are gone. I have also re-introduced color on this site, by using slightly different shades of blue as accent or background colors. A dark mode color scheme is also supported and displayed when requested by the browser.

For the first time, I made my own theme, rather than use one readily available on the Internet, as I now know enough CSS and SASS to be dangerous. However, my work was loosely based on the Hugo Bootstrap v4 Blog theme. It uses Bootstrap 4 for the layout, Fortawesome for the symbols, and Lato as the main font.

Another change is the lack of Google Analytics, as I have grown incredibly frustrated with Google’s approach to privacy, and their lack of innovation. In addition, many laws require to show cookie banners to end users, which is hardly conducive to a good user experience. I am currently pondering relying solely on the NGINX access logs and running either a self hosted Matomo or Plausible, but currently I have no analytics.

What’s Next?

Obviously, I intend to blog a lot more in 2020 than I did I the past few years.

Aside from inevitable tweaks and fixes, I don’t have any major plans for this blog with the exception of search.

Another change in the pipeline is to complete review the hosting which has been on Kimsufi since 2014.

It is a clean reboot of the .NET Framework that is both open-source and cross-platform, targeting Windows, macOS and Linux.

For a company that has historically been an enemy of open-source software, it is difficult not to be amazed at the amount of openness that Microsoft is demonstrating with .NET Core. Not only have they released the entire stack as open source software but the development is also being done in the open on Github.

The issue with .NET Core is that it can easily be confused with older closed source products that Microsoft has released for the past 15 years. The problem is compounded by Microsoft’s willingness to re-use similar names to designate radically different underlying technologies.

In this post, I will provide you with an overview of .NET Core, ASP.NET Core and Visual Studio, and I will explain how the open-source flavors differ from their closed source relatives.

The Different Flavors of .NET Framework

Different flavors of the .NET framework have coexisted for some time. Typically each flavor combines a different runtime and a different set of standard libraries.

• The .NET framework denotes the classical Windows-only .NET framework. This is the oldest flavor of .NET which shipped with Windows XP.

• The .NET Core framework is the new open-source framework released in 2016. It works on Linux, macOS and Windows. Generally speaking, it can be considered as a subset of the older .NET Framework.

• Xamarin Mono is a third party framework developed by open source Linux developers. It predates .NET Core by more than 10 years. It is also considered a subset of .NET Framework, because the Xamarin developers have never managed to quite catch up with Microsoft. However Mono also features special support for iOS, Android, Game Consoles, and Linux UI frameworks, which is entirely absent from Microsoft’s flavors.

• .NET Standard is a recent initiative from Microsoft which formalizes the APIs shared by the different flavors of .NET. If your code targets .NET Standard, it should be able to run without recompiling on all flavors of .NET runtimes: .NET, .NET Core and Xamarin Mono.

The Different Versions of .NET Core

Microsoft’s download page for .NET Core shows many different options:

• Microsoft offers both a Long Time Support (LTS) version and a regular version. The LTS version is older but Microsoft promises to support it for about 3 years. The downside is that it lags behind in terms of features.

• A runtime and SDK versions. The SDK includes the runtime necessary for running .NET Core executables, but also all the development tools (i.e. the compiler and dotnet command line tool). Thus you only need to install one of them.

• The Windows Server Hosting includes the runtime, but also a special module necessary to run ASP.NET Core sites within IIS. Since IIS is only available on Windows the Windows Server Hosting is not available on Linux nor macOS.

The Different Versions of Visual Studio

You can develop for .NET Core by using the text editor of your choice and the command line dotnet tool. However, I would highly recommend that you have a look at the different versions of Visual Studio that Microsoft provides.

• Visual Studio 2017 for Windows. This is the traditional version of Visual Studio. The Community edition is free and fully featured when it comes to .NET and .NET Core development. The only downside is that you will probably have to use the paid-for Professional or Enterprise editions in a commercial setting.

• Visual Studio Code is an open-source text editor based on Atom. It is a very capable text editor which also provides advanced IDE features, such as refactoring tools, compiler and debugger support for common languages including C#. It competes against Atom and Sublime Text.

• Visual Studio 2017 for Mac. Although it shares the same name as the Windows version, it is a radically different piece of software based on Mono Develop. It is a bit rough and has definitely not achieved the level of maturity as the Windows counterpart. That being said, Visual Studio Code was in a similar state when it was introduced in 2015, and I am confident that Microsoft will iterate and make Visual Studio for Mac a lot better soon.

The Different Versions of ASP.NET

This the area where the marketing at Microsoft has gone wrong. The name ASP.NET Core refers to old technologies which gives this very modern and compelling framework a very bad press, in my opinion.

• ASP, Active Server Pages is a very old technology released in the late 1990s. Think of it as a clone of PHP but using Visual Basic instead. It is also similar to what JSP for Java used to be. It was abandoned long ago and is not suitable for modern web application development.

• ASP.NET superceded ASP in the early 2000s, and development shifted from Visual Basic to C#. It is a framework based on Web Forms, with a page template system powered by XML and uses the *.aspx file extension. The framework was invented before AJAX became mainstream, and a lot of effort seems to be devoted to make sure developers never write any JavaScript. It also integrates with the System.Web pipeline of IIS which, even though is a step up from CGI, makes use of global variables to represent the state of requests and responses. Even though it is still maintained, it is a technology that is clearly showing its age.

• ASP.NET MVC is a modern framework that Microsoft introduced in 2009. It is inspired by the successful Ruby on Rails framework, and as the name indicates, features a clear separation between Models, Views and Controllers. It also introduced a better view template system named Razor and improved routing. Even though it is still integrated with the same System.Web IIS centric pipeline, it goes out of its way to hide that fact from developers, and wrap HTTP requests and responses into non-global variables, which can be mocked for the purpose of unit testing.

• ASP.NET Web API is another framework Microsoft introduced in 2012. It is very similar to ASP.NET MVC, to the point that entire class hierarchies between the two frameworks share identical names and functions. But unlike ASP.NET MVC which focuses on generating dynamic HTML pages, ASP.NET Web API as the name indicates, targets the creation of Web APIs. It is a pretty good framework to write JSON RESTful APIs for instance.

• Finally ASP.NET Core is a complete rewrite and a merge of both ASP.NET MVC and ASP.NET Web API. It was released in 2016 and is developed in the open on Github. It makes great use of dependency injection and provides support for writing middleware. ASP.NET Core web applications can be deployed to Linux servers or Docker containers.

The Different Versions of ASP.NET Core

As I indicated in the previous section, ASP.NET Core is a complete reboot of the Microsoft framework for writing web application software.

But the moniker ASP.NET Core denotes some radically different realities:

• The version of ASP.NET Core which targets Kestrel. This is the main version that Microsoft is promoting. Kestrel is an open-source asynchronous HTTP server with very good performance characteristic. If you target .NET Core as your runtime, this is the only available option. Kestrel and .NET Core can be deployed to Linux (possibly using nginx or Apache as a reverse proxy). You can also target .NET Standard or .NET 4.7 for Windows. In this flavor of ASP.NET Core, the entire HTTP pipeline has been rewritten from scratch, and you can inject your own middleware, as in any modern web application framework.

• The version of ASP.NET Core which integrates with the IIS pipeline. This flavor of ASP.NET Core integrates with the classic IIS pipeline introduced for ASP.NET and it relies on the System.Web classes and their global variables. If you adopt this flavor of ASP.NET Core you have to target .NET 4.7 and run the web application on IIS on a Windows Server. In addition some features, such as writing your own custom middleware, are not available. The advantage of this flavor of ASP.NET Core, is that it can be viewed as an incremental update from the old ASP.NET MVC and ASP.NET Web API frameworks. Unless you have some existing ASP.NET code, you should probably stay away from this configuration for new projects.

• Finally you can use ASP.NET Core with Kestrel inside IIS. So this is the option where you are essentially using IIS as a reverse proxy for the Kestrel server. In this version you sort of get the best of both worlds: your Windows centric IT team can continue managing IIS sites as before, and you get all the features and performance improvements available to the Kestrel stack. This is achieved by installing a custom IIS module provided by Microsoft, which is called the ASP.NET Core Module. This is the recommended way to deploy ASP.NET Core on Windows servers. On Linux Microsoft recommends using nginx or Apache in similar reverse proxy configurations.

Conclusion

.NET Core and ASP.NET Core are very compelling open source technologies promoted by Microsoft. The frameworks, libraries, compilers and runtime are published on Github. You can use Visual Studio Code, another open-source piece of software to write C# code and deploy it to Linux, possibly via docker containers.

But make no mistake, Microsoft still has ways to make money:

• Microsoft sells access to Azure, their own cloud computing platform which competes with the likes of AWS and Google Cloud. Naturally Azure integrates very well with ASP.NET Core.

• Microsoft also sells Visual Studio 2017, which unlike Visual Studio Code is closed source software. Although it may be used for free under some circumstances, for most commercial settings you will have to pay a subscription to use it.

• Microsoft is also strongly pushing SQL Server, their closed source relational database server. It has better integration than any other SQL databases with the .NET Core stack. Traditionally SQL Server used to run on Windows Servers only, but since the last release of SQL Server 2017 last October, the software can now be deployed to Linux servers as well.

One of the major hurdles hampering the deployment of HTTPS on smaller websites like this one, has always been the price of certificates. As much I would have liked to get one, I could hardly justify the cost. That’s why a year ago, when the Let’s Encrypt project was announced, with the promise of free domain certificates, I was particularly excited. I decided to migrate my websites as soon as the project reached the public beta phase, last week.

Even if your site does not require HTTPS for security reasons, it is worth considering:

• to provide additional privacy to your users. Unscrupulous ISPs have started to inject javascript or cookies into third party HTTP pages. They cannot do this with HTTPS pages.

• to benefit from the performance improvements of HTTP/2. Even though in theory HTTP/2 does not require TLS, all major browsers have decided to boycott the “plaintext” version of the protocol.

In this blog post, I intend to explain how I migrated this Octopress blog hosted with nginx from HTTP to HTTPS and obtained an A+ Grade from SSL Labs.

Intall Let’s Encrypt

This part of the process is very well covered by the Let’s Encrypt documentation. Since there are no packages for Ubuntu Server LTS at the moment, I used the source code approach. In this case, we end up using the letsencrypt-auto command instead of letsencrypt directly. letsencrypt-auto is a wrapper script that ensures that the tool and its dependencies are up to date, prior to running any letsencrypt commands.

# Optionally install git
sudo apt-get install git-core

# Checkout the let's encrypt git repository into /srv/letsencrypt
git clone https://github.com/letsencrypt/letsencrypt /srv/letsencrypt

# Run the tool at least once
cd /srv/letsencrypt
./letsencrypt-auto --help

When I did this, I happened to have some kind of InsecurePlatform Python warning.

Creating virtual environment...
Updating letsencrypt and virtual environment dependencies...../home/aymericb/.local/share/letsencrypt/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
./home/aymericb/.local/share/letsencrypt/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning

This warning comes from urllib3. I ignored it and everything was fine!

Getting the Certificate

At this stage you are ready to request a certificate for the website. Let’s Encrypt uses the ACME protocol, which requires serving a bespoke generated file on the web server to confirm ownership of a domain. The letsencrypt tool aims to automate the whole setup, so that writing ./letsencrypt-auto --apache -d blog.barthe.ph should be sufficient to get the entire web server up and running, if you happen to use Apache.

Unfortunately, the plugin for nginx is not stable yet and cannot be used. The setup will be slightly more complicated. We will use the --webroot switch, telling letsencrypt where to find our web server files, and afterwards, we will have to modify the nginx configuration files ourselves.

Getting the Let’s Encrypt certificate is the easy part:

./letsencrypt-auto certonly --webroot -w /srv/blog.barthe.ph/www -d blog.barthe.ph

You are asked to provide an email address (used to warn you when your certificate is about to expire), and agree to the terms and conditions.

Afterwards, you need to manually edit the nginx configuration files using your favorite editor (e.g. nano -w /etc/nginx/sites-available/blog.barthe.ph). The TLS certificate and the private keys are located in /etc/letsencrypt/live/blog.barthe.ph.

# Add the following to redirect HTTP requests to HTTPS
server {
        server_name blog.barthe.ph;
        listen 80;
        return 301 https://$server_name$request_uri;
}
# Modify the existing HTTP setup to use HTTPS
server {
        server_name blog.barthe.ph;
        listen 443 ssl;

        # Basic TLS setup
        ssl_certificate /etc/letsencrypt/live/blog.barthe.ph/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/blog.barthe.ph/privkey.pem;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

        # keep the rest of the site configuration unchanged
        # [...]
 }

You’ll notice that in the configuration, I chose to support TLS 1.0 and greater. I wanted to be more aggressive but TLS 1.1 is not that well supported. It does not work on OS X 10.8 for instance. However, I did drop support for the older SSLv3 protocol (which preceded TLS 1.0) and is still supported by most websites. The consequence is that my site is no longer compatible with older versions of Internet Explorer (IE6 to IE8). Supporting SSLv3 without weakening the security of modern TLS 1.2 browsers is becoming increasingly difficult because of downgrade attacks such as POODLE.

In order for the changes to take effect you should invoke sudo service nginx reload. If you botched the configuration file, the site will continue running with the existing configuration and you should get an error in /var/log/nginx/error.log.

Once that’s done, you should have a web server running with HTTPS, and that redirects older HTTP URLs to their HTTPS equivalents. W00t!

A+ Grade Security

When testing the Octopress server on different browsers, I got some warnings about mixed content. It means that although my website was configured to serve its content through HTTPS, it still referenced URLs that used HTTP, either for internal URLs or externals URLs (such as Google Fonts, etc…). Chrome in particular was the most strict of all the browsers about this. After spending some time searching and replacing all references to HTTP URLs, I managed to fix all the warnings.

The next step was to edit /etc/nginx/sites-available/blog.barthe.ph to improve performance, by enabling TLS session caching:

ssl_session_cache shared:SSL:10m;
ssl_session_timeout 20m;

Let’s now focus on improving security and get an A+ grade on SSL Labs.

Let’s start by overriding the default prime used for Diffie Hellman. The default prime is usually too small, and also since it is a shared prime, some pre-computed attacks like Logjam are possible. Beware, the openssl command will take a long time; from 20 minutes to a few hours.

# Be patient…
openssl dhparam -out /etc/ssl/private/dhparams_4096.pem 4096

# Add "ssl_dhparam /etc/ssl/private/dhparams_4096.pem;" to nginx config

Now, let’s add support for OCSP stapling. OCSP is a mechanism that can be used by the browser to confirm that a certificate has not been revoked. This normally involves an extra connection to the certificate authority, unless the website uses stapling.

ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/letsencrypt/live/blog.barthe.ph/fullchain.pem;

Let’s enable HSTS. This prevents a browser that connected to the website at least once before, from ever accepting an HTTP connection from the same domain.

add_header Strict-Transport-Security max-age=31536000;  # Valid for 1 year

Finally let’s massage the cipher suite to only use safe and secure ciphers.

ssl_prefer_server_ciphers on;
ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:... SEE BELOW';

In order to obtain the list of ciphers separated by :, use openssl ciphers -V to dump all available ciphers. Then do as follow:

• Delete all references to DES, 3DES and RC4, and just keep AES ciphers.

• Delete all references to MD5.

• Put all ciphers that use SHA1, whose name ends with _SHA and are SSLv3 at the bottom.

• Put ciphers with larger key lengths on top.

• Put ciphers with forward secrecy on top. That would be ECDHE or DHE, the DH stands for Diffie-Hellman, and the E for Ephemeral, which means a new key is negotiated for each connection. Stealing the private key will not allow an attacked to decrypt past TLS sessions.

• Favor GCM over CBC as a mode of operation. This is faster, and CBC brought security issues like POODLE in the past.

• Favor Elleptic Curves (anything with EC) over plain RSA, DSA, or DHE. Elliptic curves are faster because they require smaller primes than traditional crypto, and as far as we know the security behind the maths is solid.

Here’s my list:

0xC0,0x2C - ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(256) Mac=AEAD
0xC0,0x30 - ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(256) Mac=AEAD
0xC0,0x24 - ECDHE-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA384
0xC0,0x28 - ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AES(256)  Mac=SHA384
0x00,0xA3 - DHE-DSS-AES256-GCM-SHA384 TLSv1.2 Kx=DH       Au=DSS  Enc=AESGCM(256) Mac=AEAD
0x00,0x9F - DHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(256) Mac=AEAD
0x00,0x6B - DHE-RSA-AES256-SHA256   TLSv1.2 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA256
0x00,0x6A - DHE-DSS-AES256-SHA256   TLSv1.2 Kx=DH       Au=DSS  Enc=AES(256)  Mac=SHA256
0xC0,0x2B - ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(128) Mac=AEAD
0xC0,0x2F - ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(128) Mac=AEAD
0xC0,0x23 - ECDHE-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(128)  Mac=SHA256
0xC0,0x27 - ECDHE-RSA-AES128-SHA256 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AES(128)  Mac=SHA256
0x00,0xA2 - DHE-DSS-AES128-GCM-SHA256 TLSv1.2 Kx=DH       Au=DSS  Enc=AESGCM(128) Mac=AEAD
0x00,0x9E - DHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(128) Mac=AEAD
0x00,0x67 - DHE-RSA-AES128-SHA256   TLSv1.2 Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA256
0x00,0x40 - DHE-DSS-AES128-SHA256   TLSv1.2 Kx=DH       Au=DSS  Enc=AES(128)  Mac=SHA256          
0xC0,0x2E - ECDH-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AESGCM(256) Mac=AEAD
0xC0,0x32 - ECDH-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AESGCM(256) Mac=AEAD
0xC0,0x26 - ECDH-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AES(256)  Mac=SHA384
0xC0,0x2A - ECDH-RSA-AES256-SHA384  TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AES(256)  Mac=SHA384
0xC0,0x31 - ECDH-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AESGCM(128) Mac=AEAD
0xC0,0x2D - ECDH-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AESGCM(128) Mac=AEAD
0xC0,0x29 - ECDH-RSA-AES128-SHA256  TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AES(128)  Mac=SHA256
0xC0,0x25 - ECDH-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AES(128)  Mac=SHA256
0x00,0x3D - AES256-SHA256           TLSv1.2 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA256
0x00,0x9C - AES128-GCM-SHA256       TLSv1.2 Kx=RSA      Au=RSA  Enc=AESGCM(128) Mac=AEAD
0x00,0x3C - AES128-SHA256           TLSv1.2 Kx=RSA      Au=RSA  Enc=AES(128)  Mac=SHA256
0xC0,0x0A - ECDHE-ECDSA-AES256-SHA  SSLv3 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA1
0xC0,0x14 - ECDHE-RSA-AES256-SHA    SSLv3 Kx=ECDH     Au=RSA  Enc=AES(256)  Mac=SHA1
0xC0,0x09 - ECDHE-ECDSA-AES128-SHA  SSLv3 Kx=ECDH     Au=ECDSA Enc=AES(128)  Mac=SHA
0xC0,0x13 - ECDHE-RSA-AES128-SHA    SSLv3 Kx=ECDH     Au=RSA  Enc=AES(128)  Mac=SHA1
0x00,0x39 - DHE-RSA-AES256-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA1
0x00,0x38 - DHE-DSS-AES256-SHA      SSLv3 Kx=DH       Au=DSS  Enc=AES(256)  Mac=SHA1          
0x00,0x33 - DHE-RSA-AES128-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA1
0x00,0x32 - DHE-DSS-AES128-SHA      SSLv3 Kx=DH       Au=DSS  Enc=AES(128)  Mac=SHA1          
0xC0,0x22 - SRP-DSS-AES-256-CBC-SHA SSLv3 Kx=SRP      Au=DSS  Enc=AES(256)  Mac=SHA1
0xC0,0x21 - SRP-RSA-AES-256-CBC-SHA SSLv3 Kx=SRP      Au=RSA  Enc=AES(256)  Mac=SHA1
0xC0,0x20 - SRP-AES-256-CBC-SHA     SSLv3 Kx=SRP      Au=SRP  Enc=AES(256)  Mac=SHA1
0xC0,0x1F - SRP-DSS-AES-128-CBC-SHA SSLv3 Kx=SRP      Au=DSS  Enc=AES(128)  Mac=SHA1
0xC0,0x1E - SRP-RSA-AES-128-CBC-SHA SSLv3 Kx=SRP      Au=RSA  Enc=AES(128)  Mac=SHA1          
0xC0,0x1D - SRP-AES-128-CBC-SHA     SSLv3 Kx=SRP      Au=SRP  Enc=AES(128)  Mac=SHA1
0xC0,0x05 - ECDH-ECDSA-AES256-SHA   SSLv3 Kx=ECDH/ECDSA Au=ECDH Enc=AES(256)  Mac=SHA1
0xC0,0x0F - ECDH-RSA-AES256-SHA     SSLv3 Kx=ECDH/RSA Au=ECDH Enc=AES(256)  Mac=SHA1
0xC0,0x0E - ECDH-RSA-AES128-SHA     SSLv3 Kx=ECDH/RSA Au=ECDH Enc=AES(128)  Mac=SHA1
0xC0,0x04 - ECDH-ECDSA-AES128-SHA   SSLv3 Kx=ECDH/ECDSA Au=ECDH Enc=AES(128)  Mac=SHA1
0x00,0x35 - AES256-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA1
0x00,0x2F - AES128-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(128)  Mac=SHA1	

Automatic Updates of the Certificate

As a matter of policy the certificates emitted by Let’s Encrypt are only valid for a short period of 90 days. This explains why the project focuses so much on automation.

It is possible to renew the certificate by executing the following commands, which I added to a /srv/letsencrypt/cron-update.sh script

#!/bin/sh
/srv/letsencrypt/letsencrypt-auto certonly --webroot -w /srv/blog.barthe.ph/www -d blog.barthe.ph --renew-by-default  --agree-tos
service nginx reload

Unfortunately, you’ll find out that the letsencrypt-auto command fails with the following error:

Failed authorization procedure. blog.barthe.ph (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Error parsing key authorization file: Invalid key authorization: 231 parts

I found out that the ACME protocol attempts to validate the ownership using HTTP and is unable to follow the HTTP to HTTPS 301 redirection we set up at the beginning. So I had to slightly change my setup in /etc/nginx/sites-available/blog.barthe.ph.

# The following lines prevent Let's Encrypt ACME protocol from working 
#server {
#        server_name blog.barthe.ph;
#        listen 80;
#        return 301 https://$server_name$request_uri;
#}
#
# Replace by these lines, to continue serving /.well-known/ files on port 80
server {
        server_name blog.barthe.ph;
        listen 80;
        location /.well-known/ {
                root /srv/blog.barthe.ph/www;
                try_files $uri $uri/ =404;
        }
        location / {
                return 301 https://$server_name$request_uri;
        }
}

The new version let all ACME requests to /.well-known/ be served normally over HTTP by nginx, but redirects all the others to HTTPS. Once that change is done and nginx reloaded, executing the letsencrypt-auto successfully updates the certificate.

The next is to add a CRON job, by typing crontab -e (as root). I set up mine to update the certificate once a month, as follow:

# contrab -e
# m h  dom mon dow   command
0 11 7 * * /srv/letsencrypt/cron-update.sh

Conclusion

So far I’m relatively happy with Let’s Encrypt. I have chosen to run the letsencrypt-auto run with root privileges, but if that bothers you, there is a project named Let’s Encrypt no sudo which aims to prevent that.

For websites with more traffic, you could delay the HTTP to HTTPS redirection until you have fully tested the HTTPS version of the site. This is a good opportunity to fix all mixed URL scheme content issues.

Finally, I published my final /etc/nginx/sites-available/blog.barthe.ph config file as a Gist.

Edit. I initially made a mistake and wrote return 301 https://blog.barthe.ph instead of return 301 https://$server_name$request_uri in the nginx configuration. The consequence is that URLs starting with http would be redirected to the home page of the blog, instead of the https counterpart page. This defect was somehow masked by HSTS, because after browsing once to the site, the browser would redirect to the correct pages.

After considering many options, I chose a Dedibox SC gen 2, a dedicated server solution from Online.net a French server manged hosting provider. The machine features a fairly low spec CPU, a 500GB hybrid SSD-HDD disk, 2GB of RAM and unlimited bandwidth with guaranteed throughput.

All that for €12 a month! It is much cheaper that all other hardware of VPS solutions I could find, and in fact about the same price as the Dreamhost hosting plan it replaced.

France seems to have two companies (Online with Dedibox and OVH with Kimsufi) dedicated to offer very low cost hosting solutions. The prices can be low thanks to custom dedicated hardware that is both small and power efficient. As you can see below, a Dedibox SC is barely larger than a 3.5 inch HDD. My gut feeling is that the emergence of these offers in France is linked to the three strike piracy law and the disappearance of Megaupload. A lot of these servers were used as seed boxes, at least in the beginning…

The hosting on Dedibox turned out to be very good and sufficient for my needs. A properly configured nginx server, that serves mostly static pages, has no problem serving the 20,000 hits per month my sites require. I initially hesitated using Amazon S3 for some of the most popular pages, but dropped the plan entirely. I also have a few dynamic pages, but these are mostly for personal use. The PHP code of Exposition, my unfinished photo gallery software, works a lot faster than on Dreamhost.

The support on Dedibox is not great though. I had some billing issue at the beginning and it took forever for it to be resolved. I’m not sure I would trust them for something critical, like a business.

In the past few days, I made the switch to their competitor Kimsufi. I was running out of space on the 500GB disk of the Dedibox, because of my large photo collection. I opted for a KS-2, which is the exact same price price as the Dedibox SC Gen 2. It is normally fitted with a 1TB disk, but I got 2TB disk instead (surprise!). The KS-2 is also fitted with 4GB of RAM instead of the 2GB of the Dedibox.

The rest is worse though. On Dedibox, the bandwidth is guaranteed, whereas on Kimsufi it’s best effort. In practice, I have not noticed any differences. The disk on the Dedibox is a hybrid SDD; Kimsufi uses a traditional slower spinning disk. Dedibox offers free monitoring services, like email notifications when the machine is turned on or off. Kimsufi offers nothing. OVH has a full featured iOS app that used to support Kimsufi users, but is now restricted to the pro tier of OVH hosting. Shame on you OVH! So I ended up configuring a free pingdom.com account for basic monitoring.

In the end, it’s a pretty good deal if you need the storage or the RAM. 2TB of “Cloud Storage” (it should be 1TB) for €12 month and similar fees with no limits in bandwidth is pretty good. It’s more than what you would get from Amazon S3 or Dropbox for the same price. But if you’re fine with 500GB or only 2GB of RAM, I would recommend the Dedibox offer instead.

Let’s hope the new hosting is as reliable as the previous one…

As I have explained in great details in a previous post, the balance of a Bitcoin wallet differs from the balance that can be computed by looking at the blockchain. That’s because every time you make a transaction, the Bitcoin software sends back to you, the fraction of the funds that were not sent. These funds are sent back to a new address, invisible to the end users. These addresses are pre-allocated and constitute the key pool.

In this article, I explain how you can compute the balance of a BitcoinQt wallet.dat file using node.js and the JSON API of blockchain.info. I also show how to extract the public addresses contained in the wallet, whether encrypted or not, and demonstrate how to use the blockchain.info API efficiently.

Extracting the Keys

Bitcoin relies on public/private key cryptography. The public key is what is turned into a public address through hashing (detailed below) and the private key is the one used to sign the transactions. In order to compute the balance of a wallet, we need to extract all the public keys from the key pool.

The wallet.dat is a Berkley DB database. This is an old NoSQL embedded database from long before the term was coined. Unlike relational databases, such as SQLite, it is not possible to make complicated queries in SQL. The wallet uses a single key/value dictionary to store all public and private keys, the last transactions, and all other account informations.

The keys from the key pool are stored directly into the database as individual records. The database keys are split into two parts. The first part of the DB key is some kind of identifier, that describes the type of record. Non encrypted Bitcoin keys are identified with key while encrypted keys are identified with ckey. The second part of the DB key is the Bitcoin public key itself, preceded by a single byte that indicates the length in bytes of the public key. The database record associated to the DB key is the private Bitcoin key, which may or may not be encrypted. This splitting of the DB key into two parts may sound odd, you have to remember that the DB keys must be unique to identify a record.

Here’s a drawing, in case you’re starting to be confused between the DB keys and the Bitcoin public and private keys.

You can see details about the key loarding algorithm in walletdb.cpp from the source code of Bitcoin. Look at the CWalletDB::ReadKeyValue() method.

I had no luck using node-bdb for reading the wallet with node.js, because cursors are not supported. Instead, I used a crude algorithm looking for the ckey and key strings within the wallet. This works because BerkeleyDB does not use any compression, but that’s not ideal.

Computing Bitcoin Public Addresses

Once you have obtained the Bitcoin public keys from the pool, you can turn them into a Bitcoin addresses through a complicated process of hashing, which is well documented.

However, I found a discrepancy with the documentation. The length of the keys I read from my wallet were all 33 bytes instead of the expected 65 bytes. Bitcoin uses ECDSA, a variant of elliptic curve cryptography. Both public and private keys are points on a specific elliptic curve (Secp256k1) whose equation is \( y^2 = x^3 + 7 \). The keys are the \(x\) and \(y\) coordinates stored as 32 bit integers preceded by one extra header byte that OpenSSL uses to store format information. Since \(x\) and \(y\) satisfy the known equation of the curve, you can store a single coordinate and a sign to compute the other coordinate with \(y = \pm\sqrt{x^3 + 7}\). This is what is called a compressed key, and it is 33 bytes long.

I initially thought that I would have to decompress the key, and convert it to the 65 bytes format in order to compute the address. But it turns out that Bitcoin hashes the OpenSSL ECDSA public key as it is. I suspect that means you could have multiple public addresses associated to the same public key: one public address for uncompressed 65 bytes key, and one public address for compressed 33 bytes keys…

You can simply hash the public keys with with the ripemd160 and sha256 algorithms as documented. The details of the hashing are strangely complicated, but easy to follow, and you end up with a 25 bytes binary address. This binary address is then converted into an ASCII string using the Base58 Check encoding. As explained in the source code and the docs, this encoding is used instead of the easier to calculate and more base64 encoding, to avoid confusion between similar characters like 0OIl.

Implementation with node.js is relatively straightforward using the crypto and bs58 modules

Computing the Balance with Blockchain.info

You can use the JSON API from blockchain.info to compute the balance of each address in the key pool, and sum them up to obtain the final balance. These APIs are public and can be used without authentication, but they are rate limited. This can be a problem if you use naively the single address API. Instead you should use the multi address API or alternatively the unspent outputs API.

Source Code

I wrote a quick and dirty sample using node.js and published it on Github.

HFS+ is seriously old

HFS+ was released in 1998, in the era of Mac OS Classic. It predates the current Unix based version of OS X by at least three years. It was created in a period where Apple’s business was in such a dire situation that Michael Dell’s uttered this now infamous quote “What would I do? I’d shut it down and give the money back to the shareholders”.

Technically HFS+ is small evolution of its predecessor HFS which dates back to 1985. The major change from HFS to HFS+ is the transition of block addresses from 16 bits to 32 bits. This change was really needed, as hard drives capacity exploded in the late 90s. On a 16 bit addressing scheme, a file containing a single byte would use 16KB on a 1GB hard drive, and around 16MB on a 1TB hard drive. The other changes included the transition to longer filenames (from 31 to 255 characters) and a switch to Unicode encoding.

By and large the rest of the design of HFS+ has remained unchanged since 1985.

HFS+ has serious limitations and flaws

When it was first released, HFS+ did not support hard links, journaling, extended attributes, hot files, and online defragmentation. These features were gradually added with subsequent releases of Mac OS X. But they are basically hacked to death, which leads to a complicated, slow and not so reliable implementation.

In the early days, the system had a hard limit to the number of files that could be written and deleted over the lifetime of the volume. It was 2,147,483,648 (i.e. 2^31). After that, the volume would stop being able to add any more files or directories. On HFS+, every entry in the filesystem is associated to a CNID (Catalog Number ID). The early implementations used a simple a global counter nextCatalogID stored in a volume header, that could only be incremented by one until the maximum value was reached. More recent versions of Mac OS X can now recycle old unused CNIDs, but this gives you an idea of the types of considerations that went into the design of HFS+.

More recently Apple added support for full disk encryption with FileVault 2 and Fusion Drive. But these features are implemented in a layer underneath the file system, by Core Storage, a logical volume manager. Additional features like Snapshotting and Versioning would probably require much tighter integration to the file system… and they would also make TimeMachine extremely efficient and reliable. Currently TimeMachine is built on top of the file system and relies on capturing I/O events, which adds overhead and complexity. Another hack.

Finally there is Bit Rot. Over time data stored on spinning hard disks or SSDs degrade and become incorrect. Modern file systems like ZFS, which Apple considered but abandoned as a replacement, include checksums of all meta data structures content [1]. That means that when the file is accessed, the filesystem detects the corruption and throws an error. This prevents incorrect data from propagating to backups. With ZFS, you can also scrub your disk on a regular basis and verify if existing files have been corrupted preemptively.

A concrete example of Bit Rot

I have a large collection of photos, which starts around 2006. Most of these files have been kept on HFS+ volumes since their existence.

In addition to TimeMachine backups, I also use two other backup solutions that I described in a previous blog post. I keep a copy of the photos on a Linux microserver using ext3, which I checksum and verify regularly using snapraid. I also keep off-site backups using ARQ and Amazon Glaciers.

Before I acquired the Linux Microserver, I used to keep a copy of all my photos on a Dreamhost account. I recently compared these photos against their current versions on the iMac and was a bit shocked by the results.

The photos were taken between 2006 and 2011, most of them after 2008. There are 15264 files, which represent a total of 105 GiB. 70% of these photos are CR2 raw files from my old EOS 350D camera. The other photos are regular JPEGs which come from the cameras of friends and relatives.

HFS+ lost a total of 28 files over the course of 6 years.

Most of the corrupted files are completely unreadable. The JPEGs typically decode partially, up to the point of failure. So if you’re lucky, you may get most of the image except the bottom part. The raw .CR2 files usually turn out to be totally unreadable: either completely black or having a large color overlay on significant portions of the photo. Most of these shots are not so important, but a handful of them are. One of the CR2 files in particular, is a very good picture of my son when he was a baby. I printed and framed that photo, so I am glad that I did not lose the original.

If you’re keeping all your files and backups on HFS+ volumes, you’re doing it wrong.

How to check for file corruptions

I used the following technique to compare the photos from the Dreamhost backup against my main HFS+ volume.

I ran the shasum commmand line tool to compute SHA1 hashes of every single file in the backup folder, except .DS_Store files. Then, I ran shasum in verify mode to check the files on my main volume against the hashes. Differences either indicate voluntary modifications (which did not apply in my case), or corruptions courtesy of HFS+ (which was my case).

# Compute checksums
find . -type f -a ! -name ".DS_Store" -exec  shasum '{}' \; > shasums.txt
# Check against checksums
shasum -c < shasums.txt  > check.txt
# Filter out differences
cat check.txt | fgrep -v OK

You can use the same technique to check for corruptions on a single volume. You need to compute checksums and verify against them from time to time. If you use clone backups, it is probably a good idea to check for corruptions before doing the clone.

Addendum - June 11th, 2014

Thanks to everyone who spent the time to send feedback. There are a few things I would like to add:

• [1] Erratum. ZFS uses checksums for everything, not just the meta-data.
• I understand the corruptions were caused by hardware issues. My complain is that the lack of checksums in HFS+ makes it a silent error when a corrupted file is accessed.
• This not an issue specific to HFS+. Most filesystems do not include checksums either. Sadly…
• Other people have written articles on similar topics. Jim Salter and John Siracusa for Ars Technica in particular.

If you try this on a Bitcoin address that belongs to you, and fire up the Bitcoin Qt client (aka Bitcoin Core), you may have noticed a discrepancy. It’s very likely for the balance displayed on the website to be less than the one displayed by the software wallet.

The discrepancy is caused by the nature of bitcoin. Instead of storing actual coins, the bitcoin protocol should be seen as a distributed public database of transactions which together form the blockchain. You “receive” bitcoins when another party decides to use their private key to sign a transaction and send some amount of bitcoins to your public address. Bitcoins only exist in the sense that you can trace the chain of valid transactions until you reach special coinbase transaction, i.e. some mined bitcoins. You can almost think of all the transactions forming a singly linked list, that stops at one end with mined bitcoins, and on the other end with unspent bitcoins… except for the fact that each transaction can have multiple inputs or outputs. (Please keep in mind this is voluntarily simplified, if you wish to know more check the protocol documentation)

One of the quirks of the protocol, is that the amount of the inputs and outputs in the transaction must match (in reality, the output can be less than the input, and the remainder then constitutes the optional transaction fee). That rule greatly simplifies the validation of transactions, since there is no need to extract the entire history of transactions to figure out how much funds are spent or unspent for a given transaction: it’s either all or nothing.

The drawback of this solution arises when you need to spend only a fraction of the amount received in a previous transaction. In that case, the wallet software automatically creates two outputs to the transaction: one output is used to send money to the intended recipient, one output is used to send the remainder to the sender.

At this stage, it’s probably simpler to reason with an example. Let’s imagine Alice wants to sent 1.2 BTC to Bob. Alice previously received 1 BTC from Chip and 0.5 BTC from Dale. The new transaction she makes has to reference both previous unspent transactions as inputs, since neither of these transactions taken individually have enough funds. One of the outputs of the transaction must be the 1.2 BTC that are sent to Bob. But Alice also need to add a 0.3 BTC output that are sent back to herself. In the future, if she could use these 0.3 BTC coins that remain in her wallet, by referencing this 0.3 BTC output as an input for a new transaction.

It would be possible to use the same public address to send the money back to the sender, but the Bitcoin Qt software sends it to a new address instead, for privacy reasons. A bitcoin wallet contains at least a hundred of such addresses which constitute the key pool. The key pool is pre-allocated (therefore many addresses will have a balance of zero) so that slightly out of date backups of wallet files result in no loss of bitcoins. Every time a transaction that requires return funds is made, these returned funds seem to “disappear” from the balance of the wallet’s public address. It’s possible to reach a balance of zero on your public address in that way.

• The HP microserver is cheaper than a real NAS. HP regularly operates a cashback offer on this hardware (which I used). You could acquire an HP ProLiant G7 N54L 2.2GHz for roughly £150 with a cashback offer last month. The cashback and the price varies but regularly comes back. The hardware is comparable to a NAS: it is small, fitted with a 4 HD bay, it has a CD-ROM bay that can be used for an additional disk, and it has E-SATA ports for adding external disks. In comparison, Synology hardware is usually north of £400 and DROBO is north of £300. The cheapest 4 day NAS I found is the Synology DS413j which currently retails at £265. But the hardware is pale in comparison of my HP microserver: single core CPU, no E-SATA and only 512MB of RAM.

• Booting from the internal USB connector. There is a connector fitted on the motherboard, and this is where I plug my USB thumbdrive (a SanDisk Cruzer Blade). Since the microserver is not connected to any screen nor keyboard, having it boot from a thumbdrive is an advantage. In case of troubles, I can take the drive, plug it to the iMac and boot using VMWare. I also regularly clone it to another thumbdrive for backup purposes. Using the USB drive as a boot disk also means that all my hard drives are allowed to spin down to save power and prolong their lifetime.

• It’s relatively quiet and low power. I measured my power consumption over the course of 1 month and it comes to about 40 Watts on average, for a monthly cost of £3.50. It is fitted with 4 hard drives, which are probably powered down about 90% of the time. I use hdparm to power down the drives when not in use. The CPU of the machine is constantly solicited though, with various services I installed, and this probably prevents the machine from lowering to 30-35 Watt, which can be observed by just booting it and doing nothing. The large fan is relatively quiet, but it got more noisy after 1 year and absorbing dust. Currently the noise is about 40dB from 1 meter away which is comparable to my late 2008 iMac.

• Use encryption on all disks. As I explained in my previous post, the goal is not to stop the NSA or GCHQ from stealing my data. There is no 4th amendment in the UK, and the government can force you to reveal your key or put you in jail for 2 or 5 years if you refuse. However, encryption is a good way to prevent identify theft, if someone breaks into your house and steal your toys. I use full disk encryption with LUKS on all hard drives, but not the thumb drive. The encrypted disks appear as regular block devices, and you can use any filesystem or utility. Since I have no keyboard to enter a password at boot time, I have SSH into the box and enter it manually after each reboot. It’s slightly inconvenient, but I only power down the machine to dust it off, about once a month.

• Use snapraid for redundancy. I fitted the server with the hard disks I initially used as external drives for the Mac. Consequently all for disks are of different sizes. My goal is to achieve 1 disk redundancy, i.e. be immune to the failure of a single disk. Traditional RAID would have a hard time to cope with this setup, and it would be hard to expand it dynamically. Mockyblog mentioned using ZFS with RAID-Z. This fulfills the redundancy goal, but unfortunately it’s not possible to add a disk to an existing zpool. There are complicated techniques to work around it though… BTRFS is a good alternative to ZFS but it not yet ready. So in the end I adopted snapraid. Despite its name, it has nothing to do with RAID. It’s a file utility than runs on Linux, MacOS X and Windows and work with any file system. It can be easily configured with 3 data disks and 1 parity disk. The parity is stored in a series of checksum files and it’s somewhat similar to PAR2. It reads chunks of files from each data disk and store the checksum on the parity disk. You need to run snapraid regularly to update the checksums or use a cron job. It’s very well suited for for NAS storage, where data seldom changes and is mostly added (rather than modified/deleted). Compared to RAID, you also do not need to wake up all 4 drives when choose to read or write data.

• Plex Media Server. This probably the service I use the most. This is a huge improvement over manually sorting files and firing up EyeTV or VLC, because Plex automatically retrieves thumbnails and meta data, and the UI is very good. I can also access my collection from all my small collection of iOS and Android devices, or via a web page. Streaming works even outside of the home network, and even though my upload link is not good enough for 1080p films, it does stream music very well. As a consequence, I have deleted almost all of my songs from the 32GB iPhone and for the first time ever, I have lot of space to spare. It’s like having your own Spotify, iCloud, and other radio things, for much cheaper and without ads and privacy concerns.

• Bittorrent Sync. As an alternative to Dropbox, not the Pirate Bay style bittorrent. Again it’s free, and I can share a lot more than with the dropbox free tier. You could add a dedicated server with 100MBits bandwidth in the mix (which you can get for about £3.50 a month for 500GB these days, but I use this instead). My phone, my office computers, my laptop all use it. I recently synchronized 20GB of photos and videos of my brother’s wedding.

• Various download services. You can queue large download (or upload) from HTTP, SSH, RSync, BitTorrent, NZB, or anything really… The advantage is that this server is low power, you can schedule for data transfer to occur in the middle of the night, so that bandwidth is not affected during the day. There are lots of software you can run on the server and use with a Web interface: Bittorrent Sync, Sabnzbd, Sickbeard, Couchpotato, Transmission to name a few.

• Private Minecraft server. Or any game really. This is really the kind of stuff that would be hard to achieve on a NAS. I actually had to put extra RAM on the server. In honesty the CPU is a bit weak for that kind of thing, but it can be done. I doubt a NAS would be able to do this…

In the end, using commodity PC hardware and Linux you can build pretty much any solution for less money than a dedicated NAS.

There is no denying that researching and implementing features that come out of the box on a commercial NAS is really time consuming. But once it’s done, maintenance does not cost any time. I hardly touch the server anymore, unless I need to dust it off or install updates.

In the 1.1 version of the API, every single call is authenticated by OAuth. The intended goal for Twitter was to shutdown third party Twitter clients, in order to monetize content by shoving ads into the face of their users. This controversial move was announced about a year ago and has already made some victims.

Jason McIntosh published on his blog a technique to replace the built-in Octopress widget by an official Twitter timeline widget. It works, but I think the widget looks very gross, and since it hurts my feelings to use any kind of official twitter “client”, I decided to fix the original Octopress widget instead.

The most obvious way to fix the problem, is to update the twitter.js file to use the new API. However, it is probably not a good idea to put your OAuth tokens in JavaScript, where anybody could grab them and abuse them. These OAuth tokens need to be kept on the server side. The solution I describe below does this in a very straightforward manner.

1. Register as a Twitter developer and get OAuth credentials

• Log into https://dev.twitter.com/apps with your Twitter credentials (eg: bartheph).
• Create a new application. (eg: blog.barthe.ph).
• Accept the controversial rules of the road.
• Click on “Create my access token”.
• Write down the following values:
  • Consumer key
  • Consumer secret
  • Access token
  • Access token secret

2. Set up a cron job to generate a static timeline.json file

• This file will contain the timeline, and be loaded by Octopress’ slightly modified JavaScript code

• Install ruby with the oauth gem. On Ubuntu, you do something like that:

  sudo apt-get install ruby rubygems
  sudo gem install oauth
  

• Create a ruby file on the server. You need to put your Twitter OAuth token and choose an output path.

  #!/usr/bin/env ruby
  require 'rubygems'
  require 'oauth'
  
  # Edit config to suit your needs
  config = {
      :consumer_key => 'xxxxx',
      :consumer_secret => 'xxxxxx',
      :oauth_token => 'xxxx-xxxxxx',
      :oauth_token_secret => 'xxxxx',
      :output_path => '/srv/blog.barthe.ph/www/timeline.json'
  }
  
  # Create OAuth context
  oauth = OAuth::Consumer.new(
      config[:consumer_key], 
      config[:consumer_secret],
      { :site => "https://twitter.com", :scheme => :header }
  )
  access_token = OAuth::AccessToken.from_hash(
      oauth, 
      { :oauth_token => config[:oauth_token], :oauth_token_secret => config[:oauth_token_secret] }
  )
  
  # Get timeline
  url = 'https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=bartheph&trim_user=true&count=22&include_entities=1&exclude_replies=1'
  response = access_token.request(:get, url)
  
  # Write response into output file with JSONP wrapper
  File.open(config[:output_path], 'w') { |file| 
      file.write('processTweeter(')
      file.write(response.body)
      file.write(');')
  }
  

• Make sure the file is executable and not readable by anyone but the web server, since it contains OAuth tokens.

  chmod 700 /srv/blog.barthe.ph/cron/update_twitter.rb
  

• Add the ruby script to crontab to run it periodically. Type crontab -e and add a line similar to what follows.

  # Update every 10 minutes
  */10 * * * * /srv/blog.barthe.ph/cron/update_twitter.rb
  

3. Modify Octopress to use timeline.json instead of the Twitter API

• Edit your local version of source/javascripts/twitter.js.
• Edit the getTwitterFeed() function. Add a jasonp attribute and change the value of url.

function getTwitterFeed(user, count, replies) {
  count = parseInt(count, 10);
  $.ajax({

//  REMOVED     url: "http://api.twitter.com/1/statuses/user_timeline/" + user + ".json?trim_user=true&count=" + (count + 20) + "&include_entities=1&exclude_replies=" + (replies ? "0" : "1") + "&callback=?"
/* ADDED */ url: 'http://blog.barthe.ph/timeline.json?callback=processTweeter'
    , type: 'jsonp'
/* ADDED */, jsonp: 'processTweeter'
    , error: function (err) { $('#tweets li.loading').addClass('error').text("Twitter's busted"); }
    , success: function(data) { showTwitterFeed(data.slice(0, count), user); }
  })
}

That’s all. As you can see, on my blog, it looks just like before the old API was shut down. Using dynamic server side code goes a bit against the spirit of Octopress, but it’s a very tiny self contained change.

Douglas Crockford presented in his book JavaScript: The Good Parts one of the earliest techniques of consistent OO programming. However, I found it quite outdated, and after some of research and experimentation, I came up with the following technique which I now use in Exposition.

// Namespace
var ph = ph || {};
ph.barthe = ph.barthe || {};

// Use strict header
(function() {
"use strict";

// Class declaration
ph.barthe.MyClass = function(ctr_param) {
	
	// Remap this
	var self = this;

	// Private variable
	var m_private_var;

	// Private method
	var privateMethod = function(param) {
		// ...
	};

	// Public method
	self.publicMethod = function(param) {
		// ...
	};

	// Constructor body
	(function() {
		// Temporary variable i, does not leak
		for (var i=0; i<ctr_param; ++i)
		   m_private_var += privateMethod(i);
		self.publicMethod('stuff');
	})();
};

// Object usage
var ph.barthe.exampleUsage = function() {
	var my_instance = new ph.barthe.MyClass(15);
	return my_instance.publicMethod('other stuff');	
};

// Use strict footer
})();

In the previous code sample, I employed the following techniques:

• Use pseudo namespaces to avoid cluttering the global namespace and avoid potential naming conflicts with third party scripts. The namespace is in reality a single global object to which properties are appended. The OR operator in var ph = ph || {}; is used to make sure the object is only created if it does not already exist. This makes the code tolerant to changes in the order <script> statements.

• Use ECMAScript 5 strict mode. This is what the seemingly useless "use strict" statement is about. Instead of adding the statement to every function, I create an anonymous function whose scope is the entire file, and execute it immediately in the static context.

• Use “classical” OO programming with constructor functions instead of prototypes. I am more familiar with classical OO programming and the upcoming ECMAScript 6 is going to add support for classes. But a more pragmatic reason for that choice, is that relying on closures within constructor functions, appears to be the only known technique to get private members in JavaScript.

• Use the new keyword instead of Object.create or some kind of custom make function. I think it signals my intent to use classical OO programming more clearly. The ECMAScript 5 strict mode fixes the main argument Crockford uses against the “new” keyword. In non-strict mode, if you forget to use new when you invoke a constructor function, this ends up pointing to the global context instead of the object itself. This can lead to silent failures of disastrous consequences. In strict mode, an exception is raised instead, making the mistake obvious. In addition, naming conventions enforced by JSHint and JSLint help mitigate this risk.

• Use self = this to avoid the confusing remapping of this. This article illustrates how this is dynamically remapped and why we need to keep a reference to the original value. This is a variant of the that = this technique also mentioned by Crockford. I prefer self because I am familiar with ObjectiveC and this and that look too similar to my eyes.

• Use an anonymous function for the constructor body. Otherwise, temporary variables can leak to clutter the constructor closure forever. I believe this pattern also increase the readability of the code, by restricting statements outside the constructor body to mere assignments. This reduces the amount of code to scan.

So obviously you have backups. But are these up to date? Are you sure your backups work? What if your house burns or someone steals your equipment? What if you primary disk gets corrupted?

It does not take much for things to go wrong. After each hard drive failure, I either barely avoided catastrophe or lost a little bit of data. Here is a few things I have learned in the process:

1. One copy is not enough. You may think you would have time to duplicate the data after one of the disks crashed, but think again. Maybe the backup has been corrupted, and a secondary backup would be useful. Maybe your backup drive will die. As unlikely as it sounds, it happened to me on my old PowerMac G4. I had two disks in software RAID-1. The second disk died a few days later while mirroring the data after I inserted a brand new disk.

2. You need off-site backups. In case your house burns. In case someone breaks in, steal your computer and your backup drive. In case a power surge fries all the equipment plugged on the same socket.

3. Encrypt your data. Not because the government wants to snoop on you. They can use a $5 wrench or the law. But because someone might steal your gear or, if if you use an online storage, their security could be breached. And even if you have nothing to hide, the data contained on your computer can be used for a lot of nefarious purposes, such as identity theft.

4. Use incremental backups instead of mirroring. Not just to save time, but also to avoid propagating errors from your primary disk. Most file systems do not include checksums of the data. So when you copy a file, and the data had been corrupted, you copy the same errors to your backup disk. On a checksumed filesystem, you would normally get an I/O error instead. I have over 100GB of pictures, and one day, I discovered a handful (10 files or so) had been corrupted. One of my backups (mirrored) was also corrupted, but fortunately, not the incremental backups.

Here is what I currently use. This is Mac OS X specific.

1. I use Filevault 2 on all my drives. Filevault 2 is real full disk encryption, and has nothing to do with its predecessor of the same name, which was an abomination. It’s very secure even though Apple has not published formal specifications.

2. I use TimeMachine. This is a no brainer. It just works and there is no better way to have up to date backups.

3. I use SuperDuper! with the smart update option instead of the regular cloning, to avoid coping mistakes from the primary. I have a Newer Tech external Voyager and a set of 2 internal hard drives. I always keep one at home and one off-site, at the office. I typically do a backup once a month, or before every MacOS X update. If things go wrong, I can directly boot from the backup.

4. I use Arq and Amazon Glaciers. This is a little bit like an insurance, a solution of last resort in case everything fails. I have about 500GB of data backed up there. To limit the number of requests and pay less, I make only one backup every month. I end up paying about $5 per month.

5. I have a local Linux Miniserver. This is my NAS. I use it to backup super critical data like photos and family videos, as I do not have sufficient storage to keep more than a couple of years on the iMac. I use snapraid to provide redundancy on my 4 disks array. Snapraid would allow me to reconstruct the data if a single disk in the array were to die. I monitor the SMART status of all the disks with smartmontools, and I have a spare disk I can swap in at any moment. A lot of the critical data is also backed up to Amazon Glaciers, before being uploaded to the NAS.

With that kind of crazy setup, even if North Korea was to nuke Oxfordshire tomorrow, my data would be safe… but nobody would be left around to decipher it.

To say that this blog needed to be refreshed, is an understatement. The blog looked so ugly and I was so ashamed of it, that I pretty much stopped writing for it.

My motivations for switching to Octopress are the following:

• Responsive design. Most people read blogs on their phones or tablets. Having a website with a hardcoded sizes optimized for the desktop is no longer acceptable in 2013.

• Static deployment. I never really understood why I needed to run dynamic code to serve a blog. I chose Wordpress back in 2009, because of its widespread use. I was not aware of the existence of Moveable Type at the time.

• Use GIT and Markdown files, instead of some clunky online editor storing data in a complicated MySQL schema. Hopefully this will make my next blog migration much easier.

• No comments. This was probably the only dynamic feature I used. It was hassle to maintain and keep spam free. Direct contact via twitter and optionally linking to your own blog post is a better way to provide feedback.

• Migrate away from Dreamhost. I have been happy with them, but I need dedicated hosting for some private projects. I can also get much better performance for my photo gallery powered by Exposition.

Ok. There is one dynamic feature I had in Wordpress, and that I am now lacking: Search.

At the moment, the migration is not completely finished. Some pages are missing from the site and I use temporary DNS redirections. Things will probably change a lot in the next few weeks.

Here is the list I came up with.

First, I would like to emphasize the facts that these rules are not set in stone. They constitute good guidelines but any developer should exercise its own judgment and discretion in using them. There are times where these points will be conflict between each others or when they cannot be achieved in a reasonable amount of time. If these rules were absolutes, then they would likely be enforced by the compiler or code analysis tools. But that’s not the case. It means that for every single rule, there must exist at least one legitimate reason for a developer to differ from it.

Design Rules

• Use object oriented principles, such as encapsulation and the principle of least knowledge, instead of relying on global states, functions, and sharing too much information.
• Aim for the simplest solution that can possibly work and yet remain maintainable.
• Aim for loose coupling between objects. Try to minimize the surface of contact and interactions between objects, as much as possible.
• Use design patterns when they make sense, and if you do, stick to well-known naming conventions.
• Do not plan for future features in your design. These features may never ship. Stick to what is needed to complete your goal.
• Try re-using patterns that already exist in the code, rather than introducing new, slightly different patterns even if they are equivalents.

Coding Rules

• Less code is always better. But less code does not mean cryptic code without any comments.
• Make the code as predictable and as obvious as possible, almost to the point of boredom. You are not the intended audience of your own code. You write code for other reasonably competent programmers to read, and they should almost be able to “guess” what comes after each line.
• Make the relationships between different parts of the code as explicit as possible. Use signal/slots, the observer design pattern, event notifications, or any such communication techniques to make relationships between objects more explicit. Avoid hidden flag variables. Avoid too much tight coupling that disseminate the relationships between objects.
• Avoid side effects. Your code should do what it says on the tin. Nothing more. It should be obvious whether a method modifies the internal state of an object, or not.
• Avoid code duplication. There should not be any copy/pasted code anywhere. If your code needs to do similar and yet slightly different tasks, then it should be refactored in such a way that the similar code is written once but used multiple times. You should refactor existing code when you identify or create new duplicates.
• Handle errors properly, and make it explicit what happens when errors occur.
  • Some errors should be thrown and handled by the caller, because you cannot decide what to do.
  • Some errors are non-critical and should be logged, and remain invisible to the end-user.
  • Some errors are critical and will have an impact on the user. In the worst case the program should exit.
• Use design by contract. Not religiously so. But use assertions for pre-conditions and post-conditions to detect problems. If you have a tricky algorithm, you should also use invariants. This is useful for detecting programming errors, i.e. errors that should never really happen in real production code. This also makes it obvious when you break or use existing code incorrectly.
• Provide just enough information in the logs. Too much information will pollute the logs. Not enough will make certain problems impossible to debug without extra logging. Try to think what you would like to know, if the feature your wrote broke.
• Use comments judiciously. In particular, you should document everything that would not be obvious for the intended audience. Focus on explaining workarounds, hacks, side effects or particularly tricky algorithms. Comments should also be used to provide good documentation about the role of top-level classes and the effect of the most complicated methods, and what happens when they fail.
• Follow the relevant coding guidelines. Even if no formal guidelines exist, try to stick to the same style as the file you edit. A particular emphasis should be given into naming things properly.

In the past week, LinkedIn, eHarmony and Last.fm got their password databases hacked and stollen. It’s hard to know how the database was compromised in the first place, but the sheer level of amateurism incompetence that these large companies have demonstrated is shocking.

So, here is how it should be done:

1. Do not store passwords in plaintext in the database

   • Duh! Well Sony did store a million passwords like that…

   • So did Savannah, a GNU website hosting free software, until they got hacked in December 2010.

   • Users tend to re-use passwords everywhere. So even if your website does not have sensitive data, if it gets hacked, your users might get compromised on other websites such as gmail, Facebook, etc…

2. Do not use MD5

   • Seriously. MD5 is dead, broken. It should be removed from every library and operating system, as it provides no more security than a CRC. MD5 is too easy to parallelize and too cheap to compute. Relatively inexpensive hardware such as high end GPUs, or FPGAs, make it possible to compute large number of hashes without spending much money nor time.

   • In fact, MD5 is so broken that we have seen a chosen prefix collision attack used in the wild to spoof Windows Update certificates and distribute the Flame malware. (And yeah you guessed it, Microsoft subsequently revoked all its remaining MD5 certificates!)

   • eHarmony and Last.fm both used MD5. But that was also the case of the so-called “experts” of HBGary, that security firm with government contracts, that threatened Anonymous last year and got hacked by them soon afterwards.

3. Use a Salt

   • Do not just hash the password and store the result into the database. Even if your hash function is better than MD5, it is possible to compute all hashes in advance for common passwords and create a rainbow table. It is trivial to download existing rainbow tables for standard hash functions, and get off-the-shelf tools to use them.

   • Instead you should salt your hashes. A salt is a random piece of data that is somehow combined to the password before it is hashed. Using a salt can be as simple as performing a XOR between the password and the salt, before using the hashing function.

   • Be sure that you use a cryptographically secure random number generator to create the salt, such as Yarrow or Fortuna with a good source of entropy, rather than the built-in rand() function which is only a pseudo random number generator (hence predictable and non-secure). Many operating systems provide this facility via /dev/random.

   • None of the sites hacked last week (LinkedIn, eHarmony and Last.fm) used a salt, nor did the so-called “experts” from HBGary. Some of these password hashes can be downloaded from the PirateBay here and here.

4. Use more than one salt

   • To make it even harder, it’s best to not re-use the same salt for every password. A good strategy is to create a unique salt for each password and to store it along side the hash in the database.

   • You can also combine this “per-password salt” with a “global salt” that you do not store in the database. This makes is virtually impossible to attack your passwords if only the database was compromised (via a SQL injection for example…)

5. Use the hash function multiple times (PBKDF2)

   • Cryptographic hashing functions are usually optimized for speed. But in the case of passwords, the amount of data to hash is very small and therefore the hash function runs very quickly. The speed plays to the advantage of the attacker not you. A good way to slow down the computation process is to re-use the same hashing function multiple times.

   • In fact, rather than re-inventing the wheel and taking risks, one can use an off-the-shelf key derivation algorithm such as PBKDF2 or scrypt.

   • For example, Apple uses PBKDF2 with 10,000 iterations on iOS devices.

6. Probably stay away from SHA-1 too

   • SHA-1 is still considered secure, but it is definitely on the way out. It’s 17 years old after all.

   • You should use the more recent SHA-2 variants (SHA-256, SHA-512, SHA-384) or, if you do not trust the NSA, something like Whirlpool.

Obviously, having a proper password “storage” system in place will not save your website from being hacked. But it will make it more difficult to exploit the password data if that were to happen. You should also test and pay attention to the implementation of the cryptographic functions. Try to re-use well established, peer reviewed implementations that have withstood the test of time.

And as a final note, do not forget to test your authentication system regularly. Recently, a vulnerability was discovered in MySQL that completely broke the authentication for some combinations of compilers and operating system. The bug was caused by a simple “loss of precision” cast from a 4 bytes to 1 byte, which probably trigged a warning in GCC but was ignored for a while. The smallest details matter.

Have you ever noticed some strange behavior when you resize a WebView hosting resizable Flash content? Does the Flash content seems to be play catch up with the resizing of the WebView?

If so you are particularly unlucky. The problem only arises if you have the following set of conditions:

• The hosted flash content is resized when the WebView itself is resized, a bit like in this website.
• Your process is running in 32 bit mode. The 64 bit mode seems to be immune, probably because Flash is hosted in an external process.
• You app is running on 10.5 or 10.6. Other versions are immune, including 10.7.
• You are running Flash 10.1 or later which includes the Core Animation rendering model. At the time I write this, the latest version of Flash is 11.2.202.235 and it still has the bug.

Caused by Core Animation?

The resizing behavior made me think of Core Animation implicit animations. Core Animation introduced the notion of “layers” to the already existing “view” hierarchy in Cocoa. Views backed by layers are rendered using a more efficient hardware accelerated rendering pipeline than “vanilla” views. Each layer is associated to a set of properties such as position, opacity, etc… and whenever a property is changed, by default, the system generates an implicit animation. For instance, if you make the layer visible, you will have a fade-in effect. And you guessed it, if you resize it, some kind of resize animation. This approach makes it possible to make eye candy animations without introducing much complexity to the code. And of course, Core Animation makes it possible to make explicit animations for more complex cases…

My suspicion was reinforced when I discovered that older versions of the Flash plugin (10.0) which do not use the Core Animation rendering model were exempt from the problem. Core Animation According to the blog of an Adobe employee, three rendering models are offered to plugins on the Mac platform: QuickDraw, Quartz and CoreAnimation. I quickly corroborated this by looking at Mozilla’s Plugin API documentation, and I discovered the notes on NPDrawingModelCoreAnimation. The drawing model of a plugin is chosen after a simple negotiation. First, the plugin checks what the browser advertises, and decides to choose the best model. Second, the browser checks what rendering model the plugin has chosen. In the case of Core Animation, it appears that the plugin creates its own CALayer and provides it to the browser via NPP_GetValue.

To confirm my suspicions, I needed to access the layer returned by the plugin. But this turned out to be more complicated than I anticipated. I could not find any public API in the WebKit that gives me access to the “plugin instance”, which is the first parameter of NPP_GetValue. And since there must be one plugin instance per Flash object in the DOM, loading the flash plugin bundle directly from “Internet Plugin” would do no good. So, after reviewing the source code of the WebKit I opted to use the pluginLayer method from the WebNetscapePluginView class. Both the method and the class and private APIs, but they looked like a fairly stable choice given the circumstances: the WebNetscapePluginView class name is hardcoded to WebNetscapePluginDocumentView via a macro because of an old bug in Acrobat’s plugin, and the pluginLayer method looks indispensable.

The next steps consisted in disabling the implicit animations. If you modify the property of a layer yourself, you can use CATransaction. However in my case, the bounds property of the layer is modified internally by the WebView, as it reacts to the resize event. So the only options available are:

• Use setAction: on CALayer to set a new array of actions, which are set to NSNull
• Associate a delegate via setDelegate: to the CALayer and implement actionForLayer:forKey: to return NSNull.
• Or override the actionForKey: method for the target layer, via some kind of dark voodoo method swizzling.

I opted to use the least intrusive methods, which is to use setActions:. I soon discovered that it did not work, and that in fact the layer given by the plugin already had Null actions. However, by digging in the layer hierarchy, I found out that 2 levels below the plugin root layer, the plugin had created a layer from a class named FP_FPCAOpenGLLayer, which did not override the default actions. There was my problem and as suspected it was a lame Flash bug.

Fix / Workaround

In my Cocoa application I intercept the webView:didFinishLoadForFrame: from the WebFrameLoadDelegate. There I run my own FixFlashResizingBug() function, which does the following:

• Browse the view hierarchy and look for WebNetscapePluginDocumentView.
• For each WebNetscapePluginDocumentView it finds, try to call the pluginLayer method
• Then drill two levels down the layer hierarchy, and look for a FP_FPCAOpenGLLayer
• Set the actions of FP_FPCAOpenGLLayer to NSNull.

Obviously this is fragile since it both relies on private WebKit API’s and manipulate internal structures of the Flash plugin directly, but I could not think of a better idea.

Disk Images have always been popular on the Mac. They play a similar role to “ISO” images on other platforms, in the sense that they can be used to clone medias or can be mounted/unmounted as a virtual disks.

However DMG images offer a LOT more functionalities than traditional ISO images:

• They can hold a partition table, and therefore contain more than one volume. (Typically a partition table with a single partition).
• They may contain any kind of filesystem. (Typically HFS+).
• Images can be compressed (via bzip2, zlib, or Apple’s own format for the old ones).
• Images can be made as read-only, but also as read-write. (Installer images are typically read-only).
• Read-write images can be sparse. That means the image file only grows as more content is put inside, rather than be pre-allocated to the full capacity of the virtual disk.
• Images can have checksums to verify their integrity (options: several CRCs, MD5, several SHAs).
• Images can embed Mac specific behaviors: contain a EULA, have a custom icon, automatically mount when downloaded by Safari, etc…
• And I probably forget a lot of other features (RTFM: man hdiutils).

The Mac OS “Classic” Era

One of the initial motivations for using disk images, during the Mac OS classic era, is that application files could not be reliably transmitted over non-Mac specific networks like the Internet. This limitation stems from the fact that files were divided into two logical parts, called forks: the “data fork” and the “resource fork”. The data fork is what is traditionally considered to be a file. The structure of the data is entirely determined by the developer. The resource fork on the other hand, contains a collection of data organized in a standard way and accessible through a dedicated API. Although it is possible to define custom resources, the OS would also provide lots of default types to hold classical information: user interface elements, rich text, pictures, etc…

On Mac OS X, resource forks are still supported for backward compatibility reasons, but all application now use bundles instead. A bundle appears as a single icon to the end user, but it is in reality a directory containing several files. The role that the resource fork used to play to organize data is now played by the file system: each resource appears as an individual file within a hierarchy of directories. Hence, a typical Mac OS X application is composed of several files, and disk images remain useful to solve the problem of their distribution across networks.

In order to safely transport data, and installers, Apple chose to favor the usage of disk images rather self extracting installers. Since the early 90s, all Apple software installation disks have been distributed as either disk images or physical medias. Early Mac users would probably remember either using Apple’s DiskCopy, or Aladdin’s ShrinkWrap, for creating or mounting disk images. The current DMG file format is a direct descendant of the New Disk Image Format (aka NDIS) that Apple introduced with Disk Copy 6 in 1996. This file format used the resource fork to store all the meta data: file version, partition table, type of compression, checksums, etc… and more importantly where to find the data for the virtual disk sectors. The data fork only contained the raw data (compressed or not) of the virtual disk.

The Mac OS X Era

The astute reader may have noticed that, since the NDIS images used forks, it was not suitable for safe redistribution across the Internet. Disk images had typically to be encoded with either BinHex or UUencode.

When Apple introduced Mac OS X in 2001, they decided to solve the problem once and for all. They designed the DMG file format, which they called Universal Disk Image Format (aka UDIF). DMG is a flat file format in the sense that it does not contain any forks, but it is basically a wrapper around the traditional NDIS format. A typical DMG file is a concatenation of the data fork, followed by the resource fork of an NDIS disk image.

Apple has never published the specifications of either the NDIS or the UDIF file format. The private DiskImage framework plays a central role in the implementation of DMG parsing and the mounting of disk images in general. The framework apparently supports plugins (see hdiutil plugins) but there is no developer documentation available. The only non-Apple plugin known was made by Connectix for VirtualPC images, during the PowerPC era (see Unlocking FireVault). However it does not appear that either VMWare nor Parallels have been able to use the private framework in recent times.

Useful Tools

Although it is sad the DiskImage framework is closed, the DMG file format itself, is a bit of an open secret. It has been successfully reverse engineered many times, and there are a few open source implementations:

• Several commercial software support it: MacDrive, PowerISO, MagicISO.
• Vultur’s dmg2iso is probably the older open source implementation. It is a quick and dirty perl script to convert a DMG file into an ISO. It is deprecated, and its replacement is a cleaner and more complete C implementation called dmg2img.
• Erik Larsson provides a series of Java tools that can parse DMG files. There is the simple DMGExtractor, but also the more complete HFSExplorer which provides a graphical interface to browse HFS+ volumes and DMG files.
• Another interesting source is libdmg from planetbeing. It is a C library with UDIF and HFS+ support. It can be used to convert DMG from/to ISOs; extract files; or create DMG images from * scratch. The project was initially started as a way to manipulate Apple’s software restore packages for iPhone devices.
• Finally, 7-zip on Windows is able to open and extract DMG files.

My company has set up a VPN access that can be used from home. This proved to be extremely useful last winter when I got snowed in, and I have used it occasionally ever since.

The VPN system we use (Sonicwall) is not supported on MacOS X, and Sonicwall only provides connection software for Windows. I have tried to use IPSecuritas but I could not find a way to configure it properly. The commercial software VPN Tracker worked like a charm, but when I saw the price (€99 excl VAT) I stopped considering it. For about half of that price ($60 excl VAT) you can get a license to VMWare Fusion.

So, I decided to VMWare Fusion for my VPN access. The obvious disadvantage is that VMWare uses a lot of resources, but this is not really a problem for me. In addition, the setup is difficult but hopefully this post should help you with that. The obvious advantage is that, of course, VMWare can do a lot more than provide access to a VPN!

Summary of the Setup

My goal is to access the company network (10.100.10.x/24) from home on my iMac running Mac OS X 10.6. Since the VPN software is only supported on Windows, I connect to the VPN via Windows XP SP3 running inside VMWare Fusion. The virtual machine Internet connection is provided via a NAT network interface connected to the iMac.

My setup consists in creating another network interface between the iMac and the VM. Then, I configure both Windows and MacOS X so that my packets get properly routed.

A picture is worth a thousand words.

Setting up VMWare Fusion

I assume that you have correctly installed the virtual machine, Windows XP, and that the VPN software works as expected. My network configuration uses NAT to provide internet access to the VM, but it is certainly possible to use other configurations.

The first thing to do is to create a new network interface to provide access to the VPN from the Mac to the VM.

Steps

• Switch off the VM
• Go to “Settings”, “Network”
• Add a network interface with the “+” button at the bottom.
• Choose “The host only” option at the bottom.

Setting up Windows XP

The next part of the setup consists in configuring the newly created “Local Area Network 2″ interface (LAN2) so that it shares the connection with the VPN. We need to share the internet connection from the “Virtual Private Network” interface to the LAN2 interface. We also need to enable IP packet routing.

Steps

• Sharing the connection
  • Go to “Control Panel”, “Network Connections”.
  • Right click on the “Virtual Private Network” interface, select “properties”, go the the “advanced” tab and select the option “Allow other network users to connect through this computer’s Internet connection“.
  • In the combo box below select “Local Area Network 2″.
  • Deselect the option “Allow other network users to control or disable the shared Internet connection”.
  • Click on “Ok” to validate the settings.
• Enabling IP routing
  • Open “regedit.exe”.
  • Go to HKEY_LOCAL_MACHINE SYSTEM\CurrentControlSet\Services\Tcpip\Parameters.
  • Locate the key IPEnableRouter and set it to 1.
  • This setting is different on older or newer versions of Windows.
• Set up Local Area Network 2 interface
  • Open the “Terminal.app” application on the Mac and type “ifconfig”.
  • Note the IP address (inet) associated to the “vmnet1″ interface and the network mask. In my case it’s 192.168.245.1 and 0xffffff00.
  • Go the “Control Panel”, “Network Conections” on the VM
  • Open the “Local Area Network 2″ interface properties window.
  • In the “general” tab, double click on the “TCP/IP” item.
  • Choose an IP address on the same network than the one from the Mac, and set the network mask. In my case, I chose 192.168.245.2 and 255.255.255.0.
  • Click “Ok” to validate the settings

At this stage, although this is not strictly necessary, you may want to enable the ping on Windows XP, to verify that your connection works.

• Go to “Control Panel”, “Windows Firewall”.
• Go to the “advanced” tab and select “settings” in the “ICMP” section.
• Enable the “allow incoming echo request” option.
• On the Mac, open “Terminal.app”
• Type ping 192.168.245.2 and the Virtual machine, should normally reply to the ping. If it does not, you may have made a configuration mistake. Use ifconfig on the Mac and ipconfig on Windows to debug the issue.

Setting up Mac OS X

At this stage we need to set up Mac OS X routing so that VPN addresses are resolved through the vmnet1 interface. We also optionally set up the company DNS. If you have not done it yet, you should connect to the VPN from the virtual machine.

Steps

• Once you are connected to the VPN on Windows, launch “cmd.exe” and type “ipconfig /all”.
• Look for the following details: your IP address on the VPN network, the mask, and the addresses of the VPN.
• In my example I have an IP of 10.100.10.212, a mask of 255.255.255.0 and two DNS servers at addresses 10.100.10.12 and 10.100.10.14
• Go back to Mac OS X.
• In the terminal type a command line similar to sudo route -n add 10.100.10.0/24 192.168.245.2. (Or alternatively if you do not wish to worry about the CIDR notation for the mask, sudo route -n * add 10.100.10.0 192.168.245.2 255.255.255.0)
• The first partial IP corresponds to the IP of your network (normally the first digits, until the mask is zero).
• The second IP is the IP of the VM on the LAN2 interface.
• This line instructs MacOS X to route all IPs from the 10.100.10.x network to the 192.168.245.2 gateway, i.e. the virtual machine.
• You must the enable IP routing. Type sudo sysctl -w net.inet.ip.forwarding=1.

At this stage you should be able to ping yourself on the VPN from the Mac. In my example ping 10.100.10.212 should reply. Now what if it does not work

• Try traceroute 10.100.10.212. If you see your message routed through your normal internet connection, then something is wrong on the Mac. Otherwise it’s on Windows.
• On the Mac netstat -rn will dump the route table.
• Verify that you still have an internet connection, on the Mac and the VM. If you do trash your route table you may lose it entirely. (Try removing your incorrect route with “route delete …” and/or plug in and off your modem connection).

Optional Steps to Set Up the DNS

• Go to “System Preferences” on the Mac.
• Select “Network”.
• Click on “Advanced” at the bottom of the interface that provides your internet connection.
• Go to the “DNS” tab. In the “DNS server” sections add the DNS servers of your VPN, on top of the list. In my case it’s 10.100.10.12 and 10.100.10.14.
• Close, validate and do not forget to “apply” the settings.

Going Further…

• You can snapshot the Windows virtual machine to retrieve your settings next time.
• On the Mac you can put the combination of the route and sysctl commands in a script to further automate the process. Maybe it’s possible to write a launchd script and do it at startup
• I’m not sure how to configure the DNS from the command line, nor how to make the DNS of the company be used only for domains unknown from your regular DNS server.

WhichSize DiskWave is a freeware for MacOS X 10.6 (hence Intel Macs only for now) that can recursively size the directories on your drive. This allows to quickly identify and reclaim wasted space.

It is freely available here: http://diskwave.barthe.ph/

DiskWave was basically hacked in a weekend, but this is not a new idea. I already worked on a similar project, back in 2004, after discovering OmniDiskSweeper. I was appalled by the terrible performance of OmniDiskSweeper and quickly realized I could do a better job. So I learned Cocoa, ObjectiveC, the Carbon APIs, and came up painfully with a prototype only to discover that another software named WhatSize was free and worked very well. I also had a tricky memory leak due to a bug in Cocoa API. So I quitted.

Fast forward to 2010. OmniDiskSweeper is no longer a shareware, but its performance is still bad. WhatSize is no longer a freeware. The old freeware version of WhatSize I have is a PowerPC version, but since I moved to Snow Leopard, I felt like I did not want to install Rosetta. So here is DiskWave.

So this time, I did my homework, and it does not look like there is a suitable alternative. Here is the list of similar tools (on MacOS X) I came up with:

• OmniDiskSweeper
  • Freeware (formerly shareware).
  • Slower because it relies on Cocoa APIs. It also consumes a lot of memory.
  • Does not have many more features than DiskWave. I hope to catch up pretty soon…
• WhatSize
  • Shareware (formerly freeware).
  • Same speed. Consumes a little bit more of memory, because it supports another feature that requires it
  • Does have many more features than DiskWave. Some of them, I never intend to get: fat binaries removal, locale removals, etc…
  • The thing that annoys me is that you can size/browse only one drive at a time.
• DaisyDisk
  • Shareware.
  • New kid on the block. The visualization is really awesome but… I do not find it that practical.
  • If I ever wish to play with CoreAnimation, I may add an animated sunburst view.
• GrandPerspective
  • Opensource
  • Seems to work well and is properly maintained. But I do not like this kind of visualization.
• DiskInventory X
  • Opensource
  • It looks like it is no longer maintained. I do not fancy the GUI. Same as GrandPerspective.
  • DiskWave being in its early infancy is somewhat limited in terms of features. I hope to gradually upgrade it. If you have ideas of features you would like to see added, just let me know.

Reference:

• DiskWave homepage: http://diskwave.barthe.ph/

Update April 12th 2010:

• Renamed WhichSize into DiskWave after receiving complaints from id design, inc.

When you programmatically delete a shortcut from the Desktop folder, Windows is usually smart enough to update the Desktop and the recently used programs section of the Start Menu. But sometimes, it does not work, and you are left with a “ghost” icon.

As a regular end user, you can hit the F5 key to force a refresh of the Desktop, but for the Start Menu you are left with no alternative than killing/restarting explorer.exe.

I was looking for a better solution for a custom installer application. And after a lot of MSDN scattering and googling around, Eureka, I finally found a magic incantation to programmatically force a refresh of the Desktop.

SHChangeNotify (SHCNE_ASSOCCHANGED, 0, 0, 0);

This works well, except for one small detail. When the CRT is not installed, on Windows 2000 ~~~only~~~, the LoadLibrary error would also cause a MessageBox to be displayed. This stops the flow of the installer and of course, is not very elegant…

But there is a simple trick to get rid of the MessageBox, using the very obscure SetErrorMode system call.

::SetErrorMode(SEM_FAILCRITICALERRORS);
HMODULE hDll = ::LoadLibrary("CrtCheck.Dll"); // Won't bark
if (hDll)   ; // CRT is installed
else        ; // CRT is NOT installed

Do you guys know a better (yet simple) way to determine if the CRT is installed?

Update: this is not a Windows 2000 specific issue (thanks Ferruccio). It’s also true for older versions of Windows. As far as I know WinXP, Vista and 7 do NOT display a MessageBox under the same circumstances.

As of Xcode 3.2.1, there is no way to create a command line or Swing application project which is fully managed by Xcode. There is still support for the Java syntax in the text editor and the java compilation tools are also installed, though.

One of the default project templates, JNI library, does actually let you create a project containing java files. However these files are compiled with ANT. This demonstrates that Xcode can still be used for Java development through the use of “external build systems”, but that’s it… Java is no longer a first class citizen in Xcode.

I guess this decision makes sense for Apple. The Cocoa bindings for Java have not been supported since MacOS X 10.4 after all.

Object Initialization

In C++, there are two common idioms with regards to object construction and initialization:

• the constructor fully initializes the object: this idiom makes it impossible to construct an object which is not correctly initialized and ready to use.

• the constructor does not initialize (or partially initialize) the object: the initialization is performed at a latter stage by calling some init() method. This, in effect, splits the object allocation and initialization. It is common to encounter this pattern whenever the class mimics some transactional behavior like opening/closing a file, connection/disconnecting from a server…

One may have legitimate reasons to use one idiom over the other, but as a rule of thumb, the first approach (full initialization in constructor) should be favored. This is because it makes it hard to use the interface of the object incorrectly. In the second case it is possible to construct the object and call a method on it before the initialization was performed.

There are two ways to handle this improper usage:

• do nothing and state that this is an “undefined behavior”. It will probably make the class a bit faster to write, but harder to debug. Not a good idea…

• carry an internal state to remember whether or not the object was initialized. This make it possible to report errors in a much more graceful way. On the other hand it puts the burden on maintaining an internal state and checking against it. Maintaining an internal state looks very easy but my experience is that it is error prone in the long run…

So in summary, unless you have a good reason not to construct a fully initialized object, then you should avoid doing it. It will make your code harder to use incorrectly and much simpler. And you probably will have to write less.

There’s one issue though: error handling.

Error Handling at Initialization

There are three general ways to handle errors:

• exceptions: this is the recommended and the best possible way.

• return error code on each function: this is the classical C idiom. In practice it means every function you call returns an error code and that you MUST! check against it.

• global error state: there is a function or method that you need to call after each method of the public API. This is similar to the role of the GetLastError() function in Win32. It’s quite unpractical, and very seldom used. Therefore, I will not discuss it any further…

In the case of a fully initializing constructor, it is not possible to use the “return error code” idiom, because the constructor does not return anything. That makes it mandatory to use an exception based system.

I worked in a large organization that essentially banned the use of exceptions in their code, for rather dubious reasons. The consequence of this move was that people were forced to use the “non initializing constructor” idiom almost everywhere. This made the code more verbose (init/terminate functions everywhere), the interface easy to use incorrectly (many bugs were related to calling methods on a non initialized object) and put the burden on maintaining an internal state on each object (many bugs were related to failing to maintain this internal state correctly). But the main source of bug was by far, people “forgetting” to check against the error code, leading to much more subtle bugs long after the point of error.

Use RAII to avoid leaks

It sounds like a good idea to favor the fully initializing constructor isn’t it? If you use this idiom, then you must know how to use exceptions properly. Techniques like RAII and smart pointers are mandatory. Here’s one reason:

If an exception is throw in the constructor of an object, the destructor of that object is not called.

So if you do something like in the (counter)example below, you are in troubles:

// class Child1 and Child2 defined earlier...
class Parent {
public:
    Parent() : m_child1(new Child1()), m_child2(new Child2()) { }
    ~Parent() {
        if (m_child1)
             delete m_child1;
        if (m_child2)
             delete m_child2;
    }
private:
     Parent(const Parent& ); // non copyable
     Parent &operator=(const Parent &); // non assignable
     Child1* m_child1;
     Child2* m_child2;
};

In this example if the constructor of Child2 throws and exception, then the constructor of Parent also throws. Since the destructor of Parent is not called, the newly constructed Child1 is not destroyed. It leaks!

However if you rely on smart pointers, the problem disappears. While the destructor of a class is not called if an exception is thrown in its constructor, the destructor of each instance variable is called. The following code would fix the leak:

// class Child1 and Child2 defined earlier...
class Parent {
public:
    Parent() : m_child1(new Child1()), m_child2(new Child2()) { }
private:
     Parent(const Parent& ); // non copyable
     Parent &operator=(const Parent &); // non assignable
     std::auto_ptr m_child1;
     std::auto_ptr m_child2;
};

That’s all folks!

The Bug

Sometimes, and at random, our application was crashing on exit. The crash was always some dodgy memory access violation. This happened after calling a SendMessage() to HWND_BROADCAST… I did not quite understand what was wrong with it, but calling SendMessage was stupid enough in the first place. So, I removed the SendMessage, replaced it by something much better, and promptly claimed to have fixed the bug. Voilà! Not surprisingly, on the very next day, QA found a similar crash in DestroyWindow(), albeit much harder to reproduce.

Time for a proper proper investigation!

The Cause

A few days later, after unrolling tons and tons of code, I found miles away from the code that was crashing this little jewel:

FrameWidget::~FrameWidget()
{
  m_hWnd = 0;
  if (IsWindow())
    DestroyWindow();
  // etc...
}

Suddenly, the whole picture got clearer.

The first line resets the window handle. The next one checks if the handle represents a window. Of course not! So no need to destroy the window object associated to the handle, right?! Then, by the time you get out of the destructor, your message map/callback functions have been destroyed and point to garbage memory… So, all it takes to crash it now, is one message routed to this window. And thanks to Murphy’s law, it indeed does happen, sometimes!

What SendMessage() to HWND_BROADCAST and DestroyWindow() have in common, is that they both actually send a message to this phantom window. In case you did not know, DestroyWindow() does purge the message loop before destroying the window…

Why was the handle set to null in the first place is beyond me, but it sure was a bitch to find!

Reference

• DestroyWindow (MSDN)

Today, my colleague and I, have found a bug in OpenSceneGraph. Everybody knows that singletons are evil and may cause cancer, but that does not prevent OpenSceneGraph from using them all over the place. One of them, in particular, is initialized in the static context, and happens to call the standard library function getenv(). This was the beginning of all our troubles: from time to time, this seemingly benign initialization would cause a deadlock in our software.

Static initialization occurs, for instance, when you construct an object outside of any function body by writing something like static ObjectBlah myGlobalVar = new ObjectBlah(). The constructors are called before the entry point of your program (i.e. the main() function). In the case of a shared library, the static initialization generally occurs when the shared library is loaded. On Windows, the DllMain callback function is used for that purpose.

The rest of this post will detail the specific behavior on Windows, and explain why it is prone to deadlocks. I think it is generally a bad practice to call standard library functions on other platforms too, although I do not have a detailed proof to back my words (yet!).

The Dangers of DllMain

If you read the MSDN documentation of DllMain, you should realize that this function is quite dangerous to use. I extracted and highlighted some parts below:

Warning There are serious limits on what you can do in a DLL entry point. To provide more complex initialization, create an initialization routine for the DLL. You can require applications to call the initialization routine before calling any other routines in the DLL.

Because Kernel32.dll is guaranteed to be loaded in the process address space when the entry-point function is called, calling functions in Kernel32.dll does not result in the DLL being used before its initialization code has been executed. Therefore, the entry-point function can call functions in Kernel32.dll that do not load other DLLs. For example, DllMain can create synchronization objects such as critical sections and mutexes, and use TLS. Unfortunately, there is not a comprehensive list of safe functions in Kernel32.dll.

Windows 2000: Do not create a named synchronization object in DllMain because the system will then load an additional DLL.

One of the reasons why it is so critical to not use LoadLibrary() or any kernel32.dll functions susceptible to call LoadLibrary() (like user32.dll) is because the process private critical section is locked while DllMain is running. This “loader lock” is taken any time a library is loaded but also when functions like GetModuleHandle() or GetModuleFileName() are used.

At this point you might think it is safe to call a standard library function, as long as it does not appear to require anything more than kernel32.dll functions, and is not using GetModuleHandle() nor GetModuleFileName(). This relies on implementation details of the standard library, so it’s a bit border line, but still might work, right? No… wrong! Now all you need to cause a deadlock is another lock. Guess what? There are plenty of locks taken in the standard library functions…

Principle: do not call any standard library functions that acquire locks in DllMain().

It’s quite another thing to know whether a function uses advanced kernel functions (unlikely to change) than to know whether it does acquire internal locks (subject to change every time Microsoft releases a new CRT). Therefore the following corollary can be deduced:

Corollary: do not use any standard library functions at all in DllMain() (because you really have no idea whether it will change and acquire a lock in the future).

The Deadlock

If you use CRT functions in your DllMain(), the following events might occur in the wrong preemption order and, consequently, cause a deadlock.

Imagine that you have one thread that calls LoadLibrary() on your DLL. It acquires the loader lock, then executes DllMain(), which finally executes your static initializer. At this stage if you use a CRT function, you will try to acquire an internal CRT lock…

Meanwhile, in the rest of you program, you might want to access another CRT function that happens to do the following: attempt to acquire the same CRT lock, then attempt to call LoadLibrary (because it uses an advanced kernel function that requires a new DLL to be loaded). LoadLibrary will attempt to acquire the loader lock as well.

Boom! You now have two thread that are trying to acquire two access locks in opposite orders.

In practice, the bug we discovered involved the following race condition:

• Thread A
  1. LoadLibrary() called. Acquire loader lock.
  2. DllMain() executes getenv(). Acquire _ENV_LOCK.
• Thread B
  1. stat() is called. Acquire _ENV_LOCK
  2. stat() calls GetTimeZoneInformation(), which requires ntdll.dll to be loaded.
  3. Then calls LoadLibrary() to get ntdll. Acquire loader lock.

If your threads are preempted such as 1, 3, 4, 2, 5 are executed in sequence, then you have a deadlock!

I discovered a posteriori that Richard Chen and Larry Osterman documented the problem. Although in their case they do not explicitly mention the C runtime, it is obvious that their remarks are relevant once you know that the CRT uses internal locks.

References

• Another reason not to do anything scary in your DllMain: Inadvertent deadlock, Raymond Chen, The Old New thing blog
• Best practices for DllMain, Larry Osterman’s blog

Correction

I previously wrote “the rest of this post will detail the behavior on Windows, but the same general principles are true for other platforms as well.”. This was a gross exaggeration. I’m still convinced that using libc in static initializers is prone to troubles on other platforms as well. Having it all work correctly depends on so many implementation details: kernel (how are shared objects loaded, how system calls works), libc (how does it interact with the kernel), and the linker (does it load stuff in the right order w.r.t. static initializers).

It’s not rocket science, but you’d be surprised to see how many people get this wrong. Here’s a code snippet of what people usually do:

const HICON hicon = ::LoadIcon(
  ::GetModuleHandle(0),
  MAKEINTRESOURCE(IDR_MAINFRAME)
);
if (hicon)
{
  ::SendMessage(hwnd, WM_SETICON, ICON_BIG, reinterpret_cast(hicon));
  ::SendMessage(hwnd, WM_SETICON, ICON_SMALL, reinterpret_cast(hicon));
}

The code above would set the large icon (as displayed in the ALT-TAB switch window dialog) and the small icon (as displayed in the window title bar or the task bar) of the window (whose handle is hwnd).

Except for the error checking which is not particularly thorough, most people would consider this code to be correct. You could also have used LoadImage with LR_DEFAULTSIZE.

Why this is wrong

The problem reveals itself with an icon which has a slightly different design for the small 16×16 size compared to the large 32×32 size. The 16×16 icon will look like a scaled down version of the 32×32 one. The reason why the smaller icon looks different in the first place is usually because the large icon’s design is a bit overloaded and cannot be downscaled without becoming unrecognizable; so you will notice!

What’s wrong? If you dig into the documentation you will found the following information:

• LoadIcon loads the most appropriate icon, at one fixed size, which is SM_CXICON by SM_CYICON pixels, i.e. the size of a large icon.
• The small icon should be SM_CXSMICON by SM_CYSMICON pixels.
• WM_SETICON for ICON_SMALL would downscale a larger icon to SM_CXSMICON by SM_CYSMICON pixels automatically.
• This odd behavior is due to backward compatibility with legacy versions of Windows, which only had one size of icons.

How to use WM_SETICON

Now that we understand the problem (the wrong size of the icon is loaded), we know how to fix it. The solution is to call LoadImage twice and explicitly provides the size of the icon to avoid the downscaling.

const HANDLE hbicon = ::LoadImage(
  ::GetModuleHandle(0),
  MAKEINTRESOURCE(IDR_MAINFRAME),
  IMAGE_ICON,
  ::GetSystemMetrics(SM_CXICON),
  ::GetSystemMetrics(SM_CYICON),
  0);
if (hbicon)
  ::SendMessage(hwnd, WM_SETICON, ICON_BIG, reinterpret_cast(hbicon));

const HANDLE hsicon = ::LoadImage(
  ::GetModuleHandle(0),
  MAKEINTRESOURCE(IDR_MAINFRAME),
  IMAGE_ICON,
  ::GetSystemMetrics(SM_CXSMICON),
  ::GetSystemMetrics(SM_CYSMICON),
  0);
if (hsicon)
  ::SendMessage(hwnd, WM_SETICON, ICON_SMALL, reinterpret_cast(hsicon));

References

Links to MSDN:

• LoadImage
• LoadIcon
• Window classes, icon section
• WM_SETICON

The problem with that rarely updated large repository is that performing an svn update on it can take quite a while even though there are no updates to perform. The reason is that subversion checks that no files are missing on the local checkout directory, because you can perform non recursive partial checkouts or updates, or potentially alter the whole structure after a checkout.

Nevertheless, if all you do is mirror the subversion directory content on your drive, you may have a faster option than svn update. You can use svn info to compare the local revision of your checkout to the latest available version on the server. If the revisions match it means your repository is up to date and you can avoid the costly svn update.

I regularly use a powershell script to automate that process.

Powershell Script

# Set your svn path
$Svn = "svn.exe"

trap {
    Write-Error "$_.Exception.GetType().FullName + $_.Exception.Message"
}

if ($args.length -ne 1) {
  throw "Invalid arguments"
}

$source_path = $args[0]
$local_info = & $Svn info $source_path
if ($LastExitCode -ne 0) {
  throw "Local svn info failed"
}
$local_revision = $local_info | %{ if ($_.startsWith("Revision")) { Write-Output $_ } } | %{ Write-Output $_.substring($_.indexOf(": ")+2) }
$url = $local_info | %{ if ($_.startsWith("Repository Root")) { Write-Output $_ } } | %{ Write-Output $_.substring($_.indexOf(": ")+2) }
$remote_info = & $Svn info $url
if ($LastExitCode -ne 0) {
  throw "Remote svn info failed"
}
$remote_revision = $remote_info | %{ if ($_.startsWith("Revision")) { Write-Output $_ } } | %{ Write-Output $_.substring($_.indexOf(": ")+2) }

if ($local_revision -ne $remote_revision)
{
  svn update $source_path
}
else
{
  Write-Host "Up to date"
}

Here are my few tips for a spamless life:

1. Do not publish your email address anywhere.

2. Do not register your email address anywhere.

3. If you need to register to a website then create a new account. Use this account just for the purpose of registering and enable mail forwarding to your main account. As soon as you start receiving spam from that account, or legitimate but annoying commercial content, disable email forwarding but leave the account alive (so that spammers waste resources).

4. Never click on “un-register” links from emails.

5. Disable HTML viewing especially the viewing of images. You may set up a whitelist from known senders.

6. Use state of the art anti-spam. I really like google gmail spam filters. I forward it to my final email provider that does use good spam-assassin rules.

7. Try to have a core of 2 or 3 different email addresses shared among the people you know (no websites!). When you need to change your email, you will be upsetting fewer people. An idea is to set up groups like: family, friends from university, other friends, colleagues or former colleagues… groups of people you are unlikely to email simultaneously.

8. Update your core email addresses when they are compromised. Do it gradually over an entire year by keeping email forwarding on and adding an auto reply rule informing your contacts they should stop using the old email. Do not write your new email in clear inside the automated message because it could be harvested by bots.

9. If your address is publicly available and spammed in huge amounts, make it mandatory for people to add some text in the title in order to contact you. Filter out everything not complying to the rule.

10. If your address is publicly available and the previous tip does not work (someone is deliberately targeting you) remove the contact email address from public space and replace it with a contact form using captcha.

However, the compiler, emulator and code signing tools work perfectly on older PPC Macs provided you work around the arbitrary limitations of the installer.

My iMac broke down so I had to revert to using my old and venerable PowerBook using Mac OS X 10.5.6 (9G55). I used the first public SDK (md5=96849e4a17674d55d5af75b2d1d6579f). The following tricks might not work with the latest versions.

Installation Steps

The installation steps are simple:

1. Install the iPhone SDK. The installer will install XCode but it will not let you select the iPhone components.
2. Install the missing iPhone packages manually.
   • Those packages are located inside the Packages subdirectory of the SDK installation disk image.
   • You can simply double click on them to start installing.
   • You need to select the /Developer directory as the install location for each of them.
   • The packages to install manually are: DeveloperDiskImage.pkg and all the packages starting by iPhone.
3. Change iPhone Simulator Architectures.xcspec.
   • This allows to compile iPhone apps and to use the simulator on PPC machines.
   • The file is located into “/Developer/Platforms/iPhoneSimulator.platform/Developer/Library/Xcode/Specifications/”.
4. Change /usr/bin/codesign.
   • This allows to upload signed executable to the iPhone.
   • For some mysterious reasons codesign requires a universal binary, so the trick is to make it believe it got one.
   • You need to rename codesign sudo mv /usr/bin/codesign /usr/bin/codesign.orig.
   • Then replace it by the perl script attached below. Don’t forget to give it executable rights: sudo chmod +x /usr/bin/codesign.

That’s it. It should now work.

Files to Replace

• iPhone_Simulator_Architectures.xcspec
• codesign

Links to Original Authors

• http://discussions.apple.com/thread.jspa?threadID=1455699&tstart=0
• http://www.tbradford.org/2008/03/iphone-sdk-beta-2-possible-ppc-fix.html

Updates

Update: it works perfectly with the latest version of the SDK i.e.

iphone_sdk_for_iphone_os_2.2.19m2621afinal.dmg, md5=9d0a818f41be507537495920cd0ef9bc

Update: Obviously this trick no longer works for iOS 3 and more recent versions.

How it Works

Microsoft Authenticode works as follows: it computes a cryptographic hash of the executable file. The hash is then used to make a digital certificate which is authenticated by some trusted authority.

The certificate is attached to the end of the PE executable, in a dedicated section called the Certificate Table. When the executable is loaded, Windows computes the hash value, and compares it to the one attached to the Certificate table. It should “normally” be impossible to change anything in the file without breaking the digital authentication.

However three areas of a PE executable are excluded from the hash computation:

• the Checksum in the optional Windows specific header: 4 bytes.
• the Certificate Table entry in the optional Windows specific header: 8 bytes.
• the Digital Certificate section at the end of the file: variable length.

You should be able to change those area without breaking the signature. I have discovered by accident that it is possible to append an arbitrary amount of data at the end of the Digital Certificate. The data are ignored by both the signature parsing and hash computation algorithms. It works on all version of Windows I tested (2000, XP, Vista), as long as the length of the Certificate Table is correctly increased. The length is stored in two different location: the PE header and the beginning of the certificate table.

How to Add a Payload to a Signed Executable

1. Locate beginning of PE header (PE)
2. Skip COFF header (+=28 bytes)
3. Go to Certification Table Entry in the Windows specific optional PE header (+=120 bytes after COFF; total +=148 bytes)
4. Change size of Certificate Table as defined in IMAGE_DATA_DIRECTORY.Size to add the size of the payload.
5. Go to location defined IMAGE_DATA_DIRECTORY.VirtualAddress. This is the absolute location of the Certificate Table within the file.
6. Change again the size of the header, inside the PKCS1_MODULE_SIGN.dwLength
7. This should normally be the last section in the executable; so go to the end and add payload
8. Possibly calculate the new checksum of the file

Caution: the previous constants are true for the 32 bit x86 versions of Windows. Payload needs to be 64 bits aligned. All the 32 bits constants are of course little endians woo woo!

Documentation Links

• http://www.thehackerslibrary.com/?p=377
• http://msdn.microsoft.com/en-us/library/ms904424.aspx
• http://msdn.microsoft.com/en-us/library/aa448396.aspx

Sources

• AppendPayLoad.tar.bz2