• They can hold a partition table, and therefore contain more than one volume. (Typically a partition table with a single partition).
• They may contain any kind of filesystem. (Typically HFS+).
• Images can be compressed (via bzip2, zlib, or Apple’s own format for the old ones).
• Images can be made as read-only, but also as read-write. (Installer images are typically read-only).
• Read-write images can be sparse. That means the image file only grows as more content is put inside, rather than be pre-allocated to the full capacity of the virtual disk.
• Images can have checksums to verify their integrity (options: several CRCs, MD5, several SHAs).
• Images can embed Mac specific behaviors: contain a EULA, have a custom icon, automatically mount when downloaded by Safari, etc…
• And I probably forget a lot of other features (RTFM: “man hdiutils“).

One of the initial motivations for using disk images, during the MacOS classic era, is that application files could not be reliably transmitted over non-Mac specific networks like the Internet. This limitation stems from the fact that files were divided into two logical parts, called “forks”: the “data fork” and the “resource fork”. The data fork is what is traditionally considered to be a file. The structure of the data is entirely determined by the developer. The resource fork on the other hand, contains a collection of data organized in a standard way and accessible through a dedicated API. Although it is possible to define custom resources, the OS would also provide lots of default types to hold classical information: user interface elements, rich text, pictures, etc…

On MacOS X, resource forks are still supported for backward compatibility reasons, but all application now use bundles instead. A bundle appears as a single icon to the end user, but it is in reality a directory containing several files. The role that the resource fork used to play to organize data is now played by the file system: each resource appears as an individual file within a hierarchy of directories. Hence, a typical MacOS X application is composed of several files, and disk images remain useful to solve the problem of their distribution across networks.

In order to safely transport data, and installers, Apple chose to favor the usage of disk images rather self extracting installers. Since the early 90s, all Apple software installation disks have been distributed as either disk images or physical medias. Early Mac users would probably remember either using Apple’s DiskCopy, or Aladdin’s ShrinkWrap, for creating or mounting disk images. The current DMG file format is a direct descendant of the “New Disk Image Format” (aka “NDIS”) that Apple introduced with Disk Copy 6 in 1996. This file format used the resource fork to store all the meta data: file version, partition table, type of compression, checksums, etc… and more importantly where to find the data for the virtual disk sectors. The data fork only contained the raw data (compressed or not) of the virtual disk.

The astute reader may have noticed that, since the NDIS images used forks, it was not suitable for safe redistribution across the Internet. Disk images had typically to be encoded with either BinHex or UUencode. When Apple introduced MacOS X in 2001, they decided to solve the problem once and for all. They designed the DMG file format, which they called “Universal Disk Image Format” (aka “UDIF”). DMG is “flat” file format in the sense that it does not contain any forks, but it is basically a wrapper around the traditional “NDIS” format. A typical DMG file is a concatenation of the data fork, followed by the resource fork of an NDIS disk image.

Apple has never published the specifications of either the NDIS or the UDIF file format. The private DiskImage framework plays a central role in the implementation of DMG parsing and the mounting of disk images in general. The framework apparently supports plugins (see “hdiutil plugins”) but there is no developer documentation available. The only non-Apple plugin known was made by Connectix for VirtualPC images, during the PowerPC era (see Unlocking FireVault). However it does not appear that either VMWare nor Parallels have been able to use the private framework in recent times.

Although it is sad the DiskImage framework is closed, the DMG file format itself, is a bit of an open secret. It has been successfully reverse engineered many times, and there are a few open source implementations:

• Several commercial software support it: MacDrive, PowerISO, MagicISO.
• Vultur’s dmg2iso is probably the older open source implementation. It is a quick and dirty perl script to convert a DMG file into an ISO. It is deprecated, and its replacement is a cleaner and more complete C implementation called dmg2img.
• Erik Larsson provides a series of Java tools that can parse DMG files. There is the simple DMGExtractor, but also the more complete HFSExplorer which provides a graphical interface to browse HFS+ volumes and DMG files.
• Another interesting source is libdmg from planetbeing. It is a C library with UDIF and HFS+ support. It can be used to convert DMG from/to ISOs; extract files; or create DMG images from scratch. The project was initially started as a way to manipulate Apple’s software restore packages for iPhone devices.
• Finally, 7-zip on Windows is able to open and extract DMG files.

My company has set up a VPN access that can be used from home. This proved to be extremely useful last winter when I got snowed in, and I have used it occasionally ever since. The VPN system we use (Sonicwall) is not supported on MacOS X and Sonicwall provides a Windows only connection software. I have tried to use IPSecuritas but I could not find a way to configure it properly. The commercial software VPN Tracker worked like a charm, but when I saw the price (€99 excl VAT) I stopped considering it. For about half of that price ($60 excl VAT) you can get a license to VMWare Fusion. So I decided to go for that instead. The obvious disadvantage in using VMWare is that it uses a lot of resources, but this is not really a problem for me, and it’s kind of hard to configure, but this post should help you with that. The obvious advantage of VMWare is of course that it does much more than provide access to a VPN…

Summary of the setup

My goal is to access the company network (10.100.10.x/24) from home on my iMac running Mac OS X 10.6. Since the VPN software is only supported on Windows, I connect to the VPN via Windows XP SP3 running inside VMWare Fusion. The virtual machine internet connection is provided via a NAT network interface connected to the iMac.

My setup consists in creating another network interface between the iMac and the VM. I then configure both Windows and MacOS X so that my packets get properly routed.

A picture is worth a thousand words.

Overview of the VPN sharing setup

Setting up VMWare Fusion

I assume that you have correctly installed the virtual machine, Windows XP, and that the VPN software works as expected. My network configuration uses NAT to provide internet access to the VM, but it is certainly possible to use other configurations.

The first thing to do is to create a new network interface to provide access to the VPN from the Mac to the VM.

Steps

• Switch off the VM
• Go to “Settings”, “Network”
• Add a network interface with the “+” button at the bottom.
• Choose “The host only” option at the bottom.

Create a network interface in VMWare

Setting up Windows XP

The next part of the setup consists in configuring the newly created “Local Area Network 2″ interface (LAN2) so that it shares the connection with the VPN. We need to share the internet connection from the “Virtual Private Network” interface to the LAN2 interface. We also need to enable IP packet routing.

Steps

Setup of network interfaces on Windows XP

At this stage, although this is not strictly necessary, you may want to enable the ping on Windows XP, to verify that your connection works.

• Go to “Control Panel”, “Windows Firewall”.
• Go to the “advanced” tab and select “settings” in the “ICMP” section.
• Enable the “allow incoming echo request” option.
• On the Mac, open “Terminal.app”
• Type “ping 192.168.245.2″ and the Virtual machine, should normally reply to the ping. If it does not, you may have made a configuration mistake. Use “ifconfig” on the Mac and “ipconfig” on Windows to debug the issue.

Setting up Mac OS X

At this stage we need to set up Mac OS X routing so that VPN addresses are resolved through the vmnet1 interface. We also optionally set up the company DNS. If you have not done it yet, you should connect to the VPN from the virtual machine.

Steps

• Once you are connected to the VPN on Windows, launch “cmd.exe” and type “ipconfig /all”.
• Look for the following details: your IP address on the VPN network, the mask, and the addresses of the VPN.
• In my example I have an IP of 10.100.10.212, a mask of 255.255.255.0 and two DNS servers at addresses 10.100.10.12 and 10.100.10.14
• Go back to Mac OS X.
• In the terminal type a command line similar to “sudo route -n add 10.100.10.0/24 192.168.245.2“. (Or alternatively if you do not wish to worry about the CIDR notation for the mask, “sudo route -n add 10.100.10.0 192.168.245.2 255.255.255.0“)
  • The first partial IP corresponds to the IP of your network (normally the first digits, until the mask is zero).
  • The second IP is the IP of the VM on the LAN2 interface.
  • This line instructs MacOS X to route all IPs from the 10.100.10.x network to the 192.168.245.2 gateway, ie the virtual machine.
• You must the enable IP routing. Type “sudo sysctl -w net.inet.ip.forwarding=1“.

At this stage you should be able to ping yourself on the VPN from the Mac. In my example “ping 10.100.10.212″ should reply. Now what if it does not work

• Try “traceroute 10.100.10.212″. If you see your message routed through your normal internet connection, then something is wrong on the Mac. Otherwise it’s on Windows.
• On the Mac “netstat -rn” will dump the route table.
• Verify that you still have an internet connection, on the Mac and the VM. If you do trash your route table you may lose it entirely. (Try removing your incorrect route with “route delete …” and/or plug in and off your modem connection).

Optional steps to set up the DNS

• Go to “System Preferences” on the Mac.
• Select “Network”.
• Click on “Advanced” at the bottom of the interface that provides your internet connection.
• Go to the “DNS” tab. In the “DNS server” sections add the DNS servers of your VPN, on top of the list. In my case it’s 10.100.10.12 and 10.100.10.14.
• Close, validate and do not forget to “apply” the settings.

Going further…

• You can snapshot the Windows virtual machine to retrieve your settings next time.
• On the Mac you can put the combination of the “route” and “sysctl” commands in a script to further automate the process. Maybe it’s possible to write a launchd script and do it at startup
• I’m not sure how to configure the DNS from the command line, nor how to make the DNS of the company be used only for domains unknown from your regular DNS server.

WhichSize DiskWave is a freeware for MacOS X 10.6 (hence Intel Macs only for now) that can recursively size the directories on your drive. This allows to quickly identify and reclaim wasted space.

It is freely available here: http://diskwave.barthe.ph/

DiskWave was basically hacked in a weekend, but this is not a new idea. I already worked on a similar project, back in 2004, after discovering OmniDiskSweeper. I was appalled by the terrible performance of OmniDiskSweeper and quickly realized I could do a better job. So I learned Cocoa, ObjectiveC, the Carbon APIs, and came up painfully with a prototype only to discover that another software named WhatSize was free and worked very well. I also had a tricky memory leak due to a bug in Cocoa API.  So I quitted.

Fast forward to 2010. OmniDiskSweeper is no longer a shareware, but its performance is still bad.  WhatSize is no longer a freeware. The old freeware version of WhatSize I have is a PowerPC version, but since I moved to Snow Leopard, I felt like I did not want to install Rosetta. So here is DiskWave.

So this time, I did my homework, and it does not look like there is a suitable alternative. Here is the list of similar tools (on MacOS X) I came up with:

• OmniDiskSweeper.
  • Freeware (formerly shareware)
  • Slower because it relies on Cocoa APIs. It also consumes a lot of memory.
  • Does not have many more features than DiskWave. I hope to catch up pretty soon…
• WhatSize
  • Shareware (formerly freeware)
  • Same speed.  Consumes a little bit more of memory, because it supports another feature that requires it
  • Does have many more features than DiskWave. Some of them, I never intend to get: fat binaries removal, locale removals, etc…
  • The thing that annoys me is that you can size/browse only one drive at a time.
• DaisyDisk
  • Shareware
  • New kid on the block. The visualization is really awesome but… I do not find it that practical.
  • If I ever wish to play with CoreAnimation, I may add an animated sunburst view.
• GrandPerspective
  • Opensource
  • Seems to work well and is properly maintained. But I do not like this kind of visualization.
• DiskInventory X
  • Opensource
  • It looks like it is no longer maintained. I do not fancy the GUI. Same as GrandPerspective.

DiskWave being in its early infancy is somewhat limited in terms of features. I hope to gradually upgrade it. If you have ideas of features you would like to see added, just let me know.

Reference:

• DiskWave homepage: http://diskwave.barthe.ph/

Update April 12th 2010:

• Renamed WhichSize into DiskWave after receiving complaints from id design, inc.

When you programatically delete a shortcut from the Desktop folder Windows is “generally” smart enough to update the Desktop and the “recently used programs” section of the Start Menu. But sometimes it does not work and you are left with a “ghost” icon.

As a regular end user, you can hit the F5 key to force a refresh of the Desktop, but for the Start Menu you are left with no alternative than killing/restarting explorer.exe.

I was looking for a better solution for a custom installer application. And after a lot of MSDN scattering and googling around, Eureka, I finally found a magic incantation to programmatically force a refresh of the Desktop.

SHChangeNotify (SHCNE_ASSOCCHANGED, 0, 0, 0);

This works gloriously well, except for one simple detail. When the CRT is not installed, on Windows 2000 only, the LoadLibrary error would also cause a MessageBox to be displayed. This stops the flow of the installer and of course, is not very elegant…

But there is a simple trick to get rid of the MessageBox, using the very obscure SetErrorMode system call.

::SetErrorMode(SEM_FAILCRITICALERRORS);
HMODULE hDll = ::LoadLibrary("CrtCheck.Dll"); // Won't bark
if (hDll)   ; // CRT is installed
else        ; // CRT is NOT installed

Do you guys know a better (yet simple) way to determine if the CRT is installed?

Update: this is not a Windows 2000 specific issue (thanks Ferruccio). It’s also true for older versions of Windows. As far as I know WinXP, Vista and 7 do NOT display a MessageBox under the same circumstances.