Aymeric on software

Have you ever noticed some strange behavior when you resize a WebView hosting resizable Flash content? Does the Flash content seems to be play catch up with the resizing of the WebView?

If so you are particularly unlucky. The problem only arises if you have the following set of conditions:

• The hosted flash content is resized when the WebView itself is resized, a bit like in this website.
• Your process is running in 32 bit mode. The 64 bit mode seems to be immune, probably because Flash is hosted in an external process.
• You app is running on 10.5 or 10.6. Other versions are immune, including 10.7.
• You are running Flash 10.1 which includes the Core Animation rendering model. At the time I write this, the latest version of Flash is 11.1.102.55 and it still has the bug.

The resizing behavior made me think of Core Animation implicit animations. Core Animation introduced the notion of ‘layers’ to the already existing ‘view’ hierarchy in Cocoa. Views backed by layers are rendered using a more efficient hardware accelerated rendering pipeline than ‘vanilla’ views. Each layer is associated to a set of properties such as position, opacity, etc… and whenever a property is changed, by default, the system generates an implicit animation. For instance, if you make the layer visible, you will have a fade-in effect. And you guessed it, if you resize it, some kind of resize animation. This approach makes it possible to make eye candy animations without introducing much complexity to the code. And of course, Core Animation makes it possible to make explicit animations for more complex cases…

My suspicion was reinforced when I discovered that older versions of the Flash plugin (10.0) which do not use the Core Animation rendering model were exempt from the problem.

According to the blog of an Adobe employee, three rendering models are offered to plugins on the Mac platform: QuickDraw, Quartz and CoreAnimation. I quickly corroborated this by looking at Mozilla’s Plugin API documentation, and I discovered the notes on NPDrawingModelCoreAnimation. The drawing model of a plugin is chosen after a simple negotiation. First, the plugin checks what the browser advertises, and decides to choose the best model. Second, the browser checks what rendering model the plugin has chosen. In the case of Core Animation, it appears that the plugin creates its own CALayer and provides it to the browser via NPP_GetValue.

To confirm my suspicions, I needed to access the layer returned by the plugin. But this turned out to be more complicated than I anticipated. I could not find any public API in the WebKit that gives me access to the “plugin instance”, which is the first parameter of NPP_GetValue. And since there must be one plugin instance per Flash object in the DOM, loading the flash plugin bundle directly from “Internet Plugin” would do no good. So, after reviewing the source code of the WebKit I opted to use the pluginLayer method from the WebNetscapePluginView class. Both the method and the class and private APIs, but they looked like a fairly stable choice given the circumstances: the WebNetscapePluginView class name is hardcoded to ‘WebNetscapePluginDocumentView‘ via a macro because of an old bug in Acrobat’s plugin, and the ‘pluginLayer‘ method looks indispensable.

The next steps consisted in disabling the implicit animations. If you modify the property of a layer yourself, you can use CATransaction. However in my case, the ‘bounds’ property of the layer is modified internally by the WebView, as it reacts to the resize event. So the only options available are:

• Use setAction: on CALayer to set a new array of actions, which are set to NSNull
• Associate a delegate via setDelegate: to the CALayer and implement actionForLayer:forKey: to return NSNull.
• Or override actionForKey: method for the target layer, via some kind of dark voodoo method swizzling.

I opted to use the least intrusive methods, which is to use setActions. I soon discovered that it did not work, and that in fact the layer given by the plugin already had Null actions. However by digging in the layer hierarchy, I found out that 2 levels below the plugin root layer, the plugin had created a layer from a class named ‘FP_FPCAOpenGLLayer‘, which did not override the default actions. There was my problem and as suspected it was a lame Flash bug.

Fix / Workaround

In my Cocoa application I intercept the webView:didFinishLoadForFrame: from the WebFrameLoadDelegate. There I run my own FixFlashResizingBug() function, which does the following:

• Browse the view hierarchy and look for “WebNetscapePluginDocumentView“.
• For each “WebNetscapePluginDocumentView” it finds, try to call the “pluginLayer” method
• Then drill two levels down the layer hierarchy, and look for a “FP_FPCAOpenGLLayer“
• Set the actions of “FP_FPCAOpenGLLayer“ to NSNull.

Obviously this is fragile since it both relies on private WebKit API’s and manipulate internal structures of the Flash plugin directly, but I could not think of a better idea.

Aymeric Uncategorized

Disk images have always been popular on the Mac. They play a similar role to “ISO” images on other platforms, in the sense that they can be used to clone medias or can be mounted/unmounted as a virtual disks. However DMG images offer a LOT more functionalities than traditional ISO images:

• They can hold a partition table, and therefore contain more than one volume. (Typically a partition table with a single partition).
• They may contain any kind of filesystem. (Typically HFS+).
• Images can be compressed (via bzip2, zlib, or Apple’s own format for the old ones).
• Images can be made as read-only, but also as read-write. (Installer images are typically read-only).
• Read-write images can be sparse. That means the image file only grows as more content is put inside, rather than be pre-allocated to the full capacity of the virtual disk.
• Images can have checksums to verify their integrity (options: several CRCs, MD5, several SHAs).
• Images can embed Mac specific behaviors: contain a EULA, have a custom icon, automatically mount when downloaded by Safari, etc…
• And I probably forget a lot of other features (RTFM: “man hdiutils“).

One of the initial motivations for using disk images, during the MacOS classic era, is that application files could not be reliably transmitted over non-Mac specific networks like the Internet. This limitation stems from the fact that files were divided into two logical parts, called “forks”: the “data fork” and the “resource fork”. The data fork is what is traditionally considered to be a file. The structure of the data is entirely determined by the developer. The resource fork on the other hand, contains a collection of data organized in a standard way and accessible through a dedicated API. Although it is possible to define custom resources, the OS would also provide lots of default types to hold classical information: user interface elements, rich text, pictures, etc…

On MacOS X, resource forks are still supported for backward compatibility reasons, but all application now use bundles instead. A bundle appears as a single icon to the end user, but it is in reality a directory containing several files. The role that the resource fork used to play to organize data is now played by the file system: each resource appears as an individual file within a hierarchy of directories. Hence, a typical MacOS X application is composed of several files, and disk images remain useful to solve the problem of their distribution across networks.

In order to safely transport data, and installers, Apple chose to favor the usage of disk images rather self extracting installers. Since the early 90s, all Apple software installation disks have been distributed as either disk images or physical medias. Early Mac users would probably remember either using Apple’s DiskCopy, or Aladdin’s ShrinkWrap, for creating or mounting disk images. The current DMG file format is a direct descendant of the “New Disk Image Format” (aka “NDIS”) that Apple introduced with Disk Copy 6 in 1996. This file format used the resource fork to store all the meta data: file version, partition table, type of compression, checksums, etc… and more importantly where to find the data for the virtual disk sectors. The data fork only contained the raw data (compressed or not) of the virtual disk.

The astute reader may have noticed that, since the NDIS images used forks, it was not suitable for safe redistribution across the Internet. Disk images had typically to be encoded with either BinHex or UUencode. When Apple introduced MacOS X in 2001, they decided to solve the problem once and for all. They designed the DMG file format, which they called “Universal Disk Image Format” (aka “UDIF”). DMG is “flat” file format in the sense that it does not contain any forks, but it is basically a wrapper around the traditional “NDIS” format. A typical DMG file is a concatenation of the data fork, followed by the resource fork of an NDIS disk image.

Apple has never published the specifications of either the NDIS or the UDIF file format. The private DiskImage framework plays a central role in the implementation of DMG parsing and the mounting of disk images in general. The framework apparently supports plugins (see “hdiutil plugins”) but there is no developer documentation available. The only non-Apple plugin known was made by Connectix for VirtualPC images, during the PowerPC era (see Unlocking FireVault). However it does not appear that either VMWare nor Parallels have been able to use the private framework in recent times.

Although it is sad the DiskImage framework is closed, the DMG file format itself, is a bit of an open secret. It has been successfully reverse engineered many times, and there are a few open source implementations:

• Several commercial software support it: MacDrive, PowerISO, MagicISO.
• Vultur’s dmg2iso is probably the older open source implementation. It is a quick and dirty perl script to convert a DMG file into an ISO. It is deprecated, and its replacement is a cleaner and more complete C implementation called dmg2img.
• Erik Larsson provides a series of Java tools that can parse DMG files. There is the simple DMGExtractor, but also the more complete HFSExplorer which provides a graphical interface to browse HFS+ volumes and DMG files.
• Another interesting source is libdmg from planetbeing. It is a C library with UDIF and HFS+ support. It can be used to convert DMG from/to ISOs; extract files; or create DMG images from scratch. The project was initially started as a way to manipulate Apple’s software restore packages for iPhone devices.
• Finally, 7-zip on Windows is able to open and extract DMG files.

Aymeric MacOS X, UDIF, dmg

Rationale

My company has set up a VPN access that can be used from home. This proved to be extremely useful last winter when I got snowed in, and I have used it occasionally ever since. The VPN system we use (Sonicwall) is not supported on MacOS X and Sonicwall provides a Windows only connection software. I have tried to use IPSecuritas but I could not find a way to configure it properly. The commercial software VPN Tracker worked like a charm, but when I saw the price (€99 excl VAT) I stopped considering it. For about half of that price ($60 excl VAT) you can get a license to VMWare Fusion. So I decided to go for that instead. The obvious disadvantage in using VMWare is that it uses a lot of resources, but this is not really a problem for me, and it’s kind of hard to configure, but this post should help you with that. The obvious advantage of VMWare is of course that it does much more than provide access to a VPN…

Summary of the setup

My goal is to access the company network (10.100.10.x/24) from home on my iMac running Mac OS X 10.6. Since the VPN software is only supported on Windows, I connect to the VPN via Windows XP SP3 running inside VMWare Fusion. The virtual machine internet connection is provided via a NAT network interface connected to the iMac.

My setup consists in creating another network interface between the iMac and the VM. I then configure both Windows and MacOS X so that my packets get properly routed.

A picture is worth a thousand words.

Setting up VMWare Fusion

I assume that you have correctly installed the virtual machine, Windows XP, and that the VPN software works as expected. My network configuration uses NAT to provide internet access to the VM, but it is certainly possible to use other configurations.

The first thing to do is to create a new network interface to provide access to the VPN from the Mac to the VM.

Steps

• Switch off the VM
• Go to “Settings”, “Network”
• Add a network interface with the “+” button at the bottom.
• Choose “The host only” option at the bottom.

Setting up Windows XP

The next part of the setup consists in configuring the newly created “Local Area Network 2″ interface (LAN2) so that it shares the connection with the VPN. We need to share the internet connection from the “Virtual Private Network” interface to the LAN2 interface. We also need to enable IP packet routing.

Steps

At this stage, although this is not strictly necessary, you may want to enable the ping on Windows XP, to verify that your connection works.

• Go to “Control Panel”, “Windows Firewall”.
• Go to the “advanced” tab and select “settings” in the “ICMP” section.
• Enable the “allow incoming echo request” option.
• On the Mac, open “Terminal.app”
• Type “ping 192.168.245.2″ and the Virtual machine, should normally reply to the ping. If it does not, you may have made a configuration mistake. Use “ifconfig” on the Mac and “ipconfig” on Windows to debug the issue.

Setting up Mac OS X

At this stage we need to set up Mac OS X routing so that VPN addresses are resolved through the vmnet1 interface. We also optionally set up the company DNS. If you have not done it yet, you should connect to the VPN from the virtual machine.

Steps

• Once you are connected to the VPN on Windows, launch “cmd.exe” and type “ipconfig /all”.
• Look for the following details: your IP address on the VPN network, the mask, and the addresses of the VPN.
• In my example I have an IP of 10.100.10.212, a mask of 255.255.255.0 and two DNS servers at addresses 10.100.10.12 and 10.100.10.14
• Go back to Mac OS X.
• In the terminal type a command line similar to “sudo route -n add 10.100.10.0/24 192.168.245.2“. (Or alternatively if you do not wish to worry about the CIDR notation for the mask, “sudo route -n add 10.100.10.0 192.168.245.2 255.255.255.0“)
  • The first partial IP corresponds to the IP of your network (normally the first digits, until the mask is zero).
  • The second IP is the IP of the VM on the LAN2 interface.
  • This line instructs MacOS X to route all IPs from the 10.100.10.x network to the 192.168.245.2 gateway, ie the virtual machine.
• You must the enable IP routing. Type “sudo sysctl -w net.inet.ip.forwarding=1“.

At this stage you should be able to ping yourself on the VPN from the Mac. In my example “ping 10.100.10.212″ should reply. Now what if it does not work

• Try “traceroute 10.100.10.212″. If you see your message routed through your normal internet connection, then something is wrong on the Mac. Otherwise it’s on Windows.
• On the Mac “netstat -rn” will dump the route table.
• Verify that you still have an internet connection, on the Mac and the VM. If you do trash your route table you may lose it entirely. (Try removing your incorrect route with “route delete …” and/or plug in and off your modem connection).

Optional steps to set up the DNS

• Go to “System Preferences” on the Mac.
• Select “Network”.
• Click on “Advanced” at the bottom of the interface that provides your internet connection.
• Go to the “DNS” tab. In the “DNS server” sections add the DNS servers of your VPN, on top of the list. In my case it’s 10.100.10.12 and 10.100.10.14.
• Close, validate and do not forget to “apply” the settings.

Going further…

• You can snapshot the Windows virtual machine to retrieve your settings next time.
• On the Mac you can put the combination of the “route” and “sysctl” commands in a script to further automate the process. Maybe it’s possible to write a launchd script and do it at startup
• I’m not sure how to configure the DNS from the command line, nor how to make the DNS of the company be used only for domains unknown from your regular DNS server.

Aymeric MacOS X, Windows, vmware