That Strange Flash Resizing Bug when using 32 bits WebViews on Mac OS X

   Programming, Mac

The Problem

Have you ever noticed some strange behavior when you resize a WebView hosting resizable Flash content? Does the Flash content seems to be play catch up with the resizing of the WebView?

If so you are particularly unlucky. The problem only arises if you have the following set of conditions:

  • The hosted flash content is resized when the WebView itself is resized, a bit like in this website.
  • Your process is running in 32 bit mode. The 64 bit mode seems to be immune, probably because Flash is hosted in an external process.
  • You app is running on 10.5 or 10.6. Other versions are immune, including 10.7.
  • You are running Flash 10.1 or later which includes the Core Animation rendering model. At the time I write this, the latest version of Flash is 11.2.202.235 and it still has the bug.

Caused by Core Animation?

The resizing behavior made me think of Core Animation implicit animations. Core Animation introduced the notion of “layers” to the already existing “view” hierarchy in Cocoa. Views backed by layers are rendered using a more efficient hardware accelerated rendering pipeline than “vanilla” views. Each layer is associated to a set of properties such as position, opacity, etc… and whenever a property is changed, by default, the system generates an implicit animation. For instance, if you make the layer visible, you will have a fade-in effect. And you guessed it, if you resize it, some kind of resize animation. This approach makes it possible to make eye candy animations without introducing much complexity to the code. And of course, Core Animation makes it possible to make explicit animations for more complex cases…

My suspicion was reinforced when I discovered that older versions of the Flash plugin (10.0) which do not use the Core Animation rendering model were exempt from the problem. Core Animation According to the blog of an Adobe employee, three rendering models are offered to plugins on the Mac platform: QuickDraw, Quartz and CoreAnimation. I quickly corroborated this by looking at Mozilla’s Plugin API documentation, and I discovered the notes on NPDrawingModelCoreAnimation. The drawing model of a plugin is chosen after a simple negotiation. First, the plugin checks what the browser advertises, and decides to choose the best model. Second, the browser checks what rendering model the plugin has chosen. In the case of Core Animation, it appears that the plugin creates its own CALayer and provides it to the browser via NPP_GetValue.

To confirm my suspicions, I needed to access the layer returned by the plugin. But this turned out to be more complicated than I anticipated. I could not find any public API in the WebKit that gives me access to the “plugin instance”, which is the first parameter of NPP_GetValue. And since there must be one plugin instance per Flash object in the DOM, loading the flash plugin bundle directly from “Internet Plugin” would do no good. So, after reviewing the source code of the WebKit I opted to use the pluginLayer method from the WebNetscapePluginView class. Both the method and the class and private APIs, but they looked like a fairly stable choice given the circumstances: the WebNetscapePluginView class name is hardcoded to WebNetscapePluginDocumentView via a macro because of an old bug in Acrobat’s plugin, and the pluginLayer method looks indispensable.

The next steps consisted in disabling the implicit animations. If you modify the property of a layer yourself, you can use CATransaction. However in my case, the bounds property of the layer is modified internally by the WebView, as it reacts to the resize event. So the only options available are:

  • Use setAction: on CALayer to set a new array of actions, which are set to NSNull
  • Associate a delegate via setDelegate: to the CALayer and implement actionForLayer:forKey: to return NSNull.
  • Or override the actionForKey: method for the target layer, via some kind of dark voodoo method swizzling.

I opted to use the least intrusive methods, which is to use setActions:. I soon discovered that it did not work, and that in fact the layer given by the plugin already had Null actions. However, by digging in the layer hierarchy, I found out that 2 levels below the plugin root layer, the plugin had created a layer from a class named FP_FPCAOpenGLLayer, which did not override the default actions. There was my problem and as suspected it was a lame Flash bug.

Fix / Workaround

In my Cocoa application I intercept the webView:didFinishLoadForFrame: from the WebFrameLoadDelegate. There I run my own FixFlashResizingBug() function, which does the following:

  • Browse the view hierarchy and look for WebNetscapePluginDocumentView.
  • For each WebNetscapePluginDocumentView it finds, try to call the pluginLayer method
  • Then drill two levels down the layer hierarchy, and look for a FP_FPCAOpenGLLayer
  • Set the actions of FP_FPCAOpenGLLayer to NSNull.

Obviously this is fragile since it both relies on private WebKit API’s and manipulate internal structures of the Flash plugin directly, but I could not think of a better idea.