Before it can be used, the Composite extension has to be initialized. This is done in the CompositeExtensionInit() function. Beyond that however, the extension also appears to need a PictureScreen (provided by RENDER), this is done by calling fbPictureInit(). Originally I had tried to use the mfbPictureInit()/cfbPictureInit() series of functions, but they did not support Compositing. This is provied in fb/fbpict.c as fbComposite().
mfb and cfb are going away anyway. Don't use them, they have known performance and correctness issues in adddition to being unreadable. -- AdamJackson
Xnest appears to have the same bug as Xvnc (which is loosely based on Xnest and Xvfb), in that it never calls CompositeExtensionInit() and thus crashes when the Composite extension is turned on.
One interesting thing which was discovered was that XAACompositeSrcCopy() was basically an optimized version of XAAComposite() (it seems a bug had been filed for the slow down caused by always using the code in XAAComposite() directly).
Some Render operations can be reduced to simpler forms based on the format of the source, mask, and destination pictures. This work should be landing in the generic Render code shortly, which will make the XAA special-casing redundant. -- AdamJackson
There appears to be more code involved when moving windows around. Specifically, when running under Xvnc rendering errors occur in the pixmap used to store the images drawn onto the window. Currently the workaround is to re-expose the window every time it is moved.