The effect is provided by the ‘unity dialog handler’ plugin.
Unlike the Unity 2D version of the dialogs which look visible ’different’ from a regular dialog window the current Unity build retains the ‘native’ Ambiance/GTK themed look. One would imagine that this is likely to change at some point to match the original mock-up/Unity 2D.
Various aspects of the modal dialog effect can currently be tweaked using the Compiz Configuration Settings Manager, including the strength of the ‘dim’ applied to to parent window.
As ever all Ubuntu 11.10 development updates the features you see and read about are subject to change before the release of Ubuntu 11.10, due mid-October.