I also have problems trying to activate fullscreen mode.
When I press Enter after setting the fullscreen flag to 1 in the options menu, it tries to go fullscreen... It goes black, I can see the little mouse pointer and move it around... Then it blips away for a single frame like I closed it, returns to black, no cursor for a while...then the cursor reappears and the cycle repeats.
I have an nVidia xfx GeForce 8800 GT 512mb video card, 638MB RAM, running windows xp sp2...