It's likely that this is partly just luck.
If both buffers are identical, there will be no flickering. However, to make them both identical, you would have to update both identically, which would involve drawing and swapping twice in one frame. This is unreliable; I have not tested this theory.
There doesn't seem to be any reason to attempt this, though. As mentioned above, refreshing everything every frame is not really a problem. However, if it's because you want to only draw things once (rather than updating the window once), you can do this with a render texture. Update the render texture (you draw to it in the same way you draw to a window) whenever something changes, then draw the current render texture to the window each frame. Drawing to a render texture and then drawing that render texture to the window is a common technique as it's required for full game post effects.
I suppose the question that should be asked is:
are you having some trouble with the frame rate by drawing directly to the window every frame?