Requesting a Stencil Buffer on sf::RenderTexture?

[Clairvoire]

Hello!  I use stenciling a lot for lighting effects.  I can request a stencil buffer through sf::RenderWindow by passing it a ContextSettings instance (8bits stencil, 24bits depth).  However, this same functionality doesn't seem to be present with sf::RenderTexture, unless I'm mistaken of course! 

I was wondering if there was any work around?

RenderTextures do allow requesting a depth buffer, though only as a boolean (can't specify the bitplane count).  Took a look to see if I could easily add a stencil request myself, but it lead me into 2 difference -Impl.cpp type files, which I'm hesitant to mess with, aha...

[Clairvoire]

Went through the source with a crayon scrawling recklessly.  I think I have my work around now!  At least, it doesn't crash the program by some miracle while doing what I wanted.  Thought I'd share it here if anyone was interested.

Made 2 small edits in sfml, and added a small namespace named 'clair' (so I can search it easily if something explodes) that contains a boolean that activates the code needed for stencils.  I just flip the bool to true before creating a RenderTexture, then turn it back off after it's done.  Yeah... it's pretty ugly 

The destructor seems to clean up the FBO version regardless of which branch is taken (but I might be fatally mistaken).

RenderTextureImplFBO.cpp

bool RenderTextureImplFBO::create(unsigned int width, unsigned int height, unsigned int textureId, bool depthBuffer)
{
        ...
       
        if (depthBuffer)
        {
                ...
               
                glCheck(glBindRenderbufferEXT(GL_RENDERBUFFER_EXT, m_depthBuffer));
                if(clair::enableStencilOnRenderTextureCreation){
                        // edited to allow stencils  --clair
                        glCheck(glRenderbufferStorageEXT(GL_RENDERBUFFER_EXT, GL_DEPTH24_STENCIL8_EXT, width, height));
                        glCheck(glFramebufferRenderbufferEXT(GL_FRAMEBUFFER_EXT, GL_DEPTH_ATTACHMENT_EXT, GL_RENDERBUFFER_EXT, m_depthBuffer));
                        glCheck(glFramebufferRenderbufferEXT(GL_FRAMEBUFFER_EXT, GL_STENCIL_ATTACHMENT_EXT, GL_RENDERBUFFER_EXT, m_depthBuffer));
                }else{
                        // original code moved to else branch
                        glCheck(glRenderbufferStorageEXT(GL_RENDERBUFFER_EXT, GL_DEPTH_COMPONENT, width, height));
                        glCheck(glFramebufferRenderbufferEXT(GL_FRAMEBUFFER_EXT, GL_DEPTH_ATTACHMENT_EXT, GL_RENDERBUFFER_EXT, m_depthBuffer));
                }
                ...
        }
        ...
}
 

RenderTextureImplDefault.cpp

bool RenderTextureImplDefault::create(unsigned int width, unsigned int height, unsigned int, bool depthBuffer)
{
    ...

        // Create the in-memory OpenGL context
        if(clair::enableStencilOnRenderTextureCreation){
                // edited to allow stencils  --clair
                m_context = new Context(ContextSettings(24, 8, 0), width, height);
        }else{
                // original code moved to else branch
                m_context = new Context(ContextSettings(depthBuffer ? 32 : 0), width, height);
        }
        return true;
}
 

[Clairvoire]

Sorry to triple post!  Just wanted to report my results after testing it out. 

It works as intended it seems!  It does require using raw openGL calls to manipulate the stencil testing.  glStencilOp(), glStencilFunc(), and glEnable/glDisable(GL_STENCIL_TEST).  And I had to add a small overload to sf::RenderTexture::Clear() so it could take a second argument for clearing the stencil buffer.  But that's not too bad!  It doesn't leak any resources either, thankfully! 

I was messing around inside SFML's shader example to test some things with it.  Got some interesting-ish effects by just mixing a few of the shading examples.



They're the result of having one shader draw to the stencil buffer, then having two other shaders draw normally, one inside the stencil, and one outside the stencil.  So they're pretty random admittedly, but I was just mixing them at random to see what would happen.  With some thought though, it should make lighting, shadows, masking, and reflections a piece of cake (hopefully)!

[BlueCobold]

I just wanted to say thanks for helping me where to add the proper stencil support for RenderTextures. I guess I would have missed the FBO-version.
Some more stuff is required to work properly though. The "clear" method needs to also check for enable stencil buffering to set the right flag (GL_STENCIL_BUFFER_BIT) and to set the correct StencilMask before clearing (typically 0xFF).
Great post nonetheless!