Capture image data from portion of the screen?

[Pheonix]

It's okay, I don't want anyone to be unhappy here. 

Maybe I should have pointed out that I am building my first C++ application (if I exclude 'cout << Hello world' and 'SFML Works!').

It's not that I want people to do everything for me and I just lay back. I would just like someone to explain in layman terms how to do things so I can both get the job done as well as understand what I've done.

[Grimshaw]

I have it rendering the screen but it seems to be glitchy, aspect-wise. Let me fix this and I give you the code.

It's not that the concepts present in this problem are hard, but there are tons of details you need to understand first, so you can understand the whole thing. Windows API is like that. It will have a function ready for anything you want to do, but sometimes it will just tricky to get it working because there are little details to using it that are, to say the least, obscure.

[Grimshaw]

       
        // (1) get the device context of the screen
        // Definition: The CreateDC function creates a device context (DC)
        //  ..for a device using the specified name.
        HDC hScreenDC = CreateDCA("DISPLAY", NULL, NULL, NULL);  

        // (2) and a device context to put it in
        // Definition: The CreateCompatibleDC function creates a memory
        //  ..device context (DC) compatible with the specified device.
        HDC hMemoryDC = CreateCompatibleDC(hScreenDC);

        // (3) Gets the users screen size in x and y dimensions
        // Definition: The GetDeviceCaps function retrieves device-specific
        //  ..information for the specified device.
        int x = GetDeviceCaps(hScreenDC, HORZRES);
        int y = GetDeviceCaps(hScreenDC, VERTRES);

        // (4) maybe worth checking these are positive values
        // Definition: The CreateCompatibleBitmap function creates a bitmap
        //  ..compatible with the device that is associated with the specified
        //  ..device context.
        HBITMAP hBitmap = CreateCompatibleBitmap(hScreenDC, x, y);

        // (5) get a new bitmap
        // Definition: The SelectObject function selects an object into the
        //  ..specified device context (DC). The new object replaces the previous
        //  ..object of the same type.
        HBITMAP hOldBitmap = (HBITMAP)SelectObject(hMemoryDC, hBitmap);

        // Definition: The BitBlt function performs a bit-block transfer of the
        //  ..color data corresponding to a rectangle of pixels from the specified
        //  ..source device context into a destination device context.
        BitBlt(hMemoryDC, 0, 0, x, y, hScreenDC, 0, 0, SRCCOPY);

        BITMAP bm;
        ::GetObject( hBitmap , sizeof(bm) , &bm );

        BITMAPINFO bmpInfo;
        bmpInfo.bmiHeader.biSize = sizeof(BITMAPINFOHEADER);
        bmpInfo.bmiHeader.biWidth = bm.bmWidth;
        bmpInfo.bmiHeader.biHeight = -bm.bmHeight;
        bmpInfo.bmiHeader.biPlanes = 1;
        bmpInfo.bmiHeader.biBitCount = 32;
        bmpInfo.bmiHeader.biCompression = BI_RGB;        
        bmpInfo.bmiHeader.biSizeImage = 0;  
        bmpInfo.bmiHeader.biClrImportant = 0;

        COLORREF* pixel = new COLORREF [ bm.bmWidth * bm.bmHeight ];
        GetDIBits( hMemoryDC , hBitmap , 0 , bm.bmHeight , pixel , &bmpInfo , DIB_RGB_COLORS );

        Image captureImage;
        captureImage.create(bm.bmWidth, bm.bmHeight, Color::Black);
        int j = 0;
        for(int y = 0; y < bm.bmHeight; ++y)
        {
                for(int x = 0; x < bm.bmWidth; ++x)
                {
                        COLORREF thisColor = pixel[j++];
                        captureImage.setPixel(x,y, Color(GetBValue(thisColor), GetGValue(thisColor), GetRValue(thisColor)));
                }
        }

        Texture captureTexture;
        captureTexture.loadFromImage(captureImage);

 

And then you may render a sprite normally with captureTexture set. This solution worked flawlessly in my computer. Hope it works in yours too.

[Pheonix]

YAAAY!  Thanks very much Grimshaw 

It works on mine too! Now I just have to get it streaming - though I should hopefully be able to figure that one out on my own.

I do wonder however how efficient it is to load bitmap into an image, then load the image into a texture, then load the texture into a sprite before drawing? (not saying it's wrong... but is this okay?)

[Pheonix]

YAY I got it streaming!!!!!!

Could you please explain how the code works?

EDIT: Updated code. Actually runs better now. Still a tiny bit choppy, but I'm not sure if that's a code problem or maybe I'm maxing out my computer (the capture is after all 1600x900).

Seems like there's many lines of code that I don't need in the stream. Here's my stream code:

/////////// OUR LOOP TO STREAM ///////////////////////////////
                BitBlt(hMemoryDC, 0, 0, x, y, hScreenDC, 0, 0, SRCCOPY);
                GetDIBits( hMemoryDC , hBitmap , 0 , bm.bmHeight , pixel , &bmpInfo , DIB_RGB_COLORS );

                j = 0;
                for(y = 0; y < bm.bmHeight; ++y){
                        for(x = 0; x < bm.bmWidth; ++x){
                                COLORREF thisColor = pixel[j++];
                                captureImage.setPixel(x,y, sf::Color(GetBValue(thisColor), GetGValue(thisColor), GetRValue(thisColor)));
                        }
                }

                captureTexture.loadFromImage(captureImage);
                //////////////////////////////////////////////////////////////

EDIT: Checked the frame-rate, I get about 15FPS. I was hoping to get at least 30-60FPS. Does anyone have any suggestions in how I could speed it up?

[eXpl0it3r]

Screen capture is a rather advanced topic and I don't think you can just call an OS screenshot function and hope to stream that at 30-60 fps. If I'd wanted to stream a screen capture on Windows, I'd most likely start by looking at the source code of Open Broadcaster Software, it's the only software I know that gets a good screen capture going (and it's open source).

[Pheonix]

Why does it slow down? For example...

If I disable updating the texture, I get 30FPS then gradually decreases to about 20FPS
If I enable updating the texture, I get 20FPS > 15FPS.

I think perhaps the way I am streaming it is wrong? (FPS should be stable IMO)
Also, updating the texture seems to drop FPS by 5-10FPS. Maybe there's an alternative? Can I set the texture to some kind of dynamic/streaming mode?

(and yes, this is in 'Release' mode.)

[eXpl0it3r]

Why does it slow down?

Looking at the outcome doesn't help much, instead you should just run a profiler and find out where the time is lost.

Also, updating the texture seems to drop FPS by 5-10FPS. Maybe there's an alternative? Can I set the texture to some kind of dynamic/streaming mode?

Updating the texture is a heavy operation and you can't do anything else with SFML directly. What you actually want to do, to get max performance is locating the "texture" of the screen and copying parts of that to your own texture. I've no idea if it's possible and even less how. I just know it's possible for game capturing, where you inject the game with your code and capture the hook yourself into the back buffer updating.

[Pheonix]

In other words, instead of loading image into texture, actually write the pixel info directly into the already existing texture?

I don't see why it wouldn't be possible... Surely the texture is made up of pixel information..? Just need to find the address in memory and write it in.

EDIT: There's two lines that severely drop the FPS (from a base of 1,300 FPS)

(When commenting out all other code in the loop...)

Drop to about 30FPS

BitBlt(hMemoryDC, 0, 0, x, y, hScreenDC, 0, 0, SRCCOPY);

Drop to about 65FPS

captureImage.setPixel(x,y, sf::Color(GetBValue(thisColor), GetGValue(thisColor), GetRValue(thisColor)));

(this one is in two for loops scanning every pixel)

Drop to about 250FPS

captureTexture.loadFromImage(captureImage);

EDIT: I managed to increase the FPS of the streaming from 15FPS to about 40FPS by disabling the Windows Aero theme!? I suppose this means bitblt has less work to do. But I still must use bitblt otherwise it won't work.

[Laurent]

To get maximum performances you should do the pixel conversions into your own array (std::vector<sf::Uint8>) and then use Texture::update. In other words, drop sf::Image, which is not suitable for realtime image manipulation.

[eXpl0it3r]

In other words, instead of loading image into texture, actually write the pixel info directly into the already existing texture?

I can only speak for Windows and even then I've very limited knowledge. With Windows Vista and thus Aero the whole "desktop" is hardware-accelerated which means that the stuff you see is actually rendered somewhere on the graphics card and thus it should be possible to hook directly into DWM and grab the texture information, but since Microsoft obviously uses DirectX stuff, I've no idea if the textures can get converted on the fly, otherwise you'd still have to go via CPU/RAM, which then would still have a big impact on the performance.

Drop to about 30FPS

Drop to or by?

BitBlt(hMemoryDC, 0, 0, x, y, hScreenDC, 0, 0, SRCCOPY);

BitBlt is known to be slow, especially with Aero being active (see here).

captureTexture.loadFromImage(captureImage);

As Laurent said, you should use your own vector/array of pixels and update the texture with update();

[Pheonix]

To get maximum performances you should do the pixel conversions into your own array (std::vector<sf::Uint8>) and then use Texture::update. In other words, drop sf::Image, which is not suitable for realtime image manipulation.

Do you mean create my own array, then pass a pointer to the texture update function? Is the formatting top left to bottom right? RGBARGBARGBA? What if I don't need alpha channel?

Drop to or by?

Drop to. (Yes, I know it's a very significant drop from 1,300!)

[Laurent]

Do you mean create my own array, then pass a pointer to the texture update function? Is the formatting top left to bottom right? RGBARGBARGBA? What if I don't need alpha channel?

Yes. Yes. Yes. Set alpha components to 255.

[Grimshaw]

Glad you got it working. Your major bottleneck for that application should be the streaming part. Sending over the network such a huge amount of data in realtime is nearly impossible. That's why so much work is put into creating amazing decoders to stream video even in the worst environments. Its all about making the frames as small as possible to send via network, while still being able to decompress them quickly.

Play with it, but don't expect to achieve a top-notch application for this, unless you use good  middleware.

[eXpl0it3r]

Your major bottleneck for that application should be the streaming part. Sending over the network such a huge amount of data in realtime is nearly impossible.

Maybe I got the wrong impression, but with streaming I think he rather meant local "streaming", from screen grab, via SFML to the GPU and the to the screen.