Http error 1001 in multithreaded application

[AB]

I'm using the Parallel Patterns Library to run several http clients in parallel in this manner:

concurrency::parallel_for_each(begin(queries), end(queries), [&](const std::pair<std::string, std::string> &p){
      try
      {
         sf::Http http("some_host_name");
         sf::Http::Request request("some_path_name" + p.second);
         auto response = http.sendRequest(request, sf::seconds(30));
         if (response.getBody().find(found_substring) != std::string::npos)
         {
            results.push(p.first);
         }
                        // some debug info gathering stuff omitted for brevity
      }
      catch (const std::exception &e)
      {
         // error handling code omitted for brevity
      }

   });

Note that both the "queries" and "results" are concurrency safe containers from the PPL.

The problem is that sometimes the above code gets stuck (it doesn't crash, just doesn't work properly). I've done some error checking and I've observed the following:
The problem is related to the SFML http client, which returns error code 1001 and times out. Once the problem occurs, the same thread will continue reporting error 1001 from its http clients. The problem takes some time to appear, usually after several thousand queries have been processed successfully.

Any tips on how I can diagnose and solve the problem?

[binary1248]

One thing I don't understand is... why are you using PPL to run multiple sf::Http requests at the same time? The benefit of PPL is to execute code on accelerators and generally exploit the parallel compute capabilities of the underlying hardware.

Because sf::Http really relies on waiting on a remote server to answer your request, most of the time will be spent waiting anyway and I don't see the benefit of using PPL over a self-made thread pool for instance.

I also vaguely remember that PPL isn't meant for everything and it should only be used with code that is "PPL-compatible" for lack of a better term. Perhaps sf::Http is something that is not that compatible with the way PPL works.

As for the error you are experiencing, it is neither an HTTP status code (they only go up to 599) nor an SFML error code (SFML doesn't use error code numbering) so the only source I can think of is Windows. Checking the System Error Code catalogue on MSDN, 1001 corresponds to ERROR_STACK_OVERFLOW which might be another indicator that PPL and sf::Http don't play nice with each other.

If I were you, I would just drop the parallel_for_each call in favour of something more home-made and probably also more efficient. You can issue HTTP requests yourself using non-blocking sockets or even a thread pool. The HTTP protocol isn't even that hard to understand since everything is transmitted in plain text. If you want even more performance (since you are issuing a large amount of requests) you can even implement support for a persistent HTTP connection and even request pipelining if you feel you can benefit from it.

[AB]

I'm using PPL so I don't have to write my own code to safely divide the queries between several threads.

By the way, error 1001 is sf::Http::Response::ConnectionFailed

http://sfml-dev.org/documentation/2.1/classsf_1_1Http_1_1Response.php

[binary1248]

Hmm... I must have overlooked that SFML defines its own HTTP status codes...

In that case you can probe for that code and connect your own sf::TcpSocket in the thread that fails and see what error code the socket returns from the connect. It's basically what sf::Http does anyway except that you have access to the return value of the connect() call.

Chances are, since you issue so many requests on separate connections to the same server, either it doesn't want to process any more of your requests, or a firewall kicked you out. If you are using your own server for testing it is a different story...

[AB]

Hmm... I must have overlooked that SFML defines its own HTTP status codes...

In that case you can probe for that code and connect your own sf::TcpSocket in the thread that fails and see what error code the socket returns from the connect. It's basically what sf::Http does anyway except that you have access to the return value of the connect() call.

Chances are, since you issue so many requests on separate connections to the same server, either it doesn't want to process any more of your requests, or a firewall kicked you out. If you are using your own server for testing it is a different story...

I'm testing on a public server. It has always been cooperative and returns status code 200, if a connection can be established. Maybe it imposes a limit on simultaneous connections? In any case, analyzing the socket sounds like a good idea.

[Joshua Flynn]

1001 is an internal SFML error code (it's not valid HTTP error code).

You might be having a similar problem I had with SFML HTTP code in that if a connection doesn't exist/website down/internet connection down on a blocking or non-blocking socket, the program can either freeze (blocking) or have trouble connecting. I had to modify SFML 1.6's HTTP code so it returned true if a connection already existed on a non-blocking socket (SFML 1.6 classifies all WSAERROR as errors: the name is misleading, though, because one such 'error' is 'already connected': WSAEISCONN).

I don't know if it's been fixed for 2.0 and 2.1, but if you're running 1.6 with non-blocking sockets (especially if you're using threaded processes) you'll run into problems.

Also, be aware that winsockets (I'm assuming it's window sockets, if not, disregard this), on the lowest level, can have a 'maximum number of connections' (I can't tell you much about it off the top of my head) that can operate at any one time, and the setting has to be changed on the lowest level to allow it to handle more. You could also be inadvertently triggering this but it'd be hard to say. Only throwing it out there in-case it helps.