Welcome, Guest. Please login or register. Did you miss your activation email?

Author Topic: Thread creation starts to fail.  (Read 6151 times)

0 Members and 1 Guest are viewing this topic.

TMKCodes

  • Newbie
  • *
  • Posts: 11
    • View Profile
Thread creation starts to fail.
« on: June 07, 2011, 02:59:08 pm »
I wrote a threaded multiplayer networking, but my threads stopped being created when client connected so i wrote test which reproduces the problem, but with just creating threads code.

Code: [Select]

#include <iostream>
#include <SFML/System.hpp>

void thread_one(int* counter) {
for(int i = 5; i < 100; i++) {
sf::Sleep(500);
}
}

void thread_two(int* counter) {
for(int i = 5; i < 100; i++) {
sf::Sleep(500);
}
}

int main() {
int counter = 2;
while(true) {
sf::Sleep(1000);
sf::Thread * threadOne = new sf::Thread(&thread_one, &counter);
sf::Thread * threadTwo = new sf::Thread(&thread_two, &counter);
threadOne->Launch();
threadTwo->Launch();
std::cout << "Two threads launched!" << counter <<  std::endl;
counter += 2;
}
}

This code starts failing to create threads after few hundred threads and i will be needing like 4000 threads.

I'm running linux so it creates pthreads and gives me the error message from the linux version of Launch() function. Oh yeah and i'm using SFML 2.

Care to help?

Laurent

  • Administrator
  • Hero Member
  • *****
  • Posts: 32498
    • View Profile
    • SFML's website
    • Email
Thread creation starts to fail.
« Reply #1 on: June 07, 2011, 03:06:18 pm »
There's a limit to the number of threads that can be created at the same time.

4000 is an insane number, your CPU will spend more time managing the threads than executing the code they contain.
Laurent Gomila - SFML developer

TMKCodes

  • Newbie
  • *
  • Posts: 11
    • View Profile
Thread creation starts to fail.
« Reply #2 on: June 07, 2011, 03:11:17 pm »
Yes, but my kernel limits the thread count to 15 thousand and not just to few hundred.

Laurent

  • Administrator
  • Hero Member
  • *****
  • Posts: 32498
    • View Profile
    • SFML's website
    • Email
Thread creation starts to fail.
« Reply #3 on: June 07, 2011, 03:18:42 pm »
There's probably another limit. Anyway, SFML simply calls pthread_create without any additional check, so if it fails it's a limit from the OS, not from SFML.

I've found this after searching a little bit:
Quote
The maximum number of threads available is determined by the minimum of:

      -  The user processes setting (ulimit -u) in /etc/security/limits.conf

      -  The limit MAX_TASKS_PER_USER defined in /usr/include/linux/tasks.h  (change requires Linux kernel to be recompiled)

      -  The limit PTHREAD_THREADS_MAX defined in libpthreads.so (change requires Linux kernel to be recompiled)
Laurent Gomila - SFML developer

TMKCodes

  • Newbie
  • *
  • Posts: 11
    • View Profile
Thread creation starts to fail.
« Reply #4 on: June 07, 2011, 03:27:31 pm »
ulimit -u returns 1024. Well maybe i have to use less threads then. :P

Laurent

  • Administrator
  • Hero Member
  • *****
  • Posts: 32498
    • View Profile
    • SFML's website
    • Email
Thread creation starts to fail.
« Reply #5 on: June 07, 2011, 03:30:04 pm »
Definitely not a bad idea... :P
Laurent Gomila - SFML developer

Groogy

  • Hero Member
  • *****
  • Posts: 1469
    • MSN Messenger - groogy@groogy.se
    • View Profile
    • http://www.groogy.se
    • Email
Thread creation starts to fail.
« Reply #6 on: June 07, 2011, 10:11:55 pm »
What kind of computer will you be sitting on? IF you don't got enough cores then 4000 threads will have a huge impact on the performance and actually time slicing/green threading like 2-10 threads would be faster.
Developer and Maker of rbSFML and Programmer at Paradox Development Studio

Reiv

  • Newbie
  • *
  • Posts: 13
    • View Profile
Thread creation starts to fail.
« Reply #7 on: June 11, 2011, 03:53:43 pm »
Since you are talking about threads than hypothetically what is better?
To make 1 thread for every area of mmorpg to ensure that every area is handled "at the same time" (+-100 threads) or to make few threads that would handle the same job? (I thinked concept in which main thread writes to shared proms and other threads can only read from them, so teoreticcaly scenario in which deadlock isnt problem)?

edit: on eg. 4-core 2+Ghz processor

Groogy

  • Hero Member
  • *****
  • Posts: 1469
    • MSN Messenger - groogy@groogy.se
    • View Profile
    • http://www.groogy.se
    • Email
Thread creation starts to fail.
« Reply #8 on: June 11, 2011, 04:19:08 pm »
Since you only have 4 cores things won't be handled at the same time. If the CPU got Hyperthreading then a maximum of 8 hardware threads can be run at the same time with 4 cores.

Look only way to have so many threads at the same time is by having a super computer with enough cores. Those are pretty expensive since they are not out in the nearby computer shop.

Also having so many threads... how do you synchronize them? There is a law by some computer scientist. Don't remember his name but the law is based pretty much as the application can't be faster than it's slowest synchronization point. Pretty soon the application will hit a roof on how much faster it can go just by paralleling using threads. Because at some point the threads has to stop up and synchronize the data.

So if we have a thread which takes 25% of the time, another thread takes 10% of the time and a third that takes 30% of the time. And all these threads synchronize, the second thread will have to wait 20% before it can synchronize. And this only scale up according to the law. I'll see if I can find it.

I think the accepted number of threads is: (available hardware threads) + (1-2)
Why we add 1-2 is because we will need one or two extra small threads that does nearly nothing. Like the main thread is a common one. I for instance use main to synchronize the threads and pull input. This process is really small and does nearly nothing so it hogs no CPU time from the other threads.
Developer and Maker of rbSFML and Programmer at Paradox Development Studio

Ceylo

  • Hero Member
  • *****
  • Posts: 2325
    • View Profile
    • http://sfemovie.yalir.org/
    • Email
Thread creation starts to fail.
« Reply #9 on: June 11, 2011, 04:50:12 pm »
Quote from: "Groogy"
Look only way to have so many threads at the same time is by having a super computer with enough cores. Those are pretty expensive since they are not out in the nearby computer shop.


Actually you can already find hundreds of cores in GPUs. But programming for the GPU is not the easiest task, though there are some new API like OpenCL now. But anyway the synchronization issue is still the same.
Want to play movies in your SFML application? Check out sfeMovie!

Reiv

  • Newbie
  • *
  • Posts: 13
    • View Profile
Thread creation starts to fail.
« Reply #10 on: June 11, 2011, 05:12:58 pm »
Quote from: "Groogy"
...


Thanks for reply, I am just at the phase of making design, and learning what is best(also testing some ideas on examples I create), so I will know which way will be the best for me.

And when i saw how easiest it is to work with threads than to work with processes (at least for me, shared memory, mutex etc.. i hate that  :? ) I tried to make design in which threads works with their own proms and communicate with other threads by pipes/sockets etc.
But now I get that 1 thread that makes 1000 loop  != 10 threads making 100 loopes each