Process forking for rbSFML?

[Groogy]

Well since Ruby's native threading isn't perfect. I got to thinking, if something like Distributed Ruby can share objects over the network, why wouldn't I be able to do something over several processes?

It's way in the future for me to get time to do something major like this. Who knows maybe someone beats me to it. But was interested in hearing peoples thoughts on this. Also Laurent if you like the idea and if I should include it directly into the bindings to compensate real native threading, or if I should keep it separated from the bindings.

For you Laurent that isn't familiar with Ruby. Previously ruby(1.8.x) had green threads so they were managed internally by the VM. But in the latest ruby(1.9.x) they have switched to native threads, BUT there's a GIL(Global interpreter lock) in place which enforces only 1 thread to be running ruby code(though any extensions to ruby using threads can still run). So more or less, if you create a thread in Ruby you do get parallelism, but not the benefits of it. That's what I want to try and achieve.

Anyway my idea is something like:

Code: [Select]


thread = SFML::Thread.new do |thread|
  # This is run in another process, so it isn't really a thread.
  thread[:var] = 5
  thread[:obj] = Object.new
  while thread.continue?
    thread[:var] += 1
    thread.yield # When yielding, the process will look if it got any pending messages.
  end
end

puts thread[:var] # Send a message to the thread that we would like to look at :var
puts thread[:obj].to_s # Any method calls to objects also will go trough inter-process messaging.
object = thread[:obj] # Actually only returning a virtual object wrapping any calls on to the process.


The only problem with this that I can't get around is that there will be A LOT of blocking between the processes. Every time we want data from the "thread" the parent will have to wait for the child to yield and any pending messages processed. Then again, if you need to exchange a lot of data between threads, then things aren't really paralleling.

Also worth noting, this is already more or less supported. You can use distributed ruby, but then all "messaging" will go trough the network in order to reach your process.

[Mon ouïe]

Well since Ruby's native threading isn't perfect.

Notice some Ruby implementations have real threading (JRuby, but no C extensions there ; MacRuby, but it's not portable ; Maybe Rubinius too, I'm unsure about that one)

Code: [Select]

thread.yield # When yielding, the process will look if it got any pending messages.
So, something like fibers on a native process ?

[Groogy]

Yeah exactly, combining Distributed Ruby and fibers ^^
Unless I come up with an idea on how to have the process take care of messages without the developer telling it when there's time for it. Though that might be pretty good for the overall system that the developer is in control of that?

I might be able to do so that when the application sleeps, it first try and process messages, check how much time was used for that and sleep the remaining time.

Edit: Actually, not like fibers. Fibers might have an interface that looks like it but it wouldn't be the same. The process doesn't yield to another process or anything like that. The same process would be working and would continue as soon as the handling of messages are done... Maybe yield is a bad name? People might misinterpret it... Should I call it like process_messages instead?

Also still waiting for Laurent's opinion

Note: Cool thing is that this thread class can be written entirely in ruby code

[Laurent]

Also still waiting for Laurent's opinion

If it's not a strict binding of the C++ features, I don't think it should be part of the official binding. An extension would be better.

[Groogy]

Aight, it will be an extension then. Just need to come up with an awesome name ^^

Played around a little with the idea now while I'm drinking my morning coffee. So far I got this:

NOTE: I'll change names when I come up with something better

Code: [Select]



module SFML
  class Thread
    def run(&block)
      start_process(&block)
    end
   
  private
    def initialize(&block)
      @process = nil
      self.run(&block) if block_given?
    end
   
    def start_process(&block)
      @process = IO.popen('-', 'w+') do |pipe|
        if pipe.nil?
          block.call(self)
        end
      end
    end
   
  end
end

$stderr.puts "#{Process.pid} is the parent!"
t1 = SFML::Thread.new do |thread|
  $stderr.puts "#{Process.pid} is here"
end


On my computer I got:

8371 is the parent!
8373 is here

So just getting the parallel running is simple. Now I just have to create an usable interface and implement the object reference sharing.

[Groogy]

One thing that will be a problem is that stdout and stdin will be piped to the parent process. So the developer won't be able to directly printout text... Maybe I can somehow reroute the $stdout and $stdin in the child process to the parent process streams...

Edit: Some more experimentation showed me that using blocks don't work... The parent process sleeps at the end of the block so I'll have to create an entirely new process without the block and somehow get it to create a local thread object and call the "run" method... I'll try to mimic the SFML C++ thread object.

[Groogy]

I'm wondering Laurent, is inheriting from sf::Thread supported anymore in SFML? The documentation isn't clear on that and I would like to create an interface that mimics the SFML one.

[Laurent]

I'm wondering Laurent, is inheriting from sf::Thread supported anymore in SFML? The documentation isn't clear on that and I would like to create an interface that mimics the SFML one.

Not anymore, the constructor directly takes the function to execute. I guess it's very easy to pass a function to another in Ruby (at least much easier than in C++), so you should do the same and forget inheritance.

[Groogy]

Well since you can create so called anonymous functions easily and method calls are dynamic message dispatch, yeah it's simple  Currently on the bus to the university but I'll create a public interface draft when I get there. I think I'll just call them: Groogy::Thread.

[Groogy]

The public API:

Code: [Select]

def process_meth1(process)
  # do stuff
end

def process_meth2(process, *args)
  # do stuff with arguments. You are not forced to use variable-length arguments.
end

class ProcessObject
  def process_instance_method1(process)
    # do stuff
  end

  def process_instance_method2(process, *args)
    # do stuff with arguments
  end
end

obj = ProcessObject.new

processes = []
processes << Groogy::Process.new(:process_func1)
processes << Groogy::Process.new(:process_func2, arg1, arg2, ..., argN)
processes << Groogy::Process.new(:process_instance_method1, obj)
processes << Groogy::Process.new(:process_instance_method2, obj, arg1, arg2, ..., argN)

for process in processes
  process.launch()
end

running = true
while running == true
  all_finished = true
  for process in processes
    if process[:running] == true
      all_finished = false
    end
  end

  running = false if all_finished == true
end


# Useless at this point but just demonstrating the public API
for process in processes
  process.wait()
end

for process in processes
  process.terminate()
end

Behind it there will be a Groogy::Process::Protocol module defining all message values and how to interpret them. A Groogy::Process::Manager instance for each thread in both child and parent that will handle all virtual calls to another process. A Groogy::Process::VirtualObject that will wrap references to any objects in another process.

Or well that's the big-picture idea for the whole system.