Best way to structure a continuous running, high load process?

[tibberous]

I have a web app that needs to continuously upload large amounts of data to a server. What to upload and when to upload it is stored in a database, but it needs to do this continuously, and needs to be pretty fault resistant.

 

I was thinking cron jobs, but it really needs to run all the time, not at regular intervals. Is there some way so I can make it so it runs constantly, processing it's queue? Also, could it use a new process to handle the queue items? So if there are three items in the queue, it will make three processes, but still keep processing?

 

Thanks for any help, I'm going to look at multithreading...

[btherl]

I don't think php does multithreading, but it does do multi-process.  I have written several php daemons which spawn children to do jobs that need to be done simultaneously.  You have to be careful, as resources held by the parent will be closed if the child exits cleanly, as the child also believes it holds those resources.  The solution here is NOT to exit cleanly.  Instead you can exit with this:

 

pcntl_exec('/bin/true');

 

This will kill your child without closing any resources shared with the parent.

 

My main loop is like this:

 

while (true) {
  Any kids exited?  If no, wait a while and continue the loop
  Kid has exited!  Finish up after child, and fetch a new job of the same type from the appropriate queue
  pcntl_fork() to start new child
  Make sure child exits using pcntl_exec(), not exit()
}

 

The script ensures there is a child running one job of each type at any time (unless there are no jobs in the queue for that job type, in which case it just waits).

[btherl]

Here's some clarification on the forking process, in case you're not used to such stuff:

 

      $pid=pcntl_fork();
      if ($pid == -1) {
        die("Could not fork.  Got a knife?\n");
      } else if ($pid) {
        // we are the parent
        $running[$job]['pid'] = $pid;
      } else {
        // we are the child.  Do child stuff
        pcntl_exec('/bin/true'); # Goodbye cruel world
      }

[tibberous]

Thanks Got me going anyway.

 

Is it better to fork, or to use PHP to exec PHP command line? That seems a little simpler to me, any downside to it I'm not seeing?

 

Thanks

[btherl]

When you fork, you get the pid of your child.  And you also get notification via wait() for when the child is finished.  If you put a process into the background through exec(), then you won't get notification when it's done.

 

I use this to check if any children have finished (edited a bit for brevity):

 

    foreach ($running as $job => $v) {
      if ($v['pid'] === false) continue;
      $res = pcntl_waitpid($v['pid'], &$status, WNOHANG);
      if ($res === -1) {
        print "Error waiting on children (may be none left)\n";
      } elseif ($res === 0) {
        # Child not exited yet
      } else {
        print("$job child {$v['pid']} has exited\n");
      }
    }

[tibberous]

This doesn't work on Windows =(

 

No, I'm not running it on Windows, but I was hoping to test it here.

 

Maybe just wait and code that part last on the server?

 

Any alternatives?

 

Thanks

[btherl]

Oops.. yes that will make testing difficult   Unless you install a develeopment environment like cygwin (or even linux in a virtual server).  I'm not 100% sure that cygwin would work for this purpose anyway.

 

Well, you can test the framework on the server, and then introduce the real scripts later.