DOM Traversing

[gingerninja]

Dear All,

 

I have been trying to speed up a script I wrote that takes the content of a webpage (it's a SAP BI Web Query if anyone's wondering) and parses through it reading the content of a particular table and inserting into a mysql database table.

 

Now before anyone asks, the problem has nothing to do with mysql.

 

What I have found to be a performance problem is this bit of code :-

 

   $doc = new DOMDocument();
   @$doc->loadHTMLFile("/tmp/test.html");

   $tables = $doc->getElementsByTagName('table');
   for ($i=0; $i < $tables->length; ++$i) {
    if ($tables->item($i)->getAttribute('name') == "GR1Table")
     {
       $table = $tables->item($i);
       $find_all_tables_time = time() + microtime();
       break;
     }
   }

   // get all rows from the first table
   $rows = $table->getElementsByTagName('tr');
   set_time_limit(1200);

   $rowcount = $rows->length;
		$insert_limit = 1000;

   // iterate over all but the first row
for ($i = 1; $i <= $insert_limit; ++$i)
{
$row = $rows->item($i)->textContent;
if ($i == $insert_limit)
{
   $loop_1 = time() + microtime();
   echo "Rows = " . $insert_limit . "\tTime:\t " . round($loop_1 - $start_time, 4) . "<br/>";
   if ($insert_limit < $rowcount)
   {
      $insert_limit += 500;
   }
}
}

 

I'll try to explain as best I can.  The html document contain 8163 rows in the table I am parsing and using PHP just to loop over the rows and assigning the textContent to a variable becomes slower the more rows you process.  Here is the resulting output of the above code.

 

Rows = 1000 Time: 2.1187

Rows = 1500 Time: 3.5387

Rows = 2000 Time: 5.7027

Rows = 2500 Time: 8.5453

Rows = 3000 Time: 12.0353

Rows = 3500 Time: 16.1604

Rows = 4000 Time: 20.917

Rows = 4500 Time: 26.3189

Rows = 5000 Time: 32.3477

Rows = 5500 Time: 39.0064

Rows = 6000 Time: 46.2941

Rows = 6500 Time: 54.2183

Rows = 7000 Time: 62.8111

Rows = 7500 Time: 72.1436

Rows = 8000 Time: 81.9869

Rows = 8500 Time: 92.3004

 

Now that's slow, the time is in seconds!

 

Here's the same code written in Javascript together with it's results :-

function rowloop()
{
var mytabs = document.getElementsByTagName("TABLE");
for (i=0;i<mytabs.length;++i)
{
   if (mytabs[i].getAttribute("name") == "GR1Table")
   {
     mytab = mytabs[i];
     break;
   }
}

var myrows = mytab.getElementsByTagName("TR");
var rowcount = myrows.length;
var insert_limit = 1000;
var startTime = new Date();
for (i=1;i <= insert_limit;++i)
{
var myrow = myrows[i].textContent;
  if (i == insert_limit)
  {
    var endTime = new Date();
    var totalTime = endTime-startTime;
    document.write("Rows = " + insert_limit + " Time: " + totalTime + "ms<br/>");
    if (insert_limit < rowcount)
    {
       insert_limit += 500;
    }
  }
}
}

And here's the results for the javascript code :-

 

Rows = 1000 Time: 16ms

Rows = 1500 Time: 643ms

Rows = 2000 Time: 650ms

Rows = 2500 Time: 658ms

Rows = 3000 Time: 666ms

Rows = 3500 Time: 674ms

Rows = 4000 Time: 682ms

Rows = 4500 Time: 690ms

Rows = 5000 Time: 698ms

Rows = 5500 Time: 706ms

Rows = 6000 Time: 789ms

Rows = 6500 Time: 797ms

Rows = 7000 Time: 805ms

Rows = 7500 Time: 813ms

Rows = 8000 Time: 821ms

 

Can anyone explain this for me as it's driving me up the wall?

 

I am running php 5.2.5 but have also tried 5.2.4 with no difference to the result.

 

Thanks in advance.

Craig