Jump to content

Assistance with understanding memory limit and why?

sw9

By sw9
in PHP Coding Help

 Share

Recommended Posts

sw9 Member

sw9

Posted
Posted

Forgive me for my limited knowledge as I ask this question. I am in a shared hosting environment, and I'm trying to run a PHP script. My script opens a text file that has about 2000 records in it. It then loops through the records and creates/updates nodes in Drupal based on them. So essentially it does DB inserts. I also have it print to the screen which record it's at, if it's updating or inserting, etc.

 

So when I run my script via command line (it times out via browser), it goes through about 630 records and then gives me the memory error:

Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 1987 bytes)

 

So my host says that I'm at the max of 32M. I just don't understand why after 600 records it errors out? I also don't understand why an import script needs more than 32M.

 

The other thing that's baffling to me is that I essentially converted a perl script that runs on this same host and does the same thing to PHP. In perl it runs in about 2 minutes and there's no issue. But in PHP it is taking more than 4 minutes and then gives me that error.

 

Can anyone give me some general info on what's going on exactly?

 

Just in case, I am posting my entire script. Perhaps I am just not optimizing it. I am still pretty newbie at this, so I might be the problem.

 

Thanks, I appreciate any insight you can give.

 

<?php 
// bootstrap drupal
include_once 'includes/bootstrap.inc';
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);

print "Importing programs, msn information to Drupal nodes now.....<br><br>";

$fields = array('title', 'episode_number', 'episode_name', 'secondary_name', 'series_id', 'category', 'subcategory', 'first_run', 'active', 'length', 'notes', 'ep', 'source','pdate', 'adate', 'purchase_date', 'exp_date', 'distributor', 'msn', 'msn_description', 'msn_program_type', 'msn_status', 'msn_filename');
$handle = fopen("VLM_master.txt", "r");
$counter = 1;
$last_id = null;
while (($data = fgetcsv($handle, 10000, "	")) !== FALSE) {
   
  $data = array_combine($fields, $data);
  
  // If the title number is the same, then we can skip most of the import, hopefully reducing load time
  if($data['title'] != $last_id) {
    $last_id = $data['title'];
    

// -------------VCAM CATEGORIES-------------------------------- //

// Do not do anything if it is a VCAM program!
if ($data["category"] != 'VCAM Pub' && $data["category"] != 'VCAM Gov' && $data["category"] != 'VCAM Filler' ){

// -------------FORMAT DATES-------------------------------- //

// SET ALL THE DATES CORRECTLY SINCE SYNERGY DOESN'T PROVIDE THE RIGHT FORMAT
   $pdate_year = substr($data["pdate"], 0, 4);
   $pdate_month = substr($data["pdate"], 4, 2);
   $pdate_day = substr($data["pdate"], 6, 2);
   $pdate = $pdate_year."-".$pdate_month."-".$pdate_day;
   
   $fr_year = substr($data["first_run"], 0, 4);
   $fr_month = substr($data["first_run"], 4, 2);
   $fr_day = substr($data["first_run"], 6, 2);
   $first_run = $fr_year."-".$fr_month."-".$fr_day;
   
   $adate_year = substr($data["adate"], 0, 4);
   $adate_month = substr($data["adate"], 4, 2);
   $adate_day = substr($data["adate"], 6, 2);
   $adate = $adate_year."-".$adate_month."-".$adate_day;
   
   if($data["exp_date"] == '00000000'){ $exp_date = ''; 
   }
   else{
   $exp_date_year = substr($data["exp_date"], 0, 4);
   $exp_date_month = substr($data["exp_date"], 4, 2);
   $exp_date_day = substr($data["exp_date"], 6, 2);
   $exp_date = $exp_date_year."-".$exp_date_month."-".$exp_date_day;
   }
   
// -------------AQUISITION / PRODUCTION -------------------------------- //
   
   // Set Acquisition Date to be Production Date in Drupal IF it is greater than pdate (meaning if it isn't 0 and pdate is 0)
   if ($data["adate"] > $data["pdate"]) {$pdate = $adate;}
   
// ---------------SOURCE---------------------------- //
   
   // See if Source has already been input
   if($data["source"] != ''){
   $source = taxonomy_get_term_by_name($data["source"]);
   
   // if we did not return a result for the source, then we create new terms in the Source taxonomy
   if($source[0]->tid == ''){
   $sources = array('name' => $data["source"]); 
   drupal_execute('taxonomy_form_term', $sources, '4');
   }
   // Now we get the the source tid again so that we can submit it with node->taxonomy
   $source = taxonomy_get_term_by_name($data["source"]);
   }
// ---------------PRODUCER---------------------------- //
   if($data["ep"] != ''){
   // See if ep has already been input
   $ep = taxonomy_get_term_by_name($data["ep"]);
   
   // if we did not return a result for the ep, then we create new terms in the Producer taxonomy
   if($ep[0]->tid == ''){
   $eps = array('name' => $data["ep"]); 
   drupal_execute('taxonomy_form_term', $eps, '5');
   }
   // Now we get the the source tid again so that we can submit it with node->taxonomy
   $ep = taxonomy_get_term_by_name($data["ep"]);
   }
   
// ------------------SERIES------------------------------------ //
  
   // We need to get the node reference id for our series based on the synergy series id
   $sSeries = "SELECT nid FROM {content_type_series} WHERE field_synergy_id_value = '".$data["series_id"]."'";
   $seriesResult = db_query($sSeries);
   while ($seriesData = mysql_fetch_assoc($seriesResult)) {
  $series = $seriesData["nid"];
  }
// ---------------------------------------------------------------- //

   // Check to see if the node already exists in drupal
   $sql = "select nid from {content_type_program} where field_tnum_value = '".$data['title']."'";
   $result = db_query($sql);
   // die ($sql);
   $count = mysql_num_rows($result);
   // if there is a result, then we are going to update the drupal node
   if($count > 0){
   while ($node_exist = mysql_fetch_assoc($result)) {
     // If the category is delete or dns, and it exists in drupal, then we delete the drupal node
     if($data["category"] == 'DELETE' || $data["category"] == 'DNS'){
     print "delete ".$data["episode_name"]."<br>";
 node_delete($node_exist['nid']);
 }
 else{
   
       print $counter.": ";
       print "update program ".$node_exist['nid'].", ";
       $node1 = node_load($node_exist['nid']);
     
       $node = new StdClass();
       $node->nid = $node1->nid;
       $node->changed = time(); 
       $node->title = $data['episode_name'];
       $node->field_tnum[0]['value'] = $data['title'];
       $node->field_episode_number[0]['value'] = $data['episode_number'];
       $node->field_active[0]['value'] = $data['active'];    
       $node->field_secondary_name[0]['value'] = $data['secondary_name'];
       $node->field_first_run[0]['value'] = $first_run;
       $node->field_expiration_date[0]['value'] = $exp_date;  
       $node->field_pdate[0]['value'] = $pdate;
       $node->field_approximate_length[0]['value'] = $data['length'];
       $node->field_description[0]['value'] = $data['notes'];
       $node->field_series[0]['nid'] = $series;
       if ($data["category"] == 'ADMIN'){
       $node->field_program_preferences[0]['value'] = "Do Not Show";
       }
       if ($data["category"] == 'ARCHIVE'){
       $node->field_program_preferences[1]['value'] = "Archive";
       }
   
   $node->taxonomy = array($ep[0]->tid, $source[0]->tid);
       // Prepare the node and then save it
       node_prepare($node);
       node_save($node);
     
       $programid = $node->nid;
	} // End if delete
      }  // End While
     } // End if count, drupal node doesn't exist, so in that case we create a new node
     else{
     
     // if it's a delete, don't do anything
     if($data["category"] == 'DELETE' || $data["category"] == 'DNS'){
     print $counter.": ";

     print "delete ".$data["episode_name"]."<br>";

 }
 else{
   print $counter.": ";

       print "insert program, ";
       // start building the node object
       $node = new StdClass();
       $node->type = 'program';
       $node->status = 1;
       $node->created = time();
       $node->comment = 0;
       $node->promote = 0;
       $node->sticky = 0;
       $node->format = 1;
       $node->uid = 5;
       $node->name = 'programs';

       $node->title = $data['episode_name'];
       $node->field_tnum[0]['value'] = $data['title'];
       $node->field_episode_number[0]['value'] = $data['episode_number'];
       $node->field_active[0]['value'] = $data['active'];    
       $node->field_secondary_name[0]['value'] = $data['secondary_name'];
       $node->field_first_run[0]['value'] = $first_run;
       $node->field_expiration_date[0]['value'] = $exp_date;  
       $node->field_pdate[0]['value'] = $pdate;
       $node->field_approximate_length[0]['value'] = $data['length'];
       $node->field_description[0]['value'] = $data['notes'];
       $node->field_series[0]['nid'] = $series;

        if ($data["category"] == 'ADMIN'){
       $node->field_program_preferences[0]['value'] = "Do Not Show";
       }
       if ($data["category"] == 'ARCHIVE'){
       $node->field_program_preferences[1]['value'] = "Archive";
       };
       $node->taxonomy = array($ep[0]->tid, $source[0]->tid);

       // Prepare the node and then save it
       node_prepare($node);
       node_save($node);
     
    
       } // End if delete
     } // End if count
   
   // Now we need to check and see if the media serial number node exists in drupal
   $smsn = "SELECT nid FROM {content_type_msn} WHERE field_msn_0_value = '".$data["msn"]."'";
   $msnResult = db_query($smsn);
   $msn_count = mysql_num_rows($msnResult);
   
   if($msn_count > 0){
     while ($node_exist = mysql_fetch_assoc($msnResult)) {
     
       if($data["category"] == 'DELETE' || $data["category"] == 'DNS'){
       print "delete ".$data["episode_name"]." msn <br>";
   node_delete($node_exist['nid']);
   }
   else{
     
         print " update msn \n<br />".$data["episode_name"]."<br><br>";
         $node1 = node_load($node_exist['nid']);
         $node = new StdClass();
         $node->nid = $node1->nid;
         $node->field_msn_0[0]['value'] = $data["msn"];
         $node->field_program_0[0]['nid'] = $programid;
         $node->title = $data["msn"]." - ".$data["episode_name"];
         $node->field_msn_description[0]['value'] = $data["msn_description"];
         $node->field_program_type[0]['value'] = $data["msn_program_type"];
         $node->field_status[0]['value'] = $data["msn_status"];
         $node->field_filename[0]['value'] = $data["msn_filename"];
         node_prepare($node);
         node_save($node);
       } // End if delete
     } // end while
   
   } // end if count
   else{
    if($data["category"] == 'DELETE' || $data["category"] == 'DNS'){
    print "delete ".$data["episode_name"]." msn <br><br>";
 }
 else{
   
     print " insert msn \n<br />".$data["episode_name"]."<br>";
     // start building the node object
     $node = new StdClass();
     $node->type = 'msn';
     $node->status = 1;
     $node->created = time();
     $node->comment = 0;
     $node->promote = 0;
     $node->sticky = 0;
     $node->format = 1;
     $node->uid = 5;
     $node->name = 'programs';
     
     $node->field_msn_0[0]['value'] = $data["msn"];
     $node->field_program_0[0]['nid'] = $programid;
     $node->title = $data["msn"]." - ".$data["episode_name"];
     $node->field_msn_description[0]['value'] = $data["msn_description"];
     $node->field_program_type[0]['value'] = $data["msn_program_type"];
     $node->field_status[0]['value'] = $data["msn_status"];
     $node->field_filename[0]['value'] = $data["msn_filename"];
     node_prepare($node);
     node_save($node);
     
     } // End if delete

   } // end else

} //  End if VCAM (skip it)  
else {
print $counter.": Skipping ".$data["episode_name"]."<br><br>";
}

} // End if $data['title'] != $last_id, we will skip most options and just update what's necessary
else{
print "Skipping most entry...";

$sql = "select nid from {content_type_program} where field_tnum_value = '".$data['title']."'";
$result = db_query($sql);
// die ($sql);
$count = mysql_num_rows($result);
  if($count > 0){
    while ($node_exist = mysql_fetch_assoc($result)) {
      if($data["category"] == 'DELETE' || $data["category"] == 'DNS'){
      print "delete ".$data["episode_name"]."<br>";
  node_delete($node_exist['nid']);
  }
  elseif ($data["category"] == 'ADMIN' || $data["category"] == 'ARCHIVE'){
       
      print $counter.": ";
      print "update program, ";
      $node1 = node_load($node_exist['nid']);
     
      $node = new StdClass();
      $node->nid = $node1->nid;
      $node->changed = time(); 
   
        if ($data["category"] == 'DNS'){
        $node->field_program_preferences[0]['value'] = "Do Not Show";
        }
        if ($data["category"] == 'ARCHIVE'){
        $node->field_program_preferences[1]['value'] = "Archive";
        }
   
   // Prepare the node and then save it
       node_prepare($node);
       node_save($node);
     
      
  } // End if categories
    }  // End While
    } // End if count
    
$smsn = "SELECT nid FROM {content_type_msn} WHERE field_msn_0_value = '".$data["msn"]."'";
$msnResult = db_query($smsn);
$msn_count = mysql_num_rows($msnResult);
   
  if($msn_count > 0){
    while ($node_exist = mysql_fetch_assoc($msnResult)) {
     
      if($data["category"] == 'DELETE' || $data["category"] == 'DNS'){
      print "delete ".$data["episode_name"]." msn <br>";
  node_delete($node_exist['nid']);
  }
  else{
     
      print " update msn \n<br />".$data["episode_name"]."<br><br>";
      $node1 = node_load($node_exist['nid']);
      $node = new StdClass();
      $node->nid = $node1->nid;
      $node->field_msn_0[0]['value'] = $data["msn"];
      $node->field_program_0[0]['nid'] = $programid;
      $node->title = $data["msn"]." - ".$data["episode_name"];
      $node->field_msn_description[0]['value'] = $data["msn_description"];
      $node->field_program_type[0]['value'] = $data["msn_program_type"];
      $node->field_status[0]['value'] = $data["msn_status"];
      $node->field_filename[0]['value'] = $data["msn_filename"];
      node_prepare($node);
      node_save($node);
       } // End if delete
     } // end while
   
   } // end if count
   else{
    if($data["category"] == 'DELETE' || $data["category"] == 'DNS'){
    print "delete ".$data["episode_name"]." msn <br><br>";
 }
 else{
   
     print " insert msn \n<br />".$data["episode_name"]."<br>";
     // start building the node object
     $node = new StdClass();
     $node->type = 'msn';
     $node->status = 1;
     $node->created = time();
     $node->comment = 0;
     $node->promote = 0;
     $node->sticky = 0;
     $node->format = 1;
     $node->uid = 5;
     $node->name = 'programs';
     
     $node->field_msn_0[0]['value'] = $data["msn"];
     $node->field_program_0[0]['nid'] = $programid;
     $node->title = $data["msn"]." - ".$data["episode_name"];
     $node->field_msn_description[0]['value'] = $data["msn_description"];
     $node->field_program_type[0]['value'] = $data["msn_program_type"];
     $node->field_status[0]['value'] = $data["msn_status"];
     $node->field_filename[0]['value'] = $data["msn_filename"];
     node_prepare($node);
     node_save($node);
     
     } // End if delete

   } // end else

}
$counter++;
} // end in file
fclose($handle);

?>

Link to comment
Share on other sites
sw9 Member

sw9

Posted
  • Author
Posted

Yes, it can successfully process a quarter of the file. If I cut it down to 600 records, it works fine. Any suggestions on how I could approach getting the whole file to complete or how I might trick it into parsing it in chunks? Also, any insight on why it works so quickly in perl? Thanks again for the help!

Link to comment
Share on other sites
This thread is more than a year old. Please don't revive it unless you have something important to add.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
 Share
Go to topic listing
×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.