Creating New Array Inside A Foreach Loop And Returning Its Value

deadliver · September 26, 2012

I am doing some web scraping.

I start with an array:

$CategoriesMajor = array("Battles Participated","Victories","Defeats","Battles Survived","Destroyed","Detected","Hit Ratio","Damage","Capture Points","Defense Points");

I am getting the html using curl. and get substrings based on my categories from my array.

Up to this point its all good.

My Get_Stats() is not returning anything in the array. I know the $str is getting a value as it loops. I assume I am having an issue adding it into the array properly or returning it from the function properly.

 
//this function is fine just included in post for reference
function get_part($str,$from,$to) {
$value = substr($str,strpos($str,$from)+strlen($from),300);
$value = substr($value,0,strpos($value,$to));
return $value;
}

function GetStats($curlstr,$array){

//array is $Categories Major
   foreach($array as $key =>$value)
   {

  $str = '<td class=""> '.$value.': </td>';
  $str = get_part($curlstr,$str,'</tr>'); 
  $str = get_part($str,'<td class="td-number-nowidth">','</td>');  //grab the part of string with the numerical value
  //Replace spaces
  $str = str_replace(' ',' ',$str);
  //It has %, replace the original value with %
  if (strpos($str,'(')) {
    $str = get_part($str,'(',')');
  }

  $str = strip_tags($str);




  $Stat[$value] = $str;

}
  return $Stats;
}

I am trying to get the information returned using this code, but nohing is loaded into the array

$Stats[] =  GetStats($string,$CategoriesMajor);
foreach($Stats as $key=>$value){
echo $key." ".$value;
}

Christian F. · September 26, 2012

You really should be using the DOMdocument class for this, as it'll help you properly accessing the elements and retrieving the details. Without having to rely upon the string parsing, which does not understand the HTML syntax or structure.

Though, your problem is that you're adding the returned array into the $Stats array at the first available index, so when you're trying to loop through the results you'll get the actual stats array as $value. Which is why you get "# Array" when you try to print it, where # is a number (most likely 0).

deadliver · September 26, 2012

Christian I understand what you said about the index and it is 0, but with an associative array I do not think i understand the concept of how to put it in a different index.

I assumed the way this code would work is that each time I called

 
[color="#000000"]$Stat[/color][color="#666600"][[/color][color="#000000"]$value[/color][color="#666600"]][/color] [color="#666600"]=[/color][color="#000000"] $str[/color][color="#666600"];[/color]

that the $value would become the index and the next $value in the loop would add to the array and be the next index ?

If that is not the case. I am not sure how to proceed. If you could advise me on some reading etc. I have taken a look

at a few reference books, but am still at a loss.

Thank You.

Psycho · September 26, 2012

Why you are using the full, unprocessed $value as the key when building array makes no sense to me, but whatever.

As to your problem, take a closer look at the last few lines of the function and see if you can spot the problem.

	 $Stat[$value] = $str;
}
return $Stats;
}

Hint "Stats" != "Stat"

Edited September 26, 2012 by Psycho

Barand · September 26, 2012

try

$Stats =  GetStats($string,$CategoriesMajor);    // not $Stats[]
foreach($Stats as $key=>$value){
echo $key." ".$value;
}

deadliver · September 26, 2012

Why you are using the full, unprocessed $value as the key when building array makes no sense to me, but whatever.

My apologies for being a novice, perhaps this is too advanced of a forum for me to be asking questions.

Pyscho, I am not sure what you mean when you say unprocessed $value. I come here in all sincerity and respectfully sincerely looking for guidance not ridicule.

If I had a clear understanding of what I was doing I would not be seeking assistance and I would also be a php guru like yourself.

Barand..thank you for your information. I will try your recommendation.

Psycho · September 27, 2012

Pyscho, I am not sure what you mean when you say unprocessed $value. I come here in all sincerity and respectfully sincerely looking for guidance not ridicule.

Ridicule? I have no clue what you are ultimately trying to accomplish. I saw something that doesn't look right, but for all I know it is what you need. Rather than provide a lengthy explanation that may or may not apply I simply asked the question for the purpose of you to possibly take a second look to see if it is, in fact, what you intended.

What did I mean by it? You have a loop that is extracting the index and the value of each element of the array into the varaibles $key and $value

foreach($array as $key =>$value)

Then within that loop you are apprently doign some processing of the $value variable which is then stored in a separate variable called $str, correct? Then at the end of the loop you are taking that processed value ($str) and putting it into a new array. But, you are using the value from the variable $value (i.e. the unprocessed, original value) as the key

$Stat[$value] = $str;

Based upon the processing that you are doing to get the result for $str, the variable $value contains HTML code. That would seem like odd content to be using as the index for an array element. But again, how the hell am I to know what you are ultimately using the data for? So, I simply asked the question.

Also, you're welcome for me finding one of the two errors causing the underlying problem, Barand found the other.

Edited September 27, 2012 by Psycho

deadliver · September 27, 2012

Psycho, my apologies if I misread your intent. What I am doing is just getting the all the data I need into arrays then I will be putting them into a database for later use.

I just thought if I use an associative array ($Stats) that it would be easier when it comes time to write the code to dump the information into the db. And of course I do appreciate you finding the error as well. Oh..you are right $value is kind of the html code. Really its the text I know is inside the code of the html table, but then once

I find that text I know where to get the numerical value based on positioning.

When I get a chance I will switch that all to DOM.

Would something like this work better? or you know put me on the path to being better.

 
$CategoriesMajor = array(
array("Battles Participated","Victories","Defeats","Battles Survived","Destroyed","Detected","Hit Ratio","Damage","Capture Points","Defense Points")
array("0","0","0","0","0","0","0","0","0","0")
);


function GetStats($curlstr,$array){		
  for ($i=0; $i < 11; $i++)
  {
    $str = '<td class=""> '.$CategoriesMajor[0][$i].': </td>';  
    $str = get_part($curlstr,$str,'</tr>');   
    $str = get_part($str,'<td class="td-number-nowidth">','</td>'); 
    $str = str_replace(' ',' ',$str);	   
    if (strpos($str,'(')) {			   
    $str = get_part($str,'(',')');		
}

   $CategoriesMajor[0][$i]$i] = $str;
  }
}

Psycho · September 27, 2012

Would something like this work better? or you know put me on the path to being better.

You accept you are a novice - and there is nothing wrong with that, we all have to start somewhere. So, with that understanding it is best, IMO, to lay out what you are really trying to accomplish. You may be taking the wrong approach entirely and we can help point you into a more efficient and elegant solution. Can you please provide some details such as:

Sample of the originating data ($curlstr). I assume you are reading a single file and then reading parts of it based upon the "tags" in the array. You are taking multiple steps to get that data. So, it would be helpful to see a sample of the data showing the "tag" as well as the specific data piece you are trying to get. Although string functions are typically faster than regek - I think this might be a perfect problem for a regex solution.

Psycho · September 27, 2012

OK, I created some sample data by working backwards from the processes you were using. Not sure if it is accurate or not - but it should be close. I then created a small function to extract the data you are apparently after.

Here was the sample text I used for $curlstr

<tr><td class="">Battles Participated</td>
<td class="td-number-nowidth">99</td>
</tr>
<tr><td class="">Victories</td>
<td class="td-number-nowidth">55</td>
</tr>
<tr><td class="">Defeats</td>
<td class="td-number-nowidth">44</td>
</tr>
<tr><td class="">Battles Survived</td>
<td class="td-number-nowidth">35</td>
</tr>

Here is the code I created to process that text:

function GetStats($curlstr, $statList)
{
$results = array();
foreach($statList as $stat)
{
 $pattern = "#<td class=\"\">{$stat}</td>.*?<td class=\"td-number-nowidth\">([^<]*)#is";
 echo $pattern . "<br>\n";
 if(preg_match($pattern, $curlstr, $match))
 {
	 $results[$stat] = $match[1];
 }
}
return $results;
}

$CategoriesMajor = array(
"Battles Participated", "Victories", "Defeats", "Battles Survived",
"Destroyed", "Detected", "Hit Ratio", "Damage", "Capture Points", "Defense Points");

print_r(GetStats($text, $CategoriesMajor));

Here is the output of the print_r() for the results:

Array
(
[battles Participated] => 99
[Victories] => 55
[Defeats] => 44
[battles Survived] => 35
)

A couple notes: 1) If the array of categories is not used anywhere else I would probably define them inside the function. But, if the categories you need are the only patterns in the content that you need. You probably don't need to define the categories at all and can have the RegEx pattern just grab all the relevant data. For example, this gives the same result as above with my test data without having to define the categories:

function GetStats($curlstr, $statList)
{
$results = array();
$pattern = "#<td class=\"\">([^<]*)</td>.*?<td class=\"td-number-nowidth\">([^<]*)#is";
if(preg_match_all($pattern, $curlstr, $matches, PREG_SET_ORDER))
{
 foreach($matches as $match)
 {
	 $results[$match[1]] = $match[2];
 }
}
return $results;
}

print_r(GetStats($text, $CategoriesMajor));

deadliver · September 27, 2012

This is the knowledge I am starting with as far as the patterns of the HTML code curl returns from the website.

Here are the formats:

<h1>usernamer</h1>

If the user is in a clan it will have a link

<a class="b-link-clan" href="/community/clans/1000005799-B-O-W/"> <span class="tag">[b-O-W]</span>

Based on that this information I wrote one function


function get_login($curlstr){

$login = get_part($curlstr,'<h1>','</h1>');
/** WAS PROFIE FOUND? **/
if ($curlstr == "" || $login == "" || $login == ' PAGE NOT FOUND ')
{
 die("User Not Found");
}
//IF USER HAS CLAN
$s = '<a class="b-link-clan"';
if (strpos($curlstr,$s)!=0)
{
 $clan = substr($curlstr,strpos($curlstr,$s)+strlen($s),300);
 $clan = substr($clan,strpos($clan,'[')+1,strpos($clan,']')-strpos($clan,'[')-1);
 $login = "[$clan] $login";
}else{
$login = $login;
}
return $login;
}

Now the values in $CategoriesMajor will follow this HTML format:

<tr><td class=""> Battles Participated: </td> <
td class="td-number-nowidth"> 10 338 </td>
</tr>

And of course I was using the infamous function you have been helping me with for this.

I have another array

 $CategoriesMinor = array('Targets Destroyed','Targets Detected','Hit Ratio');

the HTML follows the same format as for $CategoriesMajor but doesnt have the colon at the end and has a second value to grab

< td><span>Targets Destroyed</span></td>
< td class="right value">7 042</td>
< td class="right value">17 556</td>

What you have not seen yet is a hole other format almost identical for a lot more information

		 <td class="value">

			 <a class="b-gray-link" href="/encyclopedia/vehicles/china/ch01_type59/">Type 59</a>

		 </td>
		 <td class="right value">1 083</td>
		 <td class="right value">539</td>

the link shows a vehicle name and the following <td></td> shows total battles in that vehicle and the next set shows total wins

I have simply been using substrings to gather information via this function, rather than DOM

function get_part($str,$from,$to) {
$value = substr($str,strpos($str,$from)+strlen($from),300);
$value = substr($value,0,strpos($value,$to));
return $value;
}

The CURL is like this

/** LOAD SETTINGS **/
$id = "1002011274";
$server = "com";
$flag = us;
$languages = 1;
$url="[url="http://worldoftanks.$server/community/accounts/$id/"]http://worldoftanks.$server/community/accounts/$id/[/url]"; // break;
/** DOWNLOAD DATA **/
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
$string = curl_exec($ch);
curl_close($ch);
//dont start unless we get the webpage
if(!isset($string)){
echo "no return";
}

I attached the file the curl_exec returns.

This is really all I have so far except scribble at the end where I am testing values I get returned etc.

I think I need to step back and start over with a better plan of attack, first starting with using DOM

wotsource.txt

I am going to rethink , rewrite, and respond to this thread. In the mean time if you have any additional

advice that will help me with my strategy to attack this..please post your thoughts.

Sorry again for the misunderstanding Psycho.

Psycho · September 27, 2012

OK, here you go. using the sample file you have provided this should do most everything you need. All you need to pass to the function is the location of the file or even the url to the page with the content to be parsed. The function will then return an array of all the data.

function getUserData($source_location)
{
   $source = file_get_contents($source_location);
   if(!$source) { return false; }

   $results = array();

   $major_cats = array(
    "Battles Participated", "Victories", "Defeats", "Battles Survived", 'Hit Ratio',
    "Destroyed", "Detected", "Hit Ratio", "Damage", "Capture Points", "Defense Points");
   foreach($major_cats as $cat)
   {
	 $pattern = "<td class=\"\"> {$cat}: </td>.*?<td class=\"td-number-nowidth\">([^<]*)";
	 if(preg_match("#{$pattern}#is", $source, $match))
	 {
		 $results[$cat] = trim($match[1]);
	 }
   }

   $minor_cats = array('Targets Destroyed', 'Targets Detected');
   foreach($minor_cats as $cat)
   {
	 $pattern  = "<td><span>{$cat}</span></td>[^<]*";
	 $pattern .= "<td class=\"right value\">([^<]*)</td>";
	 if(preg_match("#{$pattern}#is", $source, $match))
	 {
		 $results[$cat] = trim($match[1]);
	 }
   }

   $vehicles = array('Type 59', 'IS-3', 'IS-8', 'IS-7', 'T-150', 'IS');
   foreach($vehicles as $veh)
   {
	 $pattern  = "<a class=\"b-gray-link\" href=\"/encyclopedia/vehicles/[^\"]*\">{$veh}</a>[^<]*";
	 $pattern .= "</td>[^<]*";
	 $pattern .= "<td class=\"right value\">([^<]*)</td>[^<]*";
	 $pattern .= "<td class=\"right value\">([^<]*)";
	 if(preg_match("#{$pattern}#is", $source, $match))
	 {
		 $results['vehicles'][$veh]['total'] = trim($match[1]);
		 $results['vehicles'][$veh]['wins'] = trim($match[2]);
	 }
   }

   return $results;
}


$source_file = 'wotsource.txt';
$userData = getUserData($source_file);

print_r($userData);

Using the sample file you provided, here is what the content of the array looks like:

Array
(
   [battles Participated] => 10 338
   [Victories] => 4 963 (48%)
   [Defeats] => 5 166 (50%)
   [battles Survived] => 2 325 (22%)
   [Hit Ratio] => 61%
   [Destroyed] => 7 042
   [Detected] => 12 848
   [Damage] => 8 409 237
   [Capture Points] => 13 725
   [Defense Points] => 5 562
   [Targets Destroyed] => 7 042
   [Targets Detected] => 12 848
   [vehicles] => Array
       (
           [Type 59] => Array
               (
                   [total] => 1 083
                   [wins] => 539
               )

           [iS-3] => Array
               (
                   [total] => 588
                   [wins] => 252
               )

           [iS-8] => Array
               (
                   [total] => 563
                   [wins] => 257
               )

           [iS-7] => Array
               (
                   [total] => 379
                   [wins] => 178
               )

           [T-150] => Array
               (
                   [total] => 353
                   [wins] => 140
               )

           [iS] => Array
               (
                   [total] => 346
                   [wins] => 153
               )

       )

)

Edited September 27, 2012 by Psycho

deadliver · September 27, 2012

omg. you are sickenly talented. I was just getting ready to post that I bit off more than I can chew trying to use DOM.

I will try it of course I'm sure it works..when my domain gets fixed I guess my host is having DNS issues.

Thanks for the help on this. I should just quit because well.look at your code and then look at my babble.

deadliver · September 27, 2012

Just though you might like to know: :happy-04:

deadliver · October 6, 2012

 
 $results['vehicles'][$veh]['total'] =  str_replace(" ","",trim($match[1]));
 $results['vehicles'][$veh]['wins'] =  str_replace(" ","",trim($match[2]));

if my array is set up like this code..how would a foreach loop be coded to go through this data?

I can do a normal foreach loop, but not one like this.

I was thinking of doing it like this , but if I do I dont know how to code it correctly.

 
foreach ($tankData as $row)
{
   echo $row['vehicles'][<whatgoeshere>];
   echo $row['vehicles'][<whatgoeshere?>]['wins'] ;
  echo  $row['vehicles'][<whatgoeshere?>]['total'];
}

Barand · October 6, 2012

foreach ($results['vehicles'] as $veh => $vdata) {
   echo "$veh : Wins: {$vdata['wins']} : Total : {$vdata['total']} <br>";
}

deadliver · October 6, 2012

Thanks for the help. Can I ask for an edification.

 [color="#000000"]echo [/color][color="#008800"]"$veh : Wins: {$vdata['wins']} : Total : {$vdata['total']} <br>"[/color][color="#666600"];[/color]

In my thoughts when you used $veh in the echo it would have had to start of with at least

$vdata['vehicles'][????]

Jessica · October 6, 2012

Look up foreach in the manual.

deadliver · October 6, 2012

php manual:

the internal array pointer is automatically reset to the first element of the array

and because of this we can use $veh to get the first element of the array each time?

Jessica · October 6, 2012

No...

Did you try the code?

deadliver · October 6, 2012

I know your frustrated by me and I am sorry.

Yes, your solution works perfectly.

Let me tell you what I think I know..and maybe that is better so you can see the faulty logic I am using.

I understand each set of $vdata represents all the current element's information, both keys and values.

so by using

$results['vehicles']=>$vdata

we are putting in the values for the the name,total,wins.

That is how we can use $veh directly to because it is representing a value not a key.

That is what I think is happening anyways as I try to grasp this.

.

Sign In

Creating New Array Inside A Foreach Loop And Returning Its Value

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Join the conversation

Important Information