Jump to content

seany123

Members
  • Content Count

    1,053
  • Joined

  • Last visited

Community Reputation

1 Neutral

About seany123

  • Rank
    Prolific Member

Profile Information

  • Gender
    Not Telling
  1. is the column called 'Venue' with a capital v in the sql table?
  2. so first, you need to have a semicolon after assigning the variable, secondly you would want to append the values to the variable $venue rather than overwriting it on every run of the while loop... $venue = $venue . "<option>" . $row['Venue']. "</option>";
  3. This is a really great reply! I'll be able to get something working from what you have outlined. Thanks
  4. Thanks for your reply, we have a PayPal subscription button, which creates a recurring payment on PayPal. there are 2 buttons, which allow for 30-day recurring or 1-year recurring. Sean
  5. Hello, I want to allow for users to pay a monthly or yearly subscription for a premium service. what would be the best table structure for managing the payments and subscriptions? I already have a table which retrieves the paypal IPN and saves the details into a table and assigns the users unique id. so from there should i be updating the users table which (which holds the users username/email/pass/cookies etc) with subscription details? or should i create another table which holds the subscription details for each user? Also what date/time format should i be using to know when the user subscription starts/ends? whats the best way to handle auto-renewals? any help would be great Thanks
  6. This is thread is regarding cURL and the issue how to deal with redirections etc...
  7. No i dont. Tbh im not overly bothered about violating their tos, as what im doing perfectly legal, if they do wish to block me from accessing their website thats another story, however i highly doubt that would be the case. its going off topic a little here though.
  8. yes there was a redirect to an "enable Cookie" page but that was resolved with the cookie options. Im trying to create a script to collect the products from that website, the function im using works for majority of pages, but there are a few which the function fails with. I think the webpage will always redirect, regardless of the url. probably to stop this time of access to their website, So really the question is, how can i get the content from that webpage even if it does do all the redirecting.
  9. Thanks for your response, so the urls i am testing have been taken directly from their navigation bar, so im not sure how i can use the correct urls to begin with, when they are the urls the websites provides.. maybe there is a way to see where the url will be redirected to, before i try downloading it? looking into this more, i can see the urls have for example: krypto=VPlGaWUypwMmg17kzWYmO6EN56YvHkYVWm295zTZI%2BjXwe1Sjr6scuaUSXOQxj9j5lJ1w4SaNwnVZc6wFjyNITCK%2BjyQwvWQlIj51J6x4zZL1EOiGG4gMDFMIUQtoJY4XbiSLy%2BTjuuL4WqbXGl9B4DP0PGD8izDET1A9mVF%2BU8%3D sean
  10. Hi, I'm trying to scrape a website with the following function: function Scurl($url) { $cookie_file = "cookie.txt"; // Assigning cURL options to an array $options = Array( CURLOPT_RETURNTRANSFER => TRUE, // Setting cURL's option to return the webpage data CURLOPT_FOLLOWLOCATION => TRUE, // Setting cURL to follow 'location' HTTP headers CURLOPT_AUTOREFERER => TRUE, // Automatically set the referer where following 'location' HTTP headers CURLOPT_CONNECTTIMEOUT => 120, // Setting the amount of time (in seconds) before the request times out CURLOPT_TIMEOUT => 120, // Setting the maximum amount of time for cURL to execute queries CURLOPT_MAXREDIRS => 10, // Setting the maximum number of redirections to follow CURLOPT_USERAGENT => "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1a2pre) Gecko/2008073000 Shredder/3.0a2pre ThunderBrowse/3.2.1.8", // Setting the useragent CURLOPT_URL => $url, // Setting cURL's URL option with the $url variable passed into the function //this is for the cookie. CURLOPT_COOKIESESSION => TRUE, CURLOPT_COOKIEFILE => $cookie_file, CURLOPT_COOKIEJAR => $cookie_file, ); $ch = curl_init(); // Initialising cURL curl_setopt_array($ch, $options); // Setting cURL's options using the previously assigned array data in $options $data = curl_exec($ch); // Executing the cURL request and assigning the returned data to the $data variable curl_close($ch); // Closing cURL return $data; // Returning the data from the function } If i run this function with this url: http://www.sainsburys.co.uk/shop/gb/groceries/fruit-veg/fresh-herbs-ingredients it works fine. however if i use this url: http://www.sainsburys.co.uk/shop/gb/groceries/fruit-veg/fresh-fruit the function fails. any ideas on what the issue might be? given that they are from the same website im confused as to what might be the issue. any help given would be great! regards sean
  11. Hello, What would be the easiest way to compare rows from different databases? I have 2 tables 1 is live and the other is for newly inserted data which needs to be compared against. firstly the unique_id must match, if not then comparing the rest of the columns is irrelevant. The structure is the same for both tables, something like this: id unique_id val1 val2 val3 so i thought i could do something like this <?php $q = mysqli_query($conn, "SELECT * FROM live_table LIMIT 10"); while ($r = mysqli_fetch_array($q)) { $uniqueId = $r['unique_id']; $val1 = $r['val1']; $val2 = $r['val2']; $check_query = mysqli_query($conn, "SELECT * FROM unverified_table WHERE unique_id='$uniqueId' AND val1='$val1' AND val2='$val2'"); //from here i would check the num rows and get the result etc... } however i have around 20+ columns which would need to be checked to see if they match. also i would preferable want to be able to print out which columns matched and which didn't depending on what i need. the other idea would be to firstly check for a row with the same unique_id, return it back as an array and then loop through each individual element to check if they match... EG. <?php $q = mysqli_query($conn, "SELECT * FROM live_table LIMIT 10"); while ($r = mysqli_fetch_array($q)) { $uniqueId = $r['unique_id']; $check_query = mysqli_query($conn, "SELECT * FROM unverified_table WHERE unique_id='$uniqueId'"); while ($r2 = mysqli_fetch_array($check_query)) { if ($r[1] != $r2[1]) { //etc... } } } is there a better way to do this? thanks sean
  12. to be completely honest i haven't run into this problem before, usually i'm able to use the function to display the website then i use a simple scraping function which scrapes parts out from the source code. I thought curl downloaded the source code after the includes had been created, so didn't realize that include paths made a difference. I will try running the script locally as you suggested. thanks sean
  13. this is exactly the code i'm running: <?php //error_reporting(E_ALL); //ini_set('max_execution_time', 0); $url = "https://groceries.asda.com/"; $main_page = Acurl($url); echo $main_page; function Acurl($url) { //$cookie_file = "cookie.txt"; // Assigning cURL options to an array $options = Array( CURLOPT_RETURNTRANSFER => TRUE, // Setting cURL's option to return the webpage data CURLOPT_FOLLOWLOCATION => TRUE, // Setting cURL to follow 'location' HTTP headers CURLOPT_AUTOREFERER => TRUE, // Automatically set the referer where following 'location' HTTP headers CURLOPT_CONNECTTIMEOUT => 120, // Setting the amount of time (in seconds) before the request times out CURLOPT_TIMEOUT => 120, // Setting the maximum amount of time for cURL to execute queries CURLOPT_MAXREDIRS => 10, // Setting the maximum number of redirections to follow CURLOPT_USERAGENT => "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1a2pre) Gecko/2008073000 Shredder/3.0a2pre ThunderBrowse/3.2.1.8", // Setting the useragent CURLOPT_URL => $url, // Setting cURL's URL option with the $url variable passed into the function //this is for the cookie. CURLOPT_COOKIESESSION => TRUE, CURLOPT_COOKIEFILE => $cookie_file, CURLOPT_COOKIEJAR => $cookie_file, ); $ch = curl_init(); // Initialising cURL curl_setopt_array($ch, $options); // Setting cURL's options using the previously assigned array data in $options $data = curl_exec($ch); // Executing the cURL request and assigning the returned data to the $data variable curl_close($ch); // Closing cURL return $data; // Returning the data from the function } strange that even with your test it wasn't showing the entire webpage? just parts of the nav bar? might there be something the website is doing to block a curl connection? Ok i will give that a try and see what it returns.
  14. Thanks for the response. That's strange, im running the exact code and its redirecting. maybe this a something with my servers settings causing the redirect.
×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.