Jump to content

seany123

Members
  • Posts

    1,053
  • Joined

  • Last visited

Everything posted by seany123

  1. is the column called 'Venue' with a capital v in the sql table?
  2. so first, you need to have a semicolon after assigning the variable, secondly you would want to append the values to the variable $venue rather than overwriting it on every run of the while loop... $venue = $venue . "<option>" . $row['Venue']. "</option>";
  3. This is a really great reply! I'll be able to get something working from what you have outlined. Thanks
  4. Thanks for your reply, we have a PayPal subscription button, which creates a recurring payment on PayPal. there are 2 buttons, which allow for 30-day recurring or 1-year recurring. Sean
  5. Hello, I want to allow for users to pay a monthly or yearly subscription for a premium service. what would be the best table structure for managing the payments and subscriptions? I already have a table which retrieves the paypal IPN and saves the details into a table and assigns the users unique id. so from there should i be updating the users table which (which holds the users username/email/pass/cookies etc) with subscription details? or should i create another table which holds the subscription details for each user? Also what date/time format should i be using to know when the user subscription starts/ends? whats the best way to handle auto-renewals? any help would be great Thanks
  6. This is thread is regarding cURL and the issue how to deal with redirections etc...
  7. No i dont. Tbh im not overly bothered about violating their tos, as what im doing perfectly legal, if they do wish to block me from accessing their website thats another story, however i highly doubt that would be the case. its going off topic a little here though.
  8. yes there was a redirect to an "enable Cookie" page but that was resolved with the cookie options. Im trying to create a script to collect the products from that website, the function im using works for majority of pages, but there are a few which the function fails with. I think the webpage will always redirect, regardless of the url. probably to stop this time of access to their website, So really the question is, how can i get the content from that webpage even if it does do all the redirecting.
  9. Thanks for your response, so the urls i am testing have been taken directly from their navigation bar, so im not sure how i can use the correct urls to begin with, when they are the urls the websites provides.. maybe there is a way to see where the url will be redirected to, before i try downloading it? looking into this more, i can see the urls have for example: krypto=VPlGaWUypwMmg17kzWYmO6EN56YvHkYVWm295zTZI%2BjXwe1Sjr6scuaUSXOQxj9j5lJ1w4SaNwnVZc6wFjyNITCK%2BjyQwvWQlIj51J6x4zZL1EOiGG4gMDFMIUQtoJY4XbiSLy%2BTjuuL4WqbXGl9B4DP0PGD8izDET1A9mVF%2BU8%3D sean
  10. Hi, I'm trying to scrape a website with the following function: function Scurl($url) { $cookie_file = "cookie.txt"; // Assigning cURL options to an array $options = Array( CURLOPT_RETURNTRANSFER => TRUE, // Setting cURL's option to return the webpage data CURLOPT_FOLLOWLOCATION => TRUE, // Setting cURL to follow 'location' HTTP headers CURLOPT_AUTOREFERER => TRUE, // Automatically set the referer where following 'location' HTTP headers CURLOPT_CONNECTTIMEOUT => 120, // Setting the amount of time (in seconds) before the request times out CURLOPT_TIMEOUT => 120, // Setting the maximum amount of time for cURL to execute queries CURLOPT_MAXREDIRS => 10, // Setting the maximum number of redirections to follow CURLOPT_USERAGENT => "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1a2pre) Gecko/2008073000 Shredder/3.0a2pre ThunderBrowse/3.2.1.8", // Setting the useragent CURLOPT_URL => $url, // Setting cURL's URL option with the $url variable passed into the function //this is for the cookie. CURLOPT_COOKIESESSION => TRUE, CURLOPT_COOKIEFILE => $cookie_file, CURLOPT_COOKIEJAR => $cookie_file, ); $ch = curl_init(); // Initialising cURL curl_setopt_array($ch, $options); // Setting cURL's options using the previously assigned array data in $options $data = curl_exec($ch); // Executing the cURL request and assigning the returned data to the $data variable curl_close($ch); // Closing cURL return $data; // Returning the data from the function } If i run this function with this url: http://www.sainsburys.co.uk/shop/gb/groceries/fruit-veg/fresh-herbs-ingredients it works fine. however if i use this url: http://www.sainsburys.co.uk/shop/gb/groceries/fruit-veg/fresh-fruit the function fails. any ideas on what the issue might be? given that they are from the same website im confused as to what might be the issue. any help given would be great! regards sean
  11. Hello, What would be the easiest way to compare rows from different databases? I have 2 tables 1 is live and the other is for newly inserted data which needs to be compared against. firstly the unique_id must match, if not then comparing the rest of the columns is irrelevant. The structure is the same for both tables, something like this: id unique_id val1 val2 val3 so i thought i could do something like this <?php $q = mysqli_query($conn, "SELECT * FROM live_table LIMIT 10"); while ($r = mysqli_fetch_array($q)) { $uniqueId = $r['unique_id']; $val1 = $r['val1']; $val2 = $r['val2']; $check_query = mysqli_query($conn, "SELECT * FROM unverified_table WHERE unique_id='$uniqueId' AND val1='$val1' AND val2='$val2'"); //from here i would check the num rows and get the result etc... } however i have around 20+ columns which would need to be checked to see if they match. also i would preferable want to be able to print out which columns matched and which didn't depending on what i need. the other idea would be to firstly check for a row with the same unique_id, return it back as an array and then loop through each individual element to check if they match... EG. <?php $q = mysqli_query($conn, "SELECT * FROM live_table LIMIT 10"); while ($r = mysqli_fetch_array($q)) { $uniqueId = $r['unique_id']; $check_query = mysqli_query($conn, "SELECT * FROM unverified_table WHERE unique_id='$uniqueId'"); while ($r2 = mysqli_fetch_array($check_query)) { if ($r[1] != $r2[1]) { //etc... } } } is there a better way to do this? thanks sean
  12. to be completely honest i haven't run into this problem before, usually i'm able to use the function to display the website then i use a simple scraping function which scrapes parts out from the source code. I thought curl downloaded the source code after the includes had been created, so didn't realize that include paths made a difference. I will try running the script locally as you suggested. thanks sean
  13. this is exactly the code i'm running: <?php //error_reporting(E_ALL); //ini_set('max_execution_time', 0); $url = "https://groceries.asda.com/"; $main_page = Acurl($url); echo $main_page; function Acurl($url) { //$cookie_file = "cookie.txt"; // Assigning cURL options to an array $options = Array( CURLOPT_RETURNTRANSFER => TRUE, // Setting cURL's option to return the webpage data CURLOPT_FOLLOWLOCATION => TRUE, // Setting cURL to follow 'location' HTTP headers CURLOPT_AUTOREFERER => TRUE, // Automatically set the referer where following 'location' HTTP headers CURLOPT_CONNECTTIMEOUT => 120, // Setting the amount of time (in seconds) before the request times out CURLOPT_TIMEOUT => 120, // Setting the maximum amount of time for cURL to execute queries CURLOPT_MAXREDIRS => 10, // Setting the maximum number of redirections to follow CURLOPT_USERAGENT => "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1a2pre) Gecko/2008073000 Shredder/3.0a2pre ThunderBrowse/3.2.1.8", // Setting the useragent CURLOPT_URL => $url, // Setting cURL's URL option with the $url variable passed into the function //this is for the cookie. CURLOPT_COOKIESESSION => TRUE, CURLOPT_COOKIEFILE => $cookie_file, CURLOPT_COOKIEJAR => $cookie_file, ); $ch = curl_init(); // Initialising cURL curl_setopt_array($ch, $options); // Setting cURL's options using the previously assigned array data in $options $data = curl_exec($ch); // Executing the cURL request and assigning the returned data to the $data variable curl_close($ch); // Closing cURL return $data; // Returning the data from the function } strange that even with your test it wasn't showing the entire webpage? just parts of the nav bar? might there be something the website is doing to block a curl connection? Ok i will give that a try and see what it returns.
  14. Thanks for the response. That's strange, im running the exact code and its redirecting. maybe this a something with my servers settings causing the redirect.
  15. Thanks for your response, i have just now installed and tried it with httprequester on firefox, it appears to display the website without any further redirects.
  16. Hello, so im trying to use Curl to connect to a website, but when i try to return the url, i get redirected and it ends up redirecting me back to "myowndomain.com"/back-soon. is there a way to see why the site is redirecting when using Curl, it doesnt redirect when i connect to the website through a usual web browser.. code: $url = "https://groceries.asda.com/"; $main_page = curlFunction($url); echo $main_page; function curlFunction($url) { $cookie_file = "cookie.txt"; // Assigning cURL options to an array $options = Array( CURLOPT_RETURNTRANSFER => TRUE, // Setting cURL's option to return the webpage data CURLOPT_FOLLOWLOCATION => TRUE, // Setting cURL to follow 'location' HTTP headers CURLOPT_AUTOREFERER => TRUE, // Automatically set the referer where following 'location' HTTP headers CURLOPT_CONNECTTIMEOUT => 120, // Setting the amount of time (in seconds) before the request times out CURLOPT_TIMEOUT => 120, // Setting the maximum amount of time for cURL to execute queries CURLOPT_MAXREDIRS => 10, // Setting the maximum number of redirections to follow CURLOPT_USERAGENT => "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1a2pre) Gecko/2008073000 Shredder/3.0a2pre ThunderBrowse/3.2.1.8", // Setting the useragent CURLOPT_URL => $url, // Setting cURL's URL option with the $url variable passed into the function //this is for the cookie. CURLOPT_COOKIESESSION => TRUE, CURLOPT_COOKIEFILE => $cookie_file, CURLOPT_COOKIEJAR => $cookie_file, ); $ch = curl_init(); // Initialising cURL curl_setopt_array($ch, $options); // Setting cURL's options using the previously assigned array data in $options $data = curl_exec($ch); // Executing the cURL request and assigning the returned data to the $data variable curl_close($ch); // Closing cURL return $data; // Returning the data from the function } Any help would be great. sean
  17. Thanks for your answer, I explained the way above as i thought that was the easiest way to explain, but basically the comma delimiter keywords are a set of keywords which could be 50+ long so to create a column for each keyword wouldn't really be feasible either. Thanks for your reply, I just used that as an example.
  18. Hello, Im not sure if the title makes sense or not but ill try explain it best i can. i have 2 tables, the first table which ill call "haystack" contains a varchar field called keywords which contains keywords separated with a comma delimiter eg: "keyword1, keyword5, keyword2, keyword10" the second table which ill call "needle" has for example 10 rows with a varchar called keyword, this contain only 1 of the keywords for each row eg "keyword5" the aim is to run a search on the haystack table to only return rows where keywords from the needle table match. so i could do this in php by exploding the haystack keywords field and then looping through to compare against the needle keywords, but that would require to select * rows and would make its take a long time, so i was hoping this could be done in sql? thanks sean
  19. Hi, I know this could probably be solved using either php or SQL, so i wasnt sure which board to post this on, but ive always been under the impression its best to solve the issues through SQL if possible as thats the most efficient way. I have a products table which contains product_name varchar (255) then im allowing users to search for products, with the current php code: product_name LIKE '%$searchTerms%' //searchTerm would contain ie: "red apples" it wont return results when the users type the plural for a word ie apples instead of apple, or if they type apple red instead of red apple. im wondering the best way to search products? i thought i could explode the searchTerm variable and run the like query for every word product_name LIKE '%$searchTerm[0]%' OR product_name LIKE '%$searchTerm[1]%' but then im worried about relevance, if one of the search terms is the word "and" then it might return completely irrelevant results. also how would i go about dealing with the plurals? Any suggestions on the best way to do this? thanks
  20. php designer 2008 is seriously better than anything ive ever used. you can write pages easily twice as fast thanks to the very advanced auto completion. you can see exactly what the code is doing with the function lists (which appear in a little box) when you highlight parts of your code.
×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.