Search the Community

I've made a PHP web crawler and then made a MySQL table called "dex" as in index, then I connected to the database through PDO and tweaked the code to "INSERT" websites that aren't already crawled into the table, "UPDATE" for websites that are crawled, and used URL hashes as an indicator or "id" for links. The terminal shows all the links and links related to them, the if statement works perfectly and there are no major errors, so why does it not insert the data into the "dex" table? every-time I check the table after the process I only find the row that I inserted manually to test the if statement for "UPDATE" or "INSERT". what can I do to fix this issue and insert the date the crawler retrieves? Test.html: <a href="https://google.com"></a> <a href="https://www.yahoo.com/"></a> <a href="https://www.bing.com/"></a> <a href="https://duckduckgo.com/"></a> Crawler: <?php error_reporting(E_ALL); ini_set('display_errors', 1); $start = "http://localhost/deepsearch/test.html"; $pdo = new PDO('mysql:host=127.0.0.1;dbname=deepsearch', 'root', ''); $already_crawled = array(); $crawling = array(); function get_details($url) { $options = array('http'=>array('method'=>"GET", 'headers'=>"User-Agent: howBot/0.1\n")); $context = stream_context_create($options); // Suppress warnings for HTML parsing errors libxml_use_internal_errors(true); $doc = new DOMDocument(); @$html = @file_get_contents($url, false, $context); // Load HTML content and check for parsing errors if ($doc->loadHTML($html)) { if (!empty($titleElements)) { $title = $titleElements->item(0); $title = $title->nodeValue; } else { $title = ""; } $description = ""; $keywords = ""; $metas = $doc->getElementsByTagName("meta"); for ($i = 0; $i < $metas->length; $i++) { $meta = $metas->item($i); if ($meta->getAttribute("name") == strtolower("description")) { $description = $meta->getAttribute("content"); } if ($meta->getAttribute("name") == strtolower("keywords")) { $keywords = $meta->getAttribute("content"); } } return '{"Title": "'.str_replace("\n", "", $title).'", "Description": "'.str_replace("\n", "", $description).'", "Keywords": "'.str_replace("\n", "", $keywords).'", "URL": "'.$url.'"}'; } else { // Handle the parsing error echo "HTML parsing error: " . libxml_get_last_error()->message . "\n"; return ''; // Return an empty string or handle the error as needed } } function follow_links($url) { global $pdo; global $already_crawled; global $crawling; $options = array('http' => array('method' => "GET", 'headers' => "User-Agent: howBot/0.1\n")); $context = stream_context_create($options); $doc = new DOMDocument(); @$doc->loadHTML(@file_get_contents($url, false, $context)); $linklist = $doc->getElementsByTagName("a"); foreach ($linklist as $link) { $l = $link->getAttribute("href"); if (substr($l, 0, 1) == "/" && substr($l, 0, 2) != "//") { $l = parse_url($url)["scheme"] . "://" . parse_url($url)["host"] . $l; } else if (substr($l, 0, 2) == "//") { $l = parse_url($url)["scheme"] . ":" . $l; } else if (substr($l, 0, 2) == "./") { $l = parse_url($url)["scheme"] . "://" . parse_url($url)["host"] . dirname(parse_url($url)["path"]) . substr($l, 1); } else if (substr($l, 0, 1) == "#") { $l = parse_url($url)["scheme"] . "://" . parse_url($url)["host"] . parse_url($url)["path"] . $l; } else if (substr($l, 0, 3) == "../") { $l = parse_url($url)["scheme"] . "://" . parse_url($url)["host"] . "/" . $l; } else if (substr($l, 0, 11) == "javascript:") { continue; } else if (substr($l, 0, 5) != "https" && substr($l, 0, 4) != "http") { $l = parse_url($url)["scheme"] . "://" . parse_url($url)["host"] . "/" . $l; } if (!in_array($l, $already_crawled)) { $already_crawled[] = $l; $crawling[] = $l; $details = json_decode(get_details($l)); echo $details->URL . " "; $rows = $pdo->query("SELECT * FROM dex WHERE url_hash='" . md5($details->URL) . "'"); $rows = $rows->fetchColumn(); $params = array(':title' => $details->Title, ':description' => $details->Description, ':keywords' => $details->Keywords, ':url' => $details->URL, ':url_hash' => md5($details->URL)); if ($rows > 0) { echo "UPDATE" . "\n"; } else { if (!is_null($params[':title']) && !is_null($params[':description']) && $params[':title'] != '') { $result = $pdo->prepare("INSERT INTO dex (title, description, keywords, url, url_hash) VALUES (:title, :description, :keywords, :url, :url_hash)"); $result= $result->execute($params); //if ($result) { // echo "Inserted successfully.\n"; //} else { // echo "Insertion failed.\n"; // print_r($stmt->errorInfo()); //} } } //print_r($details)."\n"; //echo get_details($l)."\n"; //echo $l."\n"; } } array_shift($crawling); foreach ($crawling as $site) { follow_links($site); } } follow_links($start); //print_r($already_crawled); ?> at first I tried different links that got me an empty value which resulted in errors and warnings then I changed the links and started writing the "UPDATE", "INSERT" if statement and started specifically writing the insert PDO first to test it out. when I executed the the file using command php I got the intended results in term of how it was supposed to look like in the terminal but then I checked on the table and found out that nothing was inserted. I want to insert these to use them in my search engine and make them searchable by query.

I'm creating a report page and can't figure out how to retrieve multiple rows from a table and display them in html. There are 5 data elements in each row and I'm thinking that using a form in the html might be the best way as I'm totally ignorant about tables in html. I'm a newbie to php and can't figure out how to accomplish this. Here's the code that I have so far: <?php $filename = NULL; session_start(); // start of script every time. // setup a path for all of your canned php scripts $php_scripts = '../php/'; // a folder above the web accessible tree // load the pdo connection module require $php_scripts . 'PDO_Connection_Select.php'; //******************************* // Begin the script here // Connect to the database if (!$con = PDOConnect("foxclone")): { echo "Failed to connect to database" ; exit; } else: { $sql = 'SELECT COUNT(IP_ADDRESS) FROM download WHERE FILENAME IS NOT NULL'; $sql1 = 'Update download t2, ip_lookup t1 set t2.country = t1.country, t2.area = t1.area, t2.city = t1.city where ((t2.IP_ADDRESS) = (t1.start_ip) OR (t2.IP_ADDRESS) > (t1.start_ip)) AND ((t2.IP_ADDRESS) = (t1.end_ip) OR (t2.IP_ADDRESS) < (t1.end_ip)) AND (t2.FILENAME is not null and t2.country is null)'; $sql2 = 'SELECT (IP_ADDRESS, FILENAME, country, area, city) from download where FILENAME is not null'; // Update the table $stmt = $con->prepare($sql1); $stmt->execute(); // Get count of rows to be displayed in table $stmt = $con->prepare($sql); $stmt->execute() ; $cnt = $stmt->fetch(PDO::FETCH_NUM); // retrieve one row at a time $i = 1; while($i <= $cnt){ $stmt = $con->prepare($sql2); $row->execute(array('')); // Do I need an array here? // from here on, I'm lost $i++; I'd appreciate any guidance you can provide or understandable tutorials you can point me to. Larry

Hi.. I have a database which works fine. I ve just started using google charts to make the results look better for the U.Experience. . The example tables are very basic like: Country Visits England 4 Usa 6 Brazil 3 I get all my results form where statements so I have no tables which actually count the entries and show them in a basic table like this. My question is : Should I be using where, count() etc statements to populate the charts or should I have mini tables in my schema that present the information neatly like the above or a counter next to the id of the user/country etc e.g. so I don't need to worry about statements in the chart code.... thx for reading.

Hi, I am trying to read in the name and student number that the user entered and compare it to what is in the mysql database. My teacher suggested using found. However I am still not getting anything to echo out. Can someone lead me in the right direction? next.php <?php require 'connect.php'; //linking up to the database $link = mysqli_connect(HOST, USER, PASS, DB) or die (mysqli_connect_error()); //making a variable from the user data $name = mysqli_real_escape_string ($link, $_POST["name"]); $number = mysqli_real_escape_string ($link, $_POST["snumber"]); $course = $_POST["pcourse"]; // select all from table student which show student name and number $squery = "SELECT * FROM students"; $sresult = mysqli_query($link, $squery); while ($srow = mysqli_fetch_array($sresult)) { if ($name == $srow['uid'] && $number == $srow['student']) { if ($found) { echo "$srow[uid] $srow[student]"; } else { echo "Student does not exist"; } } } mysqli_close ($link); ?> <html> <body> <form action="index.php" method="post"> <br> <input type = "submit" value="back" name="back"> </form> </body> </html> This is my index.php that I use as my form <!DOCTYPE html> <html> <body> <h1>Course Selection</h1><br> <form action="next.php" method="post"> Name: <input type="text" name="name" placeholder="Name" required="required" maxlength="50"> <br><br> Student Number: <input type="text" name= "snumber" required="required" maxlength="9"> <br><br> <?php //form require 'connect.php'; $link = mysqli_connect(HOST, USER, PASS, DB) or die(mysqli_connect_error()); echo "Select a course: <select name = \"pcourse\">\n"; $query = "SELECT * FROM course"; $result = mysqli_query($link, $query); while ($row = mysqli_fetch_array($result)) { echo "<option> $row[code=auto:0] $row[name] $row[max]</option><br>"; } mysqli_free_result($results); mysqli_close ($link); echo " </select>\n"; ?> <br><br> <input type = "submit" value="submit" name= "submit"> </form> </body> </html>