Search the Community
Showing results for tags 'crawl'.
Found 3 results
Need critiques and "indexing-crawling" help
eaglehopes posted a topic in Website CritiqueHello to everybody, I need critiques and "website crawling help" about my website http://enginery.freecluster.eu . My crawling question was that: I tried google search console tools to add my website's sitemap and add it : http://enginery.freecluster.eu/sitemap.xml . It says my sitemap is ok and found 312 pages but not crawl all correctly! Three weeks have passed but nothing changed. I manually request indexing some pages(about 4 pages) and google search console, after than today it only shows some of them(not all 4) when I search using "site:http://enginery.freecluster.eu". My website's all files have php extensions. Did this prevent googlebot to reach the content of my websites' pages? My robots.txt file's content is : User-agent: Googlebot Allow: / User-agent: * Allow: / Sitemap: http://enginery.freecluster.eu/sitemapv1.xml Any critiques and help is appreciated. Thanks.
cURL Crawling Data
Sessions And Crawl