PHP Optimization Help

[ciber]

I have created a website, which runs a small php script using curl every 5 minutes.

These are the steps the script uses:

 

1. Access list of RSS feeds from DB

loop {

2. Download RSS Feed

loop {

3. Does RSS entry exist in the database ?

Yes - Next Entry

No - Add to Database

}

}

 

My problem is, is that there are a number of entries.

I run a script every minute, which divides the list into 5 categories.

Ie: I run

 

10:00 - RSS Feeds 1-15

10:01 - Rss Feeds 16 - 30

10:02 - Rss Feeds 31 - ... etc etc etc

 

Things is, the script is eating up the systems load, and causing it to peak as high as 40 or 50.

 

Is there anyway to optimize this, so that it reduces the server load??

An idea I originally had was to use the generated time on the rss feeds, problem is, cause some of them are generated on the fly, it wont work.

 

Thanks

[trq]

I assume your running these tasks from cron, are you executing this php script via your http server or via php's cli?

[JonnoTheDev]

I have a program that does exactly the same thing and the server isn't heavily loaded. Steps to take:

 

1. Optimise your queries. Make sure that you have indexes on the correct fields when you are checking for duplicates.

2. Unset variables after use, i.e once you load the feed into a variable, after you have inserted into database, unset it.

3. Add more memory to the server and allocate to the script. This is the simplest option.

4. Record a date-time on the feed so that the script will not try and get news from this feed if it has already in lets say the last 24hs. This is a simple query.

 

My script obtains news from thousands of feeds and can take hours to complete however I would suggest you run this on a linux box with only the required software installed. I would say that a windows box is a bad choice as a lot of memory is already allocated to the operating system.

[ciber]

hmm, didn't think of unsetting the variables, thats a good idea!

Thanks