Handle large XML file

[ayok]

Hi,

 

I am busy creating a site which display product list from XML (rss feeds) files. In the first phase I can do a method by saving all the data's from XML into my database, and display it from database. However, this time, I need to display it directly without saving it to my database because there are a lot more data's in the second phase. So I need to display the data's directly from the XML's.

 

I've tried several methods like using SimpleXML, it works nice, however, it can't handle larger XML (like 800 MB). So I tried other method like SAX or XMLReader. Both work like a charm. It can even parsing GB's XML and display all the data's.

 

However, I need to create query or manipulate the xml to display those data's, like only display certain category or search. Then I got again the same problem with time limited error or sometimes size exceeded error. I don't know what to do now.

 

Does anybody have advice how I should do this?

 

Thank you,

ayok

[RobertP]

IMO you should parse the xml and add to some kind of database. if you parse the xml for each page load, especially with the file size you are describing, you will lag like a mother fucker..

[ignace]

I agree with RobertP, use a database. Optimize your queries with indexes.

[ayok]

What kind of database? I use mysql and there are like hundred thousands items, is this going to be fine?

 

Thanks

[ignace]

MySQL will do just fine. Just be sure to define proper indexes.

[ayok]

Ok thanks, I'll try this way..