Parse XML - Write to a database --- PLEASE HELP!

[maros174]

This will kill me, it is taking my whole work day - each day, checking for errors so my visitors don't notice...

 

BACKGROUND:

I have a financial website that display's Stock Market Data from my country's stock exchange.

Stock Market Data is read from an XML file and written to a database.

Data is refreshed every ten minutes (around 40 times per day), for 200+ stocks.

 

PROBLEM:

Most of the time everything works fine, but, several times each day, during a refresh, a data for a random stock, usually  only one, is read only partialy.

(for example. instead of HighPrice=320.99, to a database is written HighPrice=99, or, instead of DailyLow=4270.01, database contains DailyLow=70.01 )

 

PROBLEM SCRIPTS:

XML.class.php  -  parses XML data and places it in a structured PHP array

ZSE.class.php  -  reads the data and writes it to a database. (I suspect function getMarketTradingData could be an issue (on lines 167-209), everything above just writes to a database)

 

 

Potentially important facts:

• XML is allways clean of errors.
  
• Time of data collection(refresh) is not important, sometimes the error appears at 10:25, sometimes at 12:35, sometimes at 15:45, etc...
  
• Usually the error is limitied to only one stock out of 200+, rarely two or three of them have errors.
  
• Data is collected for several fields - but only one has an error: High Price, Low Price, Closing Price, Buy Price, Sell Price, Volume, Average Price. But every time the data is read partially it is limited to only one field. For example, sometimes the Closing price has an error (to a database is written 16 instead of 316), sometimes some other price like daily low price or average price.
  
• Dellimiter is not an issue, the data is read partially independent of it. For e.g. sometimes instead of Average Price=24.49 to a database is written only the decimal part 49, sometimes instead of SellPrice=3420 database contains SellPrice=20.

 

Example:

 

 

 

 

[attachment deleted by admin]

[maros174]

Anybody? .... I could really use some help on this one.

Is it possible that nobody knows what could be wrong here ?

[harristweed]

I think the problem is this line in XML.class.php

while ($file_content = fread($fp, 4096)) {

Best option ditch the class and write a custom parser using php5 and  simplexml_load_file

 

Else you could try altering the line to

while ($file_content = fread($fp, 10000)) {

 

or/and

 

line 69

$reference[$name][$this->_levelsNumbering[$i][$name]] =$reference[$name][$this->_levelsNumbering[$i][$name]]. $data;

 

 

[maros174]

harristweed,

 

thanks for taking the time to look at the code... I'll designate you as my new best friend if we make it work

 

----

 

I changed the parameter of fread  to 10000 - I am going to keep checking the data every 10 miutes and report here if the errors continue to appear or become rarer.  (In any case, the stock market is open for only two more hours, and there may be no more errors today anyway, so I'll have to keep checking tomorrow to see if something pops out...)

 

 

Just to be sure, could you please check below if I understood right the other modification you suggested.

If errors continue to appear I am going to make changes to the line 69 as well.

 

In addition to the above changes, I am going to replace this

for ($i = 1; $i <= $this->_currentDepth; $i++) {

				$name = $this->_xmlpath[$i];

				if (!isset($reference[$name])) {
					$reference[$name] = array();
				}

				if ($i == $this->_currentDepth) {
					$reference[$name][$this->_levelsNumbering[$i][$name]] = $data;
				}
				else {
					if (!isset($reference[$name][$this->_levelsNumbering[$i][$name]])) {
						$reference[$name][$this->_levelsNumbering[$i][$name]] = array();
					}
				}
				$reference = &$reference[$name][$this->_levelsNumbering[$i][$name]];
			}

 

 

with this:

for ($i = 1; $i <= $this->_currentDepth; $i++) {

				$name = $this->_xmlpath[$i];

				if (!isset($reference[$name])) {
					$reference[$name] = array();
				}

				if ($i == $this->_currentDepth) {
					$reference[$name][$this->_levelsNumbering[$i][$name]] =$reference[$name][$this->_levelsNumbering[$i][$name]]. $data;
				}
				else {
					if (!isset($reference[$name][$this->_levelsNumbering[$i][$name]])) {
						$reference[$name][$this->_levelsNumbering[$i][$name]] = array();
					}
				}
				$reference = &$reference[$name][$this->_levelsNumbering[$i][$name]];
			}

[maros174]

Damn.

 

Changing this line in XML.class.php:

while ($file_content = fread($fp, 4096)) {

to this:

while ($file_content = fread($fp, 10000)) {

 

Had no visible effect.

There were four errors in less than two hours... same as before...

 

Tomorrow morning I'm going to change line 69 as well.

[harristweed]

I think this will solve it for you...

$reference[$name][$this->_levelsNumbering[$i][$name]] =$reference[$name][$this->_levelsNumbering[$i][$name]].$data

 

notice no space before . and $data

 

I had a similar problem.....

[maros174]

Three hours... not a single error!

I'm afraid to hope... 

[maros174]

First day in a very long time that went without a single error in the data!

 

I still won't celebrate... gonna test it all through monday and report here if it is really working... but it seems to be...  :D 

[maros174]

Second day without a single error... I think I can safely say that the script is working flawlessly.

 

harristweed,

I can't thank you enough... You really helped me a lot.

 

Thanks 

[harristweed]

you are welcome!