cyberbob1uk

After typing and testing for a number of hours, the problem is lying with the amount of files that need creating. Each line in the data that starts with either LO,LI or LT is a station or timing point, removing the LO,LI or LT gives the unique reference to that station. LOLANCSTR becomes LANCSTR LIMORCMSJ becomes MORCMSJ LTBAROW becomes BAROW each of these stations timing points need their own table (containing the rest of the data on the same row), as users on the site will search by station/timing point intially, then extra info is collected from the other tables using the unique id. I ran the file creation script so all LO,LI and LT were written to one file and this resulted in 2.2 million lines of data(in a matter of 5 seconds). After eventually getting that file into a database (LOAD DATA LOCAL INFILE failed and the database table i was trying to insert on crashed, with no lines inserted), I performed a couple of test searches but was taking 10s to return just bare basic info, that was before i got the additional info from the other tables and echoed the results to the browser. There are approximatey 5,500 different timing points for which tables are needed - there has to be a better way for me to create these?

Sorry, been at work for the last 4 nights. Ive have been trying to get it to work but to no avail. Im now starting to think of another database structure that could work in the hope it will copy all the data as needed.

First i see if the new file is available, if it is then : while(!feof($handle)) { $line = stream_get_line($handle, 81); $lineID = "$line[0]$line[1]"; if($lineID == 'TI'){ ti($line); } if($lineID == 'BS'){ $time = md5(microtime()); $time = "a$time"; $details = bs($line,$time); } if($lineID == 'BX'){ bx($line,$details); } if($lineID == 'LO'){ lo($line,$details); } if($lineID == 'LI'){ li($line,$details); } if($lineID == 'LT'){ lt($line,$details); } } each custom function opens the relevant csv file, appends the needed data, closes csv file. $details returns the md5 hash of the microtime of when BS was detedted at the start of the line, therefore giving each line the same id until the next BS line-start is encountered

originally i did try to send directly to database, but this was taking an unbelievably long time. For this reason, I decided to create CSV files, load those to the database then destroy the files as they'd be no longer needed

The weekly files I have been doing on my local machine through the night at work. The original file was split into managable chunks (usually 10 - 12 separate files). Then each record in each smaller file was made into an array using an md5 hash of microtime to create a unique array key. In the next script I manually changed the name of the smaller file (now a php file containing arrays), and included it in the script which put the info into CSV files Each record in the file contains the stopping pattern for one train service operating between date_a and date_b on day(s)_x (this info is held within the first line of each record) - one train can have more than 1 pattern depending on the date and day. Because of the amount of services and service variations, each record needs a unique id so different schedules for the same train on different days dont get mixed up. From this, main CSV files service details, and stopping patterns are created. Service details containing the necessary parts for that service, stopping patterns contains the timing points for each record in order of passing (theses are the lines starting 'LO','LI' and 'LT'), then each timing point has its own csv containing each service that passes/stops at it. CSV's are then transfered to a database using LOAD DATA INFILE

Actually, after replying, I think i caught onto your thought. While testing, I have been executing the script via a browser (and not getting the expected results), however, I assume if the browser wasnt used to execute the script, all 'should' work as expected?

This is all being done on my web server and does need to be as the daily files will be placed directy on the server ready for processing

Hi guys, I am currently receiving a large text file ( > 500mb), once per week which I have been manually splitting then processing to obtain the required CSV files. However, this is taking in the region of 2 to 3 hours. Very soon, these files will be sent daily and I really dont have the time to split and process this everyday I have been playing for a while to try and parse everything properly/automatically with fopen, feof and fgets ( and other 'f' options), but the script never seems to read the file all the way to the end - I assume this is due to memory usage. The data received in the file follows a strict pattern throughout the file which is: BSNY990141112271112270100000 POO2C35 122354000 DMUS 075 O BX NTY LOLANCSTR 1132 11322 TB LIMORCMSJ 1135 00000000 LICRNFNJN 1140 00000000 H LICRNF 1141H1142H 11421142 T LISDAL 1147H1148H 11481148 T LIARNSIDE 1152H1153 11531153 T LIGOVS 1158 1159 11581159 T LIKTBK 1202 1202H 12021202 T LICARK 1206 1207 12061207 T LIULVRSTN 1214H1215H 12151215 T LIDALTON 1223 1223H 12231223 T LIDALTONJ 1225 00000000 LIROOSE 1229 1229H 12291229 T 2 LTBAROW 1237 12391 TF That is just one record of informaton (1 of around 140,000 records), each record has no fixed amount of lines but each line in each record is fixed to 80 characters and all lines in each record need to have the same unique 'id', at present, Im using an md5 hash of microtime. The first line of every record starts with 'BS' and the last line of each record starts with 'LT' terminating with 'TF'. All the other stuff between also follows a certain pattern of which I can break down effectively. The record above show one train service schedule, hence why each line in each record needs the same unique id. Anyone got any ideas on how I could process such a file effectively?? Many thanks Dave

Hi, I have just shy of 4500 records to insert into a website's database, of which I am a member. Whilst I can do this record-by-record manually, im struggling to to automate the form submission. Manually, I use the following form: <form name = "rmUpload" action = " <URL> "> <input type = "hidden" name = "username" value = "USER"> <input type = "hidden" name = "password" value = "PASSWORD"> <input type = "hidden" name = "date" value = "<?php echo $hdate ?>"> <input type = "hidden" name = "actual_dep_time" value = "00:00"> <input type = "hidden" name = "actual_arr_time" value = "00:00"> <input type = "hidden" name = "location_dep" value = "<?php echo $row[CRS_dep] ?>"> <input type = "hidden" name = "location_arr" value = "<?php echo $row[CRS_arr] ?>"><br> <input type = "hidden" name = "tops[]" value = "<?php echo $row[h1] ?>"> <input type = "hidden" name = "tops[]" value = "<?php echo $row[h2] ?>"> <input type = "hidden" name = "tops[]"> <input type = "hidden" name = "tops[]"><br> <input type = "float" name = "distance" value = "<?php echo $miles ?>.<?php echo $chains ?>"> <input type = "hidden" name = "headcode" value = ""> <input type = "hidden" name = "route" value = ""> <input type = "hidden" name = "notes" value = "<?php echo $row[notes] ?>"> <input type = "submit" value = "SEND"> Any values are correct, just couldnt be bothered editting them all out. For single items its ok, but it redirects to the remote website stated in the 'action' part of the form, this obviously doesnt work right when used in a while statement. How can I get it to work so that I can pull my details from a mysql dB and continue to submit the form until all 4500 records have been transferred? I must provide all $_POST variables as key/value pairs. I have tried experiments with cURL but just get an error saying I need authorisation to use that resource