When the script starts I go through each of the columns and create a base table of a char type, which in MySQL goes holds up to 255 characters. As I go through the data and dump it into a text file I check whether the column can be converted into a number type, and if not check to see if it has the longest string length (if it does update the length). At the end the tables are updated to optimise them for space etc.

I'm preety sure the problem isn't a type one - using the MySQL client it inserts fine. I thought it was maybe a timing issue - Looking at the last line of the shortened data shows that the last few columns are null.. is the script moving faster than it is writing? If so why did

parse_data();
sleep 10;
query('LOAD DATA INFILE "'.$temp.'" INTO TABLE '.$table.'
        FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY "\""
        LINES TERMINATED BY "\r\n"');
unlink($temp);
[download]