mysql - SQL import CSV file with PHP -
i'm trying import pretty big csv file database (locally) file 230mb , 8.8 million lines problem have isn't opening csv or dont know how import it, file opens, imports 500,000 lines , quits , trows no error or timeout or anything, see webpage.
this code:
try { $conn = new pdo("mysql:host=$servername;dbname=adresses_database", $username, $password); // set pdo error mode exception $conn->setattribute(pdo::attr_errmode, pdo::errmode_exception); echo "connected successfully"; $row = 1; if (($handle = fopen("bagadres.csv", "c+")) !== false) { while (($data = fgetcsv($handle, '', ";")) !== false) { if (!isset($write_position)) { // move line previous position, except first line $write_position = 0; $num = count($data); // $num 15 $row++; //i dont need this? $stmt = $conn->prepare("insert adresses (openbareruimte, huisnummer, huisletter, huisnummertoevoeging, postcode, woonplaats, gemeente, provincie, object_id, object_type, nevenadres, x, y, lon, lat) values (:openbareruimte, :huisnummer, :huisletter, :huisnummertoevoeging, :postcode, :woonplaats, :gemeente, :provincie, :object_id, :object_type, :nevenadres, :x, :y, :lon, :lat)"); $stmt->bindparam(':openbareruimte', $data[0]); $stmt->bindparam(':huisnummer', $data[1]); $stmt->bindparam(':huisletter', $data[2]); $stmt->bindparam(':huisnummertoevoeging', $data[3]); $stmt->bindparam(':postcode', $data[4]); $stmt->bindparam(':woonplaats', $data[5]); $stmt->bindparam(':gemeente', $data[6]); $stmt->bindparam(':provincie', $data[7]); $stmt->bindparam(':object_id', $data[8]); $stmt->bindparam(':object_type', $data[9]); $stmt->bindparam(':nevenadres', $data[10]); $stmt->bindparam(':x', $data[11]); $stmt->bindparam(':y', $data[12]); $stmt->bindparam(':lon', $data[13]); $stmt->bindparam(':lat', $data[14]); $stmt->execute(); } else { $read_position = ftell($handle); // actual line fseek($handle, $write_position); // move previous position fputs($handle, $line); // put actual line in previous position fseek($handle, $read_position); // return actual position $write_position += strlen($line); // set write position next loop } fflush($handle); // write pending change file ftruncate($handle, $write_position); // drop repeated last line flock($handle, lock_un); } fclose($handle); } } catch(pdoexception $e) { echo "connection failed: " . $e->getmessage(); } $conn = null;
i came far looking on stackoverflow , php manual, searched if mysql error. cannot figure out, (for suggestions mysql settings im using linux mint 18)
i recommend use mysql's load data infile, fastest , efficient csv data mysql table. command setup this:
load data infile 'bagadres.csv' table adresses fields terminated ',' enclosed '"' lines terminated '\n' ignore 1 rows
if fields not enclosed quotes, or enclosed other quotes, remove or modify enclosed by
clause. also, ignore 1 rows
ignore first row, make sense assuming first line of file header row (i.e. not actual data column labels).
Comments
Post a Comment