Just for fun, how long to import data quiz

This week i was playing with some Mapping stuff and wanted the centre points of a group of post codes in the UK. The only free source i could find was the UK Office of National Statistics who provided the data in a .csv as a singe UK wide file. This contained (as i now know) 2,599,813 records.

I used MySQL Workbench to import the .csv file into a database table.
The setup in use was Windows/ Maria DB community server running on a Core i9, 32GB ram, M2 1GB SSD.

How long do you think it took to import all records?
Groups exclude lower value so 48 hours means less than 48, more than or equal to 24

  • Under 2 Hours
  • Under 8 Hours
  • Under 24 Hours
  • Under 48 Hours
  • Under 72 Hours
  • Longer then 72 Hours
0 voters
2 Likes

Old gamers may remember PlanetLAN (we had sites in Germany, Austria, Switzerland, the UK and we did venture in to the US for a while). Besides the US site we imported all the Geo Data for every single postcode for each country to populate our LAN party maps. Enter your postcode set a radius (ie 25 miles) and it would return LAN parties within your specified area. This must have been around 1999 - 2000. I can remember taking a trip to Telehouse in London to one of our racks and doing this for the UK data set. I think it was about 1.5 million records... The nightmare was actually getting the data from the Post Office. Had to jump through all types of hoops and pay a small fortune to license it.

Crazy to think everything Social Media has now we did back then, and more, a real community of like-minded people. Were some fun and challenging times. Your post brings back fond memories @Hyperbytes

2 Likes

GIVEN:

  1. Assuming 100 bytes per record, the CSV would be around 300MB
  2. M.2 SSD speed of 3-5 GB/s
  3. MariaDB can handle 10K - 50K rows per second.

OUTCOME:

  1. Best case 1-2 minutes
  2. Typical case 3-10 minutes
  3. Worst case 15+ minutes

Well,the import time was 80 hours!

1 Like

Workbench must have been the bottleneck. That is crazy!

It was really fast at first then seemed to slow down exponentially as the records count increased. Final rate was about 700 records per minute!

Just done same with LOAD DATA LOCAL INFILE statement, took a couple of seconds!

2 Likes