1500 .csv files bogging down an import system due to the number of files.
Combine files into 1 file.
All files have a header row with field names.
Combine all the files while stripping off the
find . -name "*.csv" | xargs -n 1 tail -n +2 > big.csv
I’m not sure if I have all the lines I should.
Verify number of lines in the file and compare to row count from a database.
sed -n '$=' big.csv
This removed the header row that I need at the very top of the file.
Create file (header.csv) with a single line containing the field names that we stripped off earlier.
Then take header and big and combing them into our final csv file:
cat header.csv big.csv > final.csv