Situation:
1500 .csv files bogging down an import system due to the number of files.

Solution:
Combine files into 1 file.

Problem #1:
All files have a header row with field names.

Solution #1:
Combine all the files while stripping off the

find . -name "*.csv" | xargs -n 1 tail -n +2 > big.csv

Problem #2:
I’m not sure if I have all the lines I should.

Solution #2:
Verify number of lines in the file and compare to row count from a database.

sed -n '$=' big.csv

Problem #3:
This removed the header row that I need at the very top of the file.

Solution #3:
Create file (header.csv) with a single line containing the field names that we stripped off earlier.
Then take header and big and combing them into our final csv file:

cat header.csv big.csv > final.csv

How to slow your connection down, or the Mac equivalent of the Apple ][ speed command.

found here

Configure a pipe that has the appropriate bandwidth limit and delay.
sudo ipfw pipe 1 config bw 16Kbit/s delay 350ms

Attach it to all traffic going to or from port 80.
sudo ipfw add 1 pipe 1 src-port 80
sudo ipfw add 2 pipe 1 dst-port 80

Now traffic coming from or going to port 80 anywhere is limited by the pipe that you specified. Do your testing and once you get frustrated with slow access to the web remove the rules like so:
sudo ipfw delete 1
sudo ipfw delete 2

Finally, delete the now defunct pipe like so:
sudo ipfw pipe 1 delete

 

put together in one bash script – adjust the kbit/s and delay to suit your needs

sudo ipfw pipe 1 config bw 56Kbit/s delay 500ms
sudo ipfw add 1 pipe 1 src-port 80
sudo ipfw add 2 pipe 1 dst-port 80
#
read -t 1600 -p "Hit ENTER to restore"
sudo ipfw delete 1
sudo ipfw delete 2
sudo ipfw pipe 1 delete


After many failed searches and much frustration I finally found a download for the Command Line Tools for Xcode.

Visit:

https://developer.apple.com/downloads/index.action

Then search for command line tools

The only checkbox you should have checked is “Developer Tools”

Reference the following screenshot: