Combining CSV Files and Verifying Line Count

Situation:
1500 .csv files bogging down an import system due to the number of files.

Solution:
Combine files into 1 file.

Problem #1:
All files have a header row with field names.

Solution #1:
Combine all the files while stripping off the

find . -name "*.csv" | xargs -n 1 tail -n +2 > big.csv

Problem #2:
I’m not sure if I have all the lines I should.

Solution #2:
Verify number of lines in the file and compare to row count from a database.

sed -n '$=' big.csv

Problem #3:
This removed the header row that I need at the very top of the file.

Solution #3:
Create file (header.csv) with a single line containing the field names that we stripped off earlier.
Then take header and big and combing them into our final csv file:

cat header.csv big.csv > final.csv

Advertisements

The Mac equivalent of the Apple ][ speed command

How to slow your connection down, or the Mac equivalent of the Apple ][ speed command.

found here

Configure a pipe that has the appropriate bandwidth limit and delay.
sudo ipfw pipe 1 config bw 16Kbit/s delay 350ms

Attach it to all traffic going to or from port 80.
sudo ipfw add 1 pipe 1 src-port 80
sudo ipfw add 2 pipe 1 dst-port 80

Now traffic coming from or going to port 80 anywhere is limited by the pipe that you specified. Do your testing and once you get frustrated with slow access to the web remove the rules like so:
sudo ipfw delete 1
sudo ipfw delete 2

Finally, delete the now defunct pipe like so:
sudo ipfw pipe 1 delete

 

put together in one bash script – adjust the kbit/s and delay to suit your needs

sudo ipfw pipe 1 config bw 56Kbit/s delay 500ms
sudo ipfw add 1 pipe 1 src-port 80
sudo ipfw add 2 pipe 1 dst-port 80
#
read -t 1600 -p "Hit ENTER to restore"
sudo ipfw delete 1
sudo ipfw delete 2
sudo ipfw pipe 1 delete

Zend Framework – ZendMail timeout in validate hostname on a mac

I develop on a Mac. I love my Mac. I use Mamp Pro for my *amp stack. After beating my head against the wall for a couple of hours, including several trips to GoogleLand, I finally found a stackoverflow article about iconv_strlen causing execution timeouts.

I am now upgraded to Mamp Pro 2.x.

Happy news: it upgraded my mysql without losing any data – in fact I didn’t need to do anything.

Concerning news: it warned me that my “templates” had changed. I had changed my php.ini files. That might prove to be a pain down the road – we’ll see. I did take a backup of them.