I ran into this problem the other day and I scratch my head for a long time before I found this solution. I better save this one here for future reference 🙂

I just did a simple operation like this:

grep 123456789 files/*

and this resulted in a “bash: /bin/grep: Argument list too long”. The reason for this is that the files directory contains about 24000 files. This is a little to much for grep to handle like that.

but if you do like this it works (takes quite some time though)

find files/ -type f | xargs grep 123456789

Ever wondered how long time a command takes to run? I did. A colleague showed me the time command. I’m here going to try to show how you can use it.

time ls

This will output something like this (plus the ls commands output):

real    0m0.369s                                                                                                                                           
user    0m0.002s                                                                                                             
sys     0m0.007s

Where the ‘real’ value is the one to look for. Pretty easy 🙂

Sometimes in my work I need to do something really cleaver and fast. Here is a bunch of my ‘helpers’ to make that happen

Grep in gzip files

zgrep search_string filename.gz

This will return all rows in file.gz that contains search_string

Get filenames from files in current directory containing search string

grep -l search_string *

The -l option tells grep to only return the filenames of the file that contains search_string

Delete files in current directory containing search string

grep -l search_string * | xargs rm -f

Here we send a list of filenames to the xargs program which will do ‘rm -f’ on each one

Count files in directory (NOTE: this will also count . and .. as lines)

ls -la | wc -l

The wc option -l counts lines instead of words

Count lines in file (NOTE this only works if the file contains newlines otherwise it will naturally return 0)

wc -l filename

This will count all newline characters in a file

Get filnames in current directory of files that contains search_string1 OR search_string2

egrep -l 'search_string1|search_string2'  *

The -l option works the same way as for grep. It only returns the filename of the matched files. I use egrep here to be able to use the regular expression OR operator ‘|’.

Delete files in directory that does NOT contain search_string1 OR search_string2 OR search_string3

egrep -lv 'search_string1|search_string2|search_string3' dir/* | xargs rm -f

The -v option tells egrep to return everything NOT equal to the pattern

Copy files in one directory to another directory where filname matches egrep regular expresion

cp $(find fromdir | egrep 'file[0-9]{1,2}.xml' ) todir/

This fetches all files with name file0.xml and to file99.xml from ‘fromdir’ and copies them to ‘todir’. Works just fine with mv if you rather move then copy your files

Copy files from current directory containing search_string to a new directory

cp $(grep -l search_string *) toDir/

Validate all xml files in a directory

xmllint -noout dir/*

If there are any errors in any file the error will be displayed. If no errors – no output from the command. I use the –noout option because I do not want to display all the xmls which is default behaviour of xmllint

Find specific xml tags from a directory of files that contain one row xmls

xmllint --format * | grep tagname

This will first make all xml into nice print. After that we get the desired tag through grep

Pipe output from xmllint (or other program) to Vim

xmllint --format * | vim -

This will make nice print on all xml files in current directory and then put them all on one page in Vim. The trick is in the ‘-‘ sign. This will make Vim read input from stdin instead from a file.

Search and replace text in a file with sed

sed -i 's/find/replace/g' file.txt

This will find the word ‘find’ in file.txt and replace it with the word ‘replace’. The changes will be made directly to the file

Print all text between tag1 and tag2

 sed -n "/tag1/,/tag2/p" <filename>

This will print all text between tag1 and tag2 in the file <filename>. I use this to get xml messages from logfiles where many xml messages are in one file

Working with columns

ls -l | awk -F" " '{print $5, $6, $7, $8, $9}'

This will select the last 5 columns of a ls -l command (‘size’, ‘month’, ‘day’, ‘time’ and ‘filename’. Delimiter in this case is space (” “).