When searching for files using find it is somewhat common to want to exclude
certain directories from the search. E.g. we usually don’t want to
search for files inside of .git directories, or other version control- or
programming language-specific directories (like node_modules).
How to achieve this has obviously been discussed on StackOverflow (see
links below), but I had to play around with different find invocations
until it really made sense to me.
This was a great opportunity to try and understand find a bit better.
For the examples here I am in my dotfiles folder and want to search for
files while ignoring all .git folders. (My dotfiles folder contains
multiple .git directories because Vim plugins are cloned into a
subfolder).
tl;dr: find . ! \( -name .git -prune \)
Using find to list everything in that folder tells me there are 7334 things (folders, files, links, who-knows-what):
$ find . |wc -l
7334
So that’s the total including everything inside any .git directory.
Obviously what we could do to exclude search results with a certain
pattern is to refine our find invocation.
$ find . ! \( -path '*/.git/*' -o -path '*/.git' \) |wc -l
3805
This excludes matches inside of .git directories and the .git
directories themselves. It still returns other files containing .git
in their name like .gitignore and .gitconfig files.
(Notice that alternatively we could write this without the parentheses:)
$ find . ! -path '*/.git/*' ! -path '*/.git' |wc -l
3805
The problem with this solution is that while find excludes everything
inside .git folders from the results list, it still goes into those
folders and searches for files inside of them. But that’s a waste of time
and probably one of the main reasons why we want to ignore certain
folders in the first place.
We can prevent find from recursing into directories using the -prune
action. We match the folders we want to ignore – .git in our case –
and tell find to prune the search tree.
$ find . -type d -name '.git' -prune
But wait: This just prints all the .git folders, that’s the opposite
of what we want! The command returned the .git folders because they
were successful matches in our search; -prune does not change that.
And we didn’t provide any additional search parameters to specify what
we do want to match, so there won’t be any other results. We will fix
that later.
But first: How do we know whether find recursed into the pruned
directories or not? We did tell find to not step into .git
directories, but how can we verify that?
We could search for files that we know are inside of pruned subtrees.
E.g. since in our example we want to ignore .git directories and we
know that each of them contains a file called HEAD we could search for
these files:
$ find . -type d -name '.git' -prune -o -type f -name 'HEAD' |grep HEAD |wc -l
0
And we will see that we don’t find any HEAD file inside a .git folder.
(Obviously if we had files called HEAD outside of .git
folders, those would have been found).
We can also try to benchmark the find command somehow – ignoring directories can be a search performance optimization after all.
Unless the directories we ignore are relatively huge, we won’t see any
significant difference in execution time using time. So I’d rather
look at some other stats.
find has some debug output capabilities, but I felt the quickest way
to tell how many folders find actually searched, was to use strace.
With a quick little trace for openat syscalls we can see some
differences (I added wc again, to show that the commands matched the
same results):
$ strace -cU calls -e trace=openat find . -type d -name '.git' |wc -l
41
calls syscall
--------- ----------------
2056 openat
--------- ----------------
2056 total
$ strace -cU calls -e trace=openat find . -type d -name '.git' -prune |wc -l
41
calls syscall
--------- ----------------
593 openat
--------- ----------------
593 total
We can see that the resulting output is the same (here, I just
show the number of output lines, but the lines themselves are also the
same) but the number of openat syscalls was substantially lower. That
seems to indicate that find did indeed do less work!
Now let’s get rid of the pruned directories in our output. There are
multiple ways to do this so I will show a couple of examples. For these
examples let’s say we want to find all files named config. .git
directories contain such a file but I also have some other files named
config in my dotfiles directory.
$ find . -type d -name '.git' -prune ! -path '*.git' -o -type f -name 'config' |wc -l
3
The find invocation above matches things that are .git directories, pruning their
subtrees, and that do not contain .git in their path. It also matches
all files named config. It is a little more concise to combine the
negation of the .git matches with their pruning, like so:
$ find . ! \( -type d -name '.git' -prune \) -type f -name 'config' |wc -l
3
(The parentheses have to be escaped because they should be passed to
find and not interpreted by our shell.)
Alternatively, we could use an “or” with an explicit -print action
like this:
$ find . -type d -name '.git' -prune -o -type f -name 'config' -print |wc -l
3
I’m not sure, why find behaves this way, but explicitly adding the
-print action to the second clause of the pattern, suppresses the
printing of the clause that was given the -prune action.
If you know more about the logic behind which action find performs on
matches, or other neat functionality based on find features, please
let me know!
See also: