yargs -- xargs but handle spaces in filenames

I was a bit shocked (way back) when I discovered that xargs doesn't handle spaces in filenames. /bin/find writes out filenames one per line (separates filenames by newlines), so I had long assumed that xargs read arguments one-per-line and spaces wouldn't be a problem. But xargs splits the lines that it reads in on whitespace.

The standard solution (in a few tools) is a rather ugly 'find -print0' and 'xargs -0'. The fix is nice in that it is a general solution to the rather stupid fact that you can even put newlines into file names on most Unix filesystems. It is particularly ugly in that there are tons of things that don't support the idiom of "read from a pipe a list of '\0'-separated strings" and/or don't support the idiom of "write to a pipe a list of '\0'-separated strings". For example, "find ... -print0 | grep ... | sed ... | xargs -0 ..." isn't likely to work properly.

I repeatedly run into cases where somebody thought it was reasonable to include a space in the name of a file. I have never run into a case where somebody thought it was reasonable to include a newline in a filename. So I want xargs to handle spaces in filenames without having to stop using all of my tools that know how to deal with filtering lines of text. So I wrote yargs. It is terribly complicated:

#!/bin/sh

perl -pe's/ /\\ /g' | xargs "$@"
[download]

- tye

Comment on yargs -- xargs but handle spaces in filenames Download Code

Replies are listed 'Best First'.
Re: yargs -- xargs but handle spaces in filenames by Your Mother (Archbishop) on Jul 11, 2009 at 07:03 UTC
Mmmmmmmm... I don't know. Sounds like an X/Y-args problem. Ha-ha-hahahah-ha-ha. Oh, heh-heh. Uh, ++.	[reply]
Re^2: yargs -- xargs but handle spaces in filenames by Anonymous Monk on Jul 11, 2009 at 07:16 UTC
Oh Mother :->	[reply]
Re: yargs -- xargs but handle spaces in filenames by ambrus (Abbot) on Jul 11, 2009 at 09:13 UTC
You could also try `xargs -d \\n` if you're using GNU xargs. (This likely won't work on FreeBSD xargs.)	[reply] [d/l]
Re: yargs -- xargs but handle spaces in filenames by zwon (Abbot) on Jul 11, 2009 at 08:58 UTC
I have never run into a case where somebody thought it was reasonable to include a newline in a filename. What about double quotes or backslashes? I propose the following: `#!/bin/bash perl -pe'chomp; s/$/\0/' \| xargs -0 "$@"` [download]	[reply] [d/l]
Re^2: yargs -- xargs but handle spaces in filenames by ikegami (Patriarch) on Jul 13, 2009 at 18:28 UTC
Simpler: `perl -pe's/\n/\0/' \| ...` [download] Terser: `perl -pl0e1 \| ...` [download]	[reply] [d/l] [select]
Re: yargs -- xargs but handle spaces in filenames by graff (Chancellor) on Jul 11, 2009 at 14:11 UTC
I have never run into a case where somebody thought it was reasonable to include a newline in a filename. Alas, I have run into a few cases where someone managed to include a newline in a filename without intending to. It happens. For example, "find ... -print0 \| grep ... \| sed ... \| xargs -0 ..." isn't likely to work properly. But something like this is likely to work just fine: `find ... -print0 \| perl -0 -lne 'if(m{...}){s{..}{..};print}' \| xarg +s -0 ...` [download] In fact, I think that's better overall as a general solution for this sort of problem. (updated snippet to avoid using "/" as a regex delimiter, since "/" would often be used in the regex)	[reply] [d/l]
Re: yargs -- xargs but handle spaces in filenames by shmem (Chancellor) on Jul 11, 2009 at 22:38 UTC
Well, erm... why perl? `#!/bin/sh sed -e 's/ /\\ /g' \| xargs "$@"` [download] does the same. But it isn't what you want.	[reply] [d/l]
Re: yargs -- xargs but handle spaces in filenames by afoken (Chancellor) on Jul 11, 2009 at 18:27 UTC
I use a wrapper for find and xargs, see Re^4: need help for search and replace oneliner. Because it does use find -print0 and xargs -0, it has no problems with arbitary characters in file paths, all without the need for escaping. Alexander -- Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)	[reply]
Re: yargs -- xargs but handle spaces in filenames by repellent (Priest) on Jul 13, 2009 at 02:47 UTC
I have found this to be the most robust approach to wacky filenames output by find: `tr "\n" "\0" \| xargs -0 -E '' -I '{}' "$@" '{}'` [download]	[reply] [d/l]
Re: yargs -- xargs but handle spaces in filenames by leocharre (Priest) on Jul 13, 2009 at 14:56 UTC
I hear ya. It's frustrating. Using find -exec has its problems.. If you'll calling a perl script for example, it could be pretty slow to do this: find ./ -type f -iname "whatever" -exec perlscript '{}' \; Imagine you're using a db too, opening that connection can be expensive. xargs rawks.. The following is an example that tells xargs that the delimiter is not the posix standard space, but the newline.. Find all files named jpg/JPG larger than one meg, store that to a text file: find ./ -type f -iname "*jpg" -size +1M > /tmp/found Use xargs (version 4.4.0 +), tell it the delimiter is a newline, pass this to mogrify to reduce size of files.. xargs --delimiter=\\n --arg-file=/tmp/found mogrify -resize 800x600	[reply]
Re^2: yargs -- xargs but handle spaces in filenames by pingo (Hermit) on Jul 16, 2009 at 12:55 UTC
Same thing, slightly more convoluted (and I'm not sure it works outside bash): `xargs --delimiter=\\n --arg-file=<(find ./ -type f -iname "*jpg" -size + +1M) mogrify -resize 800x600` [download]	[reply] [d/l]
Re^3: yargs -- xargs but handle spaces in filenames by ikegami (Patriarch) on Jul 16, 2009 at 14:39 UTC
leocharre pointed out that `<()` is overkill here, but it's very useful elsewhere, such as with `diff`: `diff -u <( perl version1.pl 2>&1 ) <( perl version2.pl 2>&1 )` [download]	[reply] [d/l] [select]
Re^3: yargs -- xargs but handle spaces in filenames by leocharre (Priest) on Jul 16, 2009 at 14:31 UTC
`find ./ -type f -iname "*jpg" -size +1M \| xargs --delimiter=\\n mogrif +y -resize 800x600` [download] Edit by tye to replace PRE tags with CODE tags	[reply] [d/l]
Re^4: yargs -- xargs but handle spaces in filenames by pingo (Hermit) on Jul 16, 2009 at 14:46 UTC
Re: yargs -- xargs but handle spaces in filenames by JavaFan (Canon) on Jul 12, 2009 at 00:15 UTC
I have never run into a case where somebody thought it was reasonable to include a newline in a filename. I have, but that was just for the sake of doing unexpected things. But ignoring those cases, I have come across filenames containing newlines. Not intentionally, but created by accident. I've even seen file names with one char: the newline. Now, such cases you probably just want to delete or rename such files before running ?args anyway.	[reply]

xargs rawks..