Re: Getting File Type using Regular Expressions

Of course, Windows applications make a lot of assumptions about files based on their filenames (especially their extensions).

Yep, that's pretty dumb.

You may want to check file type by investigating the contents, not the filename.

That makes some sense. But not overly because it's pretty hard to do.

The first two or four bytes of most files are often a very good clue as to the file's type. These bytes are usually referred as a "magic number."

<rant>
Well, guessing the type of the content of a file based on the first two bytes (or rather, the first couple of bytes, /etc/magic allows for variable formats) is not much smarter than using the file name. Sure, you are free to choose your filename - but who takes advantage of that? Noone stores gif images in files ending in .pl, and if you put your C program in a file called "fuddly-bumps.html", chances are your compiler will not take you seriously and refuse to compile your program. Not to mention that the classical Unix build program, make entirely depends on filenames to build the targets. And yes, another advantage is that you don't need a filename to make a guess. But the disadvantage is that the magic number gets in the way. Not much of a problem for binary formats which are purely processed by programs. But annoying, and prone to error for anything edited by humans. Furthermore, it still is uncontrolled guesswork (just like file-extensions). Anyone can invent a magic number, whether it's in use or not, there's no official way of keeping track, making sure there are no collisions etc. Here's a small example of the dumbness of magic numbers:

The Netpbm project uses several (related) file formats. The magic numbers are "P1", "P2", "P3", "P4", "P5" and "P6". Looks simple. Looks extentable as well, doesn't? If more formats are needed, just continue the numbering. "P7", "P8", "P9", "P10". Right? No. If you start a file with a P followed by a 1, regardless of what follows, file thinks it's a "Netpbm PBM image text". Even if it's a simple text file that starts with the sentence "P100s of Samsung are really cool phones".

My point is that magic numbers suck as bad as file extensions. Both magic numbers and file extensions work in practise reasonably well because people follow de facto standards. Windows uses file extensions almost exclusively. Unix (and with that, I mostly mean Unix tools) rely on both. Some tools use magic numbers. Some use tools use file extensions. Some use both.
</rant>

Abigail

Comment on Re: Getting File Type using Regular Expressions

Replies are listed 'Best First'.
Re: Re: Getting File Type using Regular Expressions by halley (Prior) on Apr 21, 2004 at 14:57 UTC
All we have in any situation is context and convention. Intuition won't solve everything, and computational completeness won't solve everything. Perhaps the byte sequence "`P100s of Samsung are really cool phones`" is a perfectly well-formed 6x6 pixel GIF file. You can only guess at the intent, and more data gives a better guess. That's why they call them 'heuristics.' That said, malicious users will attack any such heuristic assumptions to their favor. `Britney.jpg.exe` If your upload code expects web-intended images and only wants to accept web-intended images, it benefits the system to expect that any available heuristic passes muster. If it's not .jpg, toss it. If it's not JPEG magic, toss it. If the ImageMagic tool says the pixel dimensions are over 10000 in either dimension, toss it. -- `[ e d @ h a l l e y . c c ]`	[reply] [d/l] [select]
Re: Getting File Type using Regular Expressions by Abigail-II (Bishop) on Apr 21, 2004 at 16:38 UTC
Perhaps the byte sequence "P100s of Samsung are really cool phones" is a perfectly well-formed 6x6 pixel GIF file. No, it isn't. I didn't pick my example lightly. It's not a valid GIF or Netpbm PBM file. But even if it would be valid, it strengthens my argument, that looking at the first couple of bytes to guess the type of content is as bug-ridden as just looking at the file name. That said, malicious users will attack any such heuristic assumptions to their favor. Britney.jpg.exe If your upload code expects web-intended images and only wants to accept web-intended images, it benefits the system to expect that any available heuristic passes muster. Interesting that you bring up the web. See, the people who initially designed HTTP got it right. Documents delivered over HTTP are tagged with their type. No inspection of a fragment of the data is required. No implied type derived from the URL. Instead, a clear separation of information (the content) and the meta-information (the content type). Simple. Logical. Not error prone. Isn't that what MacOS does with its resource and data forks? Abigail	[reply]
Re: Re: Getting File Type using Regular Expressions by halley (Prior) on Apr 21, 2004 at 17:46 UTC
The HTTP protocol offers a MIME type. Where does the server get it? In most configurations, it guesses it. Download a RedHat Package Manager (.rpm) file, and you get application/x-realplayer-media (.rpm) mime type. Oops. Why? Because it used an imperfect heuristic scheme to specify the mime type. Neither the Windows filesystem nor the Unix filesystem offers typing metadata. Resource forks and Windows .rc/.res resource files offer typing metadata inside their encoded contents. It's just another example when you and I are in "violent agreement" over some topic. Metadata and type-determination methods are weak, and usually poorly implemented. In my case, I just suggested that two heuristics are better than one, instead of ranting that it's all futile unless you have God's Gift to Omniscient Filesystems. -- `[ e d @ h a l l e y . c c ]`	[reply]
Re: Re: Getting File Type using Regular Expressions by Anonymous Monk on Apr 21, 2004 at 17:12 UTC
It is funny, out of that entire post I did not see a valid recommendation at all. All I saw was bashing of others recomendations. An easy (but not fool proof) way of distinguishing between a text file or a binary file is use the ready-made -B file check operator. Or the -T (depending on which way your flag flies). `print "File is binary\n" if (-B); print "File is a text file\n" if (-T);` [download] or if you are a "NOT" guy/girl the following might suit your fancy. `print "File is binary\n" if (!-T); print "File is a text file\n" if (!-B);` [download] Hope that helps some.	[reply] [d/l] [select]