string? Or binary garbage?

argv has asked for the wisdom of the Perl Monks concerning the following question:

I'm using Josh Carter's IPTCInfo package on cpan that reads the IPTC header from an image (e.g., jpg) and fills in fields such as "author", "location", etc. The problem is, some data may be corrupted, or perhaps unintelligible because it's written in a different language. Is there a way to know which?

At first, it seemed simple to just check for normal ascii, but then it occurs to me that I want to accept certain accented characters, like the é in café, and so on...

Before I go off writing some routine that checks for santiy in a string to see if it really is english text instead of arbitrary gobblygook, I figured maybe someone had such a thing. Even if I only look at the first N characters in a string, that'd be fine.

Again, the brute force intuitive step would be to just do something like

$string =~ /([\s\w]){25})/
[download]

but this seems like a hornet's nest of little gotchas where people have learned it ain't that simple.

Comment on string? Or binary garbage? Download Code

Replies are listed 'Best First'.
Re: string? Or binary garbage? by Albannach (Monsignor) on Dec 01, 2004 at 00:49 UTC
It looks like a good place to start would be Lingua::Identify, which 'knows' 26 languages so far. I imagine that the IPTC headers should be short enough that you could check them whole, as I expect a larger sample would give more confidence, but some experimenting may be in order on that point. -- I'd like to be able to assign to an luser	[reply]
Re^2: string? Or binary garbage? by cog (Parson) on Dec 01, 2004 at 09:30 UTC
And if you have any problem with it, you can always bug the author :-) Why don't you send him some examples so that he can tell you if Lingua::Identify is the right tool for that? :-) I'm pretty sure he'd like to help O:-) It is possible that it isn't, but it doesn't hurt trying, and L::I hasn't reached a final version yet, so maybe there's place for something like that. Bug the author!! :-) Definitely :-)	[reply]
Re: string? Or binary garbage? by davido (Cardinal) on Dec 01, 2004 at 00:28 UTC
Would File::Type be of any help? You can feed data (not just filenames) to File::Type, and it does its best job of magically figuring out what you're looking at. Not sure if that will help or not, but it might be a start. Dave	[reply]
Re^2: string? Or binary garbage? by argv (Pilgrim) on Dec 01, 2004 at 01:11 UTC
Given the expense of the application and that it has to ierate over thousands of images at a time, I don't want to look too far into other packages for heavy overhead. Now that I think of it, I just want to see if there are any unprintable control characters... Looking at the perldocs, I should be able to use `$a =~ /\p{IsC}/` [download] ...which says, "crazy control characters and such." However, that seems to be matching legitimate text in some cases. So, I try `$a !~ /\p{IsPrint}/` [download] but that doesn't work either. Then I tried: `$a !~ /\p{IsASCII}/` [download] but that's also not working. I can get into the details on why these things aren't working, but I keep thinking someone's going to pipe in with a statement like, "I ran into this ages ago, and here's a routine that finds all the exceptions..." If not, I suppose I can elaborate on how this isn't working.	[reply] [d/l] [select]
Re^3: string? Or binary garbage? by argv (Pilgrim) on Dec 01, 2004 at 01:57 UTC
Well, I fixed the problem myself. In my previous example, I had: `$a !~ /\p{IsPrint}/` [download] which tests whether $a does NOT match any printing characters. (In other words, the test fails if it DOES match a PRINTABLE character.) So, what I wanted to do is close, but not quite the same thing: `$a =~ /\P{IsPrint}/` [download] which basically tests "does $a match the compliment of a printing character?" In other words, "are there ANY non-printing characters here?" One has to look really closely and speak out the logic into a dark empty room before knowing intuitively which was the right choice ahead of time.	[reply] [d/l] [select]
Re^4: string? Or binary garbage? by hakkr (Chaplain) on Dec 01, 2004 at 15:55 UTC
Re: string? Or binary garbage? by Anonymous Monk on Dec 02, 2004 at 13:06 UTC
The problem is, some data may be corrupted, or perhaps unintelligible because it's written in a different language. Different language from what? What does IPTC say it has to be?	[reply]