AssFace has asked for the wisdom of the Perl Monks concerning the following question:

I have been running this particular script as a cron job for about 3 months now. The general concept of what it does is:
1) do some stuff that is unimportant and populates arrays
2) at one point it splits up a csv string (on the comma of course) and puts that into a temp array
3) then strip off the quotes that are around that first position and set that as a filename.


My code looked like this:
$tempArray[0] =~ s/\"//g; ;
$fileName = $tempArray[0];

[download]


And for months it worked. (notice the double ";" in the first line - I apparently didn't notice that)


Then today it iterated through nearly 2 times as much data as it had to in the past, and it died in an odd way. It expects the data to always be all caps. It writes it that way, and the data source that we talk to also writes in all caps.
But today it died and when I started putting in print statements to trace the issue, it was because the last character of a string was lowercase.


so I changed the code above to this (which I should have done in the first place):
$tempArray[0] =~ s/\"//g; ; 
$fileName = uc($tempArray[0]);

[download]


and it worked. Again - I didn't notice the extra ";" but it doesn't seem to care.


So... from that I have the following questions:


Why did it work for months and then fail today when it had to do more work? Was that unrelated? Why had it never failed in the past?
Also, is the extra ";" not an issue - I run it with perl -w and it didn't complain.


(and yes, I checked all of the places that it is getting the data from that it looks at - and they are all in caps. I had thought for sure that it was a slip up on the data provider's side and they just accidentally had a lowercase in there - but I looked with a raw nonautomated call to it and it was as it should be.)


is my code doing something outside of what I think it is. I put in the trace statements and those were the two lines at which it broke - but not on the first instance either - it would run through about 20 iterations of the loop, and then always die at the same point.


I'm totally baffled on this one - I thought for sure it was just the data provider belching - but now I want to make sure it isn't my code.
does the extra ";" do something? and if so, what?

Replies are listed 'Best First'.
Re: Case change... why did this happen?
by Mr. Muskrat (Canon) on Jan 22, 2003 at 23:41 UTC

    Why did it happen all of a sudden?
    How do we know.
    It's your system.
    It's your code.

    You only show two lines of it yet you expect us to answer your questions!

    However you did say that you are splitting comma seperated values (without using one of the fine modules from CPAN).

    So I'd have to say that you are probably splitting a quoted section that has an embedded comma.
    blah, blah,"ha ha! I've got an embedded comma,",blah,blah
    You can't do that and expect it to work right.

    I recommend Text::CSV or Text::CSV_XS for the job.

    By the way, an extra; ;
now and then shouldn't hurt your code;;
provided it's at the end of a line; ;
and not in the ; ; ; ; middle of it.

    [download]

[reply]
[d/l]
[select]
      Good to know on the ;;;;;

      As for the csv, I can guarentee that there is never a comma in the string that it is looking at. It is stock data, the first value is a ticker, the rest are dates or numbers that are in a standard format from the data provider (another reason I'd like to figure this out - so I can confront them with an issue with how they handle it if it turns out that it was very likely their data side - which is even more confusing since it then only screwed up when called programatically and not via a straight http connection by hand)

      I only showed two lines because in the debugging that I went through, nothing else had any effect on it - so it seemed to me it would just clutter the post.

      And yeah, I looked at the CSV specific modules, but I am only acting on one line of data so I just did it my way - but I'll try it out with the modules and see how it looks in the benchmarks.

      I was mainly concerned that my matching text was somehow not doing what I imagined it was up to - or that the extra ; was making something behave oddly.
      It seems that Perl has all kinds of lovely things built in that I always seem to find by accident and I was wondering if for some reason a ;space; after a regular expression substitution would then force the last character to be lowercase. Which to me seems insane, but then again, so do some of the other things perl does that make it great for golf :)
[reply]
Re: Case change... why did this happen?
by Cody Pendant (Prior) on Jan 23, 2003 at 00:58 UTC
    I have to say, your post reads a little, let's say "illogically" to me.

    First, you say you know why it failed, because there was a lowercase letter in a certain place -- that can't be the reason if it's your s/\"//g that causes the problem.

    But later on you also say that you checked the data and it's "all in caps", so, that contradicts what you wrote earlier.

    I say, if it ran through 20 iterations then died ("in a weird way"? What was the message in $!?) then I'd say it's an issue with your data on line 19.

    Please post again in more detail and more broadly. You've posted that you have problem X, and the reason you've got it is Situation Y and can someone help with Situation Y -- but all you really know is that you've got problem X.
    --
    “Every bit of code is either naturally related to the problem at hand, or else it's an accidental side effect of the fact that you happened to solve the problem using a digital computer.” M-J D

[reply]
      Okay - good point. I tend to ramble :)

      It doesn't some stuff that gets it started up (which is reading a stock ticker from a file - that stock ticker is all caps). Then it takes that ticker and hits the data feed over http, which then returns a single line of CSVs.
      The CVS line will look something like:
      "ABCD","10/19/2002",45.67,45.67,45.67,9999292922

      I want to get the first value out of there, and strip off the quotes. Then the value that is in there (that our data feed always has in all caps) is used to look up a local filename that matches that.

      The error I got after 20 iterations was showing that the data above (its equivalent) was returning "ABCd" and then when it went to lookup that filename - it wasn't finding it because the filename would be all uppercase.

      I immediately figured it was the datafeed that was screwing up, so I went in through a raw http session and not through the automated perl one and I called that ticker and it came back "ABCD"
      I then thought perhaps there was some weird thing that it just returned what you gave it, so perhaps it was getting butchered prior to that. I tried sending it a mixed case ticker, and it still returned "ABCD"
      I then thought perhaps my code was doing something screwy based on the original file that it was using to lookup via - but after looking, that too was all caps.

      So then I thought, well, perhaps it was a glitch in the data feed and now that I've been hitting it directly it seems to be working again. So I called my perl script again, and it died in the same place again, saying that the file it was looking for was "ABCd"

      I then started looking at that value higher up in the code, and it breaks at the two lines that I included in here with my question - hence why I'm baffled.

      And like I said, it was so bizarre and then I saw the double ; ; at the end of that line and I wondered if perhaps that somehow was something special that I was unaware of in Perl - it didn't make sense to me, but that isn't to say that all of Perl makes intuitive sense to me straight off as it is.

      was that a better explaination?
[reply]
        It can't be that you get "ABCd" when it's "ABCD" that is there. Try again, and this time, let Perl help you debug it. Add code like: 
        print "Oops, '$fileName' ne '\U$fileName\E' at line $." if $fileName n
+e uc($fileName);

        [download]
        Surely, it'll have to warn you at some point, otherwise using the uc() wouldn't make a difference.
[reply]
[d/l]
        >was that a better explaination?

        Much better, but I'm still baffled.

        One thing -- you don't need to escape the quotes in your s/\"//g -- if that's something other than a quote, I guess something weird could be happening...

        OK get your script to print out like this: 

        
print "first item in the array is $myArray[0] \n";
print "doing the replace: \n";

if($myArray[0] =~ s/"//g){
  print "it happened\n";
} else {
  print "nothing to replace, apparently\n";
}

print "first item in the array is now $myArray[0] \n";

        [download]

        and see what that says. It's either uppercase or lowercase, it can't be both...
        --
        “Every bit of code is either naturally related to the problem at hand, or else it's an accidental side effect of the fact that you happened to solve the problem using a digital computer.” M-J D

[reply]
[d/l]
Re: Case change... why did this happen?
by AssFace (Pilgrim) on Jan 23, 2003 at 14:32 UTC
    Well - thanks for the help guys.

    I'm going to assume at this point that it was an error on the data feed side and smack my knuckles with a ruler for not forcing the upper case on the data that I was assuming to be upper case.

    It appears that my manual/programatic tests just happened to match when the data was screwed up - the chances of that seem to be stupidly high - but this worked for months, works now - and I can no longer repeat it today with the old or the new code.

    Again - thanks for the help guys - when things like this happen to other people, I scoff at how they are obviously overlooking this or that - but when things are happening right in front of you that you know "can't" be happening, it is a weird thing.
[reply]

Back to Seekers of Perl Wisdom