Making assessments

A few months ago (end of July) we had an intrusion into an old PerlMonks database that created and continues to create an on-going discussion and strong emotions. The question I'd like to know is what material impact has the password debacle had on PerlMonks?

I'd like to know our gut feelings and I'd like to put some numbers behind that. Usually when someone has an idea for an interesting new set of site stats they code it up and then present a post to show it off. But I want to reverse the process. Before I start coding or make requests for missing information, I'd like to discuss this.

What differences have you noticed, if any? Consider site traffic volume, composition of users visiting, quality of questions and responses, number of nodes. Is there anything missing from this list? What other indicators should be considered?

Any attempt to quantify our gut observations is likely to be ambiguous at best. Numbers never tell the whole story. Worse yet, to quantify something we often need to use a proxy that only partially correlates with the behavior we want to measure.

However, observation is vulnerable to issues of salience and sometimes wishful thinking. It is human nature to see what we want to see. But even when we try to be objective, things have to catch our attention for us to observe them. We tend to give more weight to the things we care about than the things we don't. What numbers would give us objective information that we could use to counter bias in our observations?

Here are some of my preliminary ideas. I'd like to see before and after trends in the following statistics. I emphasize the word "trend" because there could be normal seasonal variations in PM traffic and I wouldn't want to confuse that with impact analysis.

rate of new user creation.
number of daily users by level, frequency of visit, and length of membership
number of daily "unique visitors"
number of anonymous monk posts
votes per day, also by level, frequency of visit, and length of membership
posts per day, by level

What I'd like to know is: did the exploit change the behavior of monks in any way? Is the impact, if any, primarily among experienced regular users or occassional and/or low level users?

If for example, the impact is primarly in the rate of non-spam new user creation, I would surmise that we likely have suffered significant PR damage (or we have an SEO problem). If long term members are visiting less often, voting less regularly, or posting less frequently I might worry about loss of trust in the established Perl community. I would hope that such numbers would show no statistically significant impact. But I think it is important to know, even if the results are scary or painful. If there is damage, we need to correct it. It is impossible to know the right course of action unless we understand the nature of that damage.

What do you want to know? What would you do to measure it? What implications would you draw from the data? What corrective action would that imply?

Best, beth

Comment on Making assessments

Replies are listed 'Best First'.
Re: Making assessments by gwadej (Chaplain) on Oct 08, 2009 at 13:41 UTC
Although some people really seemed to be upset, my impression was also business as usual. I was a little embarrassed for PerlMonks, as were several others. But, it was mostly a short-term thing. I suspect that many of the longer term members have been in cases where they tripped over a really bad mistake they made and had to deal with it before. I think people who had been in that situation in the past, got over the problem pretty quickly and continued to use the site as before. A very wise mentor of mine once told me that the main difference between a novice and an experienced programmer can be seen when they find a bug in their code. A novice will actually say I can't believe I made that stupid mistake. An experienced programmer probably wouldn't say that. We have learned that we can be that stupid, so we correct the error (as best as we can) and move on. G. Wade	[reply]
Re: Making assessments by mje (Curate) on Oct 08, 2009 at 12:55 UTC
I'd like to know our gut feelings I've noticed no real differences I can put my finger on but of course that is not to say there aren't any. I was caught out a long time ago using the same password at multiple places and learned my lesson then so my perl monks password was not used elsewhere. As a result, the worst thing that could have happened (had my password been outed as it appears it was for some) is that someone could have logged into perl monks and pretended to be me . I have difficulty in imagining that could have caused any real long term harm that could not have been put right (as a mere Pilgrim I didn't have much to lose anyway). The fact that the password is still 8 characters and may or may not be disguised/hashed whatever in the database still to this day does not overly concern me other than what the world outside might think about it. The "Users, please read the following important update: Status of Recent User Information Leak" message seems to have been on the monastery gates for a long time now and I wonder if that might put some people off joining. As far as I can see it serves little purpose for any anonymous visitors and in any case a) how many signed up members start at the monastery gates b) often you cannot see the message because a front paged article is formatted such it is off the right of the screen. "did the exploit change the behavior of monks in any way?" I'd guess most changed their password to something they do not use elsewhere ;-)	[reply]
Re^2: Making assessments by markuhs (Scribe) on Oct 08, 2009 at 16:51 UTC
I'm quite new here, so my password was not made public... The "Users, please read the following important update: Status of Recent User Information Leak" message seems to have been on the monastery gates for a long time now and I wonder if that might put some people off joining. As far as I can see it serves little purpose for any anonymous visitors I would agree with mje, that it is a bit confusing for new users. It was for me. But, it is fair! I read it and decided to join. Everybody makes mistakes, but the way things are communicated makes me confident, that this error has been taken care of seriously. And God willing it will not happen again... So leave it or take it away, I assume both will attract some and repel others. Lukas	[reply]
Re: Making assessments by CountZero (Bishop) on Oct 08, 2009 at 13:11 UTC
After the initial panic, very much business as usual I would say, but of course that is not backed-up by any scientific analysis or statistical survey. CountZero A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James	[reply]
Re: Making assessments by goibhniu (Hermit) on Oct 09, 2009 at 19:16 UTC
I'll describe my personal pattern of use rather than a possibly inaccurate assessment of use of the site as a whole. Then I'll leave it to you to decide what stats to look for to determine what might be a larger pattern. Frequency of visits: I was admittedly in a lull in my frequency of visits even before the incident. I maintained my low frequency of visits since the intrusion. Frequency of logins: I used to login every time I came. Now, I work as anonymonk unless necessary (I never post as anonymonk; my postings are my own). Necessary = posting, voting, accessing a few tools on my Personal Nodelett that I can't ever find without. What I do here: I used to religiously live in RAT and strove to catch up all the reading. Now I check Tidings, Best Nodes and Status of Recent User Information Leak first. I'm always dissapointed at the staleness of these nodes. Then I catch up on Monastery Gates. If I have a Perl related problem I always come here first, even before Google (and usually find the answer so I don't need to go to Google and if I do go to Google, it almost always points me back here). My general assessment is that perlmonks has suffered no ill reputation in my mind. I have the greatest respect for most of the monks here both as perl-ers and as general developers/computer scientists. If someone was struggling with perl I would certainly still recommend they create a new user and get involved in this community. I know that there was some technical problem that led to the password problem, and I changed my password as a result; no big deal. I've been told to look for changes and updates and I expect to change it again when those are made (in addition to my own password frequency rules). Perhaps if some technical solutions are advertised as implemented I'l revert to my logging in every time and working from the more complete RAT view instead of staying ananoymous at the monestery gates. #my sig used to say 'I humbly seek wisdom. '. Now it says: use strict; use warnings; I humbly seek wisdom.	[reply]
Re: Making assessments by NateTut (Deacon) on Oct 09, 2009 at 16:17 UTC
While I was disappointed by the security lapse, stuff happens. I have worked for a long time in "professional" environments and have seen much worse. There are folks here who put a lot of voluteer hours in to make perlmonks the great resource that it is and I for one am willing to forgive this setback.	[reply] [d/l]
Re: Making assessments by Argel (Prior) on Oct 17, 2009 at 00:12 UTC
I'm finding it harder to promote Perl now because the Perl community and PerlMonks in particular are huge pluses (along with CPAN). In the past, I would suggest Perl and then mention that if you need help you can go to PM. But recommending a site that was recently hacked doesn't exactly win you points, especially in a corporate environment. So I have found myself hesitating to recommend Perl because of it. I hate to say it, but that "Status of Recent User Information Leak" banner doesn't help -- maybe it's time to take it down or replace it with something more subtle. I'm also concerned about how many people have the time, energy, and experience with the current codebase to even keep this site up and running. Right now it feels like Co-Rion, tye, and jdporter are holding the site together, and I don't think that's fair to them or us. For these reasons, I'm having what I call Wikipedia Syndrome, where my enthusiasm for contributing has waned. I think I would feel much more comfortable if we were on a much more maintainable platform. Also, FYI, the hack was on May 20th. It just wasn't discovered until a couple months later. Update: I really meant "discovered" more in the "found out about it" sense. Though I suppose if the hackers themselves had not disclosed their handy work it would have remained unnoticed. Elda Taluta; Sarks Sark; Ark Arks	[reply]
Re^2: Making assessments by Anonymous Monk on Oct 17, 2009 at 20:36 UTC
s!discovered!disclosed!;	[reply]