From my experience it's generally good to have multiple lines of defense.

But those should be clearly communicated, documented and automatically tested.

The fact that it took several weeks before the problem became obvious is a flaw in the concept I encountered before:

Personal "war story":

hell broke out when my boss started openly complaining in team meetings, that I broke one of my own applications. I told him that it must be another factor, since neither me nor anyone else touched the code for 6 years(!). And I immediately suspected the database server.

He didn't believe me and started many desperate experimental patches in my product, degrading the codes quality little by little.

After two weeks - I had a concussion and nobody was listening anyway - I informed them that their MariaDB upgrade 4 month ago was to be blamed and SQL backwards compatibility was subtly broken in undefined edge cases.

And I provided a patch fixing that.

2 or 3 month later I got a phonecall from the main user saying

"Hey, your app stopped working!"

Turned out my boss never applied my patches believing in his hacks, and my annoyed colleagues instead continued to work around the problems. Till the data was so corrupted that my app kept raising alarms. (Thankfully I had another line of defense throwing annoying error messages.)

Took me two weeks to apply patches to identify all the data corrupted in the meantime.

Bottom line:

this showed a big management problem, which relied on "human testing". If the issues had been communicated right after the users noticed it we could have correlated it straight away with the DB update. But like in your case there was a delay.

All this communication worked in the past when our team had one third the size at just one location. We literally sat in the same office.

Now with new employees who hardly ever saw or spoke to me, these problems were kept under the radar.

The intelligent solution, better automatic test suits for our web app was declined as "too expensive and unnecessary" ¹

It's a classic combination of Peter Principle and Mythical Man Month which is biting my ex job and I doubt they'll ever fix it.

Meta analysis:

They recently brought in new external management babbling about switching to Python because of the lack of Perl experts ,without really understanding the structural problems of failing communications in a blown up team. ²

Goes without saying that the management doesn't have any qualifications in software development and prefers short term feel good Design by Committee °

And I was required to stay still and not to interfere with their decisions, because I was "only" hired as a freelancing programmer and was threatening their authority.

Cheers Rolf
(addicted to the Perl Programming Language :)
see Wikisyntax for the Monastery

°) imagine my horror when couldn't attend meetings and decisions were made, like tasking a newbie to switch to another MariaDB engine core in a big bang.

²) Adding a bunch of new Python hackers to the mix will certainly help (sarcasm)

¹) "we need you to do other stuff we don't understand either"


In reply to Re^5: DBI and FEDERATED table slowdown (war stories) by LanX
in thread DBI and FEDERATED table slowdown by misterperl

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.