Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

Re^3: I came, I saw, I ...

by RonW (Parson)
on Jul 06, 2017 at 18:54 UTC ( [id://1194393]=note: print w/replies, xml ) Need Help??


in reply to Re^2: I came, I saw, I ...
in thread I came, I saw, I ...

Fried a custom prototype .... Damage: About 100 €

Wow - you got off easy. I'm guessing your employer is enlightened enough to include plenty of spares in the order for prototypes. (My employer's purchasing department will only allow us to order the minimum quantity accepted by the board maker. This usually means we have only 2 or 3 spares.)

I hope your dad didn't get in too much trouble for blowing the channel on the scope.

The most expensive equipment I blew up wasn't my fault. The PCB layout engineer changed the orientation of a square micro-controller by 90 degrees but forgot to update the board markets. When a technician installed a socket where the micro would go, she followed the board markings. When I got the board and plugged in the in-circuit emulator, the emulator's interface "pod" belched the magic smoke when I powered the board. That was only about US$2000 to replace.

A co-worker of mine was once subject to the same "fire works" due the same mistake (by a different PCB engineer), but the damage was US$30k.

Replies are listed 'Best First'.
Re^4: I came, I saw, I ...
by afoken (Chancellor) on Jul 06, 2017 at 22:29 UTC
    I hope your dad didn't get in too much trouble for blowing the channel on the scope.

    Well, he wasn't fired, and he did not have to pay the damage. In fact, the company was sold later, including his working contract (as usual in Germany at that time). During that time, when TV sets had still a potent high voltage supply and a CRT, a scope could easily be killed by measuring at the wrong place of a TV set with the wrong settings. Nobody is stupid enough to measure the 30 kV at the anode of the CRT, but focus and gate 2 also have a voltage that would be quite unhealty for a scope with a standard probe.

    The most expensive equipment I blew up wasn't my fault. The PCB layout engineer changed the orientation of a square micro-controller by 90 degrees but forgot to update the board markets. When a technician installed a socket where the micro would go, she followed the board markings. When I got the board and plugged in the in-circuit emulator, the emulator's interface "pod" belched the magic smoke when I powered the board. That was only about US$2000 to replace.

    That's the "oh f***ing s**t" moment. Misplacing the ICE connector is also quote common on our current project, but luckily the ICE is quite robust, cheap, and some really wise man has decided to use a connector pinout for the ICE that makes it really hard to kill the ICE or the controller. We would normally use a keyed connector on the PCB, but that exact version required for the ICE is either close to unobtainium or insanely expensive. And compatible solutions are simply too large. So we placed an unkeyed connector and a tiny dot indicating pin 1 on the PCBs. So far, we have killed no ICE, and except for my 24V mistake, no board.

    A co-worker of mine was once subject to the same "fire works" due the same mistake (by a different PCB engineer), but the damage was US$30k.

    And that's the moment when you decide to call it a day, go home, and try again the next day.

    My dad had a similar experience during his first job in the R&D department of a major electronics company. Consumer hardware electronics still was full of tubes and some rare germanium and selenium semiconductors. Industry started first projects with expensive and brand-new silicium semiconductors. Bipolar transistors and diodes, with real wires, packed in metal cans. Not that black, wireless birdseed FET stuff in plastic and ceramic cases with far too many pads and unreadable markings. A single Si transistor cost 100 € or more, in current money, and it had awful specs. Si transistors were still individuals, each with its own set of properties. Finding some pairs with similar behaviour could take a day. One day, one of the hardware engineers came with a sketch of a high-end, high-power, high-frequency, all-transistor amplifier, requiring about 20 of those expensive beasts, all paired, all wired in parallel, with paired resistors. Plus a few more smaller transistors to control the big ones, also paired. About 2 or 3 k€ in modern money, maybe more, just for the parts and a few days for the selection. "The exact spec does not matter, just find similar ones", he said. My dad build the amplifier, connected it to the dummy load (capable of heating away several kW) and a supply, switched on power to start adjusting the amplifier, and just saw all of the transistors die faster that he could switch off. He went to the engineer, confessed that he had just killed hardware worth more than he would earn in two or three months. "Oh well, I kinda expected that would happen. My fault. Damn. I will have to redesign that better." And that's all the engineer said. Buring 3 k€ (in current money) and one or two weeks of work just to test an idea was completely acceptable.

    Looking at my damages over time, I see a quite good trend. The damages get smaller. Let's hope the best ...

    Alexander

    --
    Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
Re^4: I came, I saw, I ...
by afoken (Chancellor) on Jul 06, 2017 at 21:44 UTC
    Fried a custom prototype .... Damage: About 100 €

    Wow - you got off easy. I'm guessing your employer is enlightened enough to include plenty of spares in the order for prototypes. (My employer's purchasing department will only allow us to order the minimum quantity accepted by the board maker. This usually means we have only 2 or 3 spares.)

    Well, yes and no.

    We had exactly two prototype boards from the first generation, and we modified the hardware quite a bit while integrating the hardware and software. I remember a little "SMD Stonehenge" of four or six SMD resistors they were wrongly connected as pull-downs, not pull-ups. So our hardware expert mounted the resistors vertically, only on the pads of the signal lines, soldered a thin wire to the other ends, and connected that wire to a 3.3V pad somewhere nearby. It looked quite nice, but of course, it was a hack to get stuff working. Tons of other stuff went wrong, especially with a nasty serial bridge chip. Those boards ended in a box somewhere in the storage room, and IIRC, our hardware expert later ripped some spare parts from them to fix other boards.

    The second generation of prototype boards came in a batch of 10 or 20 and had most hardware bugs from the first generation fixed. So, no more SMD stonehenge. Unlike the first generation, most of them were not mounted on scrap wood, but into prototype cases, with all peripherals. This was the first time we could do actual tests with the entire hardware, not just the electrical part. We gave some of those prototype devices to the client, so they could also do some initial testing.

    Four of those quite expensive prototype devices were also used in a transport test: Put the device in its transport box, put the box in a cardboard box, and drop that from a significant height onto a concrete floor, several times per side. No mercy. The client did all of those tests in parallel with four devices, and thereby destroyed hardware costing several k€. Lesson learned, the hard way: A simple cardboard box is not sufficient, it needs some padding, and the transport box also needed some more internal padding. We would have tested with only one device at a time, but the client insisted on fast progress. Well ...

    After that, we also dropped some devices, with better padding, damaged them significantly less, and learned how to improve the mechanics so it could survice the test without breaking apart. We repaired what could be repaired, then dropped the devices again. Finally we sorted the bits and pieces, including some remaining prototype case parts, into spare parts, parts for internal testing, scrap for ripping off spare parts, and junk. We build some test devices from the second quality, and still use these devices, full of scares, scratches, glue, and discolored plastic, as test devices for development. They have a few extra holes here and there, to access parts of the circuit not available on a normal device. Wires and, in one case, tubes dangle out, with switches, connectors, and extra electronics fixed to the case with velcro tape, Some other parts that serve no purpose for us, like a sticker with the client's support hotline, a carrying handle or a cover for power supply and service connectors, are usually removed from those devices, or were never mounted. Their serial numbers, usually hidden on a small sticker on the button, are written with a black marker on the front, so we can easily identify them. No customer of our client will ever see or touch those devices, and if they would see them, they would probably run away in terror. ;-)

    Two of the second quality boards that survived the drop tests replaced the first prototype boards on the scrap wood. We changed some parts on the boards to fine-tune the electronics, but after that, those second generation prototypes were identical to the first series production boards.

    One of the scrap wood systems was extended by a bunch of switches, wires, modified peripherals, banana jumpers for current and voltage measurement, and promoted to "test rig". With that setup, it was (and still is) possible to simulate almost all hardware problems, even the exotic ones: Broken wires, dead or broken peripherals, interrupted communication, and some more that I can't detail here.

    While working with the setup, when the first few serial boards have already been mounted into serial devices, we found out that we would need some more wires patched on the board to simulate more stuff. And one of those wires touched the +24V line.

    So yes, I killed a prototype board, but it was a second-generation board modified to be extremely similar to serial production boards. That's also the reason for the quite low damage: It was - except for minor details - electrically and mechanically identical to a serial production board, and it was already produced in a small series.

    After all, killing the board was kind of luck, because that allowed us to "steal" a brand new second series board with a better controller and some electrical improvements, so we could test the new stuff with the new software. We kept the other scrap wood system with a first series board for compatibility tests. Actually, that's currently my main target for development. If the software works on a series one board, it will also work with a series two board. So I test on a series one board. The software still has to support the series one boards, because several customers of our client have bought devices from the first series. Until all of those "old" devices have been exchanged for new devices or have ther main board swapped for a series two board, we can not release software that runs only on series two boards.

    Alexander

    --
    Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

      Sounds very familiar. :)

      No customer of our client will ever see or touch those devices, and if they would see them, they would probably run away in terror.

      Yes. Real life development is a lot closer to "McGiver" than the "show case" labs featured in advertisements and company tours.

      (And, in our show case lab, don't open the doors of the rack cabinets. It's not only a tangled nest of patch cables, but also an assortment of modified prototype boards providing functionality that even the vendor of the hugely expensive, LED-bejeweled hardware simulator can't figure out how to provide.)

        Real life development is a lot closer to "McGiver" than the "show case" labs featured in advertisements and company tours.

        Yes, indeed. I can play with Lego all day and get paid for it. ;-) No, really:

        We like to use Lego for quick test setups. Take a large Lego board, click a few bricks onto the board, fit a PCB, sensor, actuator, laser pointer, prototype board, eval board between the bricks. Add the next bricks, and parts. Add a few more bricks to prevent that mess from falling apart. Add wires, tubes, whatever is needed. Pad with paper or adhesive rubber feet. That works surprisingly well.

        Larger, heavy Lego parts also work well as paperweights to temporily fix pieces of printed paper with fake screenshots on a large piece packaging paper while discussing interaction between the parts. Old Lego rails work incredible well to mark borders between the parts. We used exactly that on a ping-pong table to discuss how our software should work with our client. Different colors could be used for grouping, and the client asked for the meaning of the colors, but we did not use that. Colors were meaningless in that discussion. All of us, including the client, moved paper and Lego parts around until we found out how the client wanted the software to behave. We taped the paper pieces to the packing paper, replaced Lego borders with marker lines, and draw some notes onto the packing paper. That was our inofficial main document for the user interface development. We simply taped it to a long wall and looked at it whenever we needed details or had to discuss implementation variants.

        We later reused that trick to discuss interaction between parts of the software, without the client; and once again for a larger rework of the software internals.

        Alexander

        --
        Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1194393]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others admiring the Monastery: (6)
As of 2024-03-28 19:54 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found