Years ago, I was working with a large retail client that had a sustained network outage at a majority of their locations. Even though it was an ISP outage, someone at a particular location took it upon themselves to try and fix it, and in doing so, got the cables switched around and some unplugged - so it also became a LAN issue for them.
Since I had no remote access, I was walking the District Manager over the phone through the usual troubleshooting steps and he noticed a cable was unplugged that he swore was plugged into the switch at one point, so I had him plug it back in. It didn't bring up the network and he couldn't trace it because it was a tangled-maze of cables, so I let it ride because he was sure it should have been plugged in that spot.
Once we figured out the WAN/LAN cables were in the wrong ports everything came back up, though network speeds were much slower than usual. But as long as it was up they were happy and wanted to deal with speed issues later. I converted the spooled (faux-transactions) files and pushed them through so they got paid on those. They were also able to take live transactions at the registers, so they were very happy.
About two days later management calls me in for an emergency meeting; turns out all the transactions that had been spooling for several days and all transactions afterwards were actually charged anywhere between 5-20 times within seconds of each other; even though we only showed a single authorization file on the server. BIG time problem!
We had to take credit cards down and look at everything again, but the program, system, network settings looked fine. The network was still running slow, which was unusual because all the other stores were fine and they shared the same ISP managed at the corp level. So I went back to square one. After more phone time, followed by an on-site tech dispatch that came out of the support budget, it turns out that the cable that "looked like it should have been plugged back into the switch", was actually already plugged into the switch but not connected to anything else; so we looped the cable creating a network flap.
It just so happened that the payment software on the server would retouch the processor end-point at each up/down state, creating new and unique transaction-ID's. Because the trans ID's were unique, the processor/gateway didn't catch it and fold the duplicates into one transaction; so their system processed the duplicate transactions as unique account withdrawals. And keeping in mind that transition amounts were anywhere from 50.00 - 500.00+ USD, people's credit cards were being maxed out, banking accounts were overdrawn or locked because of suspected fraud. One family was on vacation and had no access to funds. A total nightmare.
1. I never use my bank debit card at payment terminal.
2. It pays to take the time to label cables and make sure you have them zipped-tied and well organized - and a sign that says DO NOT touch *cough*ahem*dustin*cough*.
3. It would have been great to have something like indeni to tell me exactly what was going on in the switch.