PortraIT

I work in IT. This device is a 2-way optical transceiver. It is uses fibre optic cable to connect the networks in two of our buildings together. A week ago last Friday this particular unit suffered a partial failure on the receive (RX) port. The failure was not catastrophic - most of our information systems continued to operate; albeit at greatly reduced speed. Some users complained their computers were slow and some applications were unable to function correctly. Some technical people, like myself, noticed various, seemingly unrelated symptoms throughout the week and attempted to troubleshoot the issues in isolation.

In short, a lot of people wasted a lot of time last week.

Things came to a head over the weekend. As the transceiver's performance continued to deteriorate the symptoms became more and more severe. Data processing jobs which normally execute overnight were failing. Meaning we would not be ready for business on Monday. As more people were called in on Saturday to help find the source of the problems, more clues came to light as we shared our observations from the previous week. Finally a common factor to all our issues became apparent - data traveling between buildings, in one direction only, was very very slow. A simple check of an error counter revealed the culprit pictured above.

Swapping the part for a new unit returned everything to normalcy - cross off one major tech issue. And scratch one weekend.

Failure of hardware is unavoidable.

Failure to collaborate is not.

Lesson (re-)learned: Share your failures - not just your successes.

I had a bad week. The system I was working on was not functioning correctly and every adjustment I tried to fix it seemed to make it worse. If I had shared a little more of what was going wrong we might have discovered the real source of the problem sooner. If I had been aware of the problems my co-workers were tackling we might have shed some light on each others issues.

We held a postmortem meeting this afternoon for everyone involved - almost a third of our department - and walked through the saga from start to finish. There will be more sharing/commiserating next time around. And possibly a little more thinking outside the cubicle!

* * *

In the Eighties I sold computers. Our office and sales floor were decorated with glossy posters featuring loving portraits of obscure system components like this one. The subjects were usually photographed against a star field or a grid of laser beams. I would have used graph paper as a stage for this shot, but all I had was a couple of scratch pads (an indispensable tool when troubleshooting information technology). Taken with Pureshot on an iPhone 4S. Straightened, adjusted for contrast and vignetted in Snapseed.