Saturday, December 20, 2008

The worst Christmas of my working life (so far)

I've started my annual Christmas vacation and this year I'm taking two weeks off. I'm leveraging 4 Holidays into two weeks off.

My first official actions on vacation? I emptied the sink full of dishes, got up at 6:30 to take my wife to work and started laundry. Not exactly soaking up sun and relaxing but it beats working. In fact, I have a motto about it: "I like vacation better than work".

I got into this habit of taking time off around the Holidays at my first long tenured job at a local Telephony Billing company. They had a policy of closing between Christmas and the New Year and would split the time with you. If it were 5 days between Christmas and New Years outside the normal Christmas Holidays, they would take 2 as off but not vacation days. They were freebies.

I got used to it.

There have only been two times when this hasn't been possible, when I started my current job and didn't have the required vacation days saved up and the year that local Telephony Billing company released "New Product".

In 1991, I was in Software development at Telephony Billing Co. at the time and the New Product was going to be our premier platform for gathering billing records. We had farmed out the conversion of our flagship product to New Product to a large internationally known hardware and software maker with two initials. Their theory was to take all of the old code, written in assembler and Fortran, which ran on their propitiatory operating system and convert it to C and Fortran on UNIX.

It seemed like a good idea at the time.

This is like taking a large wooden machine from the 19th century and part by part copying it into steel and titanium. I guess if you are Vice President of a small software firm this seems appealing since you can continue with the old product with in house people and contract out the conversion.

It was a disaster.

The part by part copying went OK. Large, two initialed, intergalactic hardware and software firm hired three contractors for this job and as is usually the case in such situations, one contractor was totally useless, one was so-so and the third was brilliant.

On average they were OK.

So the conversions were also on average OK. Coincidentally about a third were terrible, about a third were OK and a third were brilliant.

The real problem came at integrating all these hundreds of parts back together because no one knew how they all should fit together. The software program they were converting had some parts that were reused from older products, some as old as 15 years ago. It was well over a million lines of code. The people that wrote and understood those parts were long gone, having been fired or laid off when the company went through lean times. By the time of this conversion, I had been at the firm 2 years and the generation of programmers that wrote a lot of this code were know by us new comers as "the ancients".

I saying I developed during these times was "not all the ancients were wise". Some of the parts in the code made no sense. It was like body parts that had evolved. Ones that once had an important purpose were now confusing parts that no one knew they did. Like an appendix or that tip of the spine that "used to be a tail".

There was a lot of "what the hell is that"? and "who knew"?

There was one among us who knew the ancients and had worked among them, B.

B had moved from North Jersey when the firm folded shop up there. B. had worked at the firm for over 10 years and He was the only one with "the big picture". The rest of us had parts of the system we specialized in on this unbelievably huge software product but B, allegedly, knew how all this fit together.

Other than B, it was like three blind men and an elephant. "New Product is like a column" one would say. "New Product is like a snake" another would say.

The only problem was that B didn't actually say much. He was a deep thinker.

We spent most of the year testing all the little parts that had been converted and then by fall we were starting to integrate it. Well, large, two lettered intergalactic software maker was integrating since it was in their contract to do so.

Because they had no blueprints or documentation, some times they put things together wrong. It was like they were building Frankenstein and sewed a leg were an arm should be, put his head on backwards and used a thigh for neck. It was a mess.

Not only was it difficult to put together, the conversion was based on an emulator of the original operating system that mimicked the proprietary operating system under UNIX. And that e-e-e-mulator sucked. To this day I can't say that word without stuttering.

By late fall, we were "helping" them. By December, intergalactic software and hardware giant had all but given up and it was up to us.

Now things were way behind schedule mostly because of the integration issues and we had assumed that customers were being told that we were behind. So far behind that we were thinking that it would be sunny and warm by time our Frankenstein saw the light of day.

Around Christmas, I learned "never assume"

On the morning of our Company Christmas luncheon, I had made the first real release of the software. It was more like a test. I had no idea if all the parts worked or not I was just picking them up out of the development tree and getting them into a production environment, via DAT tape.

Tape number 1.

At the luncheon, my boss asked me:

"You know that tape you made this morning? Is there any way we can ship that to customer X?"

It was a Dilbert moment. All he needed was pointy hair.

I honestly thought to myself that he was insane. Whatever I had would never ever work. We all knew that. We had months of testing and integration ahead.

Well for customer X, a salesman had told them they were going to have New Product by years end.

Two days later, the President of the little company I worked for canceled Christmas as if He were the Sheriff of Nottingham. We were told that starting bright and early on the morning of the 26th we would be working to get customer X a tape. One that worked.

30 official tapes and 8 months of 80 hour weeks for 10-15 people later it was all over. There were many stories in between but let's just say that it was a terrible Christmas.


Anonymous said...

I remember those days. I think this was the time you learned to throw DAT tapes accross the computer room. One nearly hit me in the head. Oh ... the good old days.

suburbanstories said...

That tape missed you by a mile. Remember, it's a long testing season so you don't want to injure yourself so make sure you warm up before chucking a tape.

So many memories of that 91-92 season.

I should write down more of them.

Anonymous said...

Hey there buddy ... that blog brought back many memories as I was working there with you! As always you have given me a good laugh. I wonder what B is doing these days? Hope to see you soon ...