I was talking to !evil Phil today, and something brought up a snafu that went on at the steel mill back when I first started (which, I would like to stress, is
not where I work now, nor do !evil Phil or
craftsman). I would like to find out from any of you who have accounting experience or background if you can explain to me without using magic, smoke, mirrors, or a curare-laden dart how the decision they made makes any sense.
This was 1990. We had a PR1ME 9755 mainframe that was running the (in-house-developed
1) order entry, shipping, and financial software at the mill; and two VAX 3800 mini computers that we had purchased for the express purpose of running the Ross Financial software that our VP of finance (Dave) just
had to have.
2 And since we have this
great new hardware, why not use it for other stuff?
3 So, the idea was to get the data in the databases on the PR1ME to the VAX, because the PR1ME was going to be retired Any Day Now™.
4The mill Information Systems department consisted of !evil Phil, who at the time had a Bachelor's degree in computer science from the University of Alabama and was working on his Master's. We also had
craftsman who was nearing graduation from the University of Alabama with his Bachelor's degree in computer science. I had come on board as well with my undergrad degree from the University of Alabama in computer science, and had stopped a few hours short of a Master's degree in same (severe burn-out, if you're curious). Our boss was an Electrical Engineer-type. His boss was the president of the company, Mike Austin.
Our solution (as suggested by !evil Phil) was to spend about $8000 on an Ethernet card and TCP/IP stack for the PR1ME and FTP the data over to the VAX.
"Egad, no!" they shrieked. "That's
throwing money away! The PR1ME is going away any day now, and the VAX cost us too much money to abandon! If we spend anything
at all, it has to be on the VAX!"
Here's what they actually ended up doing.
They got this guy from Tippins, one of the parent companies of the mill, to come in and use his Years of Experience™ on the PR1ME. His name was Frank. Frank was self-trained on computers. He started as a blue-collar out in a mill somewhere in Pennsylvania and worked his way up to a white collar and an office. I say more power to him, really. He unarguably knew more about the PR1ME and its screwy operating system (PRIMOS) than did any of us. And none of us
wanted to know more than we did.
They also found these two guys who were named Ralph and Avi. I don't remember much about them except that I believe one of them started life as a literature major and the other got his degree from some sort of diploma mill. But they were from Pennsylvania and were therefore Much More Smarterer Than Any Southern Bumpkin™. They paid Ralph and Avi $40,000
in advance for what I'm about to describe.
The company purchased a $3000 9-track tape drive for the VAX 3800. The PR1ME already had one. The idea was that Ralph and Avi and Frank would use their <echo chamber>
Vast Powers of Expertise!!!</echo chamber> to copy the data from the PR1ME onto the VAX using the two tape drives and countless 9-track tapes.
Let me pause here to describe the tape drives for those of you too young or too lucky to know what a 9-track tape is. Remember those old TV shows and movies where every time you'd see computers, it'd be a room full of huge, gray consoles with blinky lights, bespectacled people standing around in lab coats with clipboards, and on every blinky-light console there'd be tapes spinning back and forth? Those are 9-track tapes. They're enormous and they're not light. And a 1G USB thumb-drive would probably hold an entire room's worth of those tapes. The drives themselves were a bit smaller than refrigerators and about as cumbersome. But back to my narrative.
So...copy the data from the databases on the PR1ME to CSV files onto tape, load that onto the VAX, and in a few hours we're good to go, right?
Right?
Oh, no. No, no, no.
Here was Frank's, Ralph's, and Avi's
brilliant plan. Prepare to be
stunned by the brilliance. Are you prepared for the brilliance? Because you really should. Seriously. You might get hurt. Don't say I didn't warn you. Here comes the brilliance.
<Brilliance>We take the database on the PR1ME, and let's say it had 20 columns of data (it had many more, but let's stick with 20 for illustrative reasons). That data might be anything from a single-bit boolean to an 80-character string. Their—brilliant!—idea was to take each column (we called them "fields") of this database and write it to another database that had just one 200-character string column.</Brilliance>
So, the value of every column of each row of the original database becomes a single row of the new database, and takes up 200 characters of space, even if it was just an integer or a boolean/bit originally. So a 1000-row database of 20 columns expands to a 20,000 row database of one column, each one guaranteed to be 200 characters wide.
<understatement>Needless to say, this vastly expanded the data.</understatement> Instead of compressing the data, Frank, Ralph and Avi
bloated it. And wrote this on tapes. Well, Frank did. He was the PR1ME guy, after all. And I'm sure his code was actually pretty tight. Frank actually did know what he was doing, up to a point.
5Now. It was our jobs—mine, !evil Phil's, and
craftsman's—to babysit the tape transfer. 24x7 for what I'm told was a week, but seems like a lot longer in my memory. We worked shifts so someone would be there at all times, just letting a tape fill up on one system and then moving the tape to the other and starting the process of loading it. We were all zombies for the entire thing.
Meanwhile, in another part of the mill, Ralph and Avi had logins on the VAX 3800. It was
their jobs to take the data Frank wrote to the tapes and extract it and put it in a corresponding database on the VAX.
Let me belabor this point: their task was to write a program that basically read a single record of 200 characters off tape, convert that into its original data type, and insert that into the proper column and row of a new table. Simple loop, right?
Right?
Oh, no. No, no, no.
Ralph and Avi—who I must repeat at this point had
already been paid $40,000—wrote a series of batch files that explicitly extracted this data from tape one column at a time. They repeated the same lines of code millions upon millions of times, but with different column names and such. And if they made a typo 200,000 lines into the batch file (I'm not joking—one of their batch files was 1 million lines long), it took hours to page down to that point, make the correction, and then start the file from that point, after repositioning the 9-track tape, as well, so it could resume reading at the point where it left off.
!Evil Phil even took a look at what they were doing at one point and suggested loops, but Ralph and Avi—who had
already been paid $40,000 and who were The Experts on the VAX™—knew better, and disdained the use of loops.
The three people who actually knew what the hell we were talking about were patted condescendingly on the head and told to just be nice little tape-swap-monkeys and not make trouble by talking about things we didn't know and couldn't possibly comprehend by two overpaid yahoos with about as much programming skill as a
stale marshmallow peep. And the mill backed up Ralph and Avi. See, they had already paid them $40,000, so their opinions must be right. Because if they weren't experts, they wouldn't have paid them $40,000 in advance, right?
Right?
So, here's my question to the accountant-types or any of you who operate in the world of finance at a level far beyond that of li'l ol' me: can you make me understand how spending $8000 on hardware and using other existing hardware, software, and personnel to accomplish task A in, let's say two weeks, is "throwing money away," but spending $3000 on new hardware, bringing in a consultant from the parent company and having him waste his time, and hiring two outside consultants for $40,000 to waste a week of three mill employees' time needlessly flipping tapes is
reasonable?
Oh! Oh! I forgot the best part. Take your explanation you were going to hit me with from the question I just asked and now factor this in: right after we finished the entire task of flipping tapes, running millions of loopless lines of code, and working 24x7 shifts...after spending $40,000 on two consultants, bringing in a PR1ME expert from the parent company, and spending $3000 on a new tape drive for the VAX...they canceled the project.
Yep. The entire thing was thrown out.
Does that change your answer?
- At one of our parent companies, Tippins, I believe.
- Dave paid DEC something like $60,000 for a study and they (of course!) recommended Ross Financials because it ran on VAX computers and was The Best Financial Software In the Known Universe™. So we bought the VAX 3800s ($250,000) and Ross ($500,000), but we never could get Ross to work on it; that would have cost an extra $400,000 for implementation. This is where they finally balked.
- There was a push to use PathWorks on the VAX computers as the main repository for all user data, including the Windows 3.0 diskless desktop workstations, but !evil Phil and
craftsman were able to demonstrate that Novell blew PathWorks and the VAX away on speed and reliability and we were spared that indignity. They wanted to use that hardware they'd invested so much in. - The PR1ME was decommissioned in 1998 or so.
- So after Dan "resigned" and Frank became our boss, he wanted to try to find something to replace Ross and the PR1ME software. So he paid DEC (we have these VAX computers we aren't using, and why waste the investment?) another $50,000 for another study to find software that would not only do financial stuff, but order entry and shipping and all that stuff. DEC recommended RollCIM, which was another $450,000, plus, I believe, $288,000 for "minor changes" to match our exact spec. But RollCIM was too much of a HAWG to run on our puny VAX 3800's, so we bought four VAX 6000's and a number of VAX 3100's to use as desktop machines for the IS Department. Then, it was determined that RollCIM needed more work to make it compliant with our spec. And more. And more. So about 7 years later and more than $1.5 million into the "minor changes" to RollCIM, I left the company, having never seen RollCIM work as advertised. The mill also ended up getting the latest version of Ross to run payroll, which required another new VAX. That was about 3 years before I left.