Did foul play cause the ERP system crash—or was it just an inept tech?
Hi, there. My name is Jack.
Er, you come down here to SOCAL often? No? That’s too bad.
"What do I do?" you ask? I work the consultant beat in north county San Diego and the Orange county area, when there’s action up there.
Yep, that’s right, action.
It’s the kind of action that causes a company to hit the horn and scream for help. Only they don’t really scream too loud because they know that it... raises the price.
One time back at the office, the dispatcher Bonnie gets a call from someplace in north county, just beyond the morning fog that comes in from the coast. Some lady says that they need SQL Server reinstalled.
“That’s odd” Bonnie says. “We have you down as one of our Dynamics Great Plains customers, and it says here in the CRM system that you have a SQL Server.”
“Please send someone,” responded the voice of an absolute femme fatale.
Bonnie hits my cell. Here’s what happened after that.
Bonnie: Jack, how’s my best man doing?
Me: Well maybe you can swing it so’s I can be your best man someday, Bonnie. How’s it going with that guy you’re hanging with?
Bonnie: Never mind that now Jack, I have a hot one. Woman called in with a SQL problem. They sound worried. Here’s the address. They need you up there.
Me: No problem, the traffic’s not so bad at this time in the morning.
So I drive up route five and pull off the highway (excuse me), freeway to a swank office park with the required fountain in the parking lot.
As I park my ride, I notice what looks to me like a fire drill rehearsal going on in front of a nice office building. There must have been over 100 people standing outside.
Not knowing what was going on, I just grabbed my laptop bag and went walking toward the main entrance.
As I got near the crowd, a classy woman’s voice with a British accent called out and said, “Are you Jack? Are you here to rescue us?”
Then another voice rang out with a distinct Windy City flair, “Are you the GURU that’s going to ‘fix’ us?” Man, I hate that word GURU.
I walked into the elevator with these people, and they informed me that we were to have a meeting in their conference room first. The elevator was shaking… and so were the passengers’ feet in their boots.
The front door of the suite did not impress me… But, the conference room did. Around me were arrayed the cream of the company and one very overdone tech who looked like he should be in a sleep-aid commercial. I looked at his hands, and he was a few centimeters from the cuticle, if you know what I mean.
Exec1: We have a problem with our Dynamics Great Plains server.
Me: What’s the problem?
Nervous Wreck Tech: It……. crashed. I was on the phone with Microsoft all night, but they couldn’t help me.
Me: Are you the company IT person?
Nervous Wreck Tech: I work as a consultant like you. Do you want to see my Linux certification?
Me: Eh! Perhaps some other time. So what’s the impact of the server crash?
Master Piece: We haven’t been able to pay our employees for three days. All 800 of them.
Femme Fatale: Some of them are of …. questionable background, so we are not sure what action they will take. We’re rather frightened.
Me: So how many servers do you have?
Nervous Wreck Tech: One. My company recommended that since they only have 35 workstations, they needed only one server.
Me: So the operating system is Small Business Server?
Nervous Wreck Tech: No just Server. It’s a standalone.
Me: And your boys recommended this, huh? Interesting!
Femme Fatale: Oh, save us, Jack, save us!
Me: Take it easy, lady, there’s only one of me. OK, this is what we are going to do. Ms. Fatale, get me all the software CDs you have in the office. Look for any papers that say “license number” on them. Those are our letters of transit out of this situation, and they’re real important. Master Piece, you get me the administrator password to the box, or any other passwords and usernames you know.
I was escorted to the server room, a 4’ x 4’ unfinished closet at the end of the office. I saw pieces of sheet rock and construction dust on the floor. The “server” was just a single (popular vendor) box, with two drives and 2GB of RAM—and all the server roles that a modern office needs today were configured on this box. There was even a pair of remote users that were waiting to connect to the box on Terminal Server sessions. The box was a file server, print server, SQL Server, Great Plains server.
I stood at the console of this hole in the wall and went to the command prompt to take a look at the boot.ini file.
I went to the root and typed
attrib b*.* -h -s –a
and in a moment I was looking at the first piece of evidence. I noticed that the boot.ini had three entries on it.
Nervous Wreck Tech: Wow, I haven’t seen that file since my certification test that I <cough> didn’t pass.
Me: I’m not surprised; most techs get sloppy and forget about the fundamentals.
Nervous Wreck Tech (who was right behind me): I had to install a third instance because I couldn’t get the other two to boot.
I felt the floor give beneath my feet for a second. Then Nervous Wreck Tech grabbed the mouse and opened a folder.
Nervous Wreck Tech: I found these files that were real big, so I think they’re important or maybe they’re just corrupted junk. I moved them to the new machine on the receptionist’s Desktop. She has the largest drive in the office.
Me: They are .bak files, SQL backup files. And yes, they’re very important. Maybe even more valuable than gold, at this point. We’ll see what’s in them later today IF we’re lucky.
Nervous Wreck Tech: There wasn’t much space on the drives last night while I was working with the Microsoft Tech, so I removed some of the folders in the Program Files folder.
Me: And who came up with that little approach?
Nervous Wreck Tech: Well… I did.
Me: Would you happen to have a TUMS on you?
I was able to boot the machine and, exploring all the volumes, we moved off to a USB drive every possible thing we could, in all the instances.
Femme Fatale found the CDs and better yet, the license codes. I installed SQL Server and the service pack while the machine was disconnected from the Internet.
Then I looked for a text file called acctfram.txt, since this file contains the account framework for their financial system that ran on Microsoft Dynamics.
Femme Fatale (batting her eyelashes): So what’s the account framework for Jack?
Me: Well, lady, the account framework defines the place segments and the account format. This format defines the account numbers that ID a “company” in the accounting system.
Everything has to be tied up nice with string and paper for the government auditors. Otherwise there could be… well, let’s just say we wouldn’t want the auditors to be unhappy now, would we?
Often a site will make a shotgun kind of choice to cover all the bases, like five segments with a max of 25 characters. But this was like a complicated safe combination. It was a good call to check all this out before attempting the restore the SQL Server database .bak files we found on the receptionist’s Windows XP machine at the front desk.
With SQL Server installed, I installed the Microsoft Dynamics ERP software and duplicated the account framework. For the uninitiated, the base program is installed, as are the modules that the client has licenses for. A look at the Microsoft Dynamics Web site gave me the client’s site name, so I could re-create the server and get the licenses installed.
After a few hours, when we restored the SQL Server data for the company databases we found that we had ducked a bullet. All the data up to a few days ago was there, including the reports and the Check templates.
Just then, the owner of the operation came in. Mr. Guttmann was his name. He was the reincarnation of Alfred Hitchcock, down to the voice.
He wanted to tell me the long-winded history of how this mess got started. He also thanked me when, a few minutes later, the accounting people were connected and they could do payroll.
Me: You know, this was never going to be a permanent solution for you. What you basically have here is a plan to fail.
Mr. Guttman: Apparently we were badly advised.
Me: Yeah, well considering how much money they charged you to make this mess, perhaps you should call the police.
Mr. Guttmann: Well, Jack, can you give us some guidance?
Me: I’ll write a project plan and a Statement of Work.
So I went on my merry way and sent him the proposal. Two months go by, and I hear nothing. Finally I get a call. It’s Femme Fatale.
Femme Fatale: Jack, we had our old IT support firm in again, and now nothing works.
Me: I’ll be up in a few hours.
This time, I took my associate Dave with me to the office. We walked in and were escorted to the server closet again. The door was opened and there stood… a brand-new SINGLE server.
Dave: So what’s this?
Me: It’s the stuff that dreams are made of.