This past weekend I went into work on Saturday morning to do some maintenance on some of the servers at work. You know the usual, install the latest hot fixes, install a new switch, work on getting some space free for one of my servers, etc.
First I get in, and backups didn't run on one of the two main servers I will be working with. Reason why they didn't run, we forgot to tell the server it had new tape in it. So it thought it had the old tapes which it couldn't write to in it. No biggie. I'll just tell it to Ineventory, and go work on other things.
Well first thing is to export the database out of Microsoft Sharepoint. Again no big deal. Then we back that up. We had been having some issues getting a full backup, and part of what I was working on was trying to fix that. Exporting the database is the MS reccomended way of backing it up. However any changes made to anything anywhere get blown away if you have to do a restore from this. *UGH*. So I then back this file up to tape, not once, but twice, and to two different servers. Just in case.
I then uninstall IBM Director from the Sharepoint Server. Reboot the Sharepoint server, and no keyboard or mouse. No way to get into the server. All services are working, but I have no way to manage anything on the server. Oh no. So after messing with it for a while, I call Microsoft. The guy I got was very helpful, and very nice. I was talking to him for several hours. Finally it came down to, reloading the server from scratch. I got a refund on the money I spent to call support since they were not able to provide a solution for me.
So I reloaded the server, and built it a bit more intelligently since I now had experience with Sharepoint. Sharepoint got completely installed to the 300+GB drive, instead of part if it being on the OS/Boot Drive.
By this point it was 11:00P.M., I had been there since 7:30A.M., and had taken an hour lunch and that was it. So I figured I'll start the restore of Sharepoint and go home. Actually, I restored the database file off tape (the second backup I did, I didn't have access to the first). Then I ran the command to restore the database. The silly thing just sat there. After a half hour of watching it not seem to do anything (hard drive lights were flickering, and the processor load was higher than normal). I figured I would go home, get some sleep and check on it in the morning.
I got back into the office at 8:30 the next morning. Checked Sharepoint, and nothing seemed to be there. I then checked the log files, and Sharepoint is indexing itself, and shows the workspaces as being there. So I call Microsoft again. I get told I will get a call back in the next 4 hours, the Sharepoint team is on call, and not there on the weekends.
After 4 hours I call in again, and ask when should I get a call. After being on the phone for an hour, I get told I will have a call back in one hour. Somewhere in the next hour I get a call back from a guy at Microsoft. He is the manager of the Sharepoint department. He used to be a tech, and now is a manager, but he is goin got try to help me.
Within 15 minutes this guy has the problem diagnosed. He needs to get me a tool, and will call me back once he finds out where it is, and e-mails it to me. I get a call back a bit later (an hour, maybe hour and a half). Check my e-mail, follow the guys directions.
This fixes 90% of the problem. We then stop the Search Service, kill the search indexes, then start the search service. This fixes 2% of the last 10%. next we have to back up the Database again, and restore it. So I get it backing up, tell him I will call him if I have problems. Now I have been in the server room since 8:30 that morning, it is now after 3:00. I had to pee so damn bad. I go do that, go get lunch (I hadn't eaten since 1:00P.M. the day before). I am gone a half hour and the backup is finished. It took over an hour the morning before to do the same thing. This time it took less than 45 minutes. I then start the restore while I eat. Little over an hour later I am done, and Sharepoint is working.
For those of you with mathskills you will figure this to be 23.5 hours in the office over the coarse of the weekend.
However, I was back in here the next morning (Monday) at 6:30A.M. just in case, I wanted to make sure no one had any problems.
According to the guy at Microsoft the reason why we were having problems backing it up might have been bad search indexes. So with this we should have fixed it. I still have another 1GB of RAM comming in for the box, and a Gig Ethernet card for it also. I hope those will fix any other problems.
Posted by ianshot at February 24, 2004 08:03 AM | TrackBack