|
Post by studio42 on Jul 27, 2004 2:21:01 GMT -5
Well, business is business, and mad money is mad money. I'm heading out on a plane tomorrow(Tuesday) and will be back on Saturday.
Customer had a major failure and their gear is out of warranty and out of maintenance contract, but either way, it's stuff that's gotta get fixed. I'm working on the maintenance paperwork to get their contract rolling again, but they are offering ridiculous money for me to get their stuff working the way it was before the failure. Since I'm the sole source of income for my day job(since I own that), I'm hoping a plane and making 5-digits for 3 days of work, PLUS they are covering all expenses including the plane tickets, hotel and rental car. All I can say is thank goodness I've been to this site before.
I got some phone calls to make tomorrow. I'm sure some folks are gonna be pissed, but hey, I'm not stupid either. I have to take things while the going is good because it will keep things rolling during slim times.
|
|
|
Post by Professor Pimpstatus on Jul 27, 2004 15:11:21 GMT -5
umm I can think of 4 guys who are going to be pissed.
but money is money I guess.
|
|
|
Post by studio42 on Jul 28, 2004 0:48:06 GMT -5
Back on the ground, I could care less. I get 50K to be out of town for 4 days.
Considering I own and operate a data communications company and am soley responsible for all income this company makes, I have to follow where the business takes me. Considering this is one of my top contracted customers, you simply don't say "no". But alas, I've talked too much business numbers.
I did leave a message for Nick as soon as I could. Kind of tough being on the phone for 18 hours a day the past few days. Thank goodness I got some sleep on the plane.
I already know there is no money from the show on Saturday. Hey, such is life on the club scene.
|
|
|
Post by studio42 on Jul 28, 2004 22:10:13 GMT -5
What a day.
Started at 5AM and was rushed to the facility. Thank goodness we had the initial meeting during breakfast in their cafeteria., and at least the food is good.
Scenario: Backup server has failed, flat out deader than crap. Emergency back-up server, also dead. Main production server limping along. Two senior engineers who designed the software: gone, retired, non-reachable, can't find them. Last engineer bailed after getting a "better" job offer elsewhere, also non-reachable. Legacy issues of non-documentation are now the main source of issues. However, the equipment I sold them is running rock-solid as best as it can.
Call equipment(what I sold them) handles on average 160K calls per 24/hr period, way up from 40K calls/night. Considering these are all coming via various toll free numbers, well, those pennies add up fast. considering a failed call takes 2 minutes, a success takes 3-5. Order confirmations are not being dialed-out, which isn't a major issue.
HOWEVER: in regards to the call equipment, there are some badly configured channelized T1 circuits, which is hosing things up. Couple that with server crippling and calls aren't making it to where they need to be. All circuits busy messages are being recieved or other calls are just not landing anywhere, coupled with failed calls.
So, they have arranged for me to meet their best and brightest. Some not so best, some not so bright. OK, one is a retard but at least he isn't drooling.
Step 1 for me is to reverse telnet to all ports and see what is alive and what isn't. Considering over 160 ports per chassis X2 chassis and 96 on another two chassis combined, this took quite some time. This was to see what is workng or not on the server. After that task was done, I was on the line with the telcos getting the T1's properly re-provisioned, one at a time as to minimize interuption of service. After each one was re-provisioned, I would get a bank of 24 direct dial numbers so I could test. If I could at least terminate the dial-up session, that was a success. Remember, on the backside, some ports were not landing on a server. After having reconfigured the circuits for the dial-out ports(that was faster) and testing, I was able to get most of the dial-in circuits fixed.
Status update on the server: Not good. Back-up server? No status. Disaster recovery: no status at all. I'll have additional status in the morning.
Management is defecating bricks, money is being lost and so far my work has gotten the most done and getting things pointed in the right direction. Sales and distribution on a national scale is being disrupted so if you don't see your drugs when you want them or need them at drug stores and some pharmacies, well, now you know why.
Non-functioning channels on T1's have been busied out. Channels with non-functioning server ports are also busied out. Call volume is expected to drop 25% because of these changes as less failures should be occuring. Dial-out calls are not being done and those ports are being re-assigned for dial-in applications in the meantime. The call-backs are a warm-fuzzy anyways.
$40K in new hardware is being Fed-Ex'd to site tomorrow. The older technology used for dial-out is being replaced now that the butthead engineer has left and that hurdle has been bridged. Management has been on my side for 2 years on this issue. Another $40K in hardware is arriving to replace additional out of date management cards, but that won't arrive until Friday.
So, what, it's 3 hours ahead, so I worked from 2AM your time until 7PM your time. I'm pooped out. No breaks. During meals, we brainstormed and got caught up. I am apparently the saving hope of this project, which annoys me because the two main guys I am working with deserve equally as much credit as I do and are working as hard as I am. One is ensuring I get whatever resources I need, while the other one is ensuring all the resources are brought to me. I purposely am not doing much without those two working with me. Granted, they can't do the work I am doing, but this is their project and without their work, I would be unable to do the work I am now able to do.
This is a team effort. I feel after tomorrow I can delegate work back to those two individuals.
|
|
|
Post by studio42 on Jul 28, 2004 23:41:01 GMT -5
Let's tag a few things onto my already overtaxed schedule. The hard drive in my web server took the big poop on me so I'm remotely re-installing everything onto a new hard drive. I had to have someone come over and crack the machine open and swap drives and is sitting around waiting for instructions, like "put in the next disk" and crap like that.
Thank goodness I VPN into my own network and manage it constantly.
|
|
|
Post by Rob Wilds on Jul 29, 2004 1:24:22 GMT -5
Chris, Chris Ward is gonna be in town monday. So let's have that meeting. Give me a call
|
|
|
Post by studio42 on Jul 29, 2004 3:03:11 GMT -5
Sounds like a plan. I'm supervising ftp'ing files back into my "new" web server. Still not perfect, but at least now things are running again!!
I'll be back in town Saturday and am immediately heading to San Jose, then will be back in Sac around, if lucky, 7-8PM. After much needed rest, I will call on Sunday, but most likely will call on Saturday anyways to prove I'm still alive and breathing.
3 hours of sleep, not gonna happen..... ugh!!
I'm thinking of going T-Mobile for that Sidekick thing. I'll be full-time on AIM if I go that route. Just a thought. I like AIM a bit more than the other chat clients.
|
|
|
Post by studio42 on Jul 29, 2004 14:59:23 GMT -5
Break in the storm. I've gotten all the circuits fixed and all ports blocked out that needed to be blocked. I informed the telco to NEVER busy out a DS0, we'll take care of it. The concept was embraced by management who are working with their circuit providers for these circuits to get that in WRITING!! I over-stepped, but upper suits LOVED it.
Server progress is going good. The primary might be fully back up today, and if so, they can get the back-up mirrored over in a matter of hours and the disaster recovery done tonight. Of course, testing will take time. Seems someone documented the heck out of something, so must have been a moment of weakness. I documented all my stuff. Anyhow, there are a staff of people watching everything being done and documenting everything except my work since that has already been done.
New hardware goes in tomorrow. Reconfiguration will be brief, I can work on the configs off-line, tftp them into the routers and then invoke them.
Now I'm being asked to engineer Phase 2, which will take at least a year to roll-out, mainly THOUSANDS of clients will need to be replaced for this to happen. To be honest, the phase 2 rollout is easier anyways, but by doing this via PPP(typical internet dial-up) as opposed to VT100/TTY direct to Unix port stuff, it takes a tad longer. Plus we're going to implement security and accounting for more extensive tracking, which are things currently being lost. We can still implement accounting without security but they want to hold off.
So, now it's a matter of contracts and others getting their acts together. Call volume dropped to around 100K calls last night, too many ports out of service, but still, big improvement. If the servers can be brought back up, we should be back to around 40K calls per night. The call flood starts in 2 hours, so server folks have to get moving. Fortunately it's Unix so nothing should need to be restarted outside of application processes. Ah, protected memory, gotta love it.
My butt is hurting from pulling this big miracle out of it. Couple that with still working on my web server remotely and it's a long pair of days.
|
|
|
Post by Professor Pimpstatus on Jul 29, 2004 19:55:11 GMT -5
shoot 50K, I would pimp my mom!!!
yeah no worries, they bought the new PA so it's all good now on their front.
Thank you for doing that show, it did really sound great, the video was perfect, and the best part was that it was loud enough that when I turned the camera away from the stage, I did not loose sound quality like at so many bar shows.
|
|
|
Post by studio42 on Jul 29, 2004 21:56:45 GMT -5
OK. Good. Sorry to have them spend some money on what I'm sure is some junk, but as long as it gets them by. My full system costs over $80K, and I brought over $50K worth of stuff to the last gig.
But, I don't run other folks stuff, so I guess Maven and Giang Tien Audio will be parting ways. I'll keep the door open for them. Maybe I do their big shows or something.
|
|
|
Post by studio42 on Jul 30, 2004 13:06:46 GMT -5
Update: Well, last night went much better. Calls were down into the mid 50K, dial-out is coming back online today as well as the rest of the locked out server ports.
Dial-in and out testing will start today to ensure it works for later tonight. 4 hours from the east coast call flood.
So far, let me recap a few things: Unix admins leaving forces panic. I suspect some deliberate sabotage. Hosed up DS0's are being fixed.
Basically, I could have done everything from my home and gotten the same stuff accomplished, however management granted me god-powers which I apparently can't get over the phone. In a sense, I re-stated the obvious to upper management. I should also note that the two guys I've been directly working with here for the past 2 years, ALSO stated the same thing to management. I am meeting with management in an hour to find out WHY those two were not listened to, despite the fact they came to the same conclusion I did over the phone. I mean, we're intelligent people. Some of the underlings/overlings, not so.
I'm making good scratch, I'm getting loads of wireless connectivity, but I do miss my wife, my baby and my two pugs.
My linux server is back online, maybe at 95%. can't get 100% until I get back, too much tweaking of CGI's, but I am functional again, and that's all that matters.
So, I leave in the middle of the night, return on Saturday, have to drive to San Jose and back and then clean out my garage for an open house on Sunday(I'm selling my house, need a larger one with NO POOL). I await throngs of pissed off people(Maven) and a long work week as I have to make up for lost time.
I can't wait to return to doing sound again. Check my events calendar.
|
|
|
Post by studio42 on Jul 31, 2004 0:01:25 GMT -5
Done. My hardware is working perfectly everywhere. Backup is being rebuilt and brought back to life. Disaster: same deal.
I get onna plane to come home in the middle of the night, then go right to San Jose with the wife and kid to see some of my wife's friends and not understand anything they say for 4 hours, then drive back. Ah, what a reward.
Sunday: Showing the house. No rest for me.
|
|
|
Post by studio42 on Jul 31, 2004 23:16:11 GMT -5
What a day. Got off the plane, into a car with my wife and kid and drove 2 hours to San Jose. Had a good time, took 3 hours to drive back. A combination of a non-scheduled side-trip, a dangerous U-Turn, some wrong directions and then the 205 to 5 exchanging sucking major wind, and well, I'm home, feeling anti-social and am diving into cleaning up the house in preparation for an open hour to get this place sold.
Tomorrow, I'm outta the house and at my mother's house(my dad died on Jan 1 of 2003) for a party for a nephew who is nearly 2. Taking the wife, kid, 2 pugs and trailer. Mainly taking the trailer so it does not detract from the house.
I'll provide alternate means to contact me as my Cingular phone doesn't work worth a crap at my mom's place in Orangevale.
I am jet-lagged and need rest that isn't going to take place. Ah well.
|
|