Poor I.T. arrangement....

[RipplecutBuddha]

Okay, Fresh off of a bad night at work, I gotta get it out of me, so here it goes.

 

I actually love my job. Really, I do. My manager is one of the finest I've ever worked for. I can't say I have a problem with any of my fellow employees either, which I have found to be a rare situation. It's even a decent company to work for that owns the hotel franchise where I work.

 

But....they totally fumbled on the final inches of a touchdown when it comes to I.T. and last night I, and *every other desk clerk that works there* caught it in the teeth at the same time.

 

We have an old server that was due to be replaced in the first quarter of the coming year. Only problem is that it has either crashed, overheated and shut down, or just decided to restart itself out of the blue at least twice a month since last June. The tech support company, and the hardware support company have both been excellent until yesterday. Corporate's second in command, a really good guy as well, sent us the company "temporary server" to hold us over until our new one comes in next month. He gave us one day's notice...the day before our manager was to head to San Diego and her Father for Christmas for two weeks. (Our issues had apparently moved us to the front of the line)

 

So, the Front Desk Supervisor was at the property for 13 1/2 hours with Hardware Support trying to get the 'temp' server online as our current server could not stay powered up. Once it was online, he had to change calls to tech support to get all the files, histories, authorizations, and security permissions transferred and/or established on the new machine so we could keep running. We had over 40 rooms check in during the day, all of which had to be done the old fashioned way, by hand, as the computers were flat useless until 10 minutes before I showed up.

 

At that point, another night auditor had been called in to begin entering all of them into the system. That way at least the computer would accurately show who was in-house for the night.Yay data entry!!! I did the cash count so the swing shift clerk could go home for the night. I began prepping for the regular night audit process, which I wasn't convinced would actually happen at the time. I ended up getting most of it done, but more on that in a bit. Then the swing shift clerk came back in and asked the FD Supervisor to help him as he had locked himself out of his truck. The FD Supervisor was small enough to crawl through the open back window of the truck to open a door from the inside, then get back on the phone with Tech Support.

 

The Assistant General Manager was there by corporate rule, which sucked for her as both her children are autistic and would not go to bed until she went with them. They all checked into a room at the hotel with the AGM's mother as well...oh, and their little dog too.

 

The 'temp' server was now online and we could begin the night audit process. Good. Then the FD Supervisor appeared to try and strangle a phantom in front of him as he talked with tech support. For some reason, the software that allows us to charge credit card accounts had not been installed on the 'temp' server. Before I get to that, the other night auditor had finished checking everyone in, but then faced an even worse task. To explain, when you check into a hotel on a credit or debit card, the computer sends an authorization request to your bank to verify that adequate funds are available to cover the expected charges. As soon as it comes back as good to go, the check-in process finishes. Your account isn't actually charged until check-out, or the middle of that night, as with our system.

 

Well, we didn't have the interface to do the authorizations during check-in, so she had to get on a cordless phone, and manually authorize each and every check in for total expected charges, being room rate, taxes, and related fees, times the number of nights they expected to stay. She needed two hours to do that little task.

 

But, back to the FD Supervisor on the phone. He had it on speaker in the manager's office.

 

"Okay, so can't you pull it from our server and install it on the temp?"

 

"No we cannot. It seems that your version of that software is three updates old. By law we can't install an older version of currently available software, especially when dealing with direct financial access to personal bank and/or credit accounts."

 

"Okay, so then if you have the new version, can you just download it to the temp?"

 

"No we cannot. You have to contact the owner of the software and request it from them."

 

*ends the call with tech support in a friendly and grateful tone of voice and then furiously dials the new number*

 

"Yes, it seems we need the current version of your software downloaded to our new server here at the hotel."

 

"Certainly, sir. I'll send you an email. Included with that email is a PDF that you need to print, complete and sign by hand, then scan and email, or fax back to us. The email will have all the details for you."

 

*call ends*

 

Only when we read the email, did we discover that the form was merely and application for permission to buy the software....oh, and it came with a $300.00 processing fee.  Only when they were certain we weren't some shaky fraudsters would they consent to sell us the software (for who knows how much more money) and download it onto our system for tech support to tie into the system.

 

Now, the reason this is serious is because the software we use for managing the guests in our hotel, from reservations, to past guest stay history, does not actually charge credit cards. That step requires an interface with the software we apparently didn't have. If the system can't process a credit card, and close out a guest's folio to zero due, it won't let us check them out of the room in the database. Even when the guest leaves, the next guest to come into that room can't be checked in, because the computer thinks the former guest is still there.

 

What this also means is that since the guests that have checked out are still seen as in-house, until we can check them out, their room rate has to be set to 0.00 so as to avoid charging them for another stay. The charges are posted every night automatically.

 

And finally, once we've booked someone into every 'clean/vacant' room in the hotel in this situation, on the computer we are then sold out. Every clean/vacant room on the property (in the real world) is seen by the computer as occupied by guests that aren't really there because they left but we can't check them out. We can't check them out because the computer can't verify the CC authorization for the charge. It can't verify the charge because we don't have the current version of the software needed for the interface. Without that we can't close out the balances due on the guest's folios. Until we can close them out, we can't open a new folio for the room to check a new guest into a room that is, in reality, ready for occupation.

 

So, what to do? Well the immediate solution may have to be to revert to completely manual paperwork and just leave the computers out of it until everything is in place to get back on track. Let me tell you how fun that sounds to a night auditor for a 108 room hotel...not very. I had to do it for a 60 room hotel on Microsoft Excell for 5 1/2 years and I'd still almost rather have my teeth pulled out by rabid alligators.

 

Corporate is in the loop on the whole situation, so I doubt it will be long before the situation is cleared up. If worse case scenario happens, we are 'sold out' and missing nearly half the property in revenue daily until it is corrected. Like I said, I needed to vent.

 

I have observed that I'm a brooder by nature. I've learned to take comfort in that awareness. I immediately reason out the worst case scenario and then before long I calm down with the assurance that it is most likely going to be much better than what I fear. And believe me, I can come up with some pretty bad scenarios.

 

Thanks for your time. Carry on I'm going to bed now.

[Vort]

So sorry for your pain. This is a classic example of why software vendors generally try to say that they will not be responsible for incidental damages, only for repair or replacement of the software itself.

 

Here's a bit of good news, though. Rabies afflicts mammals and only mammals, so you can put to rest your nightmarish scenario of dentistry being performed by a rabid reptile.

[Mahone]

First question, what does your IT disaster recovery policy/document state regarding that particular system, and how long it can realistically be offline for in the event of a disaster before company operations are seriously affected?

 

It sounds to me like whatever period of time is specified, it's clearly too long. Either that, or the person who set this up didn't pay any attention to it. From what you have said, it sounds to me like the company as a whole was very seriously affected by this event.

 

When a system is initially implemented and configured, disaster recovery policies and procedures should always be taken into account, and the questions of how long it should take to get this system back online in the event of any disaster, and how that process will occur should be asked. Desktop walkthroughs and real life simulations should be performed. In other words, someone in the IT department should be going through the steps that would be required to re-build such a system from scratch, or get it back online in the event that it was obliterated, and making sure that it can happen in a timely manner. This applies for both small and large companies.

 

Where are the failover systems located? Surely such a critical system isn't located on a single physical server in a cupboard? Someone in the IT department needs to be re-thinking this setup.

Edited December 20, 2014 by Mahone

[Crypto]

I feel for you, sounds quite frustrating.

 

This is why so many companies are virtualizing. Clone the needed setup, and simply maintain your virtual infrastructure.

(vmware ftw?)

Edited December 21, 2014 by Crypto

[RipplecutBuddha]

The hotel industry is a penny-pinching one to a rather embarrassing degree. We are, perennially, the #1 hotel in quality and customer satisfaction for our franchise brand nationwide. Having said that, at the first hotel I worked for, I learned night audit from a lady that had held the position for three years before I was hired. That's the property that did everything on Microsoft Excell. When she was trained, all the property's info/paperwork was still being kept and maintained strictly on paper ledger books only!! Let's see, that was 5 1/2 years ago, so that would put her training at around 2004 or 2005!!! And this was a hotel chain with 35 locations nationwide. It's not like a decent I.T. investment would have cost a fortune at that scale. Pricey, yes, but not astronomical.

 

It's going to get resolved, and yes, other properties have been affected in similar ways. The bad news is, when we get our new server next year, we get to go through this all over again. It would be fantastic if it were set up to link directly to servers at a single location on a remote status, but then, the US corporate model for problem solving is to figure out the right way to solve the problem, and then try everything else possible first.

 

 

Mahone:

Where are the failover systems located? Surely such a critical system isn't located on a single physical server in a cupboard? Someone in the IT department needs to be re-thinking this setup.

 

hah! It is on a server 20"x20"x32" under a table on the floor in the office behind the front desk itself. We have a DROBO file backup as well as regular daily backups on the server itself....apparently not good enough.  It only needs to operate five terminals on the property, so there's that. At the same time, it's got to be set up better. I just don't think many companies take I.T. investment seriously enough.

 

I assume by next weekend at the latest we'll be good to go, if not by the end of business day on Monday. Last night went much better, even though the main computer system was down for a few hours during the day again. I'm worried that the 'temp' server has its own issues we get to discover in the coming weeks....

[slamjet]

The first lesson I learned is that most management will not spend money on something they don't understand.  Thus, the crappiest and oldest servers out there tend to be the ones that are the most $$$ critical while priorities are to make sure everyone has colored staplers.

[Mahone]

The first lesson I learned is that most management will not spend money on something they don't understand.  Thus, the crappiest and oldest servers out there tend to be the ones that are the most $$$ critical while priorities are to make sure everyone has colored staplers.

 

Correct, although unfortunately it's part of our job to convince the relevant people of the importance of what we are asking. Some people are better at this than others - in a previous position I probably wasn't as good at doing this as I should have been, and I won't make that mistake again. At least I'll try not to.

 

All we can do after than is ensure we minute and document all of our conversations where our requests for additional budget are denied due to "not having any money".

 

All systems at some point will fail. It's not a case of if, it's merely a case of when. When a system fails, and critical services fall offline for unacceptable periods of time as a result, the next step is producing the documents where these denied budget requests are minuted and showing them to management. It's interesting how quickly this non-existent money suddenly materializes at that point.

Edited December 22, 2014 by Mahone

[Mahone]

 

It's going to get resolved, and yes, other properties have been affected in similar ways. The bad news is, when we get our new server next year, we get to go through this all over again.

 

Let me guess, IT will set the new server up in exactly the same way as the old one, and no lessons will be learnt from this incident. Or the plan will be to modify the setup at some point in the future, which never happens due to lack of budget...

Edited December 22, 2014 by Mahone

[RipplecutBuddha]

Let me guess, IT will set the new server up in exactly the same way as the old one, and no lessons will be learnt from this incident. Or the plan will be to modify the setup at some point in the future, which never happens due to lack of budget...

Heh, possibly, however we were already due to get a new server next year before any issues with this one began. The whole system went through two or three variations since I've been hired. We started out with a satellite internet connection, but that equipment was ancient seven years ago when the property was expanded and renovated. Then we went to cable internet briefly with horrible results due to a certain provider here that doesn't know what they're doing.

 

Currently we're still on that set-up, but with an additional VPN connection for management. Oh, to hear my manager express her joy with that arrangement. Apparently when the corporate I.T. set that all up, they didn't obtain enough certificates for all the properties to be on the system at the same time (something that would only make sense from my eyes) so she has to pick and choose when to get her work done by guessing when another manager or two are done doing theirs first....

 

The internet cable setup for the main system has stabilized enough for now that we finally don't miss the satellite system at all. For my part, I think we would benefit from a single outside IT support company coming in and spec'ing out a single complete system from scratch. The guest internet is completely separate from all the mess, but it's just getting uglier with band-aids overlapping each other at this point.

 

Oh, well. Supposedly on Monday we'll be getting the rest of the 'temp' server up. We'll see how solid it is I guess.