It's good to be a big fish in a small pond
Posted by NotYourNanny@reddit | talesfromtechsupport | View on Reddit | 22 comments
This one reminded me of our first experience at converting to VOIP.
When the day arrived to start replacing the 40 year old phone systems in our stores (due to replacements for broken phones costing $300 each, refurbished, at best, with a 50/50 chance they'd work), we talked to our phone (and internet) company about what was available. They were transitioning from providing analog service (over the internet, but converted on-site) to VOIP. OK, makes sense, it's the right choice.
But they were new to VOIP, and still getting a handle on how to make things work the way the customer wants, and most of their customers are offices, not brick & mortar retail. They don't know what we need, and we don't know what to ask for. OK, we'll work with them on that. They've always taken pretty good care of us. (Not the cheapest, by any means, but reliability is more important to us, because without the computers, we've got about 3 days offline before it all starts to fall apart.)
The problem was, the phones they sold us came out of the box configured for an office. The receptionist's phone rings, they answer, and transfer the call to the person it's for by extension number. Retail stores don't work that way. We need all phones to ring, be able to pick up the call anywhere, put it on hold, pick it up from another phone, lather, rinse, repeat as the associate runs around the store answering questions.
On the old phones, this is one button to put a call on hold, and one button to pick it up from any phone. The default way to do this on the new phones involves multiple buttons (including a four digit extension number) to park the call, and an equal number of buttons to pick it up. (And there's a button marked "Hold" that actually puts the call on hold - but it can only be picked up on the same phone.) Needless to say, this won't fly, even with a running start from a tall cliff. They knew this going in.
So they configured the phones to move the call to a park line with two buttons, and pick it up from any phone with two buttons. Worked . . . Okay, if not ideal. Except it relied on a feature that was deprecated, and when they did a software upgrade, it disappeared.
Then they found another way to do the same thing, but with three or four buttons. Not ideal at all, and it tended to result in dropped calls. And the particular store we converted first is in a beach city full of people rich enough to live at the beach, but not rich enough to live in the very rich beach city next door, which makes them . . . cranky. These people won't call back to finish their call, they will call back to scream obscenities at you. The only reliable way to put calls on hold such that they could be picked up on a different phone was the "official" way involving waaaay too many buttons. Plus, it also relied on a deprecated feature, and was never going to have the bugs addressed.
One of the cashiers - who has been with the company for a couple of decades and was the best they had for working with customers - simply refused to answer the phone. And her manager didn't blame her. The manager (also one of our best) was ready to quit over it. And we didn't blame him.
So a conversation was had, between me, who makes all the technical decisions, my boss, who has signature authority, our account rep (who was sympathetic, but had the technical savvy of a turnip), a senior VP from the phone company, and the lead technical guy for the phone company. It started off with my boss telling the VP "You have 30 days to make this work the way we want, or we're finding a new phone company." We were, at the time, about a quarter million dollar a year account (we're more now). Not their biggest, by any means, but big enough to get their attention. (Our account rep told me he had bigger accounts, but not many, and not by much. He was very concerned. I believe his bonuses were heavily based on total revenue generated.)
Took 'em two weeks, but they discovered some kind of macro feature ("key system emulation"?) that let them reprogram the buttons to do the complicated sequence to move the call around. One button to put a call on hold, one button to pick it up anywhere, and it worked (and we got CallerID to boot!) almost exactly like the old phones. Only difference was it was a different button to pick up the call than to put it on hold. I requested the tech guy write detailed instructions on how he did it and include it in the records for that store, so that I could refer future project engineers to that when we got to other stores. And he did, and I believe they still use those instructions to this day. And we're not their only brick & mortar customer these days, either.
alohawolf@reddit
The lack of multiple appearance/multiple pickup in VoIP phone systems is galling - key systems or hybrid key systems were more than half of the business phone system market for 50+ years.
My belief is, if the product doesnt work the way the user expects, the workflow is wrong, and the product is wrong - not the user.
NotYourNanny@reddit (OP)
They got it straightened out. They just needed some incentive to dig in when they were figuring out new tech.
alohawolf@reddit
Oh yes, I agree - it just boggles my mind why people dont design that in from the get go.
I am kinda curious which PBX system you were replacing?
NotYourNanny@reddit (OP)
The system came out of the box configured for an office, where one person is going to answer incoming calls, find out who it's for, and transfer it to their extension. Would have worked fine for that, and I suspect that's a far more common setup. Just didn't work for a retail store.
The old system was a 40 year old Comdial system that was a real workhorse, but replacement parts (like phones) hadn't been manufactured for 20+ years. We could get refurbished ones off of Amazon for about twice what brand new VOIP phones cost, with about a 50/50 chance of them actually working, or we could reliably get refurbished ones from our regular phone guy for about three times what the VOIP units cost. And the VOIP service is cheaper than the old analog conversion setup anyway.
The only reason we were slow to pull the trigger was that the old system was on either CAT3 or, more often, actual phone wire, so we had thousands of dollars worth of cabling to do up front per location (and in some cases, several times that in stores we needed to redo the entire network anyway, and one location that need a complete, brand new PA system). Between that and WiFi upgrades, we're about $300,000 into it so far.
alohawolf@reddit
I worked on basically every PBX maker but Comdial, ironically.
I get the issue with legacy PBXen though, seen the same issue with Meridian and Northstar - I have a CA customer who has enough spares for their province wide meridian system to last two decades.
NotYourNanny@reddit (OP)
Two decades is about how long we lasted after they stopped making spares, I guess (our phone guy at the time had a bunch on hand, and we were a favorite customer).
But time marches on, and standing in the way of progress is a good way to get run over.
kandoras@reddit
Good call. A project without proper documentation is broken, it's just waiting for a really bad time to announce that fact.
Sir_Jimmothy@reddit
Even back in the day, this should have been an easy feature - these days our company would set up a queue with all extensions being members, but back then a regular ring group would suffice, and was incredibly common.
Call parking was even more common back then than it is now, where you just park the call against line 1, 2, 3, etc, and another person presses that line button to pick it up. There's a reason all those fictional shows had something like "oi, call for you on line 3".
Glad they found an answer, but if they were only just getting into VoIP and hadn't hired someone specifically for it, I'd have gone elsewhere immediately until they'd matured in their offerings.
NotYourNanny@reddit (OP)
Yeah, it wasn't a matter of what the equipment could do so much as they were still figuring it all out.
That would be a major undertaking, since they also provide internet service for almost all our locations. Updating nearly 20 firewalls and VPNs would have been a bigger burden for everyone, especially me. (And we'd have been hard pressed to find another provider for internet service that was a reliable.)
Sir_Jimmothy@reddit
Oh that's fair, I only meant not going with them for VoIP :]
NotYourNanny@reddit (OP)
Unfortunately, their package deals have (so far) been impossible to beat if split between vendors. And overall, they do take good care of us.
Throwaway_Old_Guy@reddit
I can't imagine how this would have gone if you hadn't been a big enough Customer.
Loko8765@reddit
I can tell you. OP wrote:
This happened to me with a software product. We were dealing directly with the editor, but they were big enough that they didn’t really care about our measly 10k users for $200k/year contract. The feature wasn’t deprecated to begin with, and it was well understood by their salespeople that the feature was essential to our use of their product, but in an upgrade it disappeared. Luckily we tested each release, found it didn’t work, said WTF, opened a case, and the reply was basically sorry not sorry, it seems the feature was added for just one customer when the company and product were small, it was a pain to maintain, we lost that client so nobody was using the feature any more. WTF redux, we are using it! Yeah, well.
So we didn’t upgrade… some time later we found a critical bug that crashed our system in specific conditions related to the volume of data stored, which was not going to improve. Their awesome front end support analysed and diagnosed the bug in two hours, going down to the specific two lines of code that had to be changed, an error so basic that I’ve literally taught a CS M.Sc. class containing the correct algorithm. The version’s end of support was two weeks away, escalate to development support… sorry not sorry, we have already pushed the fix for the current version but we’re not going to do a bugfix release of the old version, deal with it.
After evaluating options we brought in an integrator with deep knowledge of the product who hacked together an implementation of the feature for the new version, basically making a front-end skin of the whole product and intercepting clicks when needed. Obviously, that only lasted until we could “upgrade” to a different product.
Stryker_One@reddit
Sounds oddly similar to an issue we had where I worked, wherein a piece of hardware didn't work after an update. Calls to the manufacturer landed on deaf ears. A workaround was created by us and we proceeded as usual. The kicker? A few weeks later we got a call from said manufacturer asking if they could use our workaround, turns out we weren't the only ones impacted by the "update".
Sorry if this post seems vague, it has to be in this environment.
NotYourNanny@reddit (OP)
And was the answer "no"? Or, alternately, how much did you charge them for it?
Loko8765@reddit
I was vague too, I don’t mind shaming the company in question, but I would be severely doxxing myself :D
NotYourNanny@reddit (OP)
We would have found a new phone company, and the old one wouldn't have cared. But as it was, we taught them how to properly service a a new type of account.
Arokthis@reddit
God bless you.
I'll bet the tech guy tried to take the credit but his manager got there first.
NotYourNanny@reddit (OP)
Hard to say. When we're starting an upgrade to a new store, I always tell them to look at the notes for that first location, and they have a hard time finding it all if I don't include his name. Credit up the food chain, I have no idea, but his name is apparently all over the notes.
Stryker_One@reddit
I worked for a medical device engineering company that need a servo controller to be spec'd in. I recommended one and they went with it. When it came in, I discovered that it didn't do a very simple thing that the documentation claimed it did do. After days of back and forth with them on the phone to prove that in fact it didn't do what they claimed, I finally received some brand new firmware with instruction on how to re-flash the units, a process normally ONLY done in the factory.
KelemvorSparkyfox@reddit
Well done for getting a record of the modifications in writing.
The manager threatening to walk reminds me of a failed implementation in my first office job. After the old, working, reliable system was taken away and the new, shiny, ineffective system was imposed, we struggled for a fortnight. The third Monday, my manager walked into the MD's office and said, "I cannot work with Trilogy. Either it goes, or I go." The MD was in conference with one of the other company directors at that point. This other director walked to the IT Director's office and told him to re-enable the AS/400 for us.
My manager stayed; our divisional director was given marching orders and left a year later.
NotYourNanny@reddit (OP)
Purely for selfish reasons. That was the first of a couple of dozen locations. So far (we're still buying more stores). I had no intention of going through that again.
There were (and are) other managers who wouldn't have had the same pull as this one. He's a pain in the ass in many ways, but he runs a tight ship and his store does very well.