Short Stories from the Offices of Big Brother Vol. 1

> "April the 4th, 1984. To the past, or to the future. To an age when support is free. From the Age of Big Brother, from the Age of the Corporatre Policy, from a weary engineer - greetings!" **— George Orwell, 1984 (maybe)** Since I spend most of my working hours dealing with a lot of separate issues of various degrees of complexity, not everything is always big and fancy enough to be worthy of a separate posts, but they might be funny or interesting enough to be shared still. As such, here is the first of a possibly ongoing series, **"Short Stories from the Offices of Big Brother"**. -------------------------------------------------- > "I just dropped in to see what condition my condition was in" - **Jeffrey 'The Dude' Lebowski, The Big Lebowski** As part of our complete line of *"Cameras to watch you everywhere"* and our ever expanding selection of *"Access Control solutions to lock it down good"*, we offer an intercom unit, that can be wired to a door strike to remotely unlock a door. The intercom itself can be setup to call either to our **$VMS Client** or via SIP to any SIP PBX. The following takes places when one of our Tech Support Specialists (**TSS**) was trying to assist the customer in figuring out why the call was not being placed to the **$VMS Client**. > **TSS**: So the customer is pressing the call button on the Intercom, but it's not calling in to the Client. **Me**: Okay, so you have confirmed they have setup the software to initiate the call, and it's done with the right settings? **TSS**: Yep. I checked the config and it's all done as per our *Knowledge Base Article*. **Me**: Okay good. They are logged in as one the users defined in the setup? **TSS**: Yes, we only have that user in the config for the moment. **Me**: Can you share a screenshot of the config please? At which point they, do, and this is when I notice the issue. See, our intercom works like this: the call button on the front is connected to a Digital Input that is just a simple Normally Open circuit, which when pressed down, causes a closure. The config is essentially built on "When the Input is Active (circuit closed) then start a call to Users". > **Me**: Ah see, there is a problem here. Do you see the option they checked on step 4? *"When digital input is active"* ? **TSS**: Yes I do. Is that the problem? **Me**: Sure is. Setup like this, it essentially only will start the call when the button is pressed down, ***ONLY IF*** the button is already pressed down. **TSS**: Ah yes I see. And since the button can only be pressed down or not, and can't be pressed down twice at the same time, it won't work. So we just remove that condition from the setup and we're good? **Me**: Pretty much yes. Make the change, and them test it once more, and let me know if any other issues are reported. And that's how we figured that with the right conditions, magical things can happen (or in this case, problems can be solved). -------------------------------------------------- > "Documentation, motherlover, do you read it?" **- Jules Winnfield, Pulp Fiction** Since we're not the only company in the market, we do offer a variety of integrations with some of our competitors' products. Some of them from our **$VMS** to competitors Access Control solutions. This story is about one of them. At this point, when I got roped into this problem, the case had been opened for over 6 months, and no one either on the support or Engineering side had found a solution yet. The problem everyone was stuck on was essentially this: the customer was able to install our connector software, and login to our **$VMS** using the connector, but it always failed to connect to the competitor software's database. I get pulled into a remote session along with a voice chat with our Engineering Specialist **(ENG)** and one of our Tech Support Specialists^1 **(TSS)**. > **Me**: Okay, so run me through what the issue is TSS. **TSS**: Well, as you can see, we open the connector software, and we can connect/login to our **$VMS**, but then we try to connect/login to **$Competitor**, it fails. **ENG**: From what we can see, the connector software is working as expected from the logs, the issue is with **$Competitor** software. I'll drop off, just poke me if you need help from our team. **Me**: All right. So let's take this from the top, and start from beginning. Have we checked the credentials for **$Competitor** software are valid? **TSS**: Yes, we can log to the DB with no problem. **Me:** Cool. Next step: let's see the connection configuration. We pull open the configuration for **$Competitor** software DB, and it hits me right in the face. I put myself on mute, pull up the install and configuration guide to double check my finding^2 and then get back to the voice chat. > **Me**: TSS, did the customer actually follow the installation and setup instructions as per our document? Was this reviewed at any point since this case was opened? **TSS**: Yes, the customer said they did follow the document. **Me**: Right. Well, here is something you can add to your rules of proper troubleshooting: *"The Customer said..."* is not valid troubleshooting. Always validate everything yourself. That aside, if you look at the steps on connecting **$Competitor** Software, the requirement is to use the FQDN. Which they have not done. Let's change that and see how it goes. And so we do, and we are greeted with a different error this time, pointing to a network issue. After some testing and investigating, we realize this FQDN points to an IPv6 address. > **Me**: And there we go. The ping to the FQDN resolves to an IPv6 address. TSS, check the documentation again will you? **TSS**: [takes a minute to read through the document] Oh it says the connector software does not support IPv6. I'll let the customer know about this and update the case later. I checked on the case some times later and the customer confirmed that once they had disabled IPv6 on the server hosting **$Competitor** software, the connector worked like a charm and the issue was finally solved. After 6 months of back and forth that is. -------------------------------------------------- > "Everything is a copy of a copy of a copy." **- Narrator, Fight Club** One part of my job I really enjoy to do is troubleshooting issues related to devices^3 manufactured by various competitors and vendors. In one particular instance, I was assisting another one of our Engineers to troubleshoot an serious issue for a customer operating a major transport hub. While working on the remote session, some of the issues we kept coming back to was that all of their cameras are from one of the aforementioned 3rd party vendors, which we will call **Salty Vendor**^4 . Our main concern was that the current firmware these devices are running is that it wasn't fully ONVIF conformant, and might be the cause of multiple issues we are trying to fix on their system. We mentioned to the customer that updating the devices would be a really good thing to do, which is when they informed us they had no avenue to get any update for them anymore, due to bad blood between them and **Salty Vendor**. I poked about a bit on the ol' internets and found some pictures of the cameras. They looked oddly familiar. Too familiar actually. Terribly familiar in fact > **Me**: Say Customer, these cameras look really familiar to me. I might have an idea to help you with updating their FW. **Customer**: Well, as we said, **Salty Vendor** is refusing to provide us with any new firmware for them. The only files we have are the one for the version currently on the cameras. **Me**: That's fine. Can I get a copy of the firmware file for each model you have? I want to look at something. **Customer**: Here you go, knock yourself out. And so I did. Well, figuratively. The filename naming convention was making this so familiar, I started to suspect I might actually be related to it. So I got to work, and using a combination of various tools, I managed to rip apart and dump the firmware files^5 and start looking at the structure, code, and comments left in there. What it revealed is that **Salty Vendor** does not in fact manufacture their own cameras, but (at least for a few models) they simply OEMd cameras from an **Not Well Known Vendor**. Who in turn, had themselves OEMd that line of cameras form hardware from **Better Known Vendor**. The other line of cameras, **SV** had essentially decided to bypass **NWKV** and went straight to **BKV**. How do I know this? Well, it's because **SV** didn't even bother cleaning up any thing in either firmware files, and the model numbers from both **NWKV** and **BKV** were left in there for everyone to see. Customer was in the end very happy as no checks are really enforced when it comes to applying firmware, and they were able to get a more recent version from **BKV** that worked on their devices. -------------------------------------------------- **TL;DR**: If you lock up a thousand monkeys with a thousand typewriters, and have them type randomly, they will over the span of eternity eventually produce all the inane questions customers can hit you up with when they call for support. **^1**: TSS is not always the same person, but since each Specialist usually handle a case from start to finish, I only deal with one person per case. **^2**: Also so I can swear and not be heard. **^3**: Mostly cameras, but also LIDAR arrays, loud speakers, encoders, coffee makers, twitch streaming plugins... you get the idea. **^4**: So aptly named because they didn't like at all that the customer had decided to drop their software solution, and out of spite, decided to stonewall and gatekeep them from getting any Firmware update for the several hundreds of cameras they had bought from them at full retail price **^5**: For those curious, it was a combination of Binwalk and UBI Reader.