24/7 IT Hotline service recommendations
Posted by EmbarrassedAd4932@reddit | sysadmin | View on Reddit | 18 comments
Hey fellow sysadmins. For those of you who have IT staff on call 24/7 what do you use for your middle of the night notifications? Today our on-prem phone system will take a message and then call whomever is on call every 15 minutes until they wake up and pick up the phone. We are moving to Teams for our phones which doesn't support this natively. I know we can build a power app that can do this, but it seems clunky. Does anyone know of a hosted service that provides this functionality? Thanks!
time-for-reform@reddit
Has anyone mentioned the crisis hotline number? I feel that needs to be called out
SudoZenWizz@reddit
We're using opsgenie and jira ticketing and based on customer SLA we route only notifications that are to be treated 24/7.
Conscious-Arm-6298@reddit
Neuralink can send electric discharges directly to the technicians brain
WaizenErnter@reddit
We have a smartphone with a mobile number that is given to whoever has shift so they can get called.
chickibumbum_byomde@reddit
i would recommend not building it yourself, use a proper oncall tool.
PagerDuty or Grafana OnCall handles escalation, repeat calls/SMS, and rotations out of the box. Pair it with your monitoring some decent/proper monitoring and it will cover your current “call every 15 min until picked up” setup cleanly.
set your thresholds, configure some notifications, maybe some reporting and your good to go.
mech101v@reddit
Hiplink with their app can be set to notify for x minutes or forever. There's a whole bunch of other ways can send but the app is in my opinion the best.
Our call center takes info and then uses a form built into the app website that has escalation and round robin settings.
It's kind of niche to our line of business but it works really well.
Coldsmoke888@reddit
MSP is in India. They’re awake.
We do have on call critical situation manager with ilert app. Rarely does it go off in the middle of the night, thankfully. The noise it makes scares the shit out of me.
TechMaster212@reddit
My org does 24x7 on-call. We use Teams phone numbers, we have an on-call person listed per Team in ServiceNow. The Help Desk will escalate tickets to the on-call and if they answer great they give them the details. If no answer they have 10 minutes to acknowledge either through the back line to the Help Desk or by assigning the ticket to themselves. If they don’t acknowledge then the Help Desk escalates to the next person in line.
The person taking the on-call needs to make sure they are either not putting their phone in do not disturb or have placed an override for the number calls will be coming from
aluminumpork@reddit
SIGNL4 is pretty good
Main_Ambassador_4985@reddit
We are only 6 to 6 business hours.
We have MS Teams installed on mobile phones. We have a group call queue that can be signed in or out depending on the person on call.
The call queues ends with a M365 group voicemail that is emailed to everyone in the group, appears in Teams, and forwarded to the help desk.
sryan2k1@reddit
Pagerduty
Bubby_Mang@reddit
AnswerFirst answers the phone and then calls people on a rotating schedule I set up.
The interface sucks but it's cheap and works.
whatdoido8383@reddit
Do you have 24x7 helpdesk? If a sysadmin is on call, I'd expect someone else to field calls and monitor things before being able to call the on call. If you're allowing users to call and wake up a sysadmin, that's horrible practice.
Where I work the helpdesk takes the call and determines if they need to wake up the on call based on several factors the business has put into place. There needs to be guardrails around on call or people will treat it like a 24x7 helpdesk even for the smallest issues.
We don't get called by the helpdesk unless they're getting multiple calls about system issues, there is no workaround and it's impacting the actual business financially. Just because Susan locked herself out of something doesn't mean it's eligible to wake someone up. If they need that type of service, they need a night shift.
cwm13@reddit
We arranged our one in-person team meeting for Wednesday, which is when our call schedule rotates. Person going off-call hands the iphone over to the person going on-call.
Slyscape@reddit
We’re utilizing PagerDuty, triggering it with emails or api calls and the rotation is built/ managed on the platform. It offers an escalation tree and we’ve integrated into our alerting / monitoring platform.
halodude423@reddit
If someone wants the on call they call the switchboard who then pages the on call help desk who can escalate if need be. For a while we had overseas help desk that got overnight who were the ones that called switchboard to page.
gangaskan@reddit
We have a DID that we forward our number to when on call.
Severus157@reddit
We have a very simple solution, separate work mobile phone for on call personal. These devices all have the same phone number and whoever is oncall will activate sounds/turn it on before shift. And our nightshift guys call the number when they have a problem.
In my old company we had a pager app, that was always triggered when a ticket with a certain severity came in.
Both work, but the phone is definitely the best solution for us mostly. It's not automated, but it's definitely better than being paged because someone created a ticket with a high severity because they lost their headset.