Pour One Out for Capital One Admins
Posted by sysad82@reddit | sysadmin | View on Reddit | 220 comments
They're going on some massive backend outage for 24 hours that is effecting a large percentage of their customers. It has effected ACH amongst other things which means many people are not getting their paycheck posted to their account. Too many Americans live paycheck to paycheck so that failure is causing cascading issues for so many people such as if you have things on autopay the money may not be there.
Capital One is spending tens of millions at least in advertising during sporting events but I bet they found a way to cut corners in technology. I'd bet they even did layoffs in the past few months, H1b / Outsourcing and/or RTO mandates that drove away the best talent.
InviteScared2327@reddit
For awareness, this outage is due to FIS, which Capital One leverages for these services, having a power outage that disrupted their data center. FIS' response was slow and resulted in a significant amount of batch processing that needs to occur post-recovery. This really shines a light on the risks associated with critical third party dependencies.
jpm0719@reddit
Yup. I work in a bank. The FIS data center impacted was in Little Rock. I am in Arkansas so can confirm. Interestingly enough, we have not been impacted at all and we run some products out of the LR data center.
fullchooch@reddit
Was this proprietary or colo? If colo, I'm absolutely astounded that they weren't generator backed, and it took so long for restoration
Lokabf3@reddit
FIS has many systems in the LR data center. Only one of the mainframe environments had issues... but a "big" mainframe environment none-the-less. Lots of systems in LR are fine, and were never impacted.
jpm0719@reddit
Oh I know, I am just shocked. Historically our partition always gets impacted and we were not this time. I am quite happy to not be impacted. I have friends who work there, would not want to be them.
fullchooch@reddit
TierPoint in Little Rock Arkansas?
Lokabf3@reddit
This is accurate. It's the FIS sysadmins who are having a bad couple days.
I wouldn't say that their response was slow. The power outage triggered a hardware failure as they flipped to generator power, and it took them some time to fix it. Since then it's all been about catching up on delayed processing.
InviteScared2327@reddit
My opinion on their slow response is due to their hesitance to failover to their alternate data center. There may have been a concern of data loss, but they should've made this decision much sooner rather than troubleshooting hardware issues while customers are down.
heapsp@reddit
Lots of times the DR plan is a grift by upper and middle managers and doesn't work at all like shown on paper. its too easy to lie about testing and capabilities with something that 'should' never be used. I've seen it first hand after running complex DR scenarios for large businesses then seeing the report given to the board later that claims capabilities that they didn't even HAVE. So i can understand hesitating on a full DR failover.
frygod@reddit
DR that you haven't practiced my as well not exist. Backups you've never restored from may as well not exist. This is why my org tests fail over and back between our data centers multiple times a year on our core apps, and we do at least 2 greenfield restore drills a year. First you prove your strategy works, and then you drill it until you can run the procedure at 2am on a Saturday while still a bit drunk from the bar on Friday.
heapsp@reddit
Thats good though, that sounds like a well run IT department.
Most get thrown from priority to priority and also solve complex tickets with an understaffed and underpaid department and the leaders just 'accomplish' DR by lying about it in a powerpoint.
BatemansChainsaw@reddit
One company I did work for claimed to have a DR maintained by an overseas MSP. They didn't do the needful and when the corporate office lost power (whole city had issues for hours) the DR site failed. No one could access their accounts, purchase with their cards, and ATMs weren't able to process withdrawals.
Glorious 6 hour fuckup that resulted in that MSP being fired and sued out of existence.
CaptainWart@reddit
I'll do you one better. I took over an IT management role from a finance guy who had no IT background but just assumed the MSP they were paying was doing the things they were paying for, including DR services with fail over to a secondary data center. I start asking the MSP questions and peeling back the layers and eventually figure out that they sold us the service, charged us for it every month, told us they tested it twice a year, but had never bothered to actually set it up. Thankfully I caught that before we needed it and terminated that contract as fast as possible.
dansedemorte@reddit
just to reform under a brand new name the next day and was selling cut-rate service to all their old customers.
TEverettReynolds@reddit
Me too! I once saw my DR report that had references to our cold DR site changed by some yahoo above me to now say "hot" DR sites.
They did a find and replace, changed cold to hot, and sent it to the Board of Directors. As a PDF none the less...
Ok-Pickleing@reddit
Fuxkin capitalism. At least you got paid right?
HeliosTrick@reddit
How was this a failure of capitalism? Are you telling me that all other economic systems don't lie on reports, like China, Venezuela, the Soviet Union?
This is just lazy managers covering their asses by falsifying data, which has nothing to do with economics and everything to do with shitty people who can't take the heat or admit piss poor planning.
donbowman@reddit
this is why whistleblower laws exist. Your company should have a lighthouse type compliance reporting method.
VestibuleOfTheFutile@reddit
People here love to bitch about auditors, but this is the kind of thing that auditors would love to be notified about and absolutely investigate. Someone should be and would be (in most cases) getting fired for this.
BeginningReflection4@reddit
90% of DR plans are just that--plans, never actually tested prior to the disaster.
jpmoney@reddit
They're normally stuck (or killed) in purchase approval. The hardware is never on the ground to be tested.
ImmortalTrendz@reddit
"what do you mean duplicate hardware will run $40k?"
RepresentativeDog697@reddit
At one Job, the DR site was a data center in another building on the same campus, a 100,000+ employee company.
illicITparameters@reddit
In my experience it’s usually the C-suites who don’t want to follow the DR plan when it’s go-time.
JohnGillnitz@reddit
"We have concepts of a plan."
willtel76@reddit
In my experience there is typically much more D than R in a typical DR scenario.
spokale@reddit
This is a big issue with financial institutions. Having truly synchronous replication of ledger and transaction data across large geographic distances can be inherently problematic (best case you're adding like 60ms of latency to every write), so often the replication is asynchronous.
The result is, say, you're doing 5-minute interval transaction log backups. Great idea normally, but what if you have 5,000 transactions in the intervening 5 minutes? It may take a considerable amount of time to reconcile those, maybe even more than just waiting to get the original datacenter up.
ImmortalTrendz@reddit
Correct. I haven't worked for a bank in a long time, but I see a lot of people here not understanding the data at risk. I fully understand not failing over, especially when the main data center has an repair eta of hours.
Lokabf3@reddit
Yup. Failover to DR comes with it's own risk, and given the size of the environment we're talking about, and the number of customers who depend on it, the decision to fail over to DR was a big one. Data loss, and the complexity of returning from a DR position were absolutely part of the decision making process.
At the end of the day, they decided that the lower risk was a longer outage, but an in-place recovery.
jameson71@reddit
Their "DR" sounds utterly insufficient for the criticality of the service they are providing.
Lokabf3@reddit
When you get into large enterprises, DR is truly designed for disasters. The "smoking hole in the ground" level disaster. It is not designed for incident recovery.
Had the nature of the failure been one that would have been a multi-day outage, then I'm confident DR would have been invoked. But in this case, the actual outage was around 15 hours, with the subsequent impacts being all part of the recovery efforts (catching up their batch processing).
This is the decisions that executives have to make. Do we invoke, and deal with the consequences and risks that are associated, or take the hit for a longer recovery, but have very little risk?
jameson71@reddit
Perhaps I should have said their HA is utterly insufficient, but the conversation was around DR.
Lokabf3@reddit
This is definitely not an HA environment. It seems to be active-passive with real-time replication at the data layer.
New_Enthusiasm9053@reddit
The entire DC? At that level it's an anticipated failure if only because of the power seeking missiles known as a construction worker in a digging machine. Some DCs have had duplicated power supplies, one was undergoing maintenance the other was cut by construction workers. Another had two and both of them merged at some point upstream and that failed and it went down.
Lokabf3@reddit
No, I’m speaking about the impacted system, which I understand to be a mainframe environment. I have no idea about other systems in that datacenter, of which there are probably hundreds.
It’s not an all or nothing issue. You can have active active systems running beside active passive and other systems with no redundancy.
New_Enthusiasm9053@reddit
I mean that's fair but that's an argument against mainframes if anything, I also suspect that IBM has a solution for HA mainframes.
Either way any redundant setup ideally switches frequently purely to check if it works correctly. Much easier to deal with an outage when it's during the working day and is being closely monitored. They'd still need DR but at that kind of scale they shouldn't be on a shoestring budget.
Lokabf3@reddit
Mainframes can absolutely be fully redundant and active active , but it’s very expensive. Likely a design decision by FIS.
New_Enthusiasm9053@reddit
I fully agree with you, I guess my beef is why a multibillion dollar in profit company can't afford it, or more precisely chooses to not pay for it. At that scale it's still a negligible cost.
Lokabf3@reddit
Maybe that’s why they are a multibillion dollar in profit company :).
Reverent@reddit
Haha.
That's an interesting way to phrase "large organisations are so disorganised that a coherent DR plan is anything more than a piece of paper that says "prepare 3 envelopes".
ewokninja123@reddit
An efficient DR is to just declare bankruptcy and start a new company, especially at that size.
Sengfeng@reddit
The bank I just left had a horrendous DR plan. Replication to the cloud for critical infrastructure bits, but Fiserv had requirements like having a dedicated router connecting a Fiserv-only subnet to our datacenters. Absolutely zero guidance from them on how to do that for a SDDC without taking the actual bank systems offline.
mythlabb@reddit
FIS would rather miss every contractual SLA they have than fail over to a backup datacenter. I have to assume there's a reason for that.
InviteScared2327@reddit
A few reasons:
1. Their customers take the bulk of the reputational hit.
2. The contracts I've seen for FIS give them significant time to restore service, and minimal to no payout if they miss their SLA or RTO.
3. Don't want to risk losing data or transactions that were not replicated prior to the disruptive event.
HeliosTrick@reddit
I don't blame the admins, but why did a power outage cause a hardware failure? Critical systems should be backed by tested and monitored battery systems which are then backed up by tested and monitored generators. I feel like this is pretty basic stuff, and if my smallish company can figure it out, why can't a company with a budget way higher than mine do the same?
I also feel by not doing shut down testing of hardware annually they've failed a second time, a real DR Test should include all kinds of nonsense since you never know when the harsh hand of reality will slap you in the face.
Given how hard our clients ride us for extreme DR testing, I feel these upper echelon(in revenue and employee count if not in talent) should be doing a better job here.
work-acct-001@reddit
so that little bit of "hmmmm, interesting timing to happen on a common pay day" in the back of my head can quiet down, at least a little?
Lokabf3@reddit
100%. Hardware failures require parts to repair. Parts take time to source and ship, and then install. After installation, systems need to be brought back online, and then validated to ensure data is good, and system are operating normally.
Only then can you resume processing and start to catch up. This all takes time... and on systems as big as this one, it takes a lot of time.
rhavenn@reddit
A service of this level shouldn’t be taken out by a hardware failure. You should be running hot in multiple data centers.
The real problem is a lot of financial shit is controlled by a looot of red tape and 50 year old cobol code at the root that no one wants to touch.
work-acct-001@reddit
I agree with everything you said, and still can't help but wonder a little that a bad actor would know these things.
Time will tell, and if Capital One has had billions to spend on advertising over the years they'll have more than enough to cover anything they want.
rhavenn@reddit
A service at this level should be running hot / hot data centers. There is no way they don’t have the money for it.
A LB DNS entry that just drops their down data center for the front end and some config changes “under-the-good” to disable the backend connections.
randomlyme@reddit
In Fintech : couldn’t agree more
MBILC@reddit
What happened with redundancy and not putting all your egg's in one basket.....
edwardcactus@reddit
Lol I dodged a bullet I turned down an offer to work there
diabillic@reddit
Fiserv isn't much better unfortunately
Just_Curious_Dude@reddit
They own FIS now :)
diabillic@reddit
I don't believe that is accurate. You might be thinking of First Data, which Fiserv acquired a few years ago.
Just_Curious_Dude@reddit
Just checked, you're right. It was only a couple of pieces. They still have the 3 cores under them, my fault and good catch!
diabillic@reddit
no worries, i was almost second guessing myself too.
Just_Curious_Dude@reddit
And FIS, well parts of it anyways. I don't think FIS still holds Mercury, Miser or Systemactics under their umbrella anymore but maybe they do.
I also don't think that Capital One uses them for a core, but for payments processing. FIS total core clients is about 120 among their 3.
michaelpaoli@reddit
FIS Wins 2024 Banking Tech Awards for Excellence in Payment Solutions
Uhm ...
PCRefurbrAbq@reddit
That was 2024. I doubt they'll get it this year.
ewokninja123@reddit
Unlikely
MsAnthr0pe@reddit
They musta awarded it to themselves because from my POV, they suuuuuuuuck.
michaelisnotginger@reddit
Having been acquired by FIS, can't think of a poorer run company
BoredTechyGuy@reddit
Can confirm - I work for a larger financial institution and FIS has been causing us all kinds of grief as well.
caa_admin@reddit
Thanks. OP's rant in the middle with their speculation was cringy.
Akraz@reddit
Yeah let's see if /u/sysad82 swallows his pride and edit's out his rant in his OP. He basically shat all over CO and its not even their fault.
caa_admin@reddit
Well, it's a professional sub edit: rules but way too much stuff in here gets through. :/
foalainc@reddit
exactly!! i was gonna say this started out as factual and ended up with a lot of speculation
RegistryRat@reddit
Work in finance sector, this is correct. I started reading OP's post and connected the dots.
parkersquared@reddit
We were down due to it as well. I thought the same thing about the obvious risk.
thvnderfvck@reddit
Wait is this what you're talking about?
PMmeyourITspend@reddit
I understand you're frustrated your direct deposit was delayed by 24 hours, but you have no idea what you're talking about. They don't rely on h1b's- they hire and train domestically and literally will take anyone with a college degree and pay them to learn to be a software engineer for 2 years under their CODA program. They have invested more in technology than other financial services company by leaps and bounds.
GhostDan@reddit
They just recently went from fully outsourced to attempting to hire locally.
jonboy345@reddit
Usually how it goes. CIO says outsource everything, look how much money we'll save... Does it. Saves a bunch of money, then leaps to a new org conveiniently before the whole thing bursts into flames to do the same thing to the next org. Rine and repeat.
ErikTheEngineer@reddit
I'm actually worried the cycle will stick over on the offshore side this time, given everything moving to the cloud. It's way easier to hand over the keys to Infosys or HCL or Tata when all your stuff isn't sitting in a physical data center anymore and can be managed via a web console or API.
jonboy345@reddit
I was in the Enterprise DC Hardware world until about the middle of year last year, and I was seeing massive customers start to move their workloads back on-prem, most to Colos, but yeah. Enterprises have started to wake up to the realities of cloud spend and moving back to owning their own systems again.
Cloud absolutely makes it easier to offshore and keep it there, but I think we'll see a not insignificant uptick in cloud "repatriation".
andy1307@reddit
If this was true, AWS/GCP/Azure revenues would be down.. but they’re not
jonboy345@reddit
Their pricing isn't the same this year as it was 2 years ago. Increasing revenue doesn't mean they're not losing workload to on-prem.
_haha_oh_wow_@reddit
Classic C-suite bullshit.
GhostDan@reddit
Yeah I've seen this cycle (and been part of it) for a few companies now. I love when a few years down they reach out "hey we are rebuilding IT and need you"...
Sorry I've moved on to someplace that valued me.
Pattyis420@reddit
I know nothing about h1b’s I just know I get paid on Wednesdays at 5:15 and my car insurance wants their money
JoshsTesla@reddit
As do everyone else! It’s fucked that us normies rely on banks to do one thing and that’s to ensure our paychecks are deposited on time because bills don’t care whether you have the funds or not, they are due when they are due. Thankfully I don’t use Capital One for banking but I feel for those who do.
kelleycfc@reddit
Capital One has actually been trying to hire people locally for technology roles. They struggle because they are not “cool”. I know some people that work there and they generally like it and say the org does a good job of taking care of them.
occasional_cynic@reddit
Check out their Glassdoor reviews. It is good they are trying to hire local candidates, but their culture sounds heavy on burn/churn.
PMmeyourITspend@reddit
I know a ton of people that work there because they are a large employer in my city and the overwhelming majority of people who are let go are ones bragging about only working 15 hours a week or who when given any autonomy, just stop working.
Dexanth@reddit
I can't speak for all of the 15 hour crew but I was one of them in a gig I held for quite a while - cause I could finish everything in 15 hours or less. And management was not capable of coming up with new things they wanted me to do fast enough, because they were spending 100% of their time meeting with other managers and basically 0 time actually talking with, working with, and investing in their hands-on-keyboard.
There was one notable exception, and that manager was backstabbed and demoted instead of promoted, which is what should have happened.
Anyways I loved Work-From-Home for this reason, I got back so much of my life. One size does not fit all, and I would happily work 40 hours in a week if it were possible for corporate to give me 40 hours of interesting work to do. New upcoming job may well do that, we will see, but my guess right now is I will still have 10-15 hours of empty time, minimum, each week - and the only way I could see to possibly change that is scheduling more meetings.
feathertheclutch@reddit
RVA?
jimicus@reddit
I've seen organisations like that before. Hiring a bunch of staff that behave like children tends to breed managers who treat everyone like children - and a culture that any self-respecting adult will nope out at the earliest opportunity.
RevLoveJoy@reddit
It's this. When you step in the door and find a circus, the reasonable among us ask ourselves "how long do I want to be a clown?" and plan accordingly.
beagle_bathouse@reddit
Those reviews might be the banking side, tech at banks (except SOC and helpdesk/support) can usually be kinda low key.
ErikTheEngineer@reddit
Lots of industries have this problem. My uncool industry has mostly been transportation for my whole career. All the hotshot CS grads want to play with phone apps and containers and microservices all day, so they turn up their noses at mainframes/client-server/established languages/dev patterns. This leads to a self-fulfilling prophecy where the CIO wonders aloud, "Gee, I wonder why nO oNe WaNtS tO wOrK aNyMoRe" -- and immediately an Infosys or Tata sales team appears with half the population of India in tow. CIO signs contract, CIO gets 8-figure bonus, operations falls apart just as he's pulling the ejection handle and moving onto the next ~~victim~~ workplace.
akp55@reddit
They also apparently suck to work for. Stack ranking and a lot of backstabbing. But I guess that's what happens when you have stack ranking
ErikTheEngineer@reddit
Correct. We have a lot of Big Tech refugees where I work. Everywhere, but especially Amazon, loves stack ranking. Every 6 months, the bottom 20% of a team is put on a PIP (basically fired, nobody recovers from a PIP) and the manager has zero choice in the matter...except for choosing which of his team members to sacrifice. In an environment like that, sabotage and backstabbing is how you survive.
fl3x0@reddit
I almost applied to work at CapOne but the folks I knew who worked or had worked there all advised against it. I've heard the hiring process is heavily test-based and that it inadvertently prevents good talent from getting through.
sysad82@reddit (OP)
Good to know! Maybe I'm just bitter because I'm effected by this and missing out of my money. I'm lucky I have a savings but I run thin in my liquid assets and don't want to sell off investments to make upcoming bills if they don't figure this out soon.
lpmiller@reddit
my pay goes to one bank, any bonuses/gift money to a High interest rate savings account at another bank, that I also set up a checking account with. I can lose access to one bank and still have funds to pay bills in the short term. Also, it strangely makes me better with money, but I'm not sure why. Maybe the fact that ADHD makes me bad with money is benefited by ADHD forgetting all the time that I have a savings account.
jeezarchristron@reddit
I split mine to avoid issues like this. What sucks is both banks are having the same issue. Not sure how. My Cadence bank simply states "outside vendor issues". Guess it's time to split it three ways?
parkersquared@reddit
They both use FIS which was the cause of the outage.
joshbudde@reddit
Your employer could be using them behind the scenes--if they are, their ability to use AHCI to send money to your banks could/is impacted as well.
jeezarchristron@reddit
We do not use Capitol one and most were paid today.
hells_cowbells@reddit
I split up my stuff to avoid issues. I also use Cadence. I got the email from Cadence yesterday, and then got the email from Capitol One a couple of hours after that. Just great. Maybe it's time to just keep everything in cash under my mattress.
skipITjob@reddit
It is affecting you.
hefightsfortheusers@reddit
Its efecting me to
skipITjob@reddit
To be fair, affect and effect are often confused.
the_federation@reddit
They're affectively the same
Jose_Canseco_Jr@reddit
but not affectedly
hefightsfortheusers@reddit
Lol. I know. Thats why you got my upvote.
RegistryRat@reddit
Yeah, this isn't on Capital One. FIS is still recovering from their outage and catching up on processing.
Stonewalled9999@reddit
virtue signaling they will hire an engineer and below market wages, give them 3 heart attacks and them offshore 3 engineers to one HCL dude that doesn't speak english so they can say "see, we tried with local IT but it failed"
farva_06@reddit
I used to work for a company that shared a building with a Capital One call center. Their office was on the way to the food court, and it was fully windowed. They had ping pong, pool, and fuse ball tables, with arcade games and all sorts of other fun shit. Always got jealous walking by there, but then I remembered that their actual jobs are probably miserable as fuck.
Tymanthius@reddit
Call center is not the same as IT.
farva_06@reddit
Right, I was just trying to point out that they seem to have a pretty chill work culture all around.
Goetia-@reddit
It's a mixed bag over at Capital One, and as always the team you're on is the biggest variable. But generally, you can expect a competitive and high pressure environment due to mandatory stack ranking which results in PIPs for the lowest performer(s) on each team (note I said lowest, not low; it's all relative to the overall strength of your team).
Healthy-Poetry6415@reddit
Their AI resume shredder is the issue. I was gonna offer my services. But my resume went in and an auto response hit my mailbox 5 seconds after saying they rejected me " after our agents have reviewed your resume".
So. Fuck em. Have that AI fix the shit mess boys
jimicus@reddit
AI's just the latest buzzword; for years HR departments acted as a sort-of human-driven Google - searching for keywords and assuming that any CV that was missing those keywords was irrelevant.
This was replaced with software that did the exact same thing some years ago (and I'm absolutely 100% certain that more than one company has been absolutely strangled by their own inability to hire qualified candidates - while their HR department was merrily throwing away qualified applications for years on end).
If anything, AI might be an improvement. You might not say "CentOS" in your CV, but if you say "Red Hat" and "Linux" often enough, the AI might decide that you are a relevant candidate.
That is, unless HR are manually reviewing them and feeding back to the AI that you aren't.
Jose_Canseco_Jr@reddit
no it must be like OP said, dang foreigners taking our jerbs
Khue@reddit
Is "not cool" code for "not offering competitive pay and benefits"?
panopticon31@reddit
I live near CapOnes HQ.
The vibe here is very different. They are viewed as a meat grinder that uses and discards you.
Dal90@reddit
And they're acquiring Discover which is one of the smoothest running financial companies I've dealt with :(
RedDidItAndYouKnowIt@reddit
Hold the front door! Discover is selling itself directly to a banking entity?
papers_@reddit
The news first broke in February 2024 if memory is correct.
https://www.capitalonediscover.com
RedDidItAndYouKnowIt@reddit
Well it seems I missed that in the news cycle. Ty.
sysad82@reddit (OP)
Ugh. So many of America's issues could be fixed with strong anti-trust laws to break up large companies and being far more limiting on who we allow to merge with who.
xpxp2002@reddit
Yep. I thought that already happened a year or two ago, though.
The moment I heard that I knew Discover would probably start circling the drain. They must not have done much integration-wise yet because Discover appears to be unaffected.
papers_@reddit
They can't because legally they are two separate companies. The shareholders still have to vote and the regulatory bodies still have to approve the deal.
Until those happen (and pass), it's BAU for them as if it's not happening.
woojo1984@reddit
Really? Fuck I actually like discover!
drawnbutter@reddit
Be careful about using your card. I haven't been late on any payments on anything in at least 15 years, but 6 months after I paid off my Discover card they cancelled it for not being used. The email they sent even said it was without recourse! I haven't been late or missed a payment on anything since I was in college 30 years ago.
slick8086@reddit
Sucks for admins who work for companies that are getting caught fucking over their customers.
phillymjs@reddit
I'm sure the new administration will put the immediate kibosh on that case, but Capital One definitely did that shit. I had been an ING Direct customer and had one of the rebranded 360 Savings accounts. I only found out about the Performance Savings account with the higher interest rate by accident, I think from a post on here, in early December of 2019. As soon as I did, I instantly opened one and moved all my money over.
StellarJayZ@reddit
Looking glass says it's passing traffic.
mini4x@reddit
To be fair, as a Capital One customer, at least I was told there would be an outage.
illicITparameters@reddit
As a Capital One customer… I’m so glad I got paid on the 15th. Not even for the money, but the fucking anxiety of it.
I truly feel gutted for people who truly live paycheck to paycheck who can’t access their money that they earned.
The email I got from them basically stated it’s an issue with one of their clearing houses.
theamazingjizz@reddit
Poor bastards. Been there, I have the notable distinction of having to do an overnight at every company I worked at for more than a few years because of this kind of shit. God speed tech brothers.
VulturE@reddit
Find a local gov't job that gets federal/state money geared towards keeping things up to date. Like transit.
Stable work environment, money always available for projects.
traydee09@reddit
The trick is finding a an org that is interested in doing IT well. I worked for a City government, and thought it would be a great gig. Turned out terrible. The folks there were terrible at IT. They were convinced that DHCP is a massive security risk, so all 1000 endpoints had static addresses. They had over 200 high end Cisco wireless AP's but the guy who was in charge of it was convinced wireless is insecure (that all wireless packets are sent in the clear regardless of encryption) so they'd generally only have a peak of about 40-45 devices on the entire wireless network (a company of around 1100 employees).
They had production commercial databases running on Server 2016 RTM (no patches installed in 4+ years).
The top sys admin broke AzureAD Sync twice because he kept putting in a static address for the Azure Sync server in the hosts file... (he had been in IT for 27 years at that point).
Any computer taken used off the network could never be allowed back on the network because it was deemed "dirty". So all employees HAD to use desktops, but a few high end managers could use laptops that didnt have access to any corp resources, so they had to use Citrix for all work, which was wiped on every disconnect. Using a laptop would take about 15 minutes of setup time every day for the user to get started.
VulturE@reddit
happy cake day.
I remember these kinds of environments.....20 years ago.
Druex_Machina@reddit
I can attest - not a sysadmin, but a developer supporting operations in a transit agency for a large city. In my experience, there are so many opportunities to improve the business, update our tech stack and processes to align with industry best practices, and create system integrations (e.g. HR-driven user provisioning and RBAC).
...the expertise needed to implement these things in-house is sparse, so the Powers That Be resist change due to long-term support concerns.
VulturE@reddit
The more compliant you are for FTA State Of Good Repair, the happier your budget department will be for receiving funds/grants.
In a good transit agency, the desire for more funds drives properly implementing everything.
tiktokarchive_org@reddit
Hi I made an website for the public to archive the best tiktoks - this is your expertise. Can we talk about promoting it?
Immortal_Elder@reddit
It looks like outsourcing and layoffs came back to bite them.
sysad82@reddit (OP)
Sadly there will be no bite. The new administration wants to gut the agencies that could bite them (CFPB for one) and CEOs know this which is why they're all cozying up.
The CEO, the Board, the C-Level will face no consequences at all. They will blame a more mid to senior level manager who may get canned so they feel like they did something and tell their comms department to draft some apology letters but at the end of the day nothing at all will change and the brunt of the poor decisions will be felt by everyday Americans.
IceCubicle99@reddit
There's also a recent lawsuit about alleged mishandling of some savings accounts.
https://www.reddit.com/r/news/comments/1i1ea8l/cfpb_sues_capital_one_for_cheating_customers_out/
wangston_huge@reddit
I got bit by this one... I went to check my interest earnings and noticed that I had only been getting a 0.3% APR for the last 9+ months. Created a 360 performance savings account (what's the difference? Lol) and got back up to 4.5% before the recent rate cut.
Shady as hell.
IceCubicle99@reddit
So did I, but I think I was lucky because I noticed it relatively early. I was on their site looking at other account types (CDs, etc.) and just coincidentally noticed it.
lordkuri@reddit
They're also getting lumped into the Honey lawsuit kerfuffle for their "coupon finder" browser extension shenanigans.
https://hellopartner.com/2025/01/15/influencers-sue-capital-one-shopping-paypal-honey-for-stolen-affiliate-commissions/
narcissisadmin@reddit
The CFPB and SEC overlap and collectively have budgets of over three billion dollars. I can't for the life of me imagine why they're going to be reduced in size. 🙄
arominus@reddit
Different missions really, the SEC is more worried about the stock market, CFPB is explicitly consumer facing.
Can't imagine why the GOP hates it. No sir, can't think of a reason at all.
sysad82@reddit (OP)
Double their budget as far as I'm concerned. $3 billion is nowhere near enough.
tdhuck@reddit
This is a common scenario, unfortunately. You have a good team and everything is running well, for the most part, so the higher ups think they don't need you, you don't do anything.
Then something like this happens and you've gotten rid of the team/talent that was keeping things online.
Also, I 1000% agree that they cut corners in tech and spent more on advertising.
Jkabaseball@reddit
Can't get ACH, but let me guess, processing card transactions is working, maybe even faster than normal to get the money out. Needs to be legal requirements and SLA for this kind of stuff.
Just_Curious_Dude@reddit
There are legal processing windows they have to hit already.
Just_Curious_Dude@reddit
There are, here are the windows
https://www.nacha.org/system/files/2021-03/SDA_Schedules_and_Funds_Availability.pdf
skywalker42@reddit
Wild to assume all that about capital one. I can assure you it’s not true. They have a massive technology campus outside of dc that employs people in the states, invest a ton into building products, and are one of the largest AWS customers in North America. I know many who have worked or work there now and cost cutting has never been an issue within tech.
Happy_Kale888@reddit
Hmm all looks good from here!
https://www.capitalone.com/status/
CharacterRaise5723@reddit
It’s not fixed. I’m a capital one customer. I’ve been on the phone trying to get someone on the line for the past hour.
RegistryRat@reddit
It's their core processor, FIS. Not Capital One's fault, and there's nothing they can do to speed it up.
Just_Curious_Dude@reddit
Technically Fiserv, Fiserv purchased FIS
TheLordB@reddit
Why are you trying to get someone on the phone?
It is a general outage. There isn't anything anyone you talk to is going to be able to do.
CharacterRaise5723@reddit
I’m trying to do something even though I know I’m powerless. Don’t be insensitive
uzlonewolf@reddit
So you know you are pointlessly wasting their time and your own, and are doing it anyway?
CharacterRaise5723@reddit
So you know that asking me this question when I’ve already answered it is profoundly unhelpful and annoying and you’re doing it anyway? I’ve already hung up. Do you want to work at Capital One? Why do you care?
hefightsfortheusers@reddit
Currently affected by this.
But, also used to work in technology at a big Financial Institution.
Its Friday, and they've missed every direct deposit. They are going to be backed up, and to catch up it will take hours.
Depending on the fix, and the original problem, other problems may arise as they move through the backlog. (Often, just overloading a server or vendor)
tl;dr it might be 'fixed', but will still take a good bit of time
sysad82@reddit (OP)
Their phone lines are overwhelmed. After navigating the IVR hell (because companies want to make it next to impossible to actually talk with someone) the line just disconnects.
CharacterRaise5723@reddit
I called the credit card line and they transferred me but yeah idk if I’m ever going to reach someone
_haha_oh_wow_@reddit
Survey says that is incorrect.
CharacterRaise5723@reddit
Guess my empty bank account is lying to me then?
ranhalt@reddit
affecting
CopyEdits@reddit
affecting / affected
SilenceEstAureum@reddit
On one hand I feel the pain and hope their techs get some well deserved down time after this
ON THE OTHER HAND, I use Capital One and my direct deposit was supposed to show up yesterday
RegistryRat@reddit
It's FIS, Capital One's processor who's at fault.
SilenceEstAureum@reddit
I'm sure that distinction matters a lot to the people who bank with Capital One lol
Pyrostasis@reddit
So you've joined the mob with the torch and pitch fork, but you are also winking at the sysadmin and bringing him coffee.
SilenceEstAureum@reddit
Lmao I know some are like that but I'm more inclined to just sit in my office and seethe and get out of the way of the people actually trying to fix things.
At least that's what I tell people at my work to do when shit breaks. You wanna complain, go somewhere else and stay out of my way while I fix this fuck up
IamHydrogenMike@reddit
Workers shouldn't shit on other workers for something that might not be their fault...
north7@reddit
Same here, but I split my DD so most of my paycheck goes to my main account for bills and only a couple hundred to Cap1 for play money.
Cap1 usually hits Wed and it's still not here...
concentus@reddit
Same here on all counts - just checked, and my direct deposit that goes to capital one is nowhere to be found. Thankfully its dedicated savings and not the portion I use to pay bills, but its still annoying.
buy-american-you-fuk@reddit
What's in your wallet? nothing...lol
autumngirl11@reddit
Too soon 😂
autumngirl11@reddit
Wasn’t there a guy here a week or so ago posting about having to take their entire data center down due to power issues? Now they’re saying this is due to a power issue…. RIP my dude!
FapNowPayLater@reddit
Hopefully they fumble my brand new car loan.
twisymctwist@reddit
F Capital One. I am a victim of this. The direct deposits are not working, but debits sure as hell are.
RegistryRat@reddit
It's not their fault, it's their core processor.
etzel1200@reddit
Ransomeware? Because that’s a hell of an outage otherwise.
sysad82@reddit (OP)
I read it's related to an unplanned power outage, which honestly is even worse IMO. For a function of your business that's so critical redundancies should be in place and regularly tested.
PMmeyourITspend@reddit
Their entire server ecosystem is based in AWS so why would a power outage effect them?
hefightsfortheusers@reddit
Its a power outage that affected a vendor.
Biggest problem with big corporations is how complex the infrastructure has gotten.
100 different servers, talking to 100 different vendors, none of which does any 1 person understand completely.
heapsp@reddit
I wrote an entire paper on the dangers of silos and complexity 15 years ago while this shit was just starting. I was laughed at because many in the industry thought it was a dumb idea to have too many eggs in one basket.
techsnapp@reddit
Complexity is the enemy.
SilenceEstAureum@reddit
Not to mention, for something as critical as a major financial institution, they should have battery backups the size of small trucks to keep the equipment alive while the building fails over to a generator. I mean hell WE have a similar system to that and I work at a school.
PMmeyourITspend@reddit
They literally do not have a single server on premise. OP is talking out of their ass.
AlphaNathan@reddit
it is not worse
sysad82@reddit (OP)
We take lots of protections against ransomware in many forms. Endpoint protection, network protection, email protection, phishing exercises, applocker, PAM, network segmentation, immutable S3 backups, aggressive patch cadence and a security team that monitors logs 24/7/365.
Even with all of that we all know it's not if but when. It won't ruin or business, but it will disrupt it.
immewnity@reddit
Unexpected power outages should also have a failover plan
ziobrop@reddit
i worked for the retail side of a large investment bank, and there was a planned data center outage. All the servers got failed over to the redundant systems in the other data center.
Turns out the app groups never actually set up some of the applications at the redundant site. so they went down when the datacenter was powered down for the work period.
Pork-S0da@reddit
On a Friday too, when paychecks hit.
ronin_cse@reddit
Yeah I'm a Capital One user and my paycheck isn't in my account. I guess the nice part is it seems like transactions are down too so my scheduled transfers won't happen either ;)
Mr_Compliant@reddit
I really want H1Bs running my transactions. Lol
Sultans-Of-IT@reddit
It's stitched together by a bunch of Indians who don't fucking listen to what anyone says, so now when it breaks down, you have a bunch of Indians all telling each other what to do, but they only do what they want to do.
unethicalposter@reddit
My last experience with H1B dev don't send me bad data my system will crash and I'll blame you. I didn't send you bad data, what did I send that today was bad. Answer: my app crashed on your data so your data was bad.
RevLoveJoy@reddit
tl;dr upstream provider power outage. Oopsie!
porkchameleon@reddit
Unless you know that for a fact and can bring receipts - I highly recommend shutting your pretty little mouth about it, bleeding heart.
ScroogeMcDuckFace2@reddit
didnt know capital one VPs read r/sysadmin, sheesh.
porkchameleon@reddit
I am not associated with Capital One in any shape of form.
Additionally, OP replied to this:
Permalink.
With this:
Permalink.
Which could confirm that they have a grand total of fuck all of insider information on how things are run on the Capital One's end and are admittedly "bitter" because of their own poor money management skills taking out their frustration online by airing unfounded accusations.
iliekplastic@reddit
H1B doesn't necessarily mean subpar.
When you see some kind of major failure you shouldn't jump to blaming workers first, ask yourself why the bosses allow their environment to be messed up in the first place.
TheEndOfEgo@reddit
I love that there's always posts like this in here.
If my org has a meltdown like this y'all will probably never hear about it, but it's nice to know that y'all would feel this way for us!
yewlarson@reddit
You are assuming a lot of things. They have very competent tech team and are generally a good place to work with.
krock31415@reddit
Any details on cause of outage? We taking failed change, something just broke or worse yet a cyber attack?
Lokabf3@reddit
See my comment here
krock31415@reddit
Thanks
ceantuco@reddit
What's in your wallet?
Catfo0od@reddit
I switched to CapitalOne like a month ago 😂 luckily I'm not exactly paycheck to paycheck.
My first thought was "oh my God, I know some executive said 'why are we spending so much on backups?!?' recently"
I feel bad for the admins, feel bad for the helpdesk techs, poor guy's have the worst Friday since Crowdstrike.
Key-Calligrapher-209@reddit
I briefly tried Capital One a few years ago, and gave up on it after I discovered that an ACH transaction from bank to bank took over a week to post after leaving my other account. I see that I was right to give up on them.
CharacterRaise5723@reddit
I don’t understand any of this. I work hard and want my paycheck.
pingmachine@reddit
Ha nothing like a perfectly up to date status page. https://www.capitalone.com/status/
SilenceEstAureum@reddit
And here I thought Google's workspace status page was terrible
Material-Echidna-465@reddit
Tech isn't the only area they're cutting corners in.
https://www.cnbc.com/2025/01/14/cfpb-sues-capital-one-alleges-it-misled-consumers-on-savings-rates.html
cjrecordvt@reddit
On a long weekend with a bank holiday at the end, too.
That_Description4759@reddit
Must be those December patches! 😂
nlaverde11@reddit
Yeah this is affecting my check as well as my wife’s but fortunately we have other accounts and don’t live paycheck to paycheck. I feel really bad for the people who do and can’t pay bills or buy food right now.