It's always DNS
Posted by KingofGamesYami@reddit | talesfromtechsupport | View on Reddit | 8 comments
Some background context is needed before I begin.
The company I work for has decided, in their infinite wisdom, to split into two companies. I work on a team developing and maintaining custom internal apps which are deployed to azure, aws, gcp, and our own data centers.
As part of this move, several apps I support must be moved from our current azure tenant to a new azure tenant, which affects both hosting & entra authentication.
Now, onto the story:
We've been having a... fun and exciting time moving applications for the past 2 months. By fun and exciting, I mean submitting a lot of paperwork about how long things will take, who is going to do them, and so on. I have fielded several complaints about timelines I submitted weeks ago being invalid because by the time someone reviewed the paperwork, my timeline had us deploying the app -- and obviously no work has started yet, since the paperwork hasn't been approved!
Today, however, is different. Today I have permission to deploy. The infrastructure requests I can't handle myself have been completed. In theory, everything can work.
Everything starts out smoothly. I'm able to deploy my resources, replicate the database, and move the source code over. A slight hiccup occurs with npm package locks and custom registry auth, but nothing I can't handle with some effort.
I deploy a fresh build of the application to the new environment and... it works! I'm able to log in, get to the home page, even navigate and load some data. This is great. I'm finally going to get things done and my managers' manager will stop pestering me with pointless daily updates.
Then one page fails to load. Alright, no need to panic. This is why we have application insights. I'll just check the request logs, and... what? The logs aren't there. I double & triple check the config. The connection string is correct.
Now I'm more than a little annoyed. Observability is how we find issues, without it, we're basically flying blind. I log into KUDU and start checking things. After nearly a full day of banging my head against the wall, I recall our app service is vnet integrated, and as such has some special™ DNS behavior so it can resolve internal URLs. I run `nameresolver` on the application insights ingest URL, and... it spits out a couple aliases to azure private link and no IP address.
Now *that* is interesting. Our app does not utilize private link at all, it only uses VNET to talk to resources deployed to our on-prem datacenters. I raise this issue with our architecture team, and it turns out this is a known issue, which is actively being worked on. Excellent.
Next time I'll check DNS first.
popcornrocks19@reddit
I may work for the same company, just not in the same role. Does this company happen to have a promise that's purple?
Evlavios@reddit
My favorite haiku:
It's not DNS. There's no way it's DNS. It was DNS.
Ek4m-inhale@reddit
It’s not DNS. It couldn’t be DNS. Ah hell, it was DNS.
tenkadaiichi@reddit
Yep, this is why whenever app people say that the connection isn't working, I ask them for souce and destination IP address, and then confirm with pings or some other lookup method that what they are connecting to is actually what they think they are connecting to.
About half the time they're actually trying to connect to an IP address that they've never heard of. It's maddening.
Creative-Letter-4902@reddit
Ah yes, the classic "DNS but also not DNS because of weird VNET integration" special. Been there. The nameresolver trick is gold but it sucks you had to burn a day to get there.
Architecture team sitting on a known issue while you're in the trenches? Yeah, sounds about right.
If you ever need someone to help untangle these deployment messes so you're not the one banging your head for a whole day, I got 2-3 hours free. DM me. Otherwise, good luck with the rest of the split. You'll need it.
HecateRaven@reddit
What is vnet?
lordjippy@reddit
Azure virtual network. Equivalent to AWS VPC.
skiing123@reddit
I mean there is a reason why a shirt exists for this reason