CCH Axcess Document issues with AT&T connection

Posted by Beefcrustycurtains@reddit | sysadmin | View on Reddit | 22 comments

We have been experiencing some issues with CCH Axcess Document and extremely slow opening of documents. This is only occurring on AT&T connections, but it is occurring on 3 different AT&T connections we have tested. All other ISPs seem to be fine. The AT&T connections have 0 packet loss and no issues on anything else. Of course CCH is pushing it off as an AT&T issue and AT&T will push it off as a CCH issue. I used fiddler to see what document was reaching out to when attempting to open the documents and it was [telemetry.cchaxcess.com](https://telemetry.cchaxcess.com) and [z001commonservices.cchaxcess.com](https://z001commonservices.cchaxcess.com) both on the AT&T connection and on another ISP resolve to [107.154.75.47](https://107.154.75.47). When pinging that from an AT&T connection there is anywhere from 10-20% packet loss \~55ms response times. So on that same PC and same connection I connected it to a full tunnel VPN that has another isp and ran the same ping tests and had 0% packet loss with response around 6 ms. Trace routes both end up at the same place, but the hop before the final is a little bit different. On AT&T connection only: 7 4 ms 2 ms 3 ms [gar25.dlstx.ip.att.net](https://gar25.dlstx.ip.att.net) \[[12.122.85.233](https://12.122.85.233)\] 8 \* \* \* Request timed out. 9 4 ms \* \* [dls-b23-link.ip.twelve99.net](https://dls-b23-link.ip.twelve99.net) \[[62.115.113.84](https://62.115.113.84)\] 10 53 ms 53 ms 53 ms [imperva-svc087380-ic377630.ip.twelve99-cust.net](https://imperva-svc087380-ic377630.ip.twelve99-cust.net) \[[62.115.55.35](https://62.115.55.35)\] 11 53 ms 53 ms 53 ms [107.154.75.47.ip.incapdns.net](https://107.154.75.47.ip.incapdns.net) \[[107.154.75.47](https://107.154.75.47)\]  On Full Tunnel VPN with another Provider: 4 6 ms 6 ms 6 ms [vl113.ae1009.dllstxeqnrb02.as7819.net](https://vl113.ae1009.dllstxeqnrb02.as7819.net) \[[67.210.228.10](https://67.210.228.10)\] 5 6 ms 6 ms 6 ms [eqix-da1.imperva.com](https://eqix-da1.imperva.com) \[[206.223.118.231](https://206.223.118.231)\] 6 6 ms 6 ms 6 ms [107.154.75.47.ip.incapdns.net](https://107.154.75.47.ip.incapdns.net) \[[107.154.75.47](https://107.154.75.47)\]  I have requested a supervisor with CCH because they keep telling me its an AT&T issue and AT&T will need to fix. Anyone have any ideas for pushing this through to resolve? I think AT&T will just push it right back to CCH.

Reply to Post

22 Comments

[-]

Bobs_Ur_Uncles@reddit

Any luck on your end? None here so far. CCH said they have had other clients report that AT&T fixed it for them. I working with AT&T tier 3 support and after they looked into it they say I have to contact Impervia.

[-]

Beefcrustycurtains@reddit (OP)

Still no response from Imperva support. I had a sales rep ask me about the trial and I sent him a message with my ticket number stating my issues and he didn't respond. Hopefully they can fix their stuff. Att connections can't talk to the dallas imperva server so they are routing through Sweden instead.

[-]

Bobs_Ur_Uncles@reddit

We are no longer having this issue on our 2 AT&T enterprise fiber circuits. We also have an AT&T small business fiber circuit that still has the issue. AT&T said they have not done anything so I assume Imperva fixed something related peering with the AT&T enterprise service.

[-]

Beefcrustycurtains@reddit (OP)

Still same problem on mine both of mine are the small business fiber circuits.

[-]

Bobs_Ur_Uncles@reddit

That’s unfortunate. I haven’t attempted to contact them yet as we are able to work via the enterprise circuits. Will try contacting them and Impreva.

[-]

Beefcrustycurtains@reddit (OP)

Today is the first day that after 11 am the ping times haven't shot up and we aren't losing packets anymore.

[-]

Bobs_Ur_Uncles@reddit

That’s awesome! Finally! I’ll have to test the and fiber here. Thanks. I’ll let you know what I find out on our side.

[-]

Bobs_Ur_Uncles@reddit

Thanks for updates. Hopefully they can get it resolved quickly.

[-]

Beefcrustycurtains@reddit (OP)

cch supervisor said they have tickets open with both imperva and att.

[-]

Bobs_Ur_Uncles@reddit

I’ve got a ticket open with AT&T also. The CCH tech I spoke with said they’ve had some clients that have gotten AT&T to resolve the issue. I suspect AT@T is routing via a different carrier.

[-]

modell92@reddit

We had a similar issue with CCH Axcess Tax a few years back. Slow download speeds of the DCS cache when opening a return. It looked like a routing issue and ended up being our DNS Filtering with SafeDNS sending connections to *.cchaxcess.com through a weird series of hops. We ended up whitelisting *.cchaxcess.com to not route through SafeDNS and the download speeds drastically improved along with the performance on the download related tasks of the Axcess suite. Do you use any DNS Filtering that you can whitelist the Axcess domains and see if that helps?

[-]

Beefcrustycurtains@reddit (OP)

No, and this is also occuring on multiple AT&T connections in the DFW area including home connections.

[-]

oortad@reddit

Not sure if it's specific to Texas, but I work for a West Texas MSP and one of our customers is also having this issue. Traffic is taking a path through ATT's Dallas backbone, after which it starts dropping packets. Our band-aid fix is routing traffic to 107.154.75.47 through their backup WAN connection instead of through their primary ATT WAN.

[-]

Beefcrustycurtains@reddit (OP)

Ya, this customer declined a second WAN connection when we suggested it when setting up their office. Now they understand and we have ordered one but won't complete until after the 18th which is a little late.

[-]

Bobs_Ur_Uncles@reddit

As of yesterday and for probably the last week I have a client experiencing the same issue & CCH support also stated it was an AT&T issue which all my client sites are using. Moved one site to backup non-AT&T provider & issue was resolved at that site. I am not sure if they are having the issue today but my tracert to the servers you mentioned had no packet loss and 1-9ms response on all hops. Have you made any progress in getting this issue resolved. Thanks

[-]

Beefcrustycurtains@reddit (OP)

I just submitted this ticket with their trial support: Imperva Support, We are trying to demo some of your offerings and I’m noticing that AT&T internet connections start getting packet loss and higher than normal ping times going to your service. It seems to be getting worse throughout the day. It started out this morning with 2-6 MS response times on ping test and has creeped up throughout the day to \~55 MS with terrible packet loss. I have tested this on 4 different AT&T internet connections in the Dallas area and all of them have the same problem. We noticed the same problem yesterday as well it eventually creeped up to 15-20% packet loss yesterday. ping 7hzrru9.impervadns.net -n 150 Ping statistics for 45.223.57.67: Packets: Sent = 150, Received = 141, Lost = 9 (6% loss), Approximate round trip times in milli-seconds: Minimum = 48ms, Maximum = 60ms, Average = 53ms Also occurs on this one as well that CCH Axcess uses: m4rc6r6.impervadns.net Ping statistics for 107.154.75.47: Packets: Sent = 150, Received = 140, Lost = 10 (6% loss), Approximate round trip times in milli-seconds: Minimum = 50ms, Maximum = 58ms, Average = 53ms Right now it’s sitting around 6% packet loss but it will get worse throughout the day. Can you please investigate? We can’t possibly use this product if it’s going to have issues with one of the largest ISPs in the nation.

[-]

Beefcrustycurtains@reddit (OP)

So the issue is definitely with their ImpervaDNS provider. I have signed up for a free trial and added a website to it so I could get a cname to test with. My test cname of [7hzrru9.impervadns.net](https://7hzrru9.impervadns.net) started off with 2-6 ms of ping time on AT&T connections and the same on others, then throughout this morning it has creeped up with the same [telemetry.cchaxcess.com](https://telemetry.cchaxcess.com). I'm expecting them both to start dropping packets, and when they do i'm going to open a ticket with impervadns through my free trial to have them investigate.

[-]

CPAtech@reddit

Sounds potentially like a backbone issue, but there would be others affected. We've had similar problems when Layer 3 was having issues in the US and were causing latency to one of our cloud applications, but neither us nor the cloud provider used Layer 3. All we could do was put pressure on the cloud provider to put pressure on the ISP. Those are not fun.

[-]

Beefcrustycurtains@reddit (OP)

Thanks man. Ya, I'm straight up not having a good time with it lol.

[-]

CPAtech@reddit

Have you tried pingplotter? That may do the same as fiddler, but if you can at least show problems on one route then you can provide that to the powers that be.

[-]

Beefcrustycurtains@reddit (OP)

I haven't fiddler was just to see what it was reaching out to. Pingplotter provide a lot more detail than tracert?

[-]

CPAtech@reddit

Yes, pingplotter will show latency at each hop.