Develop Reference

ASP.Net Core + .NET Framework Web API - frequent network errors

I have an ASP.Net Core 2.2 Web API targeting .NET framework 4.7.2 running under IIS, Out of Process. I am seeing a lot of outgoing network issues. The app connects to various things (SQL databases, SSAS via XMLA etc.) and I am seeing across the board network failures to connect to various resources. At first I thought it was SQL connections only but it appears to be network in general. I get some sort of failure very regularly, around 1 in 10 requests. I have this in web.config:
<system.net>
<connectionManagement>
<add address="*" maxconnection="65535"/>
</connectionManagement>
</system.net>
and this in Startup.cs:
ServicePointManager.DefaultConnectionLimit = int.MaxValue;
However, it's not just HTTP connections that are failing, it's mostly SQL, SSAS, general TCP.
The failures are things like SQL error 26, unable to connect and similar for other non-SQL network resources. It is intermittent, there is hardly any load on the box at all. It seems to happen when API calls are made back to back perhaps.
I don't think it's a general network (router/switch) issue as I can set up, for example, a scripted console SQL connect/select/teardown to one of the remote services showing connect failures from the app and this never fails, even in a repeating loop running at the same time as I see errors in the app connecting to the same DB.
There must be some TCP/network tuning I am missing and would be grateful for any suggestions.

You have a very unusual issue. The reason to use large packets sizes is when you have a very small number of errors the more data you send the more errors you will get. But the error rate should be constant. You will always get errors. But in this case the error are not constant. The only start when data is split into more than one packet which occurs when you send data larger than 1500 bytes.
You probably have a multi-hop connection where the client and server are not directly connected by one cable. You are going through routers and server which each have their own unique IP address (and name).
Normally errors are fixed by replacing cables or pieces of hardware. You can either randomly start replacing hardware or you can use ping from/to different servers and routers to find which work and which gives error helping to isolate issue.
In you case it looks like the issues is one of your devices was not designed correctly. Errors starting at 1500 bytes are usually due to vendor not fully testing devices and not fixing errors found during alpha and beta testing. Most vendors do a very good job of testing and certifying their devices. So this could be a "Made in China" issue. Or maybe even a virus. Still to fix issue start replacing or use ping to isolate which routes work and which give errors.

WCFTransaction Exception

I am currently working on client-service architecture using WCF, c# .NET. I was stuck with well-known transaction exception for this architecture while running client and server on different machines (No problem if both on same machine). By searching on web, I resolved this issue with MSDTC settings as well as Firewall settings as shown on this link: https://www.packtpub.com/books/content/how-configure-msdtc-and-firewall-distributed-wcf-service. I made outbound settings enable on client and inbound settings enabled on server. Also set exception for DTC on client side. Now that problem is partially resolved at the cost of serious issue. When I run the app, sometime it throws exception but sometimes suddenly runs successfully w/o changing anything. I am totally clueless, what is wrong in this case. Any idea?

Dtcping test passes but still have a Communication with the underlying transaction manager has failed

DTCPing tool says everything should be fine. The actual exception is:
System.Transactions.TransactionManagerCommunicationException:
Communication with the underlying transaction manager has failed. --->
System.Runtime.InteropServices.COMException: The MSDTC transaction
manager was unable to push the transaction to the destination
transaction manager due to communication problems. Possible causes
are: a firewall is present and it doesn't have an exception for the
MSDTC process, the two machines cannot find each other by their
NetBIOS names, or the support for network transactions is not enabled
for one of the two transaction managers. (Exception from HRESULT:
0x8004D02A)
The service runs on a VM. Other servers are able to run distributed transactions, so DB box is configured properly.

There is a good article on this subject here: Troubleshooting Problems with MSDTC. It was originally written for BizTalk Server but applies on other server as well.
Since you're running off a VPC, it's possible the DTC shares some identifiers with another machine on the same network. Please check the "Ensure that MSDTC is assigned a unique CID value" chapter in this article (see also http://blogs.msdn.com/b/johnlee/archive/2008/09/09/msdtc-on-virtual-machines.aspx), although I believe DTCPing should tell you about it... You can also try msdtc -uninstall and msdtc -install to fix this.
You should also check if your server are not using some VPN or encryption technology. I kinda remember this can cause issues.

Someone has already mentioned unique IDs, make sure you do that before you do anything else, I've seen people blow up a LOT of time going down other routes only to find out that was all the problem was.
That said, I have found a tool called DTCTester to be more useful than DTCPing. http://support.microsoft.com/kb/293799
It's a little bit harder to use than DTCPing, but it also tests much closer to what you are normally looking to do which gives you a cleaner test result.
* Make sure that all possible firewall/filtering software is disabled and/or non-existant *
I can't stress that enough, I once spent almost 90 minutes trying to convince a cusotmer that his VPN sofware was acting as a personal firewall and blocking incoming sockets. Once he disabled it, the problems went away. (the software incidently was Cisco VPN Client, the setting was "Stateful Firewall (Always On)" - yep, it's ALWAYS on :) )
Remember that DTC is two-way traffic, so this has to be true on both sides, and you also need to ensure both machines have 'Enable Network DTC Access" allowed. I have also found that making sure the DTC settings on both boxes are synchronized generally ensures solid interaction as well.

Why would Fiddler fix my connection issues to my WCF server?

I have a WCF server hosting clientaccesspolicy.xml (for Silverlight clients) and I was testing access to it using Internet Explorer. When I accessed it using http://localhost/clientaccesspolicy.xml everything worked correctly, but when I used http://machinename/clientaccesspolicy.xml it wasn't retrieved.
I downloaded, installed and ran Fiddler and now the http://machinename/clientaccesspolicy.xml URL works correctly - even without Fiddler running.
Obviously this isn't a scalable solution to the problem so I was wondering if anyone know what Fiddler might have done to fix my issue, or what might have been causing my issue in the first place? Windows Firewall has been disabled throughout my testing.

It turns out our testing systems have Privoxy installed on them incorrectly - it successfully blocked the connection attempts because they weren't whitelisted, but didn't display the error message it should be configured to do so. Fiddler replaced the proxy with its own settings temporarily and hence let it work. Thanks (and an upvote) to Ladislav for the hint.

WCF communication between 2 servers crashes after IIS7 process recycle

I am kind of stumped with this one, and was hoping I could find some answers here.
Basically, I have an ASP.NET application that is running across 2 servers. Server A has all of the business logic/data access exposed as web services, and Server B has the website which talks to those services (via WCF, with net.tcp binding).
The problem occurs a few seconds after a recycle of my app pool is initiated by IIS on Server A. The recycle happens after the allotted time (using the default of 29 hours set in IIS).
In the server log (of Server A):
A worker process with process id of
'####' serving application pool
'AppPoolName' has requested a recycle
because the worker process reached its
allowed processing time limit.
I believe that this is normal behavior. The problem is that a few seconds later, I get this exception on Server B:
This channel can no longer be used to
send messages as the output session
was auto-closed due to a
server-initiated shutdown. Either
disable auto-close by setting the
DispatchRuntime.AutomaticInputSessionShutdown
to false, or consider modifying the
shutdown protocol with the remote
server.
This doesn't happen on every recycle; I assume that it happens when someone is hitting the site with a request WHILE the recycle happens.
Furthermore, my application is down until I intervene; this exception continues to occur every time a subsequent request is made to the page. I intervene by editting the web.config (by adding a space or something benign to the end of file) and saving it- I assume that that causes my application to recompile and brings the services back up. I also have experimented with running a batch file that does this for me every time the exception happens ;)
Now, I could barely find any information on this exception, and I've been looking for a while. Most of the information I did find pertains to WCF settings that I am not using.
I already read up on "DispatchRuntime.AutomaticInputSessionShutdown" and I don't think it pertains to this situation. This particular property refers to the service shutting down automatically in response to behavior on the client side, which is not what is happening here. Here, the service is shutdown because of IIS.
I did read this which went through some sort of work around to bring the service back up automatically, but I am really looking to understand what is going on here, not to hack around it!
I have started playing around with the settings in IIS7, specifically turning on/off Overlapped Recycling and increasing the process startup/shutdown times. I am wondering whether it is safe to turn off recycling completely (I believe if I put 0 for the recycling time interval?) But again, I want to know what's going on!
Anyway, if you need more information, let me know. Thanks in advance!

This is probably related to how you open and close WCF connections.
If you open a proxy when your app starts and then continue to use this, a break in the connection, which is caused by a restart on the server side. Results in a error on the client side, since the server that the proxy was talking to is no longer there.
When you restart the client side (changing the web.config) new proxies are created against a server that is running.
The way to fix this is to make sure that you close a WCF connection after you use it.
http://www.codeguru.com/csharp/.net/net_wcf/article.php/c15941/

You should also make sure that you're using the correct SessionMode for your Web Service. I remember having similar trouble with some of my Services until I sorted out the correct mode. This is especially true when you're mixing this with any other authentication mode that is not "None".
This link might have some pointer.
http://msdn.microsoft.com/en-us/library/ms731193.aspx

My suggestion is to simply stop using IIS to host your services. Unless there is something you really need from IIS, I would recommend just writing a standard Windows Service to host your WCF endpoints.
If you can't do that, then by all means turn off recycling. AppPool recycling is mainly there because web developers write crappy code. I know that sounds rather blunt, but if you have enough sense to write code that doesn't leak then there is no reason to have IIS constantly restart your program.