Sometimes it will be a SocketException:
Stack trace: at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.ServiceModel.Channels.SocketConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)
And sometimes it is throwing the following CommunicationObjectAbortedException:
Stack trace: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
The behavior is very random. It is going against static calls that generally work.
What error?
If is one of the WSA error (100053 WSAECONNABORTED, 10054 WSAECONNRESETand friends) it means the underlying IP stack has closed the socket. If is an async IO pending abort 995 ERROR_OPERATION_ABORTED it means a thread that posted an async I/O request has exited and the request was aborted by the OS.
The class which implements the interface with the ServiceContract on it is only instantiated once and the methods reused. This causes multithreading exceptions which are not passed back to the client and not logged in the event log.
Related
I have been working through a head-scratcher with WCF. I have a WCF Service (hosted as a Windows Service) that is called by a console application. It works great, but recently we have run into a timeout issue at 10 minutes when we had to run a query against a legacy system.
I use log4net in both the service and the client .exe, and what is strange is that the service actually completes the job and throws no exception. It runs the query (which takes about 12 minutes, and creates a file, and logs success).
The console application, however, logs an exception, claiming that the "host" returned a timeout or an error. I presume it's a timeout and not a buffer issue, because the exception always happens exactly 10 minutes to the second after the call is initially made.
Here is the exception/stack info from my log4net:
2015-07-28 17:35:47,364 [1]
System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:29:59.9843999'. ---> System.IO.IOException: The read operation failed, see inner exception. ---> System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:29:59.9843999'. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.ServiceModel.Channels.SocketConnection.ReadCore(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout, Boolean closing)
--- End of inner exception stack trace ---
at System.ServiceModel.Channels.SocketConnection.ReadCore(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout, Boolean closing)
at System.ServiceModel.Channels.SocketConnection.Read(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
at System.ServiceModel.Channels.DelegatingConnection.Read(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
at System.ServiceModel.Channels.ConnectionStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security.NegotiateStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.NegotiateStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
--- End of inner exception stack trace ---
at System.Net.Security.NegotiateStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.NegotiateStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.ServiceModel.Channels.StreamConnection.Read(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
--- End of inner exception stack trace ---
Server stack trace:
at System.ServiceModel.Channels.StreamConnection.Read(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
at System.ServiceModel.Channels.SessionConnectionReader.Receive(TimeSpan timeout)
at System.ServiceModel.Channels.SynchronizedMessageSource.Receive(TimeSpan timeout)
at System.ServiceModel.Channels.TransportDuplexSessionChannel.Receive(TimeSpan timeout)
at System.ServiceModel.Channels.TransportDuplexSessionChannel.TryReceive(TimeSpan timeout, Message& message)
at System.ServiceModel.Dispatcher.DuplexChannelBinder.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at FileGeneratorConsole.FileGeneratorServiceReference.IFileGeneratorService.GenerateXmlFileFromSql(String sqlServer, String sqlDatabase, String sqlQuery, Int32 commandTimeout, Boolean allowEmptyResult, String outputFileName, String xslFileName, Boolean archiveExistingFile)
at FileGeneratorConsole.Program.Main(String[] args)
Now, I thought I had done my research and figured out what the issue could be. I assumed it was the sendTimeout and receiveTimeout that are part of the binding definition, so I've added that:
<bindings>
<netTcpBinding>
<binding name="NetTcpBinding_IFileGeneratorService"
maxBufferPoolSize="2147483647"
maxBufferSize="2147483647"
maxReceivedMessageSize="2147483647"
receiveTimeout="00:30:00"
closeTimeout="00:30:00"
openTimeout="00:30:00"
sendTimeout="00:30:00">
</binding>
</netTcpBinding>
</bindings>
And, if you notice in the exception above, you'll see that the error says:
Local socket timeout was '00:29:59.9843999'
It used to say 00:09:59.... so it looks like that took.
I also added this to the service config file, just in case the binding needed to be on both sides, but that didn't help.
Lastly, out of desperation, I added a few other things. I tried adding timeouts to the service section:
and I tried adding
to the behavior itself.
I'm at a loss now, it always dies at 10 minutes.
Thanks in advance
It's always simpler than you think.
Sadly, what was missing all this time was the bindingConfiguration reference in the service.endpoint. All this time, the service was running assumedly using the default binding definition. I had a binding defined in my config, but had missed the part where you specify:
I had:
<endpoint address="" binding="netTcpBinding" bindingConfiguration="">
I needed:
<endpoint address="" binding="netTcpBinding" bindingConfiguration="NetTcpBinding_IFileGeneratorService">
Once that was in place, my actual service endpoint was looking at the right binding properties.
I have a WCF service where IGameServices contains:
[ServiceContract]
public interface IGameServices
{
[OperationContract]
DtoReturnedMessage<DtoGame> GetGame(Guid gameId);
}
AdminServices contains:
public class AdminService : IGameServices
{
private readonly BoGame _boGame = new BoGame();
public DtoReturnedMessage<DtoGame> GetGame(Guid gameId)
{
return _boGame.Get(gameId);
}
}
I have that BoGame, which is a class with business logics.
My config is:
<system.serviceModel>
<bindings>
<basicHttpBinding>
<binding name="BasicHttpBinding_IGameServices" sendTimeout="00:05:00" />
</basicHttpBinding>
</bindings>
<client>
<endpoint address="http://localhost:50380/Services/Implementations/AdminService.svc"
binding="basicHttpBinding" bindingConfiguration="BasicHttpBinding_IGameServices"
contract="IGameServices" name="BasicHttpBinding_IGameServices" />
</client>
</system.serviceModel>
When I debug it step-by-step, all the business logics works properly, and throws the exception below when the response should be sent to the client.
An error occurred while receiving the HTTP response to http://localhost:50380/Services/Implementations/AdminService.svc. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details.
Server stack trace:
at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)
at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at IGameServices.GetGame(Guid gameId)
at GameServicesClient.GetGame(Guid gameId)
Inner Exception:
The underlying connection was closed: An unexpected error occurred on a receive.
at System.Net.HttpWebRequest.GetResponse()
at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
Inner Exception:
Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)
Inner Exception:
An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
It is weird because I have a method to save some data into the DB, in the same IGameServices, which saves normally and returns the success message as expected.
Does any one know what that can be?
I solved the problem. The error was too generic, actually. The issue was with build versions of Postsharp not matching.
I found it out by trying to open the services directly on the browser. I removed Postsharp in that project and BANG.
currently I'm using Azure cache in my WCF application, and I got this exception when putting data into cache:
Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode< ERRCA0017 >:SubStatus< ES0006 >:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.)
And here's the inner exception:
System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:00:30'. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.ServiceModel.Channels.SocketConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)
--- End of inner exception stack trace ---
at System.ServiceModel.Channels.SocketConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)
at System.ServiceModel.Channels.SocketConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout, BufferManager bufferManager)
at System.ServiceModel.Channels.BufferedConnection.WriteNow(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout, BufferManager bufferManager)
at System.ServiceModel.Channels.BufferedConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout, BufferManager bufferManager)
at System.ServiceModel.Channels.FramingDuplexSessionChannel.OnSend(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.OutputChannel.Send(Message message, TimeSpan timeout)
at Microsoft.ApplicationServer.Caching.CacheResolverChannel.Send(Message message, TimeSpan timeout)
at Microsoft.ApplicationServer.Caching.WcfClientChannel.SendOnChannel(EndpointID endpoint, TimeSpan& timeout, WaitCallback callback, Object state, Boolean async, IDuplexSessionChannel channel, Message message)
--- End of inner exception stack trace ---
at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody)
at Microsoft.ApplicationServer.Caching.DataCache.ExecuteAPI(RequestBody reqMsg, IMonitoringListener listener)
at Microsoft.ApplicationServer.Caching.DataCache.InternalPut(String key, Object value, DataCacheItemVersion oldVersion, TimeSpan timeout, DataCacheTag[] tags, String region, IMonitoringListener listener)
at Microsoft.ApplicationServer.Caching.DataCache.<>c_DisplayClass19.b_18()
at Microsoft.ApplicationServer.Caching.MonitoringListenerFactory.EmptyListener.Microsoft.ApplicationServer.Caching.IMonitoringListener.Listen[TResult](Func`1 innerDelegate)
at Microsoft.ApplicationServer.Caching.DataCache.Put(String key, Object value)
The problem is this message is very general, doesn't give much details
I already tried to increase requestTimeout in dataCacheClient to 2 minutes. This doesn't solve the problem, so it should not because of connection to cache timeout
I know it's not because of amount of data too big also, because the same amount of data on production server doesn't have problem (this happens on testing server). I connect from local computer to testing database and also don't have problem
Anyone has any ideas how to solve this problem? Thanks
Below is my current configuration for Azure cache
<dataCacheClient name="default" requestTimeout="30000" isCompressionEnabled="true" >
<localCache isEnabled="true" sync="TimeoutBased" objectCount="4194304" ttlValue="80000" />
</dataCacheClient>
Please set ReceiveTimeout Property between 40-50 seconds in your code to avoid socket timeouts.
Have a retry logic in your code to retry the operation if the error suggests so.
And if this issue persists over longer period of time then take a look at the cache instance performance counters to see if you are nearing 100% limit or maxing out capacity in some sense in which case you should reassess your capacity, refer Capacity Planning Considerations for In-Role Cache (Windows Azure Cache)
I need a little help with an Exception being thrown at my WCF service. The point is: I need to develop a WPF application and I've heard that to treat data WCF is the best choice so I used it only with basic knowledge and now I have this problem.
The point is, I have a Customer class, a Supplier class and an Order class. I have then two services: the first one has all methods to add, read and edit customers and suppliers from database. This service works perfectly fine. The second service has methods to add new orders, edit orders and read them from database, and this service has a problem.
The class Customer has a list of Suppliers (the suppliers from which the customer buys), the class Supplier has a list of Customers (the customers that buy from that supplier). The class order has a Customer, a Supplier and a list of Itens (and the itens are added to the database together with the order).
The method for adding an order works fine, however, once an order has been added, the method to list the orders throws the TargetInvocationException. I searched a lot on the internet and the best answer I've found is that it's possibly a "Ciclyc Reference".
The method to list the orders is inside the repository class and is simply:
public IQueryable<Order> ListOrders()
{
return Context.Set<Order>().Include(x => x.Customer).Include(x => x.Supplier);
}
Where Context is the entity framework context. I tried to remove the includes and load the customer and supplier separately, but the result was the same exception.
Can someone please help me ? If I need to post more information just tell me. And sorry if this matter is trivial, I'm new to WPF and WCF so I'm still a little confused.
Thanks a lot in advance.
EDIT Exception below
An error occurred while receiving the HTTP response to
http://SomeService.com/OrdersService.svc. This could be due to the
service endpoint binding not using the HTTP protocol. This could also
be due to an HTTP request context being
aborted by the server (possibly due to the service shutting down). See
server logs for more details.
Server stack trace: at
System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException
webException, HttpWebRequest request, HttpAbortReason abortReason)
at
System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan
timeout) at
System.ServiceModel.Channels.RequestChannel.Request(Message message,
TimeSpan timeout) at
System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message
message, TimeSpan timeout) at
System.ServiceModel.Channels.ServiceChannel.Call(String action,
Boolean oneway, ProxyOperationRuntime operation, Object[] ins,
Object[] outs, TimeSpan timeout) at
System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage
methodCall, ProxyOperationRuntime operation) at
System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage
message)
Exception rethrown at [0]: at
System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage
reqMsg, IMessage retMsg) at
System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&
msgData, Int32 type) at IOrdersService.ReturnOrders() at
OrdersServiceClient.ReturnOrders()
Inner Exception: The underlying connection was closed: An unexpected
error occurred on a receive. at
System.Net.HttpWebRequest.GetResponse() at
System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan
timeout)
Inner Exception: Unable to read data from the transport connection: An
existing connection was forcibly closed by the remote host. at
System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset,
Int32 size) at System.Net.PooledStream.Read(Byte[] buffer, Int32
offset, Int32 size) at
System.Net.Connection.SyncRead(HttpWebRequest request, Boolean
userRetrievedStream, Boolean probeRead)
Inner Exception: An existing connection was forcibly closed by the
remote host at System.Net.Sockets.Socket.Receive(Byte[] buffer,
Int32 offset, Int32 size, SocketFlags socketFlags) at
System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset,
Int32 size)
Mark your data objects with the [DataContract(IsReference = true)] attribute. This should resolve the issue.
I am using MS Dynamics CRM api to get data (more then 5000 records) from it.
I am using XML based query to get data
EntityCollection resultSet = _orgServiceProxy.RetrieveMultiple(new FetchExpression(query));
sometime time it works and most of the time it throws exception, from logs, I am getting this long exception.
An error occurred while receiving the HTTP response to https://orgno.api.crm.dynamics.com/XRMServices/2011/Organization.svc. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details. ---> System.ServiceModel.CommunicationException: An error occurred while receiving the HTTP response to https://orgno.api.crm.dynamics.com/XRMServices/2011/Organization.svc. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size) --- End of inner exception stack trace --- at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size) at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count) at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.TlsStream.Read(Byte[] buffer, Int32 offset, Int32 size) at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size) at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead) --- End of inner exception stack trace --- at System.Net.HttpWebRequest.GetResponse() at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) --- End of inner exception stack trace
I don't know why this is happening?
CRM service doesn't like at all retrieve much data in one request.
I suggested you to use a paging cookie to retrieve your records by sets. It should resolve your issue and improve the performance of your code.
You can find a nice tutorial on msdn here.
Regards,
Kévin