I'm taking a shot in the dark with this one but I've been experiencing an issue that I'm hoping someone might have come across before...My client makes a server call which pretty much just grabs data from a Sql server DB and returns it to my client however sometimes the client never receives a response from the Server. I put a bunch of logging in and the very last line of server side code is executed but still, the client never receives the response and as a result gets stuck "PROCESSING"... I added a WCF trace and monitored it using the Service Trace Viewer.
And I'm seeing exceptions in the Activity logs that contain the following text:
<E2ETraceEvent xmlns="http://schemas.microsoft.com/2004/06/E2ETraceEvent">
<System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system">
<EventID>131075</EventID>
<Type>3</Type>
<SubType Name="Error">0</SubType>
<Level>2</Level>
<TimeCreated SystemTime="2013-06-11T19:38:52.8094768Z" />
<Source Name="System.ServiceModel" />
<Correlation ActivityID="{a3b3598e-8ce1-4afc-8cf2-89c6fa505061}" />
<Execution ProcessName="TwoFour.WCF.App" ProcessID="5056" ThreadID="111" />
<Channel />
<Computer>VA3WEB-REMEDY01</Computer>
</System>
<ApplicationData>
<TraceData>
<DataItem>
<TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Error">
<TraceIdentifier>http://msdn.microsoft.com/en-US/library/System.ServiceModel.Diagnostics.ThrowingException.aspx</TraceIdentifier>
<Description>Throwing an exception.</Description>
<AppDomain>TwoFour.WCF.App.exe</AppDomain>
<Exception>
<ExceptionType>System.ServiceModel.CommunicationException, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
<Message>The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '10675199.02:48:05.4775807'.</Message>
<StackTrace>
at System.ServiceModel.Channels.SocketConnection.EndRead()
at System.ServiceModel.Channels.TracingConnection.EndRead()
at System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(Object state)
at System.ServiceModel.Channels.TracingConnection.TracingConnectionState.ExecuteCallback()
at System.ServiceModel.Channels.SocketConnection.AsyncReadCallback(Boolean haveResult, Int32 error, Int32 bytesRead)
at System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
</StackTrace>
<ExceptionString>System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '10675199.02:48:05.4775807'. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
--- End of inner exception stack trace ---</ExceptionString>
<InnerException>
<ExceptionType>System.Net.Sockets.SocketException, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
<Message>An existing connection was forcibly closed by the remote host</Message>
<StackTrace>
at System.ServiceModel.Channels.SocketConnection.EndRead()
at System.ServiceModel.Channels.TracingConnection.EndRead()
at System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(Object state)
at System.ServiceModel.Channels.TracingConnection.TracingConnectionState.ExecuteCallback()
at System.ServiceModel.Channels.SocketConnection.AsyncReadCallback(Boolean haveResult, Int32 error, Int32 bytesRead)
at System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
</StackTrace>
<ExceptionString>System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host</ExceptionString>
<NativeErrorCode>2746</NativeErrorCode>
</InnerException>
</Exception>
</TraceRecord>
</DataItem>
</TraceData>
</ApplicationData>
I don't suppose anyone has seen something like this before and can give me some recommendations as to how I can go about fixing it? Any information would be greatly appreciated.
Thanks!
A CommunicationException with an error message like "The socket connection was aborted. This could be..." is just a general failure condition and is the side-effect of the actual error.
To find out what the actual error is, use your WCF trace output. Select the items marked as Exceptions and look at the details pane for those. One of them will most likely give you more explicit information as to the error - which really could be anything from what you've provided so far.
Establish tracing on both the client and server, and when you see an exception in the trace window, highlight it to show more details in the right-hand-side panes.
Related
I have added system.diagnostics wcf tracing to my window service that hosts couple of wcf end points.
Because this is a production service i have reduce the 'switchValue' to "Error, Critical, Warning" in order to avoid huge trace files. The problem is that now when I get an exception in the trace there is no reference as to which end point had raised the exception.
Example for such an entry in the trace file bellow.
Thanks
<System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system">
<EventID>131075</EventID>
<Type>3</Type>
<SubType Name="Error">0</SubType>
<Level>2</Level>
<TimeCreated SystemTime="2020-07-04T12:21:06.1749193Z" />
<Source Name="System.ServiceModel" />
<Correlation ActivityID="{00000000-0000-0000-0000-000000000000}" />
<Execution ProcessName="Adapter" ProcessID="3440" ThreadID="107" />
<Channel />
<Computer>XX-01</Computer>
</System>
<ApplicationData>
<TraceData>
<DataItem>
<TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Error">
<TraceIdentifier>http://msdn.microsoft.com/en-US/library/System.ServiceModel.Diagnostics.ThrowingException.aspx</TraceIdentifier>
<Description>Throwing an exception.</Description>
<AppDomain>Casino3rdPartyAdapter.exe</AppDomain>
<Exception>
<ExceptionType>System.ServiceModel.ProtocolException, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
<Message>The maximum message size quota for incoming messages (65536) has been exceeded. To increase the quota, use the MaxReceivedMessageSize property on the appropriate binding element.</Message>
<StackTrace>
at System.ServiceModel.Channels.HttpInput.ThrowHttpProtocolException(String message, HttpStatusCode statusCode, String statusDescription)
at System.ServiceModel.Channels.HttpInput.ThrowMaxReceivedMessageSizeExceeded()
at System.ServiceModel.Channels.HttpInput.GetMessageBuffer()
at System.ServiceModel.Channels.HttpInput.ParseMessageAsyncResult.DecodeBufferedMessageAsync()
at System.ServiceModel.Channels.HttpInput.ParseMessageAsyncResult.BeginParse()
at System.ServiceModel.Channels.HttpInput.BeginParseIncomingMessage(HttpRequestMessage httpRequestMessage, AsyncCallback callback, Object state)
at System.ServiceModel.Channels.HttpPipeline.EmptyHttpPipeline.BeginParseIncomingMessage(AsyncCallback asynCallback, Object state)
at System.ServiceModel.Channels.HttpPipeline.EnqueueMessageAsyncResult..ctor(ReplyChannelAcceptor acceptor, Action dequeuedCallback, HttpPipeline pipeline, AsyncCallback callback, Object state)
at System.ServiceModel.Channels.HttpPipeline.EmptyHttpPipeline.BeginProcessInboundRequest(ReplyChannelAcceptor replyChannelAcceptor, Action dequeuedCallback, AsyncCallback callback, Object state)
at System.ServiceModel.Channels.HttpChannelListener`1.HttpContextReceivedAsyncResult`1.ProcessHttpContextAsync()
at System.ServiceModel.Channels.HttpChannelListener`1.BeginHttpContextReceived(HttpRequestContext context, Action acceptorCallback, AsyncCallback callback, Object state)
at System.ServiceModel.Channels.SharedHttpTransportManager.EnqueueContext(IAsyncResult listenerContextResult)
at System.ServiceModel.Channels.SharedHttpTransportManager.OnGetContextCore(IAsyncResult listenerContextResult)
at System.ServiceModel.Channels.SharedHttpTransportManager.OnGetContext(IAsyncResult result)
at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.ListenerAsyncResult.IOCompleted(ListenerAsyncResult asyncResult, UInt32 errorCode, UInt32 numBytes)
at System.Net.ListenerAsyncResult.WaitCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
</StackTrace>
<ExceptionString>System.ServiceModel.ProtocolException: The maximum message size quota for incoming messages (65536) has been exceeded. To increase the quota, use the MaxReceivedMessageSize property on the appropriate binding element.</ExceptionString>
<DataItems>
<Data>
<Key>System.ServiceModel.Channels.HttpInput.HttpStatusCode</Key>
<Value>RequestEntityTooLarge</Value>
</Data>
</DataItems>
</Exception>
</TraceRecord>
</DataItem>
</TraceData>
</ApplicationData>
I am attempting a simple example project to get to know how to configure a Windows Azure Cloud service to use Co-located caching.
I am using VS2010 and have followed this example but when I get to the end of the section on co-located caching, I cannot run my project locally. It worked fine when
I am getting the following exceptions: (I hope it's readable..)
Error in '/' Application.
The requested name is valid, but no data of the requested type was found
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.Net.Sockets.SocketException: The requested name is valid, but no data of the requested type was found
Source Error:
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
Stack Trace:
[SocketException (0x2afc): The requested name is valid, but no data of the requested type was found]
Microsoft.ApplicationServer.Caching.AsyncResultNoResult.EndInvoke() +202
Microsoft.ApplicationServer.Caching.AsyncResult`1.EndInvoke() +40
Microsoft.ApplicationServer.Caching.SocketConnectionFactory.EndConnect(IAsyncResult asyncResult) +75
Microsoft.ApplicationServer.Caching.TcpClientChannelFactory.TcpConnectionCallback(IAsyncResult result) +182
[DataCacheException: ErrorCode:SubStatus:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.). Additional Information : The client was trying to communicate with the server: net.tcp://SessionStateDemo:24233.]
Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ErrStatus errStatus, Guid trackingId, Exception responseException, Byte[][] payload, EndpointID destination) +767
Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody, EndpointID destination) +149
Microsoft.ApplicationServer.Caching.DataCacheFactory.EstablishConnection(IEnumerable1 servers, RequestBody request, Func3 sendMessageDelegate, DataCacheReadyRetryPolicy retryPolicy) +1313
Microsoft.ApplicationServer.Caching.<>c__DisplayClass5.b__2(RequestBody req) +198
Microsoft.ApplicationServer.Caching.SocketClientProtocol.SendReceive(IVelocityRequestPacket request, Func2 delegate, EndpointID& destination) +121
Microsoft.ApplicationServer.Caching.SocketClientProtocol.Initialize(IEnumerable1 servers) +1225
Microsoft.ApplicationServer.Caching.DataCacheFactory.GetCache(String cacheName, CreateNewCacheDelegate cacheCreationDelegate, DataCacheInitializationViaCopyDelegate initializeDelegate) +1104
Microsoft.ApplicationServer.Caching.DataCacheFactory.GetCache(String cacheName) +131
Microsoft.Web.DistributedCache.DataCacheFactoryWrapper.CreateDataCacheFromFactory(DataCacheFactory factory, String cacheName) +63
Microsoft.Web.DistributedCache.CacheHelpers.RunCacheCreationHooks(CacheConnectingEventArgs fetchingEventArgs, IDataCacheFactory dataCacheFactory, Object sender, EventHandler1 fetchingHandler, EventHandler1 fetchedHandler) +306
Microsoft.Web.DistributedCache.DistributedCacheSessionStateStoreProvider.CreateInternalProvider(IHttpRuntime httpRuntime, SessionInitializationData initData, IDataCacheFactory dataCacheFactory, EventHandler1 cacheFetching, EventHandler1 cacheFetched) +399
Microsoft.Web.DistributedCache.DistributedCacheSessionStateStoreProvider.GetInternalProvider() +315
Microsoft.Web.DistributedCache.DistributedCacheSessionStateStoreProvider.CreateNewStoreData(HttpContext context, Int32 timeout) +59
System.Web.SessionState.SessionStateModule.InitStateStoreItem(Boolean addToContext) +160
System.Web.SessionState.SessionStateModule.CompleteAcquireState() +340
System.Web.SessionState.SessionStateModule.BeginAcquireState(Object source, EventArgs e, AsyncCallback cb, Object extraData) +1269
System.Web.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +12698078
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +288
Can anyone tell me what is happening here? How can I fix this?
Is there something missing in that tutorial that I need to do?
EDIT: I am using VS2010 Professional, with Windows Azure SDK 2.1. Widnows Azure Cache package is v2.1.0.
Relevant web.config sections:
<system.web>
...
<!-- Windows Azure Caching session state provider -->
<sessionState mode="Custom" customProvider="AFCacheSessionStateProvider">
<providers>
<add name="AFCacheSessionStateProvider" type="Microsoft.Web.DistributedCache.DistributedCacheSessionStateStoreProvider, Microsoft.Web.DistributedCache" cacheName="default" dataCacheClientName="default" applicationName="AFCacheSessionState"/>
</providers>
</sessionState>
...
<sytem.web>
<dataCacheClients>
<dataCacheClient name="default">
<!--To use the in-role flavor of Windows Azure Caching, set identifier to be the cache cluster role name -->
<autoDiscover isEnabled="true" identifier="SessionStateDemo" />
</dataCacheClient>
</dataCacheClients>
Based on this thread it looks like the Windows Azure Cloud Service must be a startup project to work properly. Otherwise the Windows Azure Runtime won't initialize what causes the SocketExceptions.
I have few WCF servics that work fine on our test server (Windows Server 2008 R2). When i move these servics to prod server, services returns null. I am not sure what is causing this. I am guessing these is something missing on the production server but not sure what is it. Below is teh exception log. Appreciate any feedback. Thanks
<E2ETraceEvent xmlns="http://schemas.microsoft.com/2004/06/E2ETraceEvent">
<System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system">
<EventID>131075</EventID>
<Type>3</Type>
<SubType Name="Error">0</SubType>
<Level>2</Level>
<TimeCreated SystemTime="2013-09-12T16:19:39.2467613Z" />
<Source Name="System.ServiceModel" />
<Correlation ActivityID="{00000000-0000-0000-0000-000000000000}" />
<Execution ProcessName="w3wp" ProcessID="4332" ThreadID="8" />
<Channel />
<Computer>CGWAPP268PD</Computer>
</System>
<ApplicationData>
<TraceData>
<DataItem>
<TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Error">
<TraceIdentifier>http://msdn.microsoft.com/en-US/library/System.ServiceModel.Diagnostics.ThrowingException.aspx</TraceIdentifier>
<Description>Throwing an exception.</Description>
<AppDomain>/LM/W3SVC/1/ROOT/HEIGISDataSvc-1-130234759301892473</AppDomain>
<Exception>
<ExceptionType>System.ServiceModel.ProtocolException, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
<Message>Content Type application/soap+xml; charset=utf-8 was sent to a service expecting text/xml; charset=utf-8. The client and service bindings may be mismatched.</Message>
<StackTrace> at System.ServiceModel.Channels.HttpInput.ThrowHttpProtocolException(String message, HttpStatusCode statusCode, String statusDescription)
at System.ServiceModel.Channels.HttpInput.ParseIncomingMessage(Exception& requestException)
at System.ServiceModel.Channels.HttpChannelListener.HttpContextReceived(HttpRequestContext context, Action callback)
at System.ServiceModel.Activation.HostedHttpTransportManager.HttpContextReceived(HostedHttpRequestAsyncResult result)
at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.HandleRequest()
at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.BeginRequest()
at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.OnBeginRequest(Object state)
at System.Runtime.IOThreadScheduler.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
I think this is the key to it:
Content Type application/soap+xml; charset=utf-8 was sent to a service expecting text/xml; charset=utf-8. The client and service bindings may be mismatched.
A thread on the ASP.NET forums deals with the same issue.
This is usually a mismatch in the client/server bindings, where the
message version in the service uses SOAP 1.2 (which expects
application/soap+xml) and the version in the client uses SOAP 1.1
(which sends text/xml). WSHttpBinding uses SOAP 1.2, BasicHttpBinding
uses SOAP 1.1. Try the following ways to check whether it can reslove
your issue.
Change basicHttpBinding to wsHttpBinding
Change your application pool to run under Network Service and gave Network Service the proper permissions.
Thanks for the feedback everyone. It turns out that after setting "Enable 32-bit Applications" to true in application adv setting resolved this issue.
I am getting the exception shown below. BizTalk is able to call the service when there is very light load. But once I increase the load and the number of messsages to transmit increases after a while I keep getting this message for all the calls to the service.
Event Type: Error
Event Source: BizTalk Server 2006
Event Category: (1)
Event ID: 5799
Date: 11/21/2011
Time: 12:55:38 PM
User: N/A
Computer: CLBIZBTS07S
Description:
The description for Event ID ( 5799 ) in Source ( BizTalk Server 2006 ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a
remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: sndManageServiceEndPoint_WCF,
http://service.com:8100/ManageService.svc/ws, The adapter failed to transmit message going to send port "sndManageServiceEndPoint_WCF" with URL "http://service.com:8100/ManageService.svc/ws". It will be
retransmitted after the retry interval specified for this Send Port. Details:"System.TimeoutException: Client is unable to finish the security negotiation within the configured timeout (00:01:00). The
current negotiation leg is 3 (00:00:20.6239920). ---> System.TimeoutException: The request channel timed out attempting to send after 00:00:20.6239920. Increase the timeout value passed to the call to
Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: The HTTP request to
'http://service.com:8100/ManageService.svc/ws' has exceeded the allotted timeout of 00:00:20.6230000. The time allotted to this operation may have been a portion of a longer
timeout. ---> System.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
at System.Net.HttpWebRequest.GetRequestStream()
at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()
--- End of inner exception stack trace ---
at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()
at System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout)
at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
--- End of inner exception stack trace ---
at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Security.IssuanceTokenProviderBase`1.DoNegotiation(TimeSpan timeout)
--- End of inner exception stack trace ---
Server stack trace:
at System.ServiceModel.Security.IssuanceTokenProviderBase`1.DoNegotiation(TimeSpan timeout)
at System.ServiceModel.Security.SspiNegotiationTokenProvider.OnOpen(TimeSpan timeout)
at System.ServiceModel.Security.WrapperSecurityCommunicationObject.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Security.SecurityUtils.OpenCommunicationObject(ICommunicationObject obj, TimeSpan timeout)
at System.ServiceModel.Security.SymmetricSecurityProtocol.OnOpen(TimeSpan timeout)
at System.ServiceModel.Security.WrapperSecurityCommunicationObject.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.SecurityChannelFactory`1.ClientSecurityChannel`1.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.DoOperation(SecuritySessionOperation operation, EndpointAddress target, Uri via, SecurityToken currentToken, TimeSpan timeout)
at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.GetTokenCore(TimeSpan timeout)
at System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(TimeSpan timeout)
at System.ServiceModel.Security.SecuritySessionClientSettings`1.ClientSecuritySessionChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at System.ServiceModel.ICommunicationObject.Open()
at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfClient`2.GetChannel[TChannel](IBaseMessage bizTalkMessage, ChannelFactory`1& cachedFactory)
at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfClient`2.SendMessage(IBaseMessage bizTalkMessage)". .
On the service side I have the set maxReceivedMessageSize="2147483647" in the endpoint
and in the behavior section I have set
<serviceThrottling
maxConcurrentCalls="16"
maxConcurrentInstances="2147483647"
maxConcurrentSessions="10"/>
These two settings did not help.
When I turned on the tracing on wcf service I see these type of errors
Activity Name Process action 'http://schemas.xmlsoap.org/ws/2005/02/trust/RST/SCT/Cancel'.
Time 2011-11-22 15:03:12.5301
Level Error
Source System.ServiceModel
Process w3wp
Thread 32
Computer FRA1BSHR01S
Trace Identifier/Code http://msdn.microsoft.com/en-US/library/System.ServiceModel.Diagnostics.ThrowingException.aspx
System.ServiceModel.EndpointNotFoundException, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
There was no channel that could accept the message with action 'http://schemas.xmlsoap.org/ws/2005/02/trust/RST/SCT/Cancel'.
System.ServiceModel.Dispatcher.ErrorBehavior.ThrowAndCatch(Exception e, Message message)
System.ServiceModel.Channels.DatagramChannelDemuxer`2.ProcessItem(TInnerItem item)
System.ServiceModel.Channels.DatagramChannelDemuxer`2.HandleReceiveResult(IAsyncResult result)
System.ServiceModel.Channels.DatagramChannelDemuxer`2.OnReceiveComplete(IAsyncResult result)
System.ServiceModel.Channels.DatagramChannelDemuxer`2.OnReceiveCompleteStatic(IAsyncResult result)
System.ServiceModel.Diagnostics.Utility.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
System.ServiceModel.AsyncResult.Complete(Boolean completedSynchronously)
System.ServiceModel.Channels.ReplyChannel.HelpReceiveRequestAsyncResult.OnReceiveRequest(IAsyncResult result)
System.ServiceModel.Diagnostics.Utility.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
System.ServiceModel.AsyncResult.Complete(Boolean completedSynchronously)
System.ServiceModel.Channels.InputQueue`1.AsyncQueueReader.Set(Item item)
System.ServiceModel.Channels.InputQueue`1.EnqueueAndDispatch(Item item, Boolean canDispatchOnThisThread)
System.ServiceModel.Channels.InputQueue`1.EnqueueAndDispatch(T item, ItemDequeuedCallback dequeuedCallback, Boolean canDispatchOnThisThread)
System.ServiceModel.Channels.InputQueueChannel`1.EnqueueAndDispatch(TDisposable item, ItemDequeuedCallback dequeuedCallback, Boolean canDispatchOnThisThread)
System.ServiceModel.Channels.SingletonChannelAcceptor`3.Enqueue(QueueItemType item, ItemDequeuedCallback dequeuedCallback, Boolean canDispatchOnThisThread)
System.ServiceModel.Channels.SingletonChannelAcceptor`3.Enqueue(QueueItemType item, ItemDequeuedCallback dequeuedCallback)
System.ServiceModel.Channels.HttpChannelListener.HttpContextReceived(HttpRequestContext context, ItemDequeuedCallback callback)
System.ServiceModel.Activation.HostedHttpTransportManager.HttpContextReceived(HostedHttpRequestAsyncResult result)
System.ServiceModel.Activation.HostedHttpRequestAsyncResult.BeginRequest()
System.ServiceModel.Activation.HostedHttpRequestAsyncResult.OnBeginRequest(Object state)
System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke2()
System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke()
System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ProcessCallbacks()
System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.CompletionCallback(Object state)
System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
System.ServiceModel.Diagnostics.Utility.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
[TraceRecord] Severity Error
TraceIdentifier http://msdn.microsoft.com/en-US/library/System.ServiceModel.Diagnostics.ThrowingException.aspx
Description Throwing an exception.
AppDomain /LM/W3SVC/1373337114/Root-4-129664654928269938
Although increasing the timeout on both ends of the wire (BTS and WCF Service) may help avoid the error in some instances, you might just be kicking the can further down the road and you probably need to get to the root cause (does this specific web service call NORMALLY take over 20 seconds to complete?).
It is likely that under high load, or low capacity conditions, that your WCF Web Service cannot keep up with the rate of messages that BizTalk is able to send to it.
Depending on what your WCF Service does, you might look at optimising:
Database
SQL blocking while waiting for locks
DB performance may reduced during admin tasks such as Reindexing Jobs, Backups etc
IIS / ASP.NET
See if requests are being queued
Look at any throttling on the WCF Service side (e.g. MaxConcurrentCalls)
If there is no chance of further optimisation (or scale up / out) on the WCF Service side, you may need to look at throttling the number of concurrent orchestrations (using a variant of a Singleton Orch) or limiting the number of threads on your WCF Send Port Adapter (unfortunately, this will affect all messages, so you might need to split out separate SendHost
Be wary when simply 'maxing out' all of the WCF knobs - settings like maxReceivedMessageSize (or maxBufferSize) ="2147483647" will often result in OutOfMemoryExceptions and / or increase the threat surface area.
You need to change the timeout of the WCF service.
I am not sure if that solution is correct, but it is definitely in the web.config file.
I had similar problem when using SOAP adapter(I'm still in biztalk 2006)
You should try the following :
YourMessage(SOAP.ClientConnectionTimeout) = 900000;
900000 is the default value, so increase it to fit your need, the time is in milliseconds.
If this property does not apply to WCF web services, I assume there is a WCF equivalent for that property.
I am using WCF service in my client-server application and I am facing following error while communication between server and client.
Error Message =>> System.ServiceModel.CommunicationObjectAbortedException: The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it has been Aborted.
Server stack trace:
at System.ServiceModel.Channels.CommunicationObject.ThrowIfDisposedOrNotOpen()
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at ServiceLib.ServiceCallbackInterfaces.IHostServiceCallback.SendEventAndStatus(String message, Boolean isBackupGenerated)
at ServiceLib.Services.DriversService.objDriver_EventReceived(Object sender, EventReceivedEventArgs e)
Application Scenario : Client is getting data from server through WCF service and using nettcp binding. Server is connected with real device and generating 20-30 events every seconds. Generated events are pushed as string message to all connected clients using callback interface of WCF. It's working fine if client have no any process load (just receive event and display). But it gives error when we do some loading process at client side and busy for some time.
Note: I had already set maximum all timeouts in nettcp binding as below, but still problem not solved.
NetTcpBinding tcpBinding = new NetTcpBinding(SecurityMode.None);
tcpBinding.MaxBufferPoolSize = 2147483647;
tcpBinding.MaxReceivedMessageSize = 2147483647;
tcpBinding.MaxBufferSize = 2147483647;
tcpBinding.ReaderQuotas.MaxStringContentLength = 2147483647;
tcpBinding.ReaderQuotas.MaxDepth = 2147483647;
tcpBinding.ReaderQuotas.MaxBytesPerRead = 2147483647;
tcpBinding.ReaderQuotas.MaxNameTableCharCount = 2147483647;
tcpBinding.ReaderQuotas.MaxArrayLength = 2147483647;
tcpBinding.SendTimeout = TimeSpan.MaxValue;
tcpBinding.ReceiveTimeout = TimeSpan.MaxValue;
tcpBinding.ReliableSession.InactivityTimeout = TimeSpan.MaxValue;
Please guide me is there any other settings required related to communication channel or binding ?
UPDATE : Error thrown by svc tracer :
<E2ETraceEvent xmlns="http://schemas.microsoft.com/2004/06/E2ETraceEvent">
<System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system">
<EventID>131075</EventID>
<Type>3</Type>
<SubType Name="Error">0</SubType>
<Level>2</Level>
<TimeCreated SystemTime="2011-09-30T08:37:01.5691715Z" />
<Source Name="System.ServiceModel" />
<Correlation ActivityID="{00000000-0000-0000-0000-000000000000}" />
<Execution ProcessName="ServerUtility" ProcessID="2396" ThreadID="5" />
<Channel />
<Computer>TEST</Computer>
</System>
<ApplicationData>
<TraceData>
<DataItem>
<TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Error">
<TraceIdentifier>http://msdn.microsoft.com/it-IT/library/System.ServiceModel.Diagnostics.ThrowingException.aspx</TraceIdentifier>
<Description>Throwing an exception.</Description>
<AppDomain>ServerUtility.exe</AppDomain>
<Exception>
<ExceptionType>System.ServiceModel.CommunicationException, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
<Message>The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '10675199.02:48:05.4775807'.</Message>
<StackTrace>
at System.ServiceModel.Channels.SocketConnection.EndRead()
at System.ServiceModel.Channels.DelegatingConnection.EndRead()
at System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(Object state)
at System.ServiceModel.Channels.SocketConnection.FinishRead()
at System.ServiceModel.Channels.SocketConnection.AsyncReadCallback(Boolean haveResult, Int32 error, Int32 bytesRead)
at System.ServiceModel.Channels.OverlappedContext.CompleteCallback(UInt32 error, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
</StackTrace>
<ExceptionString>System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '10675199.02:48:05.4775807'. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
--- End of inner exception stack trace ---</ExceptionString>
<InnerException>
<ExceptionType>System.Net.Sockets.SocketException, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
<Message>An existing connection was forcibly closed by the remote host</Message>
<StackTrace>
at System.ServiceModel.Channels.SocketConnection.EndRead()
at System.ServiceModel.Channels.DelegatingConnection.EndRead()
at System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(Object state)
at System.ServiceModel.Channels.SocketConnection.FinishRead()
at System.ServiceModel.Channels.SocketConnection.AsyncReadCallback(Boolean haveResult, Int32 error, Int32 bytesRead)
at System.ServiceModel.Channels.OverlappedContext.CompleteCallback(UInt32 error, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
</StackTrace>
<ExceptionString>System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host</ExceptionString>
<NativeErrorCode>2746</NativeErrorCode>
</InnerException>
</Exception>
</TraceRecord>
</DataItem>
</TraceData>
<System.Diagnostics xmlns="http://schemas.microsoft.com/2004/08/System.Diagnostics">
<LogicalOperationStack></LogicalOperationStack>
<Timestamp>4730654290080</Timestamp>
</System.Diagnostics>
</ApplicationData>
</E2ETraceEvent>
As you mentioned in your question, you have set all timeouts to maximum. So as per my opinion, there should be problem in InstanceContextMode you defined. Which mode you defined in your service ? PerSession, PerCall or Single. There might be PerCall you defined and calling service by same object (globally declared object of service) then it's possible to channel come in fault state or connection disconnected.
So If you defined PerSession mode then use single object (Globally declared) from client side to get data from service. If you defined PerCall option then create new object every time.
Hope this help to you. Please feel free to ask any doubt...