All,
I have a WCF web service (let's called service "B") hosted under IIS using a service account (VM, Windows 2003 SP2). The service exposes an endpoint that use WSHttpBinding with the default values except for maxReceivedMessageSize, maxBufferPoolSize, maxBufferSize and some of the time outs that have been increased.
The web service has been load tested using Visual Studio Load Test framework with around 800 concurrent users and successfully passed all tests with no exceptions being thrown. The proxy in the unit test has been created from configuration.
There is a sharepoint application that use the Office Sharepoint Server Search service to call web services "A" and "B". The application will get data from service "A" to create a request that will be sent to service "B". The response coming from service "B" is indexed for search. The proxy is created programmatically using the ChannelFactory.
When service "A" takes less than 10 minutes, the calls to service "B" are successfull. But when service "A" takes more time (~20 minutes) the calls to service "B" throw the following exception:
Exception Message: An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail
Inner Exception Message: The message could not be processed. This is most likely because the action 'namespace/OperationName' is incorrect or because the message contains an invalid or expired security context token or because there is a mismatch between bindings. The security context token would be invalid if the service aborted the channel due to inactivity. To prevent the service from aborting idle sessions prematurely increase the Receive timeout on the service endpoint's binding.
The binding settings are the same, the time in both client server and web service server are synchronize with the Windows Time service, same time zone.
When i look at the server where web service "B" is hosted i can see the following security errors being logged:
Source: Security
Category: Logon/Logoff
Event ID: 537
User NT AUTHORITY\SYSTEM
Logon Failure:
Reason: An error occurred during logon
Logon Type: 3
Logon Process: Kerberos
Authentication Package: Kerberos
Status code: 0xC000006D
Substatus code: 0xC0000133
After reading some of the blogs online, the Status code means STATUS_LOGON_FAILURE and the substatus code means STATUS_TIME_DIFFERENCE_AT_DC. but i already checked both server and client clocks and they are syncronized.
I also noticed that the security token seems to be cached somewhere in the client server because they have another process that calls the web service "B" using the same service account and successfully gets data the first time is called. Then they start the proccess to update the office sharepoint server search service indexes and it fails. Then if they called the first proccess again it will fail too.
Has anyone experienced this type of problems or have any ideas?
Regards,
--Damian
10 mins is the default receive timeout. If you have an idled proxy for more than 10mins, the security session of that proxy is aborted by the server. Enable logging and you will see this in the diagnostics log of the server. The error message you reported fits for this behavior.
Search your system diagnostic file for "SessionIdleManager". If you find it, the above is your problem.
Give it a whirl and set the establishSecurityContext="false" for the client and the server.
Don't call the service operation in a using statement. Instead use a pattern such as...
client = new ServiceClient("Ws<binding>")
try
{
client.Operation(x,y);
client.Close();
}
catch ()
{
client.Abort();
}
I don't understand why this works but I would guess that when the proxy goes out of scope in the using statement, Close isn't called. The service then waits until receiveTimeout (on the binding) has expired and then aborts the connection causing subsequent calls to fail.
What I believe is happening here is that your channel is timing out (as you suspect).
If I understand correctly, it is not the calls to service A that are timing out, but rather to service B, before you call your operation.
I'm guessing that you are creating your channel before you call service A, rather than just in time (i.e. before calling service B). You should create the channel (proxy, service client) just before you use it like:
AResponse aResp = null;
BResponse bResp = null;
using (ServiceAProxy proxyA = new ServiceAProxy())
{
aResp = proxyA.DoServiceAWork();
using (ServiceBProxy proxyB = new ServiceBProxy())
{
bResp = proxyB.DoOtherork(aResp);
}
}
return bResp;
I believe however, that once you get over that problem (service B timing out), you'll realize that the sharepoint app's proxy (that called service A) will timeout.
To solve that, you may wish to change your service model from a request-response, to a publish-subscribe model.
With long-running services, you'll want your sharepoint app to subscribe to service A, and have service A publish its results when it is ready to do so - regardless of how long it takes.
Programming WCF Services (O'Reilly) by Juval Lowey, has a great explanation, and IDesign (Juval's company) published a great set of coding standards for WCF, as well as the code for a great Publish-Subscribe Framework.
Hope this helps,
Assaf.
I actually triggered this error just now by doing something silly. I have a unit test that modifies the system date in order to test some time-based features. And I guess the apparent time difference between when I created the context and when I called my method (because of the changes to the system date), caused something to expire.
Related
This error is logged occasionally in the function app logs. "An exception occurred while creating a ServiceBusSessionReceiver (Namespace '<servicebus namespace>.servicebus.windows.net', Entity path '<topic>/Subscriptions/<subscription>'). Error Message: 'Azure.Messaging.ServiceBus.ServiceBusException: Put token failed. status-code: 500, status-description: The service was unable to process the request; please retry the operation."
The function app uses managed identity to connect to the service bus.
There is no impact on the regular usage but just want to know the reason for this exception.
I checked online to find the reason for the exception but didn`t find anything even on StackOverflow. I want to know the reason for this exception so I will know the impact of the failure and try to resolve it.
There is no action needed for your application and nothing that you can do to resolve. This is something that is handled by the Service Bus infrastructure internally. Intermittent failures will not impact your application, though if you're seeing this in large clusters or seeing it frequently, I'd encourage you to open a support ticket for investigation.
To add some context, this exception indicates a service-side issue when passing authorization token over the CBS link, which is a background activity. The Service Bus client sends refreshes periodically with a large enough window that failures can be retried before the current authorization expires. In the worst case, a specific connection would fault and the Service Bus client would create a new one. So long as the service issue was transient, such as is common when a service node is rebooting or being moved, things will recover without noticeable impact to the application.
I have a problem in the following code where I used an MSMQ queue:
Message m;
if (t! = TimeSpan.Zero)
{
m = q.Receive (t);
}
plus
{
m =
q.Receive ();
}
What I am doing is consuming a web service that uses this code. The problem is that if I test it locally using the Visual Studio debugger it works. But when I send the message using the service that is deployed in the IIS, when the Receive () method is executed it returns a TimeOut error, apparently due to the lack of permissions.
System.Messaging.MessageQueueException (0x80004005): The timeout for the requested operation has expired
Investigating I think it could be because when using debugger the user who accesses the queue is the one who encounters the session started on the computer. And when the consumption from the service URL the user who accesses is the IIS / DefaultAppPool. But I'm not sure, and I can't make it work, if anyone has any idea of how qualitative the solution would be, I appreciate it in advance.
I am working on Windows Server 2016 with .NetFramework 4.8.
I used WCF Data Service for CRUD operations. I have one remote service and many clients (computers). Some client reseived error on all edit operation (context have entity and we edit it now). Returned exception contains next message:"Your Browser sent a request with an unknown method (MERGE)". All clients uses Windows 7 with Framework 4.5. What can be the reason for this?
Service has a initialize method:
// This method is called only once to initialize service-wide policies.
public static void InitializeService(DataServiceConfiguration config)
{
// TODO: set rules to indicate which entity sets and service operations are visible, updatable, etc.
// Examples:
config.UseVerboseErrors = true;
config.SetEntitySetAccessRule("doc", EntitySetRights.All);
config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V3;
}
UPD1: It was found that the error disappears when client application running under an Administrator account (i.e. Administrator can edit entity without error). Perhaps the problem in the ClickOnce deployment.
UPD2: The reason in using the proxy server. Researching continues...
Solved a problem with a WCF Data Service and HTTP-requests (like HTTP MERGE) locking by proxy-server (http://msdn.microsoft.com/en-us/library/dd541471.aspx).
The decision came down to one C# line for client service proxy-class using tunneling:
docService.UsePostTunneling = true;
I'm trying to add some SOAP message logging capabilities to an old, old ATL Server web service that is running in integrated mode in IIS 7.5 on a Windows Server 2008 box, but running into a strange problem. For further background, I've added the assembly that contains the HttpModule to the modules element of the web.config for the ATL Server web service.
I've been following the answer provided here and the logging itself works great.
However, whenever I use this logging capability, the service responds with "SOAP Invalid Request", while the log file has the SOAP message as expected. I've done lots of fiddling around with it and figured out that this only happens if/when I access the request object's InputStream property in my handler for the BeginRequest event. It will fail if I even simply set a variable to the length of the InputStream like this:
private void OnBegin(object sender, EventArgs e)
{
var request = _application.Request;
//this will blow up
var foo = request.InputStream.Position;
}
If I don't touch the InputStream in my handler (which doesn't do much good when I'm only doing this to log the contents of the request, obviously), the request goes through perfectly.
I can access header values in the Request object and various other properties of the HttpApplication involved, but accessing the InputStream causes the service to choke.
Is there something intrinsic with ATL Server that will prevent me from doing this logging? Do I need to add some sort of locking or other safeguard in my BeginRequest handler to make sure this behaves? Is my handler hosing up the InputStream somehow causing it to be unusable for the service?
Another way of approaching this is to ask if there is a way to see the request as it gets to my service (i.e. after this HttpModule executes)?
It may also be worth noting that I am using SoapUI to test the service.
EDIT:
I've now done some failed request tracing in IIS and I get this error message:
ModuleName IsapiModule
Notification 128
HttpStatus 500
HttpReason Internal Server Error
HttpSubStatus 0
ErrorCode 0
ConfigExceptionInfo
Notification EXECUTE_REQUEST_HANDLER
ErrorCode The operation completed successfully. (0x0)
This comes in the handler for the ATL Server web service (i.e. the DLL for the service). Directly before that is the "GENERAL_READ_ENTITY_START" and "GENERAL_READ_ENTITY_END" messages, and the "END" has this message:
BytesReceived 0
ErrorCode 2147942438
ErrorCode Reached the end of the file. (0x80070026)
Does that mean what I think it means? That the handler isn't getting any data? Is this more evidence pointing towards my HttpModule messing with the Request's InputStream?
Are you sure your request object is valid? You're doing things slightly differently here from the sample you reference. They are extracting the stream from the sender argument whereas you obviously rely on a member variable.
So I finally determined that this wasn't a workable approach: I couldn't get the HttpModule to fire at all in IIS 6 (which I would need to have it to do for it to be an acceptable solution). I tried setting the Filter property on the Request object and all sorts of other crazy ideas, but none that led me to be able to both record the request body in the HttpModule and have the service still work.
So I did more digging and came upon this article on codeproject that talks about the inner workings of ATL Server, specifically the HandleRequest method in atlsoap.h. I mucked around in there for a while and figured out a way to get at the request body in there, and it was pretty simple to write it to a file manually from there.
For those curious, this is the final code I added to HandleRequest():
//****************************************REQUEST LOGGING**********************************************************
BYTE* bytes = pRequestInfo->pServerContext->GetAvailableData();
FILE* pFile;
pFile = fopen("C:\\IISLog\\ATL.txt", "a");
fwrite(bytes, 1, pRequestInfo->pServerContext->GetAvailableBytes(), pFile);
fclose(pFile);
//****************************************REQUEST LOGGING**********************************************************
I am going to still play around with it a bit more, but I have what appears to be a workable solution.
I have a windows service that has it's name set by an app.config.
I set the name as follows:
The ServiceBase.SerivceName is set on the Service class constructor (I have removed the setting of the ServiceName in the Service.Designer):
ServiceName = ConfigurationManager.AppSettings.Get("ServiceName");
The ServiceInstaller sets the DisplayName and ServiceName like this:
ServiceInstaller.DisplayName = config.AppSettings.Settings["ServiceName"].Value;
ServiceInstaller.ServiceName = ServiceInstaller.DisplayName;
All works as expected, so the service is installed fine alongside a duplicate service.
They have different names and different locations.
The appear as seperate entries in the Services list.
But I can only start one service at a time. The error I get on trying to start the second service is the unhelpful:
The service is not responding to the control function.
There are 2 System Events that get logged when trying to run:
A timeout was reached (30000 milliseconds) while waiting for the Blah Service service to connect.
The Blah Service service failed to start due to the following error:
The service did not respond to the start or control request in a timely fashion.
Any help gratefully received.
Thanks.
In the absence of sufficent info to offer an answer to the problem at hand, I suggest using http://topshelf-project.com/ - it makes installing, configuring and debugging windows services in .NET a breeze.