WCF Duplex Service Channel Close - c#

I have an application based around a WCF Duplex service. I have problems when the user "Restarts" the work the application does... under the hood, the client side closes the connection to the WCF service and creates another. The Service contract is defined like so...
[ServiceContract(Namespace="net.tcp://namespace.MyService",
SessionMode=SessionMode.Required,
CallbackContract=typeof(IServiceCallback))]
public interface IMyService
{
[OperationContract(IsOneWay=true)]
void DoWork();
}
public interface IServiceCallback
{
[OperationContract(IsOneWay=true)]
void SendMessage(string message);
}
The implementation is defined as:
[ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Single,
InstanceContextMode = InstanceContextMode.PerSession,
UseSynchronizationContext = false,
IncludeExceptionDetailInFaults = true)]
public class MyService : IMyService
{
public void DoWork()
{
var callback = OperationContext.Current.GetCallbackChannel<IServiceCallback>();
callback.SendMessage("Hello, world.");
}
}
The configuration for the client is as follows:
<system.serviceModel>
<bindings>
<netTcpBinding>
<binding name="net.tcp" receiveTimeout="02:00:00" sendTimeout="02:00:00" maxReceivedMessageSize="2147483647">
<security mode="None"/>
</binding>
</netTcpBinding>
</bindings>
<client>
<endpoint address="net.tcp://localhost:8000/MyService/MyService"
binding="netTcpBinding" bindingConfiguration="net.tcp" contract="ExternalServiceReference.IMyService">
</endpoint>
</client>
</system.serviceModel>
Config for the service:
<system.serviceModel>
<behaviors>
<serviceBehaviors>
<behavior name="serviceBehaviour">
<serviceMetadata />
</behavior>
</serviceBehaviors>
</behaviors>
<bindings>
<netTcpBinding>
<binding name="netTcp" sendTimeout="01:00:00" receiveTimeout="01:00:00" >
<security mode="None">
</security>
</binding>
</netTcpBinding>
</bindings>
<services>
<service behaviorConfiguration="serviceBehaviour" name="MyService.MyService">
<endpoint address="MyService" binding="netTcpBinding" bindingConfiguration="netTcp" name="net.tcp" contract="MyService.IMyService" />
<endpoint binding="mexTcpBinding" bindingConfiguration="" name="net.tcp" contract="IMetadataExchange" />
<host>
<baseAddresses>
<add baseAddress="net.tcp://localhost:8000/MyService" />
</baseAddresses>
</host>
</service>
</services>
In the client's contructor:
var callback = new CallbackImplementation();
_context = new InstanceContext(callback);
_proxy = new MyServiceProxy(_context);
I'm trying the following before I establish a new connection:
try
{
if (_context != null)
{
_context.ReleaseServiceInstance();
_context.Close();
}
}
catch (Exception ex)
{
Debug.WriteLine(ex.Message);
if (_context != null)
{
_context.Abort();
}
}
The issue I see is that the _context.Close() call always times out and throws an exception. Although I'm then aborting the channel, this feels wrong to me, and I believe it's the cause of freezing in my application. Does anybody know why the Close() call fails?
EDIT: I missed something earlier regarding my callback implementation that might be relevant. It looks something like this:
[CallbackBehavior(ConcurrencyMode = ConcurrencyMode.Single,
UseSynchronizationContext = false,
IncludeExceptionDetailInFaults = true)]
public class CallbackImplementation : IServiceCallback
{
public void SendMessage(string message)
{
// Do something with the message
}
}
The exception message is "The ServiceHost close operation timed out after 00:00:30. This could be because a client failed to close a sessionful channel within the required time. The time allotted to this operation may have been a portion of a longer timeout.". There's no inner exception.
Thanks

I don't see an issue right off, but I often find that running traces on both the client and server and examining the results usually points me to the solution. Put this in your .config files (client and server), make sure the path points to a folder that exists. Run your app, get the failure, then shut everything down and run SvcTraceViewer.exe to read the results.
<system.diagnostics>
<sources>
<source name="System.ServiceModel" switchValue="Information,ActivityTracing"
propagateActivity="true">
<listeners>
<add name="xml" />
</listeners>
</source>
<source name="System.ServiceModel.MessageLogging">
<listeners>
<add name="xml" />
</listeners>
</source>
</sources>
<sharedListeners>
<add initializeData="C:\logs\TracingAndLogging-service.svclog" type="System.Diagnostics.XmlWriterTraceListener"
name="xml" />
</sharedListeners>
<trace autoflush="true" />
</system.diagnostics>

have you tried receiveTimeout="infinite", to see if you still receive the error. You may just find that it's not the timeout creating the error. Have you thought about creating a base client class that automatically keeps the connections alive till physically closed?

At a guess, it could be a deadlock caused by the use of ConcurrencyMode.Single.
What could be happening is that the server is trying to call back to the client while it is still processing the original request (or vice versa). So server is trying to call back to client, but client is blocking because it is still waiting for a response from the server. Hey presto, deadlock and eventually a timeout.
http://msdn.microsoft.com/en-us/library/system.servicemodel.concurrencymode.aspx

The issue was not only concurrency, but the binding type.
Making sure the concurrency mode of both the service and the callback was a step in the right direction. But changing the binding from netTcpBinding to wsDualHttpBinding has finally fixed the problem

Related

Thread starvation on net.tcp binding - TCP error code 10061

I have faced a very strange error in my WCF service, which appears to somehow create a deadlock or thread starvation in socket level when I use NetTcpBinding. I have a quite simple self-hosted service:
class Program
{
static void Main(string[] args)
{
using (ServiceHost serviceHost = new ServiceHost(typeof(TestService)))
{
serviceHost.Open();
Console.WriteLine("Press <ENTER> to terminate service.");
Console.ReadLine();
serviceHost.Close();
}
Uri baseAddress = new Uri("net.tcp://localhost:8014/TestService.svc");
}
}
[ServiceContract]
public interface ITestService
{
[OperationContract]
string GetData(string data);
}
public class TestService: ITestService
{
public string GetData(string data)
{
Console.WriteLine(data);
Thread.Sleep(5000);
return "Ok";
}
}
The configuration part:
<system.serviceModel>
<bindings>
<basicHttpBinding>
<binding name="basicHttpBinding" closeTimeout="00:02:00" openTimeout="00:02:00"
receiveTimeout="00:02:00" sendTimeout="00:02:00" maxBufferSize="2000000000"
maxReceivedMessageSize="2000000000" />
</basicHttpBinding>
<netTcpBinding>
<binding name="netTcpBinding" closeTimeout="00:02:00" openTimeout="00:02:00"
receiveTimeout="00:02:00" sendTimeout="00:02:00" listenBacklog="2000"
maxBufferSize="2000000000" maxConnections="1000" maxReceivedMessageSize="2000000000">
<security mode="None">
<transport protectionLevel="EncryptAndSign" />
</security>
</binding>
<binding name="TestServiceTcpEndPoint">
<security mode="None" />
</binding>
</netTcpBinding>
</bindings>
<behaviors>
<serviceBehaviors>
<behavior name="CommonServiceBehavior">
<serviceMetadata httpGetEnabled="true" />
<serviceDebug includeExceptionDetailInFaults="true" />
<serviceThrottling maxConcurrentCalls="1000" maxConcurrentSessions="1000" maxConcurrentInstances="1000" />
</behavior>
</serviceBehaviors>
</behaviors>
<services>
<service name="ServiceLauncher.TestService" behaviorConfiguration="CommonServiceBehavior">
<endpoint address="" binding="netTcpBinding" bindingConfiguration="netTcpBinding" name="TestServiceTcpEndPoint" contract="ServiceLauncher.ITestService" />
<endpoint address="" binding="basicHttpBinding" bindingConfiguration="basicHttpBinding" name="TestServiceTcpEndPoint" contract="ServiceLauncher.ITestService" />
<endpoint address="mex" binding="mexHttpBinding" bindingName="mexHttpBinding" contract="IMetadataExchange" />
<host>
<baseAddresses>
<add baseAddress="net.tcp://localhost:8014/TestService.svc"/>
<add baseAddress="http://localhost:1234/TestService.svc"/>
</baseAddresses>
</host>
</service>
</services>
</system.serviceModel>
And I have a client which consumes this service in many threads with creating new instance for every thread (it is a requirement):
static void Main(string[] args)
{
for (int i = 0; i < 1000; i++)
{
Thread tr = new Thread(() =>
{
using (var service = new Test.TestServiceClient())
{
var result = service.GetData(i.ToString());
Console.WriteLine(string.Format("{0}: {1} {2}",
DateTime.Now,
result,
Thread.CurrentThread.ManagedThreadId));
}
});
tr.Start();
}
Console.ReadLine();
}
In this case after some requests client raises EndpointNotFoundException, TCP error code 10061, No connection could be made because the target machine actively refused it. The number of requests is different all the time, and it is not the server part because it still works in normal state. And I see it keeps recieving the requests, what is most strangest in this situation. What is also strange that it can make your client host "immortal" after the exception - so that you can't kill it by any mean, except of the reboot of the system. I'm pretty sure that the problem is in low socket level of the client, and it is somehow connected with such a large number of threads, but I didn't succeed in finding something which could explaine the problem.
Every time I've seen the error "No connection could be made because the target machine actively refused it." the problem has not been with the service. Its usually a problem reaching the service.
A couple suggestions:
Avoid using with WCF Proxies. You can pick from several reasonable work arounds.
Read my answer to WCF performance, latency and scalability. Other than starting threads the old fashioned way, its basically the same test app. The post describes all the client causes (I could find) that cause “No connection could be made because the target machine actively refused it” and offers different WCF, TCP, and thread pool settings that can be adjusted.
You could be hitting the internal limits on concurrent TCP/IP connections in windows. Have a look at this article and see if it helps:
http://smallvoid.com/article/winnt-tcpip-max-limit.html

WCF client in a universal app

Is it possible to call a WCF service from a universal application?
I added a service reference and the proxy was generated just fine.
But when creating a NetTcpBinding programmatically and passing that to the proxy's constructor the service model throws the exception PlatformNotSupported.
Both running the app in the simulator and on the local machine generates the same exception.
An exception of type 'System.PlatformNotSupportedException' occurred
in System.Private.ServiceModel.dll but was not handled in user code
"this operation is not supported"
EndpointAddress address = new EndpointAddress("net.tcp://test:9000/ServicesHost/PublishService");
NetTcpBinding binding = new NetTcpBinding();
binding.Security.Mode = SecurityMode.None;
PublishingService.PublishClient proxy = new PublishingService.PublishClient(binding, address);
Does anybody have an example of a working WCF client in a UAP?
EDIT
It has something to do with the service being a duplex service!
The original contract:
[ServiceContract(CallbackContract = typeof(IPublishCallback))]
public interface IPublish { }
After removing the CallbackContract attribute the UAP client can create a connection, so basic WCF works.
So I guess it's better to rephrase the question.
Is it possible to create a duplex WCF client in a universal application?
edit servicemodel for the host
<system.serviceModel>
<bindings>
<netTcpBinding>
<binding name="netTcpPublishService" openTimeout="00:00:10" receiveTimeout="infinite">
<reliableSession inactivityTimeout="24.20:31:23.6470000" enabled="true" />
<security mode="Transport">
<transport clientCredentialType="Windows" />
</security>
</binding>
</netTcpBinding>
</bindings>
<behaviors>
<serviceBehaviors>
<behavior name="serviceBehaviour">
<serviceMetadata httpGetEnabled="true" />
</behavior>
</serviceBehaviors>
</behaviors>
<services>
<service behaviorConfiguration="serviceBehaviour" name="PublishService.Publish">
<endpoint binding="mexHttpBinding" name="mexPublishService"
contract="IMetadataExchange" />
<endpoint address="PublishService" binding="netTcpBinding" bindingConfiguration="netTcpPublishService"
name="netTcpPublishService" contract="PublishService.IPublish" />
<host>
<baseAddresses>
<add baseAddress="http://localhost:8004/ServicesHost/PublishService" />
<add baseAddress="net.tcp://localhost:9004/ServicesHost/PublishService" />
</baseAddresses>
</host>
</service>
</services>
</system.serviceModel>
Yes, it is possible. This is how i connect in a sample app i did a while ago:
using Tradeng.Srvc.Client.WinAppSimple.SrvcRefTradeng;
private InstanceContext instanceContext;
private TradengSrvcClientBase serviceProxy;
instanceContext = new InstanceContext(this);
serviceProxy = new TradengSrvcClientBase(instanceContext);
bool result = await serviceProxy.ConnectAsync();
if (result)
{
// connected...
}
I used the binding from the config file that is generated when you add a reference to your service.
This is what the app looks like. Cutting edge stuff.... :O)
https://www.youtube.com/watch?v=YSg6hZn1DpE
The service itself is running as a WebRole on Azure, by the way.

WCF Error - Object reference not set to an instance of an object. Production service not working

I'm new to writing questions, long time reader. If I omit something please let me know.
I have looked through many different scenario's and possible fixes and have been unable to get my WCF Service working correctly.
My service is responsible for passing data from many different sets to a master repository. The client gathers the data at the local set level and passes it to the service which inserts the data and passes back a result message. Again in the test environment this worked normally.
In production I added the service to an off-site server and configured the client on a remote set. The client was able to configure and receive updates from the service. Up until now everything worked correctly. However once the client attempted to transfer data across it received the following error "Object reference not set to an instance of an object."
Through bug checking I have confirmed there are no connection string issues to the db. I ran the same data transfer again in my test environment with no issues. I was able to connect to the .svc url on the local set.
I've added logging at different points through the data contract method call and none of these logs have triggered any results. I've also tested the write functionality on a test app which confirmed there were no issues with credentials writing to the temp folder. Eg:
public Result InsertVenueRecord(Venue v)
{
System.IO.File.WriteAllText(#"C:\temp\MadeItToVenue" + DateTime.Now.Ticks.ToString() + ".log.txt", "insertVenue()\r\n\r\n");
int oldId = v.VenueId;
try
{
System.IO.File.WriteAllText(#"C:\temp\MadeItToVenue" + DateTime.Now.Ticks.ToString() + ".log.txt", "insertVenue()\r\n\r\n");
//Check Address
if (v.Address.AddressId != 0)
The client app.Config looks as follows:
<configuration>
<system.serviceModel>
<bindings>
<basicHttpBinding>
<binding name="BasicHttpBinding_placeholder" />
</basicHttpBinding>
</bindings>
<client>
<endpoint address="Removed" binding="basicHttpBinding"
bindingConfiguration="BasicHttpBinding_placeholder" contract="placeholder.placeholder"
name="BasicHttpBinding_placeholder" />
</client>
</system.serviceModel>
<appSettings>
<add key="DTFirstWave" value="Venue|Event|EventSurveyLocation|Agent|LoginHistory|LoginUsed"/>
</appSettings>
<connectionStrings>
<!-- Removed Connection Strings -->
</connectionStrings>
</configuration>
The service webconfig looks as follows:
<configuration>
<system.web>
<compilation debug="true" targetFramework="4.0"/>
<httpRuntime/>
</system.web>
<system.serviceModel>
<bindings>
<basicHttpBinding>
<binding allowCookies="true" maxBufferPoolSize="2147483647" maxBufferSize="2147483647" maxReceivedMessageSize="2147483647">
<readerQuotas maxDepth="32" maxStringContentLength="52428800" maxArrayLength="52428800" maxBytesPerRead="52428800" maxNameTableCharCount="52428800"/>
</binding>
</basicHttpBinding>
</bindings>
<behaviors>
<serviceBehaviors>
<behavior>
<!-- To avoid disclosing metadata information, set the values below to false before deployment -->
<serviceMetadata httpGetEnabled="true" httpsGetEnabled="true"/>
<serviceDebug includeExceptionDetailInFaults="false"/>
</behavior>
</serviceBehaviors>
</behaviors>
<protocolMapping>
<add binding="basicHttpsBinding" scheme="https"/>
</protocolMapping>
<serviceHostingEnvironment aspNetCompatibilityEnabled="true" multipleSiteBindingsEnabled="true"/>
</system.serviceModel>
<system.webServer>
<modules runAllManagedModulesForAllRequests="true"/>
<directoryBrowse enabled="true"/>
</system.webServer>
<connectionStrings>
<!--Removed -->
</connectionStrings>
</configuration>
I'm at a loss as to why this isn't working.
plz enable your service SVC log by just add system.daignostics in your service config.It will give you proper error on production.
It will create "App_tracelog.svclog" file in your service dir.
<system.diagnostics>
<sources>
<source name="System.ServiceModel.MessageLogging" switchValue="Warning, ActivityTracing">
<listeners>
<add name="ServiceModelTraceListener" />
</listeners>
</source>
<source name="System.ServiceModel" switchValue="Verbose,ActivityTracing">
<listeners>
<add name="ServiceModelTraceListener" />
</listeners>
</source>
<source name="System.Runtime.Serialization" switchValue="Verbose,ActivityTracing">
<listeners>
<add name="ServiceModelTraceListener" />
</listeners>
</source>
</sources>
<sharedListeners>
<add initializeData="App_tracelog.svclog" type="System.Diagnostics.XmlWriterTraceListener, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" name="ServiceModelTraceListener" traceOutputOptions="Timestamp" />
</sharedListeners>

WCF MsmqIntegrationBinding - Not picking Messages off of the queue

I have a legacy client that is writing messages to a queue (MSMQ). I wanted to use a WCF service to pick the XML messages up off of the queue. I followed some of the MSFT docs and poked around at other examples, but i can't seem to get it to work. The service host is starting, but it is not firing my process and picking messages off of the queue. Most likely user error, just not sure what.
I can see messages in the queue?
Code Example:
[ServiceContract]
[ServiceKnownType(typeof(XElement))]
public interface IMessageProcessor
{
[OperationContract(IsOneWay = true, Action = "*")]
void ProcessMessage(MsmqMessage<XElement> msg);
}
class MessageServiceClient : IMessageProcessor
{
[OperationBehavior(TransactionScopeRequired = true, TransactionAutoComplete = true)]
public void ProcessMessage(MsmqMessage<XElement> msg)
{
using (TransactionScope scope = new TransactionScope(TransactionScopeOption.Required))
{
Console.WriteLine("Processing {0} ", msg.ToString());
scope.Complete();
}
}
static void Main(string[] args)
{
Uri baseAddress = new Uri(ConfigurationManager.AppSettings["baseAddress"]);
// Create a ServiceHost for the CalculatorService type and provide the base address.
using (ServiceHost serviceHost = new ServiceHost(typeof(MessageServiceClient), baseAddress))
{
// Open the ServiceHostBase to create listeners and start listening for messages.
serviceHost.Open();
// The service can now be accessed.
Console.WriteLine("The service is ready.");
Console.WriteLine("The service is running in the following account: {0}");
Console.WriteLine("Press <ENTER> to terminate service.");
Console.WriteLine();
Console.ReadLine();
// Close the ServiceHostBase to shutdown the service.
serviceHost.Close();
}
}
}
App Config:
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<appSettings>
<!-- use appSetting to configure MSMQ queue name -->
<add key="QueueName" value=".\private$\MyMessageQueue" />
<add key="baseAddress" value="http://localhost:8000/test/message" />
</appSettings>
<system.serviceModel>
<services>
<service name="MessageServiceClient">
<!-- .Net endpoint-->
<endpoint address="msmq.formatname:DIRECT=OS:.\private$\MyMessageQueue"
binding="msmqIntegrationBinding"
bindingConfiguration="DotNetBinding"
contract="WcfServiceClient.IMessageProcessor" />
</service>
</services>
<behaviors>
<serviceBehaviors>
<behavior name="MessageServiceBehavior">
<serviceDebug includeExceptionDetailInFaults="true" />
<serviceMetadata />
<!--<serviceThrottling maxConcurrentCalls="20" maxConcurrentSessions="20" />-->
<serviceTimeouts />
</behavior>
</serviceBehaviors>
</behaviors>
<bindings>
<msmqIntegrationBinding>
<binding serializationFormat="ActiveX" name="ActiveXBinding" durable="false" exactlyOnce="false">
<security mode="None" />
</binding>
<binding serializationFormat="Xml" name="DotNetBinding" durable="false" exactlyOnce="false">
<security mode="None" />
</binding>
</msmqIntegrationBinding>
</bindings>
</system.serviceModel>
</configuration>
Not sure what I am doing wrong?
--S
In your config the following element needs to be as shown below:
<services>
<service name="WcfServiceClient.MessageServiceClient">
<!-- .Net endpoint-->
<endpoint address="msmq.formatname:DIRECT=OS:.\private$\MyMessageQueue"
binding="msmqIntegrationBinding"
bindingConfiguration="DotNetBinding"
contract="WcfServiceClient.IMessageProcessor" />
</service>
</services>
Your service name above doesnt include a namespace, it should always be a fully qualified name of the service

WCF: StackoverFlow exception

I am getting an exception when trying to servialize the .NET ServiceController class. It serializes fine when it's null but once I populate it I get a stackoverflow exception.
So this works:
[DataMember]
public ServiceController MyServiceController
{
get { return null; }
}
But this gives the error "An unhandled exception of type 'System.StackOverflowException' occurred in System.ServiceProcess.dll":
public class TestClass
{
private ServiceController _serviceController;
[DataMember]
public ServiceController MyServiceController
{
get { return ServiceController.GetServices()[0];
}
}
A strange thing is that there is no error in the logs at all. When there is an error I can see it in the logs so it's not because my logs aren't working. Here is my config file
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<system.web>
<compilation debug="true" />
</system.web>
<system.serviceModel>
<behaviors>
<serviceBehaviors>
<behavior name="MyCompany.Wcf.RdbmsServer.RdbmsServiceBehavior">
<serviceMetadata httpGetEnabled="true" />
<serviceDebug includeExceptionDetailInFaults="true" httpHelpPageEnabled="true"/>
</behavior>
</serviceBehaviors>
</behaviors>
<services>
<service behaviorConfiguration="MyCompany.Wcf.RdbmsServer.RdbmsServiceBehavior"
name="MyCompany.Wcf.RdbmsServer.RdbmsService">
<endpoint address="" binding="wsHttpBinding" contract="MyCompany.Wcf.RdbmsServer.IRdbmsService" bindingConfiguration="IncreaseMaxMessageSize">
<identity>
<dns value="localhost" />
</identity>
</endpoint>
<endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange" />
<host>
<baseAddresses>
<add baseAddress="http://localhost:8731/Design_Time_Addresses/MyCompany.Wcf.RdbmsServer/RdbmsService/" />
</baseAddresses>
</host>
</service>
</services>
<bindings>
<wsHttpBinding>
<binding name="IncreaseMaxMessageSize"
maxReceivedMessageSize="655360000">
</binding>
</wsHttpBinding>
</bindings>
</system.serviceModel>
<system.diagnostics>
<sources>
<source name="System.ServiceModel"
switchValue="All"
propagateActivity="true">
<listeners>
<add name="traceListener"
type="System.Diagnostics.XmlWriterTraceListener"
initializeData= "c:\Traces.svclog" />
</listeners>
</source>
</sources>
</system.diagnostics>
</configuration>
Here is my service interface
[ServiceContract]
public interface IRdbmsService
{
[OperationContract]
TestClass GetServiceControllerList();
}
And the implementation is:
public TestClass GetServiceControllerList()
{
return new TestClass();
}
Any ideas?
You are causing the serializer to recurse:
public class TestClass
{
private ServiceController _serviceController;
[DataMember]
public ServiceController MyServiceController
{
get { return ServiceController.GetServices()[0]; // <-- this returns your service
}
}
The service this is returning is your IRdbmsService, which returns TestClass. This then needs to be serialized, etc.
Edit: To clarify:
When TestClass is serialized, the serializer looks at all of its DataMember properties, one of which is ServiceController. It then serializes this property, which does this on the get:
return ServiceController.GetServices()[0];
Since IRdbmsService is the only service defined in your scope, it is at index 0 from the response of the GetServices call, so it then must be serialized. Since the return type of GetServiceControllerList is TestClass, TestClass must then be serialized, which brings us back to the beginning (hence, the recursion).
As far as how to go about solving this, it depends on what you are trying to do. What this service is doing at the moment is returning information about itself to the caller, which doesn't make sense to me; the caller already has this information at the time of consumption (before making the call). Can you clarify your intent?
Yes, in the constructor of ServiceController CLASS, you are problably calling something recursive which don't have a condition to stop, so every time it calls something recursive, the program push to a stack, and as I said there is no stop condition, it will add to stack untill the memory explode, so it gives this error name StackOverFlow.
A suggestion: Add a break point to debug where its calling recursive. It's so easy when debugging.

Categories

Resources