Develop Reference

Proper way to handle multiple services with polly circuit breaker

I have an application where we communicate with hundreds of HTTPs endpoints. The application is a proxy of sorts.
When testing with polly, I've noticed that if one endpoint, say api.endpoint1.com fails, the calls to api.endpoint2.com and api.endpoint3.com will also be in an open/blocked state.
This makes sense as I've only defined one policy, but what is the recommended approach to handling this scenario so that calls to unrelated endpoints are not blocked due to another having performance issues?
Do I create a collection of Policy's, one for each endpoint or is there a way to supply a context key of sorts(i.e. the hostname) to scope the failures to a given host endpoint?
I've reviewed Polly's docs regarding context keys and it appears these are a way to exchange data back and forth and not what I'm looking for here.
var policy = Policy
.Handle<TimeoutException>()
.CircuitBreaker(1, TimeSpan.FromSeconds(1));
//dynamic, large list of endpoints.
var m = new HttpRequestMessage(HttpMethod.Post, "https://api.endpoint1.com")
{
Content = new StringContent("some JSON data here", Encoding.UTF8,"application/json")
};
policy.Execute(() => HTTPClientWrapper.PostAsync(message));

Yes, your best bet is to create a separate policy per endpoint. This is better than doing it per host because an endpoint may be slow responding for a reason that's specific to that endpoint (e.g., stored procedure is slow).
I've used a Dictionary<string, Policy> with the endpoint URL as the key.
if (!_circuitBreakerPolices.ContainsKey(url))
{
CircuitBreakerPolicy policy = Policy.Handle<Exception>().AdvancedCircuitBreakerAsync(
onBreak: ...
);
_circuitBreakerPolicies.Add(url, policy);
}
await _circuitBreakerPolicies[url].ExecuteAsync(async () => ... );

Here is my alternative solution which does not maintain a collection of policies (either via an IDictionary or via an IConcurrentPolicyRegistry) rather it takes advantage of named typed clients. (Yes you have read correctly named and typed HttpClients)
The named and typed clients
Most probably you have heard (or even used) named or typed clients. But I'm certain that you haven't used named and typed clients. It is a less documented feature of HttpClientFactory + HttpClient combo.
If you look at the different overloads of the AddHttpClient extension method then you can spot this one:
public static IHttpClientBuilder AddHttpClient<TClient,TImplementation>
(this IServiceCollection services, string name, Action<HttpClient> configureClient)
where TClient : class where TImplementation : class, TClient;
It allows us to register a typed client and give a logical name to it. But how can I get the proper instance? That's where the ITypedHttpClientFactory comes into the picture. It allows us to create a typed client from a named client. Wait what??? I hope you will understand this sentence at the end of this post. :)
The typed client
For the sake of simplicity let me use this typed client as an example:
public interface IResilientClient
{
Task GetAsync();
}
public class ResilientClient: IResilientClient
{
private readonly HttpClient client;
public ResilientClient(HttpClient client)
{
this.client = client;
}
public Task GetAsync()
{
//TODO: implement it properly
return Task.CompletedTask;
}
}
The named and typed clients registration
Let suppose you have a list of downstream system urls (urls). Then you can register multiple typed client instances with different unique names and base urls
foreach (string url in urls)
{
builder.Services
.AddHttpClient<IResilientClient, ResilientClient>(url,
client => client.BaseAddress = new Uri(url))
.AddPolicyHandler(GetCircuitBreakerPolicy());
}
Here I have used the url as the unique name
So, we can get the appropriate instance based on the downstream url
The policy definition
private IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicy()
=> Policy<HttpResponseMessage>
.Handle<TimeoutException>()
.CircuitBreakerAsync(1, TimeSpan.FromSeconds(1));
I have modified the policy to support async: .CircuitBreakerAsync
I've also amended it to be suitable with the AddPolicyHandler: Policy<HttpResponseMessage>
It is defined as a function so each registered named typed client will have a different Circuit Breaker instance
The usage
This is be a bit clumsy, but I think it is okay. So, wherever you want to use one of the named typed clients you have to inject two interfaces:
IHttpClientFactory: To be able to create a named HttpClient
ITypedHttpClientFactory<ResilientClient>: To be able to create a typed client from the named HttpClient
public XYZService(
IHttpClientFactory namedClientFactory,
ITypedHttpClientFactory<ResilientClient> namedTypedClientFactory)
{
var namedClient = namedClientFactory.CreateClient(xyzUrl);
var namedTypedClient = namedTypedClientFactory.CreateClient(namedClient);
}
Please note that you have to use ResilientClient concrete class as the type parameter not the interface IResilientClient
If you would use the interface then you would receive the following runtime error:
InvalidOperationException: A suitable constructor for type 'IResilientClient' could not be located. Ensure the type is concrete and all parameters of a public constructor are either registered as services or passed as arguments. Also ensure no extraneous arguments are provided.
Summary
With the named and typed client feature of AddHttpClient we can register multiple instances of the same typed client
With the IHttpClientFactory we can retrieve a registered named client which has the proper BaseAddress and decorated with a Circuit Breaker
With the ITypedHttpClientFactory we can convert the named client into a typed client to be able to hide low-level API usage
Related sample application's github repository

Call WCF Service from another decoupled WCF service on the same site without HttpClient

I have a plugin model architecture that creates my Restful WCF services.
(It will be a couple years before we move to Web Api from WCF so, moving to Web Api isn't exactly a solution.)
I have decoupled WCF Microservices that don't reference each other.
EntityAWebService
EntityBWebService
EnityAWebService knows that a service EntityBWebService exists from a configuration, but doesn't reference it.
EntityAWebService and EntityBWebService are plugins. As such, they could be on loaded on the same site.
EntityAWebService makes a call to EntityBWebService using configuration information. The EntityBWebService could be on the same server or a different server.
- If on a different server, the code will continue to use HttpClient.
- If on the same server, go cract the message and send it through the channel without going through HttpClient, operating system's network, and IIS.
Below is the architecture. The orange is what I want to create.
Using HttpClient means EntityAWebService sends a message that is going to hit the operating systems network layer and go through IIS. Neither of which is necessary. It causes performance issues, and as the Entity plugins increase, so does the number of sockets and even using a singleton httpclient, the sockets are leaking.
The orange in the architecture is what doesn't exist yet.
The code knows the Url to call for Entity B Web Service, the message content, and the headers. How would I simulate, in the code represented by the orange box, what IIS does to forward the call through the behaviors and to the Endpoint?
FYI, my current project is too complex to post, so I will create a sample and post it soon.
Sample project: https://github.com/rhyous/DecoupledWcfServices

Turns out I didn't need to use named pipes. However, investigating how to use named pipes taught me what I needed to know. I just needed to use reflection and ChannelFactory. As the ChannelFactory for IIS hosting already exists, named pipes would be redundant.
Example Project here: https://github.com/rhyous/DecoupledWcfServices
And the appropriate snippet (the meat of the solution) is below.
using System;
using System.Collections.Specialized;
using System.ServiceModel;
using System.ServiceModel.Description;
using System.ServiceModel.Web;
namespace DecoupledWcfServices
{
/// <summary>
/// Service 1 and Service 2 are in the same namespace in this project
/// </summary>
public class MessageBus
{
public string CallOtherWcfService(string url, object content, NameValueCollection headers)
{
var service = GetServiceName(url);
try
{
var netPipeUrl = $"http://localhost:54412/{service}/{service}.svc";
var serviceContractType = typeof(IService2);
var genericChannelFactoryType = typeof(WebChannelFactory<>).MakeGenericType(serviceContractType);
var binding = new WebHttpBinding();
var channelFactory = Activator.CreateInstance(genericChannelFactoryType, binding, new Uri(netPipeUrl)) as WebChannelFactory<IService2>; // I actually won't know it is an IService2 in my project, but getting this far should be enough
var proxy = channelFactory.CreateChannel() as IService2;
using (new OperationContextScope((IContextChannel)proxy))
{
var task = proxy.GetData("some data"); // Might need more work here to know which method to call based on the Url
task.Wait();
return task.Result; // Serialized JSON
}
}
catch (Exception)
{
throw;
}
}
internal string GetServiceName(string url)
{
var index = url.IndexOf(".svc");
var sub = url.Substring(0, index);
index = sub.LastIndexOf("/") + 1;
var sub2 = url.Substring(index, sub.Length - index);
return sub2;
}
}
}

Disable metadata exchange without using web.config

I am developing a single-tenant web application that will be deployed in client data centers and for security reasons we would like to disable the metadata exchange on the applications WCF services. Is it possible to do this this programatically within our service application or another mechanism besides the web.config? We want to prevent more technically minded clients from going to the web.config and turning metadata exchange back on.

You can disable the metadata exchange programmatically by setting the HttpGetEnabled/HttpsGetEnabled to false.
First, Create a derive host from ServiceHost.
public class DerivedHost : ServiceHost
{
public DerivedHost( Type t, params Uri baseAddresses ) :
base( t, baseAddresses )
{
DisableMetadataExchange();
}
private void DisableMetadataExchange()
{
var metadata = Description.Behaviors.Find<ServiceMetadataBehavior>();
if metadata != null)
{
// This code will disable metadata exchange
metadata .HttpGetEnabled = false;
metadata .HttpsGetEnabled = false;
}
}
}
Second, Create a derived factory from ServiceHostFactory.
public class DerivedFactory : ServiceHostFactory
{
public override ServiceHost CreateServiceHost( Type t, Uri[] baseAddresses )
{
return new DerivedHost( t, baseAddresses );
}
}
Third, Create or Edit your your svc file Markup and apply your derived factory.
<% #ServiceHost Factory=”DerivedFactory” Service=”MyService” %>
Fourth, Test your service in the browser and you should see a message contain "Metadata publishing for this service is currently disabled".
If want more details about this implementation kindly visit this link.

Yes. If you code your WCF service as "self describing", which basically means using a WCF intercept layer to handle all the incoming requests to an endpoint, you can just return null from the MEX request.
To make this work is a bit tricky but in my experience leads to a much cleaner implementation than all those voluminous web.config entries. This is described here WCF Configuration without a config file.

WCF/REST Logging

I was wondering if someone could show me how to log a simple request/response from my wcf rest service.
I am self hosting with a console application on the localmachine:
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
string baseAddress = "http://" + Environment.MachineName + ":8000/Service";
ServiceHost host = new ServiceHost(typeof(RawDataService), new Uri(baseAddress));
WebHttpBinding binding = new WebHttpBinding();
//binding.Security.Mode = WebHttpSecurityMode.Transport;
host.AddServiceEndpoint(typeof(IReceiveData), new WebHttpBinding(), "").Behaviors.Add(new WebHttpBehavior());
host.Open();
Console.WriteLine("Host opened");
Console.ReadLine();
}
}
}
I was really hoping all that would be required is something added to the hosting console app. I tryed following this but it was abit confusing http://blogs.msdn.com/b/carlosfigueira/archive/2011/04/19/wcf-extensibility-message-inspectors.aspx
Just to note I am not using any app.config or web.config files.
EDIT:
I also cannot use any third party products for this.

Are you talking about logging for debugging purposes or for monitoring in a live service?
If you are debugging you can just switch on WCF tracing. It will produce a very comprehensive log and there is a good free tool for viewing the log that comes as part of the Windows SDK - I presume when you say you can't use third party product it doesn't include built-in .Net and Windows SDK features...
http://msdn.microsoft.com/en-us/library/ms733025.aspx

A common way to handle this is with Aspect-Oriented Programming (AOP) using the Castle Dynamic Proxy library. The idea is that you can decorate/proxy your service implementation with a dynamic class that intercepts every single method called on your service. No matter what method is called on your service, they'll be "intercepted" by your proxy and sent to a single method where you can log what you want and then you can finish the original call. Here's a quick sample of what that looks like:
public class LoggingInterceptor : IInterceptor
{
// No matter what service method is called, it's funneled through here.
public void Intercept(IInvocation call)
{
MyLogger.Info("Starting call: " + call.Method.Name);
// Actually invoke whatever method was originally called
call.Proceed();
MyLogger.Info("Finished call: " + call.Method.Name);
}
}
Now you need to create a proxy of your service class that uses this interceptor for all of its method calls. You can pretty up and abstract as necessary, but this is the basic jist:
using Castle.DynamicProxy;
...
// Create your service object and then create a dynamic proxy of the object
// that will inject your logging interceptor logic.
ProxyGenerator generator = new ProxyGenerator();
RawDataService service = new RawDataService();
RawDataService proxy = generator.CreateClassProxyWithTarget<RawDataService>(
service,
new LoggingInterceptor());
// Register your proxy object, not the raw service w/ WCF
WebServiceHost host = new WebServiceHost(proxy, new Uri(baseAddress));
... rest of your code as it was ...
Now any call made to your RawDataService will go through the Intercept() method first and when it calls Proceed() your actual implemented service logic will happen. You can update the interceptor to handle exceptions, include a StopWatch and log parameters as needed but that's the basic idea.
My example shows you the brute force way of setting this up. The "cleaner" solution would be to use IoC to create your service instance/proxy but this should get your code doing what you want right now. For further reading, here's a link to the Castle project's tutorial on using its AOP hooks:

Is there something like a OnClientConnected event for .NET remoting?

I'm using something like this on my server:
TcpServerChannel channel = new TcpServerChannel(settings.RemotingPort);
ChannelServices.RegisterChannel(channel, true);
RemotingServices.Marshal(myRemoteObject, "myRemoteObject");
I would like to subscribe to some kind of event so that whenever a remote client connects to myRemoteObject, I can check the Thread.CurrentPrincipal.Identity.Name to decide whether to authorize him.
Currently I'm doing the authorizing check in every exposed remote method of myRemoteObject which is a messy...

In my remoting application i defined a special object/interface where clients first need to authorize. The special object then returns, if the client successfully authorized the remote object. So you have the authorization at one place.
It looks something like this.
public interface IPortal
{
object SignIn(string name, string password);
}
public class Portal : MarshalByRefObject, IPortal
{
private object _remoteObject;
public Portal() {
_remoteObject = new RemoteObject();
}
public object SignIn(string name, string password)
{
// Authorization
// return your remote object
return _remoteObject;
}
}
In your application you host the Portal-Object
TcpServerChannel channel = new TcpServerChannel(settings.RemotingPort);
ChannelServices.RegisterChannel(channel, true);
Portal portal = new Portal()
RemotingServices.Marshal(portal , "portal");

You could use something like PostSharp to factor out the check from every method - just do it in the AOP advice. (You apply this to the class which is exposing its methods, not to the client connection.)
This approach is independent of whatever transport you use for remoting - it just factors out the cross-cutting concern of authorization across all the methods in your remoted class.