Develop Reference

HttpClient Error too many requests rate limit

I am calling HTTP requests to an API which has a limit on how many requests can be made.
These HTTP requests are done in loops and are done very quickly resulting in the HttpClient sometimes throwing a '10030 App Rate Limit Exceeded' exception error which is a 429 HTTP error for too many requests.
I can solve this by doing a Thread.Sleep between each call, however this slows down the application and is not reasonable.
Here is the code I am using:
public static async Task<List<WilliamHillData.Event>> GetAllCompetitionEvents(string compid)
{
string res = "";
try
{
using (HttpClient client = new HttpClient())
{
client.DefaultRequestHeaders.TryAddWithoutValidation("Content-Type", "application/json");
client.DefaultRequestHeaders.TryAddWithoutValidation("apiKey", "KEY");
using (HttpResponseMessage response = await client.GetAsync("https://gw.whapi.com/v2/sportsdata/competitions/" + compid + "/events/?&sort=startDateTime"))
{
res = await response.Content.ReadAsStringAsync();
}
}
JObject jobject = JObject.Parse(res);
List<WilliamHillData.Event> list = jobject["events"].ToObject<List<WilliamHillData.Event>>();
return list;
}
catch (Exception ex)
{
throw ex;
}
}
Is there a way I can find out how many requests can be made per second/minute? And possibly once the limit has been reached throttle down or do a Thread.Sleep until the limit has gone down instead of doing a Thread.Sleep each time it is been called so I am slowing down the app when it is required?
Cheers.

Is there a way I can find out how many requests can be made per second/minute?
Asking the owner of the API or reading the API Docs would help. You also can test it but you never know if there are other limits and even a Ban.
I can solve this by doing a Thread.Sleep between each call, however this slows down the application and is not reasonable.
You need to. You can filter out the 429 Error and not letting it throw. You must look whats better and faster. Slowing down your API Calls, so you stay within the limit or go full speed until you get a timeout and waiting that time.

HttpWebRequest possibly slowing website

Using Visual studio 2012, C#.net 4.5 , SQL Server 2008, Feefo, Nopcommerce
Hey guys I have Recently implemented a new review service into a current site we have.
When the change went live the first day all worked fine.
Since then though the sending of sales to Feefo hasnt been working, There are no logs either of anything going wrong.
In the OrderProcessingService.cs in Nop Commerce's Service, i call a HttpWebrequest when an order has been confirmed as completed. Here is the code.
var email = HttpUtility.UrlEncode(order.Customer.Email.ToString());
var name = HttpUtility.UrlEncode(order.Customer.GetFullName().ToString());
var description = HttpUtility.UrlEncode(productVariant.ProductVariant.Product.MetaDescription != null ? productVariant.ProductVariant.Product.MetaDescription.ToString() : "product");
var orderRef = HttpUtility.UrlEncode(order.Id.ToString());
var productLink = HttpUtility.UrlEncode(string.Format("myurl/p/{0}/{1}", productVariant.ProductVariant.ProductId, productVariant.ProductVariant.Name.Replace(" ", "-")));
string itemRef = "";
try
{
itemRef = HttpUtility.UrlEncode(productVariant.ProductVariant.ProductId.ToString());
}
catch
{
itemRef = "0";
}
var url = string.Format("feefo Url",
login, password,email,name,description,orderRef,productLink,itemRef);
var request = (HttpWebRequest)WebRequest.Create(url);
request.KeepAlive = false;
request.Timeout = 5000;
request.Proxy = null;
using (var response = (HttpWebResponse)request.GetResponse())
{
if (response.StatusDescription == "OK")
{
var stream = response.GetResponseStream();
if(stream != null)
{
using (var reader = new StreamReader(stream))
{
var content = reader.ReadToEnd();
}
}
}
}
So as you can see its a simple webrequest that is processed on an order, and all product variants are sent to feefo.
Now:
this hasnt been happening all week since the 15th (day of the
implementation)
the site has been grinding to a halt recently.
The stream and reader in the the var content is there for debugging.
Im wondering does the code redflag anything to you that could relate to the process of website?
Also note i have run some SQL statements to see if there is any deadlocks or large escalations, so far seems fine, Logs have also been fine just the usual logging of Bots.
Any help would be much appreciated!
EDIT: also note that this code is in a method that is called and wrapped in A try catch
UPDATE: well forget about the "not sending", thats because i was just told my code was rolled back last week

A call to another web site while processing the order can degrade performance, as you are calling to a site that you do not control. You don't know how much time it is going to take. Furthermore, the GetResponse method can throw an exception, if you don't log anything in your outer try/catch block then you won't be able to know what's happening.
The best way to perform such a task is to implement something like the "Send Emails" scheduled task, and send data when you can afford to wait for the remote service. It is easy if you try. It is more resilient and easier to maintain if you upgrade the nopCommerce code base.
This is how I do similar things:
Avoid modifying the OrderProcessingService: Create a custom service or plugin that consumes the OrderPlacedEvent or the OrderPaidEvent (just implement the IConsumer<OrderPaidEvent> or IConsumer<OrderPlacedEvent> interface).
Do not call to a third party service directly while processing the request if you don't need the response at that moment. It will only delay your process. At the service created in step 1, store data and send it to Feefo later. You can store data to database or use an static collection if you don't mind losing pending data when restarting the site (that could be ok for statistical data for instance).
Best way to implement point #2 is to add a new scheduled task implementing ITask (remember to add a record to the ScheduleTask table). Just recover the stored data do your processing.
Add some logging. It is easy, just get an ILogger instance and call Insert.

As far as I can see, you are making a blocking synchronous call to other websites, which will definitely slow down your site in between the request-response process. What Marco has suggested is valid, try to do it in an ITask. Or you can use an asynchronous web request to potentially remove the block, if you need things done immediately instead of scheduled. :)

how to control if web service is unavailable?

I have a webservice that gets a list from client and inserts it to database. Client has a windows service that is sending a list per 10 seconds. But there is a problem. What if it cannot reach to webservice(server). I should not lost any of the data. I decided to save data to a txt or binary if server is not reachable, and then upload them after the server starts to run. However, how can I decide whether the webservice is unavaliable. If I store the data to a file in a catch block, it will store when ever it gets an error, not only webservice unavaliable error. Any advice?

You can make an http request on the service's endpoint url and check if everything is ok :
var url = "http://....";
//OR
var url = service_object.Url;
var request = (HttpWebRequest)WebRequest.Create(url);
request.Timeout = 2000; //timeout 20 seconds
HttpWebResponse response = null;
try
{
response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode != HttpStatusCode.OK)
{
throw new ApplicationException(response.StatusDescription);
}
}
catch (ApplicationException ex)
{
//Do what you want here, create a file for example...
}

I'd introduce a queuing system (such as MSMQ or NServiceBus) so that the windows service only needs to place message(s) into the queue and something else (co-located with the web service) can dequeue messages and apply them (either directly or via the web service methods).
When everything is up and running, it shouldn't introduce much more overhead over your current (direct) web service call, and when the web service is down, everything just builds up in the queuing system.

C#: How to programmatically check a web service is up and running?

I need to create an C# application that will monitor whether a set of web services are up and running. User will select a service name from a dropdown. The program need to test with the corresponding service URL and show whether the service is running. What is the best way to do it? One way I am thinking of is to test whether we are able to download the wsdl. IS there a better way?
Note: The purpose of this application is that the user need to know only the service name. He need not remember/store the corresponding URL of the service.
I need a website version and a desktop application version of this C# application.
Note: Existing services are using WCF. But in future a non-WCF service may get added.
Note: My program will not be aware of (or not interested in ) operations in the service. So I cannot call a service operation.
REFERENCE
How to check if a web service is up and running without using ping?
C program-How do I check if a web service is running

this would not guarantee functionality, but at least you could check connectivity to a URL:
var url = "http://url.to.che.ck/serviceEndpoint.svc";
try
{
var myRequest = (HttpWebRequest)WebRequest.Create(url);
var response = (HttpWebResponse)myRequest.GetResponse();
if (response.StatusCode == HttpStatusCode.OK)
{
// it's at least in some way responsive
// but may be internally broken
// as you could find out if you called one of the methods for real
Debug.Write(string.Format("{0} Available", url));
}
else
{
// well, at least it returned...
Debug.Write(string.Format("{0} Returned, but with status: {1}",
url, response.StatusDescription));
}
}
catch (Exception ex)
{
// not available at all, for some reason
Debug.Write(string.Format("{0} unavailable: {1}", url, ex.Message));
}

This approach works for me.
I used Socket to check if the process can connect.
HttpWebRequest works if you try to check the connection 1-3 times but if you have a process which will run 24hours and from time to time needs to check the webserver availability that will not work anymore because will throw TimeOut Exception.
Socket socket
= new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
var result = socket.BeginConnect("xxx.com", 80, null, null);
// test the connection for 3 seconds
bool success = result.AsyncWaitHandle.WaitOne(3000,false);
var resturnVal = socket.Connected;
if (socket.Connected)
socket.Disconnect(true);
socket.Dispose();
return resturnVal;

Enforcing a strict timeout policy on a server side webrequest

We will need to call out to a 3rd party to retrieve a value using REST, however if we do not receive a response within 10ms, I want to use a default value and continue processing.
I'm leaning towards using an asynchronous WebRequest do to this, but I was wondering if there was a trick to doing it using a synchronous request.
Any advice?

If you are doing a request and waiting on it to return I'd say stay synchronous - there's no reason to do an async request if you're not going to do anything or stay responsive while waiting.
For a sync call:
WebRequest request = WebRequest.Create("http://something.somewhere/url");
WebResponse response = null;
request.Timeout = 10000; // 10 second timeout
try
{
response = request.GetResponse();
}
catch(WebException e)
{
if( e.Status == WebExceptionStatus.Timeout)
{
//something
}
}
If doing async:
You will have to call Abort() on the request object - you'll need to check the timeout yourself, there's no built-in way to enforce a hard timeout.

You could encapsulate your call to the 3rd party in a WebService. You could then call this WebService synchronously from your application - the web service reference has a simple timeout property that you can set to 10 seconds or whatever.
Your call to get the 3rd party data from your WebService will throw a WebException after the timeout period has elapsed. You catch it and use a default value instead.
EDIT: Philip's response above is better. RIF.