Could someone tell me if this is a good way to retrieve data from an API?
I'm new to this, and wonder if someone could recommend me some kind of setup and structure, and if this could be done in a more correct way?
string baseUrl = "https://api.data.com/getITems/
//Create a new instance of HttpClient
using (HttpClient client = new HttpClient())
{
var byteArray = Encoding.ASCII.GetBytes("username:password");
client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Basic", Convert.ToBase64String(byteArray));
using (HttpResponseMessage result = await client.GetAsync(baseUrl))
using (HttpContent content = result.Content)
{
result.EnsureSuccessStatusCode();
string data = await content.ReadAsStringAsync();
var test1 = JsonConvert.DeserializeObject<Results>(data);
foreach (var item in test1.results)
{
foreach (var item1 in item.result)
{
Console.WriteLine("Result: {0}", item1.Symbol);
}
}
}
Your code looks fine. You could use a nuget package to simplify some of the operations or turn it into a helper class. The only thing you should do differently is not create a new HttpClient every time, this is against the practices. You should only create it once and use it lots of times.
Deciding if your implementation is a good way, is out of my scope, since I do not know your requirements and genereally you can always extract some functionality into some other class, while juggeling the effort you put in against the expected value you get out of it.
That being said, I'll just go into the one thing that springs to my eye and that is your usage of HttpClient itself.
If this is a long lived application and you keep calling that part of your code, you will always new-up an new instance of HttpClient and then dispose it. This will kill the connections to your endpoint immediately, but leave the ports in TIME_WAIT state.
Do this is a couple of hundred or thousand times and you will most likely run into SocketException, because your machine has run out of available sockets to use.
From the documentation:
HttpClient is intended to be instantiated once and reused throughout
the life of an application.
If all HttpClients, share the same headers and base address, you can re-use a single instance of that client throughout your application. If you have different headers, create a new static client.
In both scenarios you want to use them as a singleton throughout your application.
Some further reading:
https://medium.com/#nuno.caneco/c-httpclient-should-not-be-disposed-or-should-it-45d2a8f568bc
https://learn.microsoft.com/en-us/aspnet/web-api/overview/advanced/calling-a-web-api-from-a-net-client
https://aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/
https://codereview.stackexchange.com/questions/69950/single-instance-of-reusable-httpclient/69954#69954
Related
Is it possible to read API data as it comes , I wrote the below Csharp code in my controller but sometimes the data takes more than 2 minutes and I was wondering if it was possible to load the data in my website as they come instead of waiting for it. Below is my current code:
private static async Task<List<Model>> GetFlightData()
{
using (var client = new HttpClient())
{
client.Timeout = TimeSpan.FromMilliseconds(Timeout.Infinite);
var content = await client.GetStringAsync(URL);
var result = JsonConvert.DeserializeObject<List<Model>>(content);
return result;
}
}
The fastest way is to save the data statically and initialize it on start up,
the problem with this solution is that IIS may restart you website when there's no traffic and the data will be lost (and will cause the next visitor to wait a whole 2 minutes).
The best suggestion I have is to save it to redis/other cache of your chosing and then just pull it from there.
my application is using Azure Service Bus to store messages. I have an Azure function called HttpTriggerEnqueuewhich allow me to enqueue messages. The problem is that this function can be invoked hundreds times in a little interval of time. When I call the HttpTriggerEnqueue once, twice, 10 times, or 50 times everything works correctly. But when I call it 200, 300 times (which is my use case) I get an error and not all messages are enqueued. From the functions portal I get the following error.
threshold exceeded [connections]
I tried both the .NET sdk and the HTTP request. Here is my code
HTTP REQUEST:
try
{
var ENQUEUE = "https://<MyNamespace>.servicebus.windows.net/<MyEntityPath>/messages";
var client = new HttpClient(new HttpClientHandler() { AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip });
var request = new HttpRequestMessage(HttpMethod.Post, ENQUEUE);
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var sasToken = SASTokenGenerator.GetSASToken(
"https://<MyNamespace>.servicebus.windows.net/<MyEntityPath>/",
"<MyKeyName>",
"<MyPrimaryKey>",
TimeSpan.FromDays(1)
);
client.DefaultRequestHeaders.TryAddWithoutValidation("Authorization", sasToken);
request.Content = new StringContent(message, Encoding.UTF8, "application/json");
request.Headers.AcceptEncoding.Add(new StringWithQualityHeaderValue("gzip"));
request.Headers.AcceptEncoding.Add(new StringWithQualityHeaderValue("deflate"));
var res = await client.SendAsync(request);
}
catch (Exception e) { }
And the code using the SDK:
var qClient = QueueClient.CreateFromConnectionString(MyConnectionString, MyQueueName);
var bMessage = new BrokeredMessage(message);
qClient.Send(bMessage);
qClient.Close();
I have the standard tier pricing on Azure.
If I call the function 300 (for example) times in a little interval of time I get the error. How can I solve?
The actual issue here isn't with the Service Bus binding (although you should follow the advice that #Mikhail gave for that), it's a well known issue with HttpClient. You shouldn't be re-creating the HttpClient on every function invocation. Store it in a static field so that it can be reused. Read this blog post for a great breakdown on the actual issue. The main point being that unless you refactor this to use a single instance of HttpClient you're going to continue to run into port exhaustion.
From the MSDN Docs:
HttpClient is intended to be instantiated once and re-used throughout the life of an application. Especially in server applications, creating a new HttpClient instance for every request will exhaust the number of sockets available under heavy loads. This will result in SocketException errors.
You should use Service Bus output binding to send messages to Service Bus from Azure Function. It will handle connection management for you, so you shouldn't be getting such errors.
I am consuming a web service provided to me by a vendor in c# application. This application calls a web method in a loop and that slows down the performance. To get the complete set of results, it takes more than an hour.
Can I apply multi threading on my side to consume this web service in multiple threads and combine the results together?
Is there any better approach to retrieve data in minutes instead of hours?
First of all you have to make sure your vendor does indeed support this or does not prohibit it (which is very probable too).
The code itself to do this is fairly straightforward, using a method such as Parallel.For
Simple Example (google.com):
Parallel.For(0, norequests,
i => {
//Code that does your request goes here
} );
Exaplanation:
In a Parallel.For loop, all the requests get executed in-parallel (as implied in the name), which could potentially provide a very significant increase in performance.
Further reading:
MSDN on Parallel.For loops
You should really ask your vendor. We can only speculate about why it takes that long or if firing multiple requests will actually yield the same results as the one that takes long.
Basically, sending one request getting one response should beat the multi-threaded variant because it should be easier to optimize on the servers side.
If you want to know why this is not the case with the current version of the service, ask the vendor.
This is only samples, if you call web services in parallel:
private void TestParallelForeach()
{
string[] uris = {"http://192.168.1.2", "http://192.168.1.3", "http://192.168.1.4"};
var results = new List<string>();
var syncObj = new object();
Parallel.ForEach(uris, uri =>
{
using (var webClient = new WebClient())
{
webClient.Encoding = Encoding.UTF8;
try
{
var result = webClient.DownloadString(uri);
lock (syncObj)
{
results.Add(result);
}
}
catch (Exception ex)
{
// Do error handling here...
}
}
});
// Do with "results" here....
}
Using Visual studio 2012, C#.net 4.5 , SQL Server 2008, Feefo, Nopcommerce
Hey guys I have Recently implemented a new review service into a current site we have.
When the change went live the first day all worked fine.
Since then though the sending of sales to Feefo hasnt been working, There are no logs either of anything going wrong.
In the OrderProcessingService.cs in Nop Commerce's Service, i call a HttpWebrequest when an order has been confirmed as completed. Here is the code.
var email = HttpUtility.UrlEncode(order.Customer.Email.ToString());
var name = HttpUtility.UrlEncode(order.Customer.GetFullName().ToString());
var description = HttpUtility.UrlEncode(productVariant.ProductVariant.Product.MetaDescription != null ? productVariant.ProductVariant.Product.MetaDescription.ToString() : "product");
var orderRef = HttpUtility.UrlEncode(order.Id.ToString());
var productLink = HttpUtility.UrlEncode(string.Format("myurl/p/{0}/{1}", productVariant.ProductVariant.ProductId, productVariant.ProductVariant.Name.Replace(" ", "-")));
string itemRef = "";
try
{
itemRef = HttpUtility.UrlEncode(productVariant.ProductVariant.ProductId.ToString());
}
catch
{
itemRef = "0";
}
var url = string.Format("feefo Url",
login, password,email,name,description,orderRef,productLink,itemRef);
var request = (HttpWebRequest)WebRequest.Create(url);
request.KeepAlive = false;
request.Timeout = 5000;
request.Proxy = null;
using (var response = (HttpWebResponse)request.GetResponse())
{
if (response.StatusDescription == "OK")
{
var stream = response.GetResponseStream();
if(stream != null)
{
using (var reader = new StreamReader(stream))
{
var content = reader.ReadToEnd();
}
}
}
}
So as you can see its a simple webrequest that is processed on an order, and all product variants are sent to feefo.
Now:
this hasnt been happening all week since the 15th (day of the
implementation)
the site has been grinding to a halt recently.
The stream and reader in the the var content is there for debugging.
Im wondering does the code redflag anything to you that could relate to the process of website?
Also note i have run some SQL statements to see if there is any deadlocks or large escalations, so far seems fine, Logs have also been fine just the usual logging of Bots.
Any help would be much appreciated!
EDIT: also note that this code is in a method that is called and wrapped in A try catch
UPDATE: well forget about the "not sending", thats because i was just told my code was rolled back last week
A call to another web site while processing the order can degrade performance, as you are calling to a site that you do not control. You don't know how much time it is going to take. Furthermore, the GetResponse method can throw an exception, if you don't log anything in your outer try/catch block then you won't be able to know what's happening.
The best way to perform such a task is to implement something like the "Send Emails" scheduled task, and send data when you can afford to wait for the remote service. It is easy if you try. It is more resilient and easier to maintain if you upgrade the nopCommerce code base.
This is how I do similar things:
Avoid modifying the OrderProcessingService: Create a custom service or plugin that consumes the OrderPlacedEvent or the OrderPaidEvent (just implement the IConsumer<OrderPaidEvent> or IConsumer<OrderPlacedEvent> interface).
Do not call to a third party service directly while processing the request if you don't need the response at that moment. It will only delay your process. At the service created in step 1, store data and send it to Feefo later. You can store data to database or use an static collection if you don't mind losing pending data when restarting the site (that could be ok for statistical data for instance).
Best way to implement point #2 is to add a new scheduled task implementing ITask (remember to add a record to the ScheduleTask table). Just recover the stored data do your processing.
Add some logging. It is easy, just get an ILogger instance and call Insert.
As far as I can see, you are making a blocking synchronous call to other websites, which will definitely slow down your site in between the request-response process. What Marco has suggested is valid, try to do it in an ITask. Or you can use an asynchronous web request to potentially remove the block, if you need things done immediately instead of scheduled. :)
We are building a highly concurrent web application, and recently we have started using asynchronous programming extensively (using TPL and async/await).
We have a distributed environment, in which apps communicate with each other through REST APIs (built on top of ASP.NET Web API). In one specific app, we have a DelegatingHandler that after calling base.SendAsync (i.e., after calculating the response) logs the response to a file. We include the response's basic information in the log (status code, headers and content):
public static string SerializeResponse(HttpResponseMessage response)
{
var builder = new StringBuilder();
var content = ReadContentAsString(response.Content);
builder.AppendFormat("HTTP/{0} {1:d} {1}", response.Version.ToString(2), response.StatusCode);
builder.AppendLine();
builder.Append(response.Headers);
if (!string.IsNullOrWhiteSpace(content))
{
builder.Append(response.Content.Headers);
builder.AppendLine();
builder.AppendLine(Beautified(content));
}
return builder.ToString();
}
private static string ReadContentAsString(HttpContent content)
{
return content == null ? null : content.ReadAsStringAsync().Result;
}
The problem is this: when the code reaches content.ReadAsStringAsync().Result under heavy server load, the request sometimes hangs on IIS. When it does, it sometimes returns a response -- but hangs on IIS as if it didn't -- or in other times it never returns.
I have also tried reading the content using ReadAsByteArrayAsync and then converting it to String, with no luck.
When I convert the code to use async throughout I get even weirder results:
public static async Task<string> SerializeResponseAsync(HttpResponseMessage response)
{
var builder = new StringBuilder();
var content = await ReadContentAsStringAsync(response.Content);
builder.AppendFormat("HTTP/{0} {1:d} {1}", response.Version.ToString(2), response.StatusCode);
builder.AppendLine();
builder.Append(response.Headers);
if (!string.IsNullOrWhiteSpace(content))
{
builder.Append(response.Content.Headers);
builder.AppendLine();
builder.AppendLine(Beautified(content));
}
return builder.ToString();
}
private static Task<string> ReadContentAsStringAsync(HttpContent content)
{
return content == null ? Task.FromResult<string>(null) : content.ReadAsStringAsync();
}
Now HttpContext.Current is null after the call to content.ReadAsStringAsync(), and it keeps being null for all the subsequent requests! I know this sounds unbelievable -- and it took me some time and the presence of three coworkers to accept that this was really happening.
Is this some kind of expected behavior? Am I doing something wrong here?
I had this problem. Although, I haven't fully tested yet, using CopyToAsync instead of ReadAsStringAsync seems to fix the problem:
var ms = new MemoryStream();
await response.Content.CopyToAsync(ms);
ms.Seek(0, SeekOrigin.Begin);
var sr = new StreamReader(ms);
responseContent = sr.ReadToEnd();
With regards to your second issue, the async/await is syntactic sugar for the compiler building a state machine where the call to to a function preceded by "await" returns immediately on the current thread...one that contains HttpContext.Current in its thread local storage. The completion of that async call can occur on a different thread...one that does NOT have HttpContext.Current in its thread local storage.
If you want the completion to execute on the same thread (thus having the same objects in thread local storage like HttpContext.Current), then you need to be aware of this behavior. This is especially important on calls from the main UI thread (if you're building a Windows application) or in ASP.NET, calls from an ASP.NET request thread where you are dependent on HttpContext.Current.
See reference docs on ConfigureAwait(false). Also, view some Channel 9 tutorials on TPL. Once the "easy" stuff is grokked, the presenter will invariably talk about this issue as it causes subtle problems that are not easily understood unless you know what the TPL is doing underneath the covers.
Good luck.
With regards to your first problem, if the caller gets a result, I'm not convinced that IIS has not completed the request. How are you determining that the ASP.NET request thread initiated by this caller is hung in IIS?