I am using .NET Core Web API. I am working with some third party web api which I need to call in my Web API (so my .NET Core Web API is kind of wrapper on third party ones).
To get my result, I need to call more then one third party api using foreach loop. Details are as below:
First will be a Web API call which gives me result of around 4000 rows (each row is object of Id and value fields).
After that I need to loop through this 4000 rows and using each Id I need to call another API. On the result of this Web API I need to check some validation and return the valid ones.
I am able to make first Web API call successfully but when I do looping for another API call it gives me timeout error.
I have tried below things
1) making batches of 4000 rows and processing in batches.
2) Adding tasks in `foreach` loop and using `Task.WhenAll`
Example :
var batchSize = 50;
var returnData = new List<Order>();
foreach (var batchedItems in inventoriesList.Batch(batchSize)) //4000 rows
{
var tasks = new List<Task<Order>>();
foreach (var item in batchedItems)
{
tasks.Add(GetOrder(item.Value)); //call to another api
}
foreach (var task in await Task.WhenAll(tasks))
{
returnData.Add(task);
}
}
private async Task<Order> GetOrder(string id)
{
var order = await GetAsync<Order>(api-url);
if (order != null
&& order.IsAvailable == false
&& ValidateOrder(order)))
{
isValidOrder = true;
}
return isValidOrder == true ? order : null;
}
I have tried with LINQ as well rather then doing foreach loop for second API
call. Like below,
tasks = batchedInventories.Select(t => GetOrder(t.Value));
var result = await Task.WhenAll(tasks);
I have also tried with increasing KeepAliveTimeout of Kestrel. But no luck.
Could anybody suggest me correct and working way to do this?
I managed to resolve this error by creating static HttpClient for GetAsync method rather then creating instance for each request. Thanks a ton to John !!
I initialized static HttpClient in constructor with cookiecontainer.
Reference : Using httpclient throughout methods without losing session and cookies
Related
I'm trying to do a get all from an Azure service and it returns an AsyncPageable. According to the doc it says
A collection of values that may take multiple service requests to iterate over.
Does that mean that it is equal to doing the request for a single item multiple times with a loop?
If a service call returns multiple values in pages it would return Pageable<T>/AsyncPageable<T> as a result. Check out Consuming Service Methods Returning AsyncPageable.
To get more clarity, have a look at below:
This shows control over receiving pages of values from the service use AsyncPageable<T>.AsPages method:
// call a service method, which returns AsyncPageable<T>
AsyncPageable<SecretProperties> response = client.GetPropertiesOfSecretsAsync();
await foreach (Page<SecretProperties> page in response.AsPages())
{
// enumerate through page items
foreach (SecretProperties secretProperties in page.Values)
{
Console.WriteLine(secretProperties.Name);
}
// get continuation token that can be used in AsPages call to resume enumeration
Console.WriteLine(page.ContinuationToken);
}
If your project doesn't have C# 8.0 enabled you can still iterate over AsyncPageable using a while loop:
// call a service method, which returns AsyncPageable<T>
AsyncPageable<SecretProperties> response = client.GetPropertiesOfSecretsAsync();
IAsyncEnumerator<SecretProperties> enumerator = response.GetAsyncEnumerator();
try
{
while (await enumerator.MoveNextAsync())
{
SecretProperties secretProperties = enumerator.Current;
Console.WriteLine(secretProperties.Name);
}
}
finally
{
await enumerator.DisposeAsync();
}
Check out Azure.Core Response samples to understand more about this.
To change page size, you can use pageSizeHint parameter to AsPages method.
I am facing an issue with two different endpoints in my single asp.net app. Basically, the issue is that one of the endpoints does not allow asynchronous methods in the page and the other endpoint does. If I run the app one endpoint will ask me to have an asynchronous asp.net page, but the other one crashes and vice versa.
public async Task<AirtableListRecordsResponse> RetrieveRecord()
{
string MyProductID = ProductID;
string baseId = "00000000000xxxx";
string appKey = "00000000000xxxx";
var records = new List<AirtableRecord>();
using (AirtableBase airtableBase = new AirtableBase(appKey, baseId))
{
Task<AirtableListRecordsResponse> task = airtableBase.ListRecords(tableName: "efls", filterByFormula: ProductID);
AirtableListRecordsResponse response = await task;
if (!response.Success)
{
string errorMessage = null;
if (response.AirtableApiError is AirtableApiException)
{
errorMessage = response.AirtableApiError.ErrorMessage;
}
else
{
errorMessage = "Unknown error";
}
// Report errorMessage
}
else
{
records.AddRange(response.Records.ToList());
var record = response.Records;
//offset = response.Offset;
//var record = response.Record;
foreach (var item in record)
{
foreach (var Fields in item.Fields)
{
if (Fields.Key == "pdfUrl")
{
string link = Fields.Value.ToString();
MyLink = Fields.Value.ToString();
}
}
}
// Do something with your retrieved record.
// Such as getting the attachmentList of the record if you
// know the Attachment field name
//var attachmentList = response.Record.GetAttachmentField(YOUR_ATTACHMENT_FIELD_NAME);
}
return response;
}
}
This is the asynchronous method which asks for an asynchronous page, the other contains a strong structure and it cannot be changed for any reason. Is there any way to make them work together?
I am using airtable.com api by the way.
Thanks in advance.
I solved by my own,
The solution I found is the following:
When a page works with two different endpoints and one of them obligates the page to be asynchronous the best solution is to split the procedures into two different sections and/or pages, one of them will call the asynchronous methods and retrieves the info and other works without being asynchronous.
How can I pass the information between the sites?
Using session variables, there are endpoints which only needs to display simple data as in this case, so the session variables will be called in the page #2 which is the non-asynchronous page.
It is a simple solution but effective.
Thank you very much to all for you answers.
Using Wait on Task, you can use synchronous method
Task<AirtableListRecordsResponse> task = Task.Run(() => airtableBase.ListRecords(tableName: "efls", filterByFormula: ProductID));
task.Wait();
AirtableListRecordsResponse response = task.Result;
Use it only when you cannot use async method.
This method is completely deadlock free as mentioned on msdn blog-
https://blogs.msdn.microsoft.com/jpsanders/2017/08/28/asp-net-do-not-use-task-result-in-main-context/
I am importing data to Dynamics CRM using C# console application. I am using following code:
public static void Main(string[] args)
{
int totalRecords = dbcon.GetDataCount();
int rowCount = totalRecords / 10;
for (int i = 1, j = 1; i <= totalRecords; i = i + rowCount, j = j + 1)
{
Task myTask = new Task(() => TestMethod(i, (rowCount * j)));
myTask.Start();
}
Task.WaitAll();
}
public static void TestMethod(int startSeqNo, int endSeqNo)
{
IOrganizationService service = getServiceProxcy();
DBConnection dbcon = new DBConnection();
DataTable dt = dbcon.GetData(startSeqNo, endSeqNo);
// Insert Commented
BulkCreate(service, dt);
}
public static void BulkCreate(IOrganizationService service, DataTable dt)
{
// Create an ExecuteMultipleRequest object.
ExecuteMultipleRequest multipleRequest = new ExecuteMultipleRequest()
{
// Assign settings that define execution behavior: continue on error, return responses.
Settings = new ExecuteMultipleSettings()
{
ContinueOnError = false,
ReturnResponses = true
},
// Create an empty organization request collection.
Requests = new OrganizationRequestCollection()
};
foreach (DataRow row in dt.Rows)
{
Entity entity = new Entity("new_dataimporttest");
entity["new_name"] = row["name"].ToString();
entity["new_telephone"] = row["telephone1"].ToString();
if (multipleRequest.Requests.Count == 1000)
{
// Execute all the requests in the request collection using a single web method call.
ExecuteMultipleResponse multipleResponse = (ExecuteMultipleResponse)service.Execute(multipleRequest);
multipleRequest.Requests.Clear();
}
CreateRequest createRequest = new CreateRequest { Target = entity };
multipleRequest.Requests.Add(createRequest);
}
// Execute all the requests in the request collection using a single web method call.
if (multipleRequest.Requests.Count > 0)
{
ExecuteMultipleResponse multipleResponse = (ExecuteMultipleResponse)service.Execute(multipleRequest);
}
}
I am using Task Parallel Library. It works fine but issue is that when following line is executed it takes time.
// Execute all the requests in the request collection using a single web method call.
// ExecuteMultipleResponse multipleResponse = (ExecuteMultipleResponse)service.Execute(multipleRequest);
I want to improve performance of code as I am importing large amount of data nearly 1 million records. Currently it takes 1h 50 mins. How do I improve code to reduce execution time.
With the ExecuteMultipleRequest data throughput can only be enhanced in a limited way. This is because the Dynamics CRM server processes the requests in it in sequential order, not parallel. Therefore your main gain is less roundtrips to the server.
Throughput can really be boosted when working with multiple threads. Every thread communicating with CRM must get its own IOrganizationService instance. By default a CRM server accepts up to 10 simultaneous connections from a client. (This is the WCF default.)
In batch processes I tend to use a BlockingCollection<T> with a Producer Consumer pattern: one thread produces the requests to be sent to the CRM server and multiple threads consume the requests by taking them off the collection and sending them to CRM.
You could have two methods execute multiplerequest in the same process(500 for each), using that you can cut the time by half.
Analize in which part it is wasting more time in the execution or in the foreach create. And write here so I can't help you in a better way
If just data import is what intended here, try using SqlBulkCopy to write data directly to server (Sample Code).
I was trying to figure out some performance values of two scenarios. I thought I was only going to declare the obvious at first. But when I got the results I got a little confused. And now I am looking for a justification for the case.
I have a library which makes couple of queries through a MongoDb database and Active Directory services, then returns the results to client, which are:
GetUserType - to MongoDb - there is a collection which has username and type fields in its all documents. In the query I give the username and ask for the type field.
LoginCheck - to Active Directory - given the username and the password from the client, I create a PrincipalContext object to access to AD server and call ValidateCredentials upon it.
This job is performing on an existing MVC application at the moment. And we are going to create a new desktop application and employ it with the same job.
We were curios about how different can these two scenarios perform? We thought that a direct call to a library without any http connection would perform better than a service request without an hesitation. But we still wondered how much difference is there, and if it was acceptable we are going to make it work through the rest MVC service - because of reasons :)
Hence we tested out the following architectures:
Scenario 1:
Scenario 2:
Basically, what I do for performance test is this:
For scenario 1:
for(var i = 0; i<10000; i++)
{
new Class1().HeavyMethod();
}
For scenario 2:
// client side
for(var i = 0; i<10000; i++)
{
using ( var client = new HttpClient() )
{
var values = new Dictionary<string, string>();
var content = new FormUrlEncodedContent(values);
var response = client.PostAsync("http://localhost:654/Home/HeavyLift", content).Result;
var responseString = response.Content.ReadAsStringAsync().Result;
}
}
// MVC rest service
public class HomeController : Controller
{
public JsonResult HeavyLift()
{
return Json(new Class1().HeavyMethod(), JsonRequestBehavior.AllowGet);
}
}
Common Class:
public class Class1
{
public string HeavyMethod ()
{
var userName = "asdfasdfasd";
var password = "asdfasdfasdf";
try
{
// this call is to MongoDB
var userType = Personnel.GetPersonnelsType(userName).Result;
// this call is to Active Directory
var user = new ADUser(new Session
{
UserType = userType.Type,
UserName = userName,
Password = password
});
return userType.Type + "-" + user.Auth();
}
catch ( Exception e )
{
return e.Message;
}
}
}
The results for 10000 consecutive calls are confusingly shocking:
Scenario 1: 159181 ms
Scenario 2: 13952 ms
Scenario 1 starts off pretty quicly for the first few dosens of calls, then it starts to slow down.
Scenario 2 though offers a constant response time through 10k calls.
What is actually happening here?
Note: I checked the memory and cpu usages of the server that this scenarios runs on(everything runs on the same server) but there is nothing interesting actually, they are behaving just the same in terms of memory and cpu resources.
I am new to the WebApi, .Net world and am totally confused with all the information available as to what approach I should take. I have created a WebService using MVC4 WebApi that Twilio calls when a text message is received. I need to respond to this text message. I am consuming a WCF method which is currently being called synchronously. Since it is possible that my process can take longer than 3-5 seconds to process a reply to the text message the connection to Twilio gets disconnected due to timeout. So I am looking for ways to call this WCF method asynchronously.
My question is to call the WCF method (I am calling the WCF using a Object Factory and using)
do I need to update the contract to say Async? I am little confused on that.
BTW my Web Service is in IIS7 and am using .Net4.5 framework and MVC4 WebApi .
My code is somewhat like this: So I would like to call the SendSms part asynchronously. How do I do that? Can I simply use Task.Run Async and Await?
using Twilio.Mvc;
using Twilio.TwiML.Mvc;
using Twilio.TwiML;
public class SmsController : ApiController
{
[HttpPost]
public HttpResponseMessage Post([FromBody]SmsRequest smsReq)
{
var response = new Twilio.TwiML.TwilioResponse();
//validation checks..
try
{
-- call to WCF to get the List of sms to be sent
if ((txtMessageResponse != null) && (txtMessageResponse.SmsMessageInfo.Count > 0))
{
_smsStagingList = txtMessageResponse.SmsMessageInfo;
foreach (TextMessageStaging prepareTextMessageResponse in _smsStagingList)
{
smsDTO textMessageItems = new smsDTO();
textMessageItems.PhoneNumber = prepareTextMessageResponse.PhoneNumber;
textMessageItems.SmsMessage = prepareTextMessageResponse.SmsMessageBody;
isTxtMessageSent = SendSms(textMessageItems);
//If the messages were sent then no need to set the flag to be updated
if (isTxtMessageSent)
{
txtMessageStatusToBeUpdated = false;
}
}
return Request.CreateResponse(HttpStatusCode.OK, twilioResponse.Element);
}
else
{
//send error response
}
catch (Exception msgProcessingError)
{
//send error response again as processing error
}
finally
{
//set the outbound flag in the table
}
}
private bool SendSms(smsDTO textMessageItems)
{
bool isTxtMessageSent = false;
PushMessageRequest txtMessageRequest = new PushMessageRequest();
PushMessageResponse txtMessageResponse = null;
txtMessageRequest.SmsMessageInfo = new SendTextMessage(); //instantiate the dto
txtMessageRequest.SmsMessageInfo.ToPhone = textMessageItems.PhoneNumber;
txtMessageRequest.SmsMessageInfo.TextMessage = textMessageItems.SmsMessage;
try
{
using (ITextService textService = ObjectFactory.SendSmsMessage())
{
txtMessageResponse = textService.SendSmsMessage(txtMessageRequest);
}
isTxtMessageSent = txtMessageResponse.IsSuccessful;
}
catch (Exception ex)
{
isTxtMessageSent = false;
}
return isTxtMessageSent;
}
Twilio evangelist here.
OK, so you have a Web API method in which you call a WCF method that is potentially long running. There are two problems to solve here:
How do you call the WCF method in a way that does not block the Web API method from returning a response
How do you get Twilio to wait until the WCF method has finished
I wrote a blog post a while ago that shows you how to create an indefinite wait loop in an IVR by leveraging .NET's Task Parallel library and the loop attribute on Twilios <Play> verb.
The gist of the post is that you can use the TPL's StartNew method to start the long running WCF method on a new thread. This lets ASP.NET continue and lets you return some TwiML so Twilio does not end the call. Then you pair that with a continuation which lets you know when the WCF service request is done and you can signal back to Twilio using the REST API to redirect the in-progress call to a new set of TwiML instructions.
Hope that helps.