Hello everyone and thanks for helping me in advance. The following question might sound stupid and incorrect but I'm a beginner about it.
I have a method that gets some information from my database and sends it to an external database using a post call and a patch call in case the information has changed. I use EF Framework. In that db table there are at least 165k rows.
My question is the following: There is a way to optimize and speed up all the process? Maybe using multi threading, parallelism? I'm a beginner about it and I hope some of you help me understand.
The method is the following:
public async Task<List<dynamic>> SyncOrdersTaskAsync(int PageSize)
{
int PageIndex = 0;
if (PageSize <= 0) PageSize = 100;
const string phrase = "The fields order, task_code must make a unique set";
var sorting = new SortingCriteria {
Properties = new string[] { "WkOpenDate ASC" } };
List<dynamic> listTest = new List<dynamic>();
using (var uow = this.Factory.BeginUnitOfWork())
{
var repo = uow.GetRepository<IWorkOrderRepository>();
var count = await repo.CountAllAsync();
count = 150;
for (PageIndex = 0; PageIndex <= count / PageSize; PageIndex++)
{
var paging = new PagingCriteria
{
PageIndex = PageIndex,
PageSize = PageSize
};
var rows = await repo.GetByCriteriaAsync(
"new {WkID, CompanyID, JobNo, JobTaskNo ,WkNumber, WkYear," +
"WkYard,WkCustomerID,CuName,WkDivisionID,DvName,BusinessUnit," +
"BusinessUnitManagerID,BusinessUnitManager,WkWorkTypeID,WtName," +
"WkActivityID,WkActivityDescription,NoteDescrLavoro,WkWOManagerID," +
"ProjectManager,IDMaster,ProjectCoordinator,WkOpenDate," +
"WkDataChiusa,Prov,CodiceSito,CodiceOffice,CodiceLavorazione," +
"CodiceNodo,DescrizioneNodo,WkPrevisionalStartDate,WkRealStartDate," +
"WkPrevisionalEndDate,WkRealEndDate,NumeroOrdine," +
"WkPrevisionalLabourAmount,TotaleCosti,SumOvertimeHours," +
"SumTravelHours,SumNormalHours,WkProgressPercentage,Stato,CUP,CIG," +
"TotaleManodopera,TotalePrestazioni,TotaleNoli,TotaleMateriali," +
"SumAuxiliaryHours,TipoCommessa,TotaleOrdine, WkPreventivoData," +
"WkConsuntivoData,TotaleFatturato,AggregateTotaleFatturato," +
"AggregateTotalePrestazioni,Contract,CustomerOrderNumber," +
"XmeWBECode,LastUpdateDate,PreGestWkID,CommercialNotes,Mandant," +
"GammaProjectName,WkInventoryDate,WkCloseFlag,WkNote," +
"TotalRegisteredLabour,TotalRegisteredPerformances," +
"TotalRegisteredLeasings,TotalRegisteredMaterials,FlagFinalBalance," +
"FinalBalance,OrderDate,TotalOrderDivision,SearchDescription," +
"TotaleBefToBeApproved,TotaleBefToBeApprovedLeasings," +
"TotaleLabourToBeApproved,AggregateLevel, AggregateTotalLabour," +
"AggregateTotalLeasings,AggregateTotalMaterials," +
"AggregateTotalRegisteredLabour," +
"AggregateTotalRegisteredPerformances," +
"AggregateTotalRegisteredLeasings," +
"AggregateTotalRegisteredMaterials," +
"AggregateTotalCost,AggregateSumNormalHours," +
"AggregateSumAuxiliaryHours,AggregateSumRainHours," +
"AggregateSumTravelHours,AggregateSumOvertimeHours," +
"AggregateWkPrevisionalLabourAmount,AggregateFinalBalance," +
"AggregateTotalOrder,AggregateTotalOrderDivision," +
"AggregateTotalBefToBeApproved," +
"AggregateTotalBefToBeApprovedLeasings," +
"AggregateTotalLabourToBeApproved,TotalProduction," +
"AggregateTotalProduction,JobTaskDescription}", paging, sorting);
String url = appSettings.Value.UrlV1 + "order_tasks/";
using (var httpClient = new HttpClient())
{
httpClient.DefaultRequestHeaders.Add("Authorization", "Token " +
await this.GetApiKey(true));
if (rows.Count() > 0)
{
foreach (var row in rows)
{
var testWork = (Model.WorkOrderCompleteInfo)Mapper
.MapWkOrdersCompleteInfo(row);
var orderIdDiv = await this.GetOrderForSyncing(httpClient,
testWork.JobNo);
var jsonTest = new JObject();
jsonTest["task_code"] = testWork.JobTaskNo;
jsonTest["description"] = testWork.JobTaskDescription;
jsonTest["order"] = orderIdDivitel.Id;
jsonTest["order_date"] = testWork.OrderDate.HasValue
? testWork.OrderDate.Value.ToString("yyyy-MM-dd")
: string.IsNullOrEmpty(testWork.OrderDate.ToString())
? "1970-01-01"
: testWork.OrderDate.ToString().Substring(0, 10);
jsonTest["progress"] = testWork.WkProgressPercentage;
var content = new StringContent(jsonTest.ToString(),
Encoding.UTF8, "application/json");
var result = await httpClient.PostAsync(url, content);
if (result.Content != null)
{
var responseContent = await result.Content
.ReadAsStringAsync();
bool alreadyExists = phrase.All(responseContent.Contains);
if (alreadyExists)
{
var taskCase = await GetTaskForSyncing(httpClient,
testWork.JobTaskNo, orderIdDiv.Id.ToString());
var idCase = taskCase.Id;
String urlPatch = appSettings.Value.UrlV1 +
"order_tasks/" + idCase + "/";
bool isSame = taskCase.Equals(testWork
.toSolOrderTask());
if (!isSame)
{
var resultPatch = await httpClient.PatchAsync(
urlPatch, content);
if (resultPatch != null)
{
var responsePatchContent = await resultPatch
.Content.ReadAsStringAsync();
var jsonPatchContent = JsonConvert
.DeserializeObject<dynamic>(
responsePatchContent);
listTest.Add(jsonPatchContent);
}
}
else
{
listTest.Add(taskCase.JobTaskNo_ +
" is already updated!");
}
}
else
{
var jsonContent = JsonConvert
.DeserializeObject<dynamic>(responseContent);
listTest.Add(jsonContent);
}
}
}
}
}
}
return listTest;
}
}
Maybe I need to apply parallelism in the for loop?
Again, really thanks to everyone in advance and I hope I was clear :)
The most handy tool that is currently available for parallelizing asynchronous work is the Parallel.ForEachAsync method. It was introduced in .NET 6. Your code is quite complex though, and deciding where to put this loop is not obvious.
Ideally you would like to call the Parallel.ForEachAsync only once, so that it parallelizes your work with a single configurable degree of parallelism from start to finish. Generally you don't want to put this method inside an outer for/foreach loop, because then the degree of parallelism will fluctuate during the whole operation. But since your code is complex, I would go the easy way and do just that. I would replace this code:
foreach (var row in rows)
{
//...
}
...with this:
ParallelOptions options = new() { MaxDegreeOfParallelism = 2 };
await Parallel.ForEachAsync(rows, options, async (row, _) =>
{
//...
});
You have to make one more change. The List<T> is not thread safe, and so it will get corrupted if you call Add from multiple threads without synchronization. You can either add a lock (listTest) before each listTest.Add, or replace it with a concurrent collection. My suggestion is to do the later:
ConcurrentQueue<dynamic> listTest = new();
//...
listTest.Enqueue(jsonContent);
//...
return listTest.ToList();
After doing these changes, hopefully your code will still work correctly, and it will be running a bit faster. Then you'll have to experiment with the MaxDegreeOfParallelism setting, until you find the one that yields the optimal performance. Don't go crazy with large values like 100 or 1000. In most cases overparallelizing is harmful, and might yield worse performance than not parallelizing at all.
I am trying to get some data from a web page with HtmlAgilityPack which gets some variables and gives some results.
I want to retrieve 3 data fields from this webpage and so far I can only get the 2 of them.
My code so far
struct Result
{
public string Description;
public string thirdCountryDuty;
public string tarifPreference;
}
private Result LoadWebPage(string url, string taric)
{
//This is the webpage which contains all three datas that I want. I just write it here as
url for testing
url = "https://ec.europa.eu/taxation_customs/dds2/taric/measures.jsp?Lang=en&SimDate=20200503&Area=SG&MeasType=&StartPub=&EndPub=&MeasText=&GoodsText=&op=&Taric=6213900010&search_text=goods&textSearch=&LangDescr=el&OrderNum=&Regulation=&measStartDat=&measEndDat=%22;"
var result = new Result();
taric = "6213900010";//This is a variable. I give it here for testing purposes
txtEditCountry.Text = "SG";//This is a variable. I give it here for testing purposes
try
{
var web2 = new HtmlWeb();
var doc2 = web2.LoadFromBrowser(url, html =>
{
// WAIT until the dynamic text is set
return !html.Contains("<div id=\"" + taric.ToString() + "\"></div>");
});
//t1 is the data that I cannot get
var t1 = doc2.DocumentNode.SelectSingleNode("//span[contains(text(),'" + txtEditCountry.Text + "')] and .//span[contains(.,'duty_rate')]]").InnerText;
//This is working
var t2 = doc2.DocumentNode.SelectSingleNode("//*[contains(#id,'"+ taric + "')]/table/tbody/tr/td[2]/table/tbody/tr/td[2]").InnerText;
//This is working
var t3 = doc2.DocumentNode.SelectSingleNode("//span[contains(#class,'duty_rate')]").InnerText;
Console.WriteLine("Text 1: " + t1);
Console.WriteLine("Text 2: " + t2);
Console.WriteLine("Text 3: " + t3);
result = new Result
{
Description = t2,
thirdCountryDuty = t3,
tarifPreference = t1
};
return result;
}
catch (Exception ex)
{
result.Description= null;
result.thirdCountryDuty = null;
result.tarifPreference = null;
MessageBox.Show("Check your data and try again \n" + ex.ToString());
return result;
}
}
The data that I cannot get is t1 as I wrote in the code. This field is visible when I put a specific country in url "&Area=country code". If I put another country it will give me another number or 0%. If I don't put anything it will give me a list with all countries.
If I use this as Xpath
var t1 = doc2.DocumentNode.SelectSingleNode("//span[contains(text(),'" + txtEditXora.Text + "')]").InnerText;
It returns the country correct for example
Singapore (SG)
I want the tarif Preference percentage for this country
This is the first time that I use XPath and I am still learning but I wan't this for my project.
You can try this. I don't have time to check if it would work with other countries.
doc2.DocumentNode.SelectNodes("//div[#id='" + taric + "']//td[#name='measure_description_search']//td")[4].InnerText
Or this:
doc2.DocumentNode.SelectNodes("//div[#id='" + taric + "']//span[#class='duty_rate']")[1].InnerText
This should work
//text()[contains(.,"preference")]/../../td[2]
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I have multiple methods that do the following for a various different objects and I was wondering if there was a way to make this generic enough so that I don't need to write repetitive code.
public UpsertEntities(IList<MyEntity> entities) {
int totalImported = 0;
int totalRecords = entities.Count();
var options = new ParallelOptions { MaxDegreeOfParallelism = 8 };
var exceptions = new ConcurrentQueue<Exception>();
var errors = new ConcurrentBag<string>();
var batches = entities.ChunkBy(100);
foreach (var batch in batches)
{
var loopResult = Parallel.ForEach(batch, options, e =>
{
try
{
using (var context = GetContext())
{
context.SpecifiedEntityUpsert(e.Prop1, e.Prop2, e.Prop3, e.Prop4);
}
Interlocked.Increment(ref totalImported);
}
catch (Exception exception)
{
exceptions.Enqueue(exception);
errors.Add("Error Import " + e.Id + " " + exception.Message);
}
if (totalImported % 1000 == 0)
LoggingEngine.Instance.Info(Thread.CurrentThread.ManagedThreadId + " - " + " Imported " + totalImported + " of " + totalRecords + " records ");
});
}
foreach (var err in errors)
LoggingEngine.Instance.Error(err);
}
Thanks for any suggestions.
The part that are unique to each method is the method name, the parameter passed in and the following block of code:
using (var context = GetContext())
{
context.SpecifiedEntityUpsert(e.Prop1, e.Prop2, e.Prop3, e.Prop4);
}
The only part in the code shown that is specific to the type is the call to SpecifiedEntityUpsert. You could abstract this out of the method and delegate it to an Action parameter instead.
Call generic method
var myList = new List<MyEntity>();
UpsertEntities(myList, (context, e) => context.SpecifiedEntityUpsert(e.Prop1, e.Prop2, e.Prop3, e.Prop4));
Generic Method
// I made a guess that context is of type DbContext
public UpsertEntities<T>(IList<T> entities, Action<DbContext, T> upsert) where T : class {
int totalImported = 0;
int totalRecords = entities.Count();
var options = new ParallelOptions { MaxDegreeOfParallelism = 8 };
var exceptions = new ConcurrentQueue<Exception>();
var errors = new ConcurrentBag<string>();
var batches = entities.ChunkBy(100);
foreach (var batch in batches)
{
var loopResult = Parallel.ForEach(batch, options, e =>
{
try
{
using (var context = GetContext())
{
// call to action parameter
upsert(context, e);
}
Interlocked.Increment(ref totalImported);
}
catch (Exception exception)
{
exceptions.Enqueue(exception);
errors.Add("Error Import " + e.Id + " " + exception.Message);
}
if (totalImported % 1000 == 0)
LoggingEngine.Instance.Info(Thread.CurrentThread.ManagedThreadId + " - " + " Imported " + totalImported + " of " + totalRecords + " records ");
});
}
foreach (var err in errors)
LoggingEngine.Instance.Error(err);
}
I am a newbie to C# & am trying to use the Lync SDK to search for a Lync user programmatically to get their status. I am not sure how to pass the results to the webservice response when the async callback gets executed.
This is the code:
Webservice Controller GET endpoint:
// GET: api/Lync/5
public String Get(int id)
{
log.Info("[GET] Search for Lync User: " + id);
Lync lync = new Lync();
////lync.signIn();
Boolean isSignedIn = lync.isUserSignedIn();
if (isSignedIn)
{
log.Info("User is Signed In");
Console.Write("Enter search key : ");
lync.Search("medina");
//lync.Search("medina",
//(lyncContacts) =>
//{
// log.Debug("Search Results Callback fired!");
// log.Info("Results found: " + lyncContacts.Count);
// return lyncContacts.ToString();
//});
//Console.WriteLine(name);
//Console.ReadLine();
return "testUser";
}
else
{
log.Info("User is not Signed In!");
// TODO: Return status 500
return "testUser";
}
//Console.ReadLine();
//return "testUser";
}
The above method calls the business service lync.search(..) which is as follows:
public void Search(string searchKey)
{
List<LyncContact> contactList = new List<LyncContact>();
//List<ContactInformationType> ContactInformationList = new List<ContactInformationType>();
//ContactInformationList.Add(ContactInformationType.Activity);
//ContactInformationList.Add(ContactInformationType.Availability);
// ContactInformationList.Add(ContactInformationType.CapabilityString);
//ContactSubscription contactSubscription = LyncClient.GetClient().ContactManager.CreateSubscription();
Console.WriteLine("Searching for contacts on " + searchKey);
LyncClient.GetClient().ContactManager.BeginSearch(
searchKey,
(ar) =>
{
SearchResults searchResults = LyncClient.GetClient().ContactManager.EndSearch(ar);
if (searchResults.Contacts.Count > 0)
{
log.Info("Search results found: " + searchResults.Contacts.Count);
Console.WriteLine(searchResults.Contacts.Count.ToString() + " found");
foreach (Contact contact in searchResults.Contacts)
{
String displayName = contact.GetContactInformation(ContactInformationType.DisplayName).ToString();
ContactAvailability currentAvailability = (ContactAvailability)contact.GetContactInformation(ContactInformationType.Availability);
Console.WriteLine(
contact.GetContactInformation(ContactInformationType.DisplayName).ToString() + " " + contact.GetContactInformation(ContactInformationType.Availability).ToString());
log.Debug("Display Name: " + displayName);
log.Debug("Availability: " + currentAvailability);
LyncContact lyncContact = new LyncContact.Builder().DisplayName("Snehil").Availability("Busy").build();
contactList.Add(lyncContact);
//done(contactList);
}
return;
}
else
{
log.Info("No Results found!");
//done(contactList);
return;
}
},
null);
} else
{
log.Info("No Results found!");
//done(contactList);
return;
}
},
null);
}
I tried to use Task+await but I am having a hard time trying to figure out how can i return the results from the callback funtion as results from the search method. How do i capture this in the controller class?
If you would like to use the async/await pattern for this use Task.FromAsync https://msdn.microsoft.com/en-us/library/system.threading.tasks.taskfactory.fromasync(v=vs.110).aspx
Google for examples. Like this:
public async Task<List<LyncContact>> SearchAsync(string searchKey)
{
List<LyncContact> contactList = new List<LyncContact>();
Console.WriteLine("Searching for contacts on " + searchKey);
var cm = LyncClient.GetClient().ContactManager;
var searchResults = await Task<SearchResults>.Factory.FromAsync<String>(
cm.BeginSearch,
cm.EndSearch, searchKey, null);
if (searchResults.Contacts.Count > 0)
{
Console.WriteLine(searchResults.Contacts.Count.ToString() + " found");
foreach (Contact contact in searchResults.Contacts)
{
String displayName = contact.GetContactInformation(ContactInformationType.DisplayName).ToString();
ContactAvailability currentAvailability = (ContactAvailability)contact.GetContactInformation(ContactInformationType.Availability);
Console.WriteLine(
contact.GetContactInformation(ContactInformationType.DisplayName).ToString() + " " + contact.GetContactInformation(ContactInformationType.Availability).ToString());
LyncContact lyncContact = new LyncContact.Builder().DisplayName("Snehil").Availability("Busy").build();
contactList.Add(lyncContact);
}
}
return contactList
}
In the controller you then can do:
public async Task<String> Get(int id)
{
log.Info("[GET] Search for Lync User: " + id);
Lync lync = new Lync();
////lync.signIn();
Boolean isSignedIn = lync.isUserSignedIn();
if (isSignedIn)
{
log.Info("User is Signed In");
Console.Write("Enter search key : ");
var lyncContacts = await SearchAsync("medina");
log.Info("Results found: " + lyncContacts.Count);
return lyncContacts.ToString();
}
else
{
log.Info("User is not Signed In!");
// TODO: Return status 500
return "testUser";
}
//Console.ReadLine();
//return "testUser";
}
Or you can use a TaskCompletionSource, see http://blog.stephencleary.com/2012/07/async-interop-with-iasyncresult.html (older post but principle is still valid.
Warning
I cannot compile this since I do not have access to the libraries you use. It should work but there might be some compile errors
What does it do
Callbacks are not a nice model to work with, as you have found out. using Tasks and the async/await model are far easier to work with. Luckily for you and me they have created several methods that can act as a bridge between the old and the new model. Using Task.FromAsync you can transform IAsyncResult methods to an easier to use Task based methodology. Stephen Cleary has written some nice blogs about them. See the post I mentioned earlier.
It looks like, by looking at your commented out code, that you tried to provide some sort of callback to your Search function. That would work, it should be of type Action<Microsoft.Lync.Model.SearchResults>
public void Search(string searchKey, Action<SearchResults> callback)
{
....
LyncClient.GetClient().ContactManager.BeginSearch(
searchKey,
(ar) =>
{
SearchResults searchResults = LyncClient.GetClient().ContactManager.EndSearch(ar);
callback(searchResults);
});
....
}
Then just uncomment the code in your controller:
lync.Search("medina",
(lyncContacts) =>
{
log.Debug("Search Results Callback fired!");
log.Info("Results found: " + lyncContacts.AllResults.Count);
});
I have a console app that calls a web api and gets a list of services. It then loops through and makes calls to each service.
I have the following:
static int Main(string[] args)
{
...
Task.WaitAll(Process());
}
private static async Task BeginProcess()
{
using(HttpClientHandler handler = new HttpClientHandler())
{
handler.UseDefaultCredentials = true;
using(var client = new HttpClient(handler))
{
var response = client.GetStringAsync(_baseUrl);
List<Service> services = new List<Service>();
services = jss.Deserialize<List<Service>>(response.Result);
client.Timeout = new TimeSpan(0,3,0);
foreach(var service in services)
{
Console.WriteLine("Running " + service.Name);
var _serviceResponse = await client.PostAsync(_baseURL + service.Id.ToString(), null);
Console.WriteLine(service.Name + " responded with " + _serviceRepsonse.StatusCode);
}
}
}
}
Unfortunately, this code processes each service sequentially rather than making the calls in parallel. The problem is, I'm not sure how to make these calls run in parallel.
The answer can be found in Concurrency in C# cookbook :
static async Task<string> DownloadAllAsync(IEnumerable<string> urls){
var httpClient = new HttpClient();
var downloads = urls.select(url => httpClient.getStringAsync(url));
Task<string>[] tasks = downloads.ToArray(); //-> tasks are started
//now that you have an array of tasks you can wait for them all to finish
string[] htmlPages = await Task.WhenAll(tasks);
return string.Concat(htmlPages);
}
The key points here are :
arrange the code to obtain an array of tasks (different containers are also ok, it does not have to be an array)
use await Task.WhenAll(array of tasks);
While I've accepted an answer, I finally landed on this which I felt was more succinct.
await Task.WhenAll(services.Select(async s => {
Console.WriteLine("Running " + s.Name);
var _serviceResponse = await client.PostAsync(...);
Console.WriteLine(s.Name + " responded with " + _serviceResponse.StatusCode);
}));
as #SLaks mentioned, you will need to replace your loop with something along these lines...
var asyncTasks = new Dictionary<Service, Task>();
foreach(var service in services)
{
Console.WriteLine("Running " + service.Name);
asyncTasks.Add(service, client.PostAsync(_baseURL + service.Id.ToString(), null));
}
// All tasks are running, so wait for all of them to finish here
await Task.WhenAll(asyncTasks);
foreach(var service in asyncTasks.Keys) {
Console.WriteLine("Service " + service.Name + " returned " + syncTasks[service].Result);
}
Hope it helps.
Change this:
foreach(var service in services)
{
Console.WriteLine("Running " + service.Name);
var _serviceResponse = await client.PostAsync(_baseURL + service.Id.ToString(), null);
Console.WriteLine(service.Name + " responded with " + _serviceRepsonse.StatusCode);
}
to this:
var serviceCallTaskList = new List<Task<HttpResponseMessage>>();
foreach(var service in services)
{
Console.WriteLine("Running " + service.Name);
serviceCallTaskList.Add(client.PostAsync(_baseURL + service.Id.ToString(), null));
}
HttpResponseMessage[] responseArray = await Task.WhenAll(serviceCallTaskList);
for(int i = 0; i < responseArray.Length; i++)
{
Console.WriteLine(services[i].Name + " responded with " + responseArray[i].StatusCode);
}