Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I have multiple methods that do the following for a various different objects and I was wondering if there was a way to make this generic enough so that I don't need to write repetitive code.
public UpsertEntities(IList<MyEntity> entities) {
int totalImported = 0;
int totalRecords = entities.Count();
var options = new ParallelOptions { MaxDegreeOfParallelism = 8 };
var exceptions = new ConcurrentQueue<Exception>();
var errors = new ConcurrentBag<string>();
var batches = entities.ChunkBy(100);
foreach (var batch in batches)
{
var loopResult = Parallel.ForEach(batch, options, e =>
{
try
{
using (var context = GetContext())
{
context.SpecifiedEntityUpsert(e.Prop1, e.Prop2, e.Prop3, e.Prop4);
}
Interlocked.Increment(ref totalImported);
}
catch (Exception exception)
{
exceptions.Enqueue(exception);
errors.Add("Error Import " + e.Id + " " + exception.Message);
}
if (totalImported % 1000 == 0)
LoggingEngine.Instance.Info(Thread.CurrentThread.ManagedThreadId + " - " + " Imported " + totalImported + " of " + totalRecords + " records ");
});
}
foreach (var err in errors)
LoggingEngine.Instance.Error(err);
}
Thanks for any suggestions.
The part that are unique to each method is the method name, the parameter passed in and the following block of code:
using (var context = GetContext())
{
context.SpecifiedEntityUpsert(e.Prop1, e.Prop2, e.Prop3, e.Prop4);
}
The only part in the code shown that is specific to the type is the call to SpecifiedEntityUpsert. You could abstract this out of the method and delegate it to an Action parameter instead.
Call generic method
var myList = new List<MyEntity>();
UpsertEntities(myList, (context, e) => context.SpecifiedEntityUpsert(e.Prop1, e.Prop2, e.Prop3, e.Prop4));
Generic Method
// I made a guess that context is of type DbContext
public UpsertEntities<T>(IList<T> entities, Action<DbContext, T> upsert) where T : class {
int totalImported = 0;
int totalRecords = entities.Count();
var options = new ParallelOptions { MaxDegreeOfParallelism = 8 };
var exceptions = new ConcurrentQueue<Exception>();
var errors = new ConcurrentBag<string>();
var batches = entities.ChunkBy(100);
foreach (var batch in batches)
{
var loopResult = Parallel.ForEach(batch, options, e =>
{
try
{
using (var context = GetContext())
{
// call to action parameter
upsert(context, e);
}
Interlocked.Increment(ref totalImported);
}
catch (Exception exception)
{
exceptions.Enqueue(exception);
errors.Add("Error Import " + e.Id + " " + exception.Message);
}
if (totalImported % 1000 == 0)
LoggingEngine.Instance.Info(Thread.CurrentThread.ManagedThreadId + " - " + " Imported " + totalImported + " of " + totalRecords + " records ");
});
}
foreach (var err in errors)
LoggingEngine.Instance.Error(err);
}
Related
I'm working on an application that reads emails, extracts data from them and prints them in DataGrid (as alerts or notifications). The data is stored in a DB.
Here's the scenario:
A new email with data arrives, if a new alert does not arrive within a specific time slot(say within 5 minutes) from the same customer, with the opposite result(e.g. 1st alert was FAIL, 2nd SUCCESS => negation), the alert in the Datagrid is just updated.
Where is my problem:
In the part where I want to update the alert, a new alert is added, which causes a duplicate and the table is confused. And it does not update when new negation alert arrives.
NOTICE:
My application is larger, to better demonstrate my problem, the example is in the consol application.
But with a few modifications, the code is the same
Here is my code:
Method, which read and extract data
public void MailKitLib(EmailParser emailParser)
{
using (var client = new ImapClient())
{
using (var cancel = new CancellationTokenSource())
{
client.Connect(emailParser.ServerName, emailParser.Port, emailParser.isSSLuse,
cancel.Token);
client.Authenticate(emailParser.Username, emailParser.Password, cancel.Token);
var inbox = client.Inbox;
inbox.Open(FolderAccess.ReadOnly, cancel.Token);
Console.WriteLine("Total messages: {0}", inbox.Count);
Console.WriteLine("Recent messages: {0}", inbox.Unread);
for (int i = 0; i < inbox.Count; i++)
{
var message = inbox.GetMessage(i, cancel.Token);
MyAlert alert = new MyAlert(message.MessageId, message.Date.DateTime, message.Subject, message.Subject);
if (!MyAlert.alerts.Any(x => x.Id.Equals(alert.Id)))
{
MyAlert.alerts.Add(alert);\\Think of it as a method that saves the object to the database
}
CheckForUpdates(MyAlert.alerts[i]);
}
}
client.Disconnect(true);
}
}
Method checking for updates
public void CheckForUpdates(MyAlert alert)
{
double result = 0;
foreach (var item in MyAlert.alerts.Select(x => x.NameOfCustomer.Equals(alert.NameOfCustomer)))
{
if (item)
{
for (int i = 0; i < MyAlert.alerts.Count(); i++)
{
result = MyAlert.alerts[i].Date.Minute - alert.Date.Minute;
Console.WriteLine("Result of:" + MyAlert.alerts[i].NameOfCustomer + " " +
"and" + " " + alert.NameOfCustomer + " " + "is:" + result);
if (result > 0 && result <= 5)
{
Console.WriteLine("You HAVE TO UPDATE this" + " " + MyAlert.alerts[i].NameOfAlert);
MyAlert.alerts[i].NameOfAlert = "UPDATED";
break;
}
else
{
Console.WriteLine("You DON'T have to UPDATE this:" + " " + MyAlert.alerts[i].NameOfAlert);
}
}
}
}
Console.WriteLine("Wave:" + MyAlert.alerts.Count);
}
This is how I figured this out with DB
public double GetInterval(DateTime a, DateTime b)
{
return a.Subtract(b).TotalMinutes;
}
public void FindUpdate(Alert newAlert, Alert matchAlert)
{
//If new Alerts arrives from same customer with same problem till five 5 minutes
if (dAOAlert.GetAll().Any(x => x.Email.Equals(newAlert.Email) && x.Problem.NameOfAlert.Equals(newAlert.Problem.NameOfAlert))
&&
GetInterval(newAlert.Date, matchAlert.Date) > 0 && GetInterval(newAlert.Date, matchAlert.Date) <= 5)
{
//Find that Alert, and update his variables from alert, else save new alert
var item = dAOAlert.GetAll().FirstOrDefault(x => x.Email.Equals(newAlert.Email) && x.Problem.NameOfAlert.Equals(newAlert.Problem.NameOfAlert));
dAOAlert.Update(item, newAlert);
dAOAlert.Delete(newAlert);
LoadAlertGrid();
}
}
And this is how I have used this in Main method
for (int i = 0; i < inbox.Count; i++)
{
var message = inbox.GetMessage(i, cancel.Token);
GetBodyText = message.TextBody;
Problem problem = new Problem(message.MessageId);
if (!dAOProblem.GetAll().Any(x => x.Message_Id.Equals(problem.Message_Id)))
{
dAOProblem.Save(problem);
Alert alert = new Alert(message.MessageId, message.Date.DateTime, message.From.ToString(), 1, problem.Id);
if (!dAOAlert.GetAll().Any(x => x.Id_MimeMessage.Equals(alert.Id_MimeMessage)))
{
dAOAlert.Save(alert);
foreach (var item in dAOAlert.GetAll())
{
FindUpdate(alert, item);
}
LoadAlertGrid();
}
else
{
MessageBox.Show("Duplicate");
}
}
}
Hello everyone and thanks for helping me in advance. The following question might sound stupid and incorrect but I'm a beginner about it.
I have a method that gets some information from my database and sends it to an external database using a post call and a patch call in case the information has changed. I use EF Framework. In that db table there are at least 165k rows.
My question is the following: There is a way to optimize and speed up all the process? Maybe using multi threading, parallelism? I'm a beginner about it and I hope some of you help me understand.
The method is the following:
public async Task<List<dynamic>> SyncOrdersTaskAsync(int PageSize)
{
int PageIndex = 0;
if (PageSize <= 0) PageSize = 100;
const string phrase = "The fields order, task_code must make a unique set";
var sorting = new SortingCriteria {
Properties = new string[] { "WkOpenDate ASC" } };
List<dynamic> listTest = new List<dynamic>();
using (var uow = this.Factory.BeginUnitOfWork())
{
var repo = uow.GetRepository<IWorkOrderRepository>();
var count = await repo.CountAllAsync();
count = 150;
for (PageIndex = 0; PageIndex <= count / PageSize; PageIndex++)
{
var paging = new PagingCriteria
{
PageIndex = PageIndex,
PageSize = PageSize
};
var rows = await repo.GetByCriteriaAsync(
"new {WkID, CompanyID, JobNo, JobTaskNo ,WkNumber, WkYear," +
"WkYard,WkCustomerID,CuName,WkDivisionID,DvName,BusinessUnit," +
"BusinessUnitManagerID,BusinessUnitManager,WkWorkTypeID,WtName," +
"WkActivityID,WkActivityDescription,NoteDescrLavoro,WkWOManagerID," +
"ProjectManager,IDMaster,ProjectCoordinator,WkOpenDate," +
"WkDataChiusa,Prov,CodiceSito,CodiceOffice,CodiceLavorazione," +
"CodiceNodo,DescrizioneNodo,WkPrevisionalStartDate,WkRealStartDate," +
"WkPrevisionalEndDate,WkRealEndDate,NumeroOrdine," +
"WkPrevisionalLabourAmount,TotaleCosti,SumOvertimeHours," +
"SumTravelHours,SumNormalHours,WkProgressPercentage,Stato,CUP,CIG," +
"TotaleManodopera,TotalePrestazioni,TotaleNoli,TotaleMateriali," +
"SumAuxiliaryHours,TipoCommessa,TotaleOrdine, WkPreventivoData," +
"WkConsuntivoData,TotaleFatturato,AggregateTotaleFatturato," +
"AggregateTotalePrestazioni,Contract,CustomerOrderNumber," +
"XmeWBECode,LastUpdateDate,PreGestWkID,CommercialNotes,Mandant," +
"GammaProjectName,WkInventoryDate,WkCloseFlag,WkNote," +
"TotalRegisteredLabour,TotalRegisteredPerformances," +
"TotalRegisteredLeasings,TotalRegisteredMaterials,FlagFinalBalance," +
"FinalBalance,OrderDate,TotalOrderDivision,SearchDescription," +
"TotaleBefToBeApproved,TotaleBefToBeApprovedLeasings," +
"TotaleLabourToBeApproved,AggregateLevel, AggregateTotalLabour," +
"AggregateTotalLeasings,AggregateTotalMaterials," +
"AggregateTotalRegisteredLabour," +
"AggregateTotalRegisteredPerformances," +
"AggregateTotalRegisteredLeasings," +
"AggregateTotalRegisteredMaterials," +
"AggregateTotalCost,AggregateSumNormalHours," +
"AggregateSumAuxiliaryHours,AggregateSumRainHours," +
"AggregateSumTravelHours,AggregateSumOvertimeHours," +
"AggregateWkPrevisionalLabourAmount,AggregateFinalBalance," +
"AggregateTotalOrder,AggregateTotalOrderDivision," +
"AggregateTotalBefToBeApproved," +
"AggregateTotalBefToBeApprovedLeasings," +
"AggregateTotalLabourToBeApproved,TotalProduction," +
"AggregateTotalProduction,JobTaskDescription}", paging, sorting);
String url = appSettings.Value.UrlV1 + "order_tasks/";
using (var httpClient = new HttpClient())
{
httpClient.DefaultRequestHeaders.Add("Authorization", "Token " +
await this.GetApiKey(true));
if (rows.Count() > 0)
{
foreach (var row in rows)
{
var testWork = (Model.WorkOrderCompleteInfo)Mapper
.MapWkOrdersCompleteInfo(row);
var orderIdDiv = await this.GetOrderForSyncing(httpClient,
testWork.JobNo);
var jsonTest = new JObject();
jsonTest["task_code"] = testWork.JobTaskNo;
jsonTest["description"] = testWork.JobTaskDescription;
jsonTest["order"] = orderIdDivitel.Id;
jsonTest["order_date"] = testWork.OrderDate.HasValue
? testWork.OrderDate.Value.ToString("yyyy-MM-dd")
: string.IsNullOrEmpty(testWork.OrderDate.ToString())
? "1970-01-01"
: testWork.OrderDate.ToString().Substring(0, 10);
jsonTest["progress"] = testWork.WkProgressPercentage;
var content = new StringContent(jsonTest.ToString(),
Encoding.UTF8, "application/json");
var result = await httpClient.PostAsync(url, content);
if (result.Content != null)
{
var responseContent = await result.Content
.ReadAsStringAsync();
bool alreadyExists = phrase.All(responseContent.Contains);
if (alreadyExists)
{
var taskCase = await GetTaskForSyncing(httpClient,
testWork.JobTaskNo, orderIdDiv.Id.ToString());
var idCase = taskCase.Id;
String urlPatch = appSettings.Value.UrlV1 +
"order_tasks/" + idCase + "/";
bool isSame = taskCase.Equals(testWork
.toSolOrderTask());
if (!isSame)
{
var resultPatch = await httpClient.PatchAsync(
urlPatch, content);
if (resultPatch != null)
{
var responsePatchContent = await resultPatch
.Content.ReadAsStringAsync();
var jsonPatchContent = JsonConvert
.DeserializeObject<dynamic>(
responsePatchContent);
listTest.Add(jsonPatchContent);
}
}
else
{
listTest.Add(taskCase.JobTaskNo_ +
" is already updated!");
}
}
else
{
var jsonContent = JsonConvert
.DeserializeObject<dynamic>(responseContent);
listTest.Add(jsonContent);
}
}
}
}
}
}
return listTest;
}
}
Maybe I need to apply parallelism in the for loop?
Again, really thanks to everyone in advance and I hope I was clear :)
The most handy tool that is currently available for parallelizing asynchronous work is the Parallel.ForEachAsync method. It was introduced in .NET 6. Your code is quite complex though, and deciding where to put this loop is not obvious.
Ideally you would like to call the Parallel.ForEachAsync only once, so that it parallelizes your work with a single configurable degree of parallelism from start to finish. Generally you don't want to put this method inside an outer for/foreach loop, because then the degree of parallelism will fluctuate during the whole operation. But since your code is complex, I would go the easy way and do just that. I would replace this code:
foreach (var row in rows)
{
//...
}
...with this:
ParallelOptions options = new() { MaxDegreeOfParallelism = 2 };
await Parallel.ForEachAsync(rows, options, async (row, _) =>
{
//...
});
You have to make one more change. The List<T> is not thread safe, and so it will get corrupted if you call Add from multiple threads without synchronization. You can either add a lock (listTest) before each listTest.Add, or replace it with a concurrent collection. My suggestion is to do the later:
ConcurrentQueue<dynamic> listTest = new();
//...
listTest.Enqueue(jsonContent);
//...
return listTest.ToList();
After doing these changes, hopefully your code will still work correctly, and it will be running a bit faster. Then you'll have to experiment with the MaxDegreeOfParallelism setting, until you find the one that yields the optimal performance. Don't go crazy with large values like 100 or 1000. In most cases overparallelizing is harmful, and might yield worse performance than not parallelizing at all.
This question already has answers here:
DbUpdateException: Which field is causing "String or binary data would be truncated"
(4 answers)
Closed 12 months ago.
How do you determine which column is the culprit when you have 80(+/-) columns to choose from? Using .Net Core (netcoreapp2.2) and EF Core 2.2.4.
Picked up some existing code and there was an attempt to track columns that failed. However, it does not work. I've looked at dozens of examples here and elsewhere and have not found a way to do this in EF Core 2.x.
public int GetColumnMaxLength(string table, EntityEntry entityEntry)
{
// Just a rough to get the right data - always returns 0 for the moment...
int result = 0;
var modelContext = entityEntry.Context;
var entityType = modelContext.Model.FindEntityType(table); // THIS IS ALWAYS NULL!
if (entityType != null)
{
// Table info
var tableName = entityType.Relational().TableName;
var tableSchema = entityType.Relational().Schema;
// Column info
foreach (var property in entityType.GetProperties())
{
var columnName = property.Relational().ColumnName;
var columnType = property.Relational().ColumnType;
var isFixedLength = property.Relational().IsFixedLength;
};
}
return result;
}
The above code is being called by this catch portion of a try/catch around the db.SaveAsync(); statement.
catch (Exception ex)
{
// -----------------------------------------
// no idea what this was really trying to
// do as it barfs out all columns...
// -----------------------------------------
var dataInfo = new DataInfo();
var strLargeValues = new List<Tuple<int, string, string, string>>();
foreach (var entityEntry in _db.ChangeTracker.Entries().Where(et => et.State != EntityState.Unchanged))
{
// -----------------------------------------
// try to get the column info for all
// columns on this table...
// -----------------------------------------
dataInfo.GetColumnMaxLength("Subscription", entityEntry);
foreach (var entry in entityEntry.CurrentValues.Properties)
{
var value = entry.PropertyInfo.GetValue(entityEntry.Entity);
if (value is string s)
{
strLargeValues.Add(Tuple.Create(s.Length, s, entry.Name, entityEntry.Entity.GetType().Name));
}
}
}
var l = strLargeValues.OrderByDescending(v => v.Item1).ToArray();
foreach (var x in l.Take(100))
{
Trace.WriteLine(x.Item4 + " - " + x.Item3 + " - " + x.Item1 + ": " + x.Item2);
}
throw;
}
So, the crux of the question is: How do I get the SQL column definition from EF Core?
I want to be able to log the specific table and column when incomingData.Length > targetColumnDefinition.Length
FINAL SOLUTION:
public override int SaveChanges()
{
using (LogContext.PushProperty("DbContext:Override:Save", nameof(SaveChanges)))
{
try
{
return base.SaveChanges();
}
catch (Exception ex)
{
var errorMessage = String.Empty;
var token = Environment.NewLine;
foreach (var entityEntry in this.ChangeTracker.Entries().Where(et => et.State != EntityState.Unchanged))
{
foreach (var entry in entityEntry.CurrentValues.Properties)
{
var result = entityEntry.GetDatabaseDefinition(entry.Name);
var value = entry.PropertyInfo.GetValue(entityEntry.Entity);
if (result.IsFixedLength && value.ToLength() > result.MaxLength)
{
errorMessage = $"{errorMessage}{token}ERROR!! <<< {result.TableName}.{result.ColumnName} {result.ColumnType.ToUpper()} :: {entry.Name}({value.ToLength()}) = {value} >>>";
Log.Warning("Cannot save data to SQL column {TableName}.{ColumnName}! Max length is {LengthTarget} and you are trying to save something that is {LengthSource}. Column definition is {ColumnType}"
, result.TableName
, result.ColumnName
, result.MaxLength
, value.ToLength()
, result.ColumnType);
}
}
}
throw new Exception(errorMessage, ex);
}
}
}
On .NET Core 3.1 and EFCore 5.0.2 this logging works with no additional extension methods needed:
try
{
await context.SaveChangesAsync();
}
catch(Exception ex)
{
foreach (var entityEntry in context.ChangeTracker.Entries().Where(et => et.State != EntityState.Unchanged))
{
foreach (var entry in entityEntry.CurrentValues.Properties)
{
var prop = entityEntry.Property(entry.Name).Metadata;
var value = entry.PropertyInfo?.GetValue(entityEntry.Entity);
var valueLength = value?.ToString()?.Length;
var typemapping = prop.GetTypeMapping();
var typeSize = ((Microsoft.EntityFrameworkCore.Storage.RelationalTypeMapping) typemapping).Size;
if (typeSize.HasValue && valueLength > typeSize.Value)
{
Log.Error( $"Truncation will occur: {entityEntry.Metadata.GetTableName()}.{prop.GetColumnName()} {prop.GetColumnType()} :: {entry.Name}({valueLength}) = {value}");
}
}
}
throw ex;
}
As mentioned in the comments, you need the full name and this can be read from the metadata.
public int GetColumnMaxLength(EntityEntry entityEntry)
{
int result = 0;
var table = entityEntry.Metadata.Model.FindEntityType(entityEntry.Metadata.ClrType);
// Column info
foreach (var property in table.GetProperties())
{
var maxLength = property.GetMaxLength();
// For sql info, e.g. ColumnType = nvarchar(255):
var sqlInfo = property.SqlServer();
};
return result;
}
I have a method like so...
static async Task GetLeads(ForceClient client)
{
Console.WriteLine("Get Leads");
var accts = new List<Lead>();
var results = await client.QueryAsync<Lead>(Lead._select);
var totalSize = results.TotalSize;
Console.WriteLine("Queried " + totalSize + " leads.");
accts.AddRange(results.Records);
Console.WriteLine("Added " + results.Records.Count + " leads...");
var nextRecordsUrl = results.NextRecordsUrl;
if (!string.IsNullOrEmpty(nextRecordsUrl))
{
Console.WriteLine("Found more records...");
while (true)
{
var continuationResults = await client.QueryContinuationAsync<Lead>(nextRecordsUrl);
Console.WriteLine("Queried an additional " + continuationResults.Records.Count + " leads.");
accts.AddRange(continuationResults.Records);
if (string.IsNullOrEmpty(continuationResults.NextRecordsUrl)) break;
nextRecordsUrl = continuationResults.NextRecordsUrl;
}
}
Upsert(accts, Lead.target);
}
I have another method like so..
static async Task GetSupplierProducts(ForceClient client)
{
Console.WriteLine("Get SupplierProduct");
var accts = new List<SupplierProduct>();
var results = await client.QueryAsync<SupplierProduct>(SupplierProduct._select);
var totalSize = results.TotalSize;
Console.WriteLine("Queried " + totalSize + " SupplierProduct.");
accts.AddRange(results.Records);
Console.WriteLine("Added " + results.Records.Count + " SupplierProduct...");
var nextRecordsUrl = results.NextRecordsUrl;
if (!string.IsNullOrEmpty(nextRecordsUrl))
{
Console.WriteLine("Found more records...");
while (true)
{
var continuationResults = await client.QueryContinuationAsync<SupplierProduct>(nextRecordsUrl);
Console.WriteLine("Queried an additional " + continuationResults.Records.Count + " SupplierProduct.");
accts.AddRange(continuationResults.Records);
if (string.IsNullOrEmpty(continuationResults.NextRecordsUrl)) break;
nextRecordsUrl = continuationResults.NextRecordsUrl;
}
}
Upsert(accts, SupplierProduct.target);
}
How can I make a method that abstracts this behavior generically?
The types Lead and SupplierProduct must somehow be related in the sense that they either implement the same interface or inherit from the same base class to make this work. Apparently the similarities are _select which apparently is a static member which cannot be included in an interface. Furthermore, the generation of human-readable strings would have to be refactored into the types.
If Base would be hypothetical base class, the signature of the generic function would have to be as follows.
static async Task Get<T>(ForceClient client) where T : Base
Assuming both Lead and SupplierProduct inherits the same Parent:
static async Task GetMyInstance<T>(ForceClient client) where T : Parent
{
Console.WriteLine("Get " + T.GetType().Name);
var accts = new List<T>();
var results = await client.QueryAsync<T>(T._select);
var totalSize = results.TotalSize;
Console.WriteLine("Queried " + totalSize + " " + T.GetType().Name +".");
accts.AddRange(results.Records);
Console.WriteLine("Added " + results.Records.Count + T.GetType().Name + "...");
var nextRecordsUrl = results.NextRecordsUrl;
if (!string.IsNullOrEmpty(nextRecordsUrl))
{
Console.WriteLine("Found more records...");
while (true)
{
var continuationResults = await client.QueryContinuationAsync<T>(nextRecordsUrl);
Console.WriteLine("Queried an additional " + continuationResults.Records.Count + " " + T.GetType().Name + ".");
accts.AddRange(continuationResults.Records);
if (string.IsNullOrEmpty(continuationResults.NextRecordsUrl)) break;
nextRecordsUrl = continuationResults.NextRecordsUrl;
}
}
Upsert(accts, T.target);
}
Please note that Parent should contain _select and target for this to work
And you call it like this:
var foo = GetMyInstance<Lead>(forceClient);
or
var foo = GetMyInstance<SupplierProduct>(forceClient);
This likely won't suffice fully, but when I want to make multiple partial-similar methods accessible through a single function, I tend to pass target-strings and use a repeating switch-statement on them.
This way we can also run multiple actions after each other by supplying an array of targets.
Note; this code isn't debugged or anything, I just wanted to point you towards how switch-statements might be of some use to you. It wasn't possible for me to give a more comprehensive answer because I can't fully understand the intent of your code.
static async Task GetRecordsFor(ForceClient client, string[] targets )
{
foreach (string target in targets){
switch ( target )
{
case 'leads':
Console.WriteLine("Get Leads");
var accts = new List<Lead>();
// more specific code for fetching leads
break;
case 'suppliers':
Console.WriteLine("Get SupplierProduct");
var accts = new List<SupplierProduct>();
// more specific code for fetching suppliers
break;
}
// Actions you want to perform on each of these.
accts.AddRange(continuationResults.Records);
}
}
I have a console app that calls a web api and gets a list of services. It then loops through and makes calls to each service.
I have the following:
static int Main(string[] args)
{
...
Task.WaitAll(Process());
}
private static async Task BeginProcess()
{
using(HttpClientHandler handler = new HttpClientHandler())
{
handler.UseDefaultCredentials = true;
using(var client = new HttpClient(handler))
{
var response = client.GetStringAsync(_baseUrl);
List<Service> services = new List<Service>();
services = jss.Deserialize<List<Service>>(response.Result);
client.Timeout = new TimeSpan(0,3,0);
foreach(var service in services)
{
Console.WriteLine("Running " + service.Name);
var _serviceResponse = await client.PostAsync(_baseURL + service.Id.ToString(), null);
Console.WriteLine(service.Name + " responded with " + _serviceRepsonse.StatusCode);
}
}
}
}
Unfortunately, this code processes each service sequentially rather than making the calls in parallel. The problem is, I'm not sure how to make these calls run in parallel.
The answer can be found in Concurrency in C# cookbook :
static async Task<string> DownloadAllAsync(IEnumerable<string> urls){
var httpClient = new HttpClient();
var downloads = urls.select(url => httpClient.getStringAsync(url));
Task<string>[] tasks = downloads.ToArray(); //-> tasks are started
//now that you have an array of tasks you can wait for them all to finish
string[] htmlPages = await Task.WhenAll(tasks);
return string.Concat(htmlPages);
}
The key points here are :
arrange the code to obtain an array of tasks (different containers are also ok, it does not have to be an array)
use await Task.WhenAll(array of tasks);
While I've accepted an answer, I finally landed on this which I felt was more succinct.
await Task.WhenAll(services.Select(async s => {
Console.WriteLine("Running " + s.Name);
var _serviceResponse = await client.PostAsync(...);
Console.WriteLine(s.Name + " responded with " + _serviceResponse.StatusCode);
}));
as #SLaks mentioned, you will need to replace your loop with something along these lines...
var asyncTasks = new Dictionary<Service, Task>();
foreach(var service in services)
{
Console.WriteLine("Running " + service.Name);
asyncTasks.Add(service, client.PostAsync(_baseURL + service.Id.ToString(), null));
}
// All tasks are running, so wait for all of them to finish here
await Task.WhenAll(asyncTasks);
foreach(var service in asyncTasks.Keys) {
Console.WriteLine("Service " + service.Name + " returned " + syncTasks[service].Result);
}
Hope it helps.
Change this:
foreach(var service in services)
{
Console.WriteLine("Running " + service.Name);
var _serviceResponse = await client.PostAsync(_baseURL + service.Id.ToString(), null);
Console.WriteLine(service.Name + " responded with " + _serviceRepsonse.StatusCode);
}
to this:
var serviceCallTaskList = new List<Task<HttpResponseMessage>>();
foreach(var service in services)
{
Console.WriteLine("Running " + service.Name);
serviceCallTaskList.Add(client.PostAsync(_baseURL + service.Id.ToString(), null));
}
HttpResponseMessage[] responseArray = await Task.WhenAll(serviceCallTaskList);
for(int i = 0; i < responseArray.Length; i++)
{
Console.WriteLine(services[i].Name + " responded with " + responseArray[i].StatusCode);
}