I have a list of headlines that I am trying to process by using a Task that updates a parameter of the headline object. The code I am trying to do it with does not actually populate the parameter properly. When I debug it, I can see that the setters are being activated and properly updating the backing fields, but when examined after Task.WhenAll , none of the properties are in fact set to their expected values.
//**Relevant signatures are:**
class Headline{
public Uri Uri { get; set; }
public IEnumerable<string> AttachedEmails { get; set; } = Enumerable.Empty<string>();
}
async Task<IEnumerable<string>> GetEmailsFromHeadline(Uri headlineUri) {
//bunch of async fetching logic that populates emailNodes correctly
return emailNodes.Select(e => e.InnerText).ToList();
}
//**Problem Area**
//Initiate tasks that start fetching information
var taskList =
postData
.Select(e => new HttpRequest() { Uri = actionUri, PostData = e })
.Select(e => Task.Run(() => GetHeadlines(e)))
.ToList();
//Wait till complete
await Task.WhenAll(taskList);
//Flatten list
var allHeadlines =
taskList
.SelectMany(e => e.Result.ToList());
//After this section of code I expect every member AttachedEmails property to be properly updated but this is not what is happening.
var headlineProcessTaskList =
allHeadlines
.Select(e => Task.Run( new Func<Task>( async () => e.AttachedEmails = await GetEmailsFromHeadline(e.Uri) ) ) )
.ToList();
await Task.WhenAll(headlineProcessTaskList);
There isn't enough code to reproduce the problem. However, we can guess that:
allHeadlines is IEnumerable<Headline>
Since this is IEnumerable<T>, it is possible that it is a LINQ query that has been deferred. Thus, when you enumerate it once in the ToList call, it creates Headline instances that have their AttachedEmails set. But when you enumerate it again later, it creates new Headline instances.
If this is correct, then one solution would be to change the type of allHeadlines to List<Headline>.
A similar problem could occur in GetEmailsFromHeadline, which presumably returns Task<IEnumerable<string>> and not IEnumerable<string> as stated. If the enumerable is deferred, then the only asynchronous part is defining the query; executing it would be after the fact - more specifically, outside the Task.Run. You might want to consider using ToList there, too.
On a side note, the new Func is just noise, and the wrapping of an asynchronous task in Task.Run is quite unusual. This would be more idiomatic:
var headlineProcessTaskList =
allHeadlines
.Select(async e => e.AttachedEmails = await GetEmailsFromHeadline(e.Uri) )
.ToList();
or, if the Task.Run is truly necessary:
var headlineProcessTaskList =
allHeadlines
.Select(e => Task.Run( () => e.AttachedEmails = (await GetEmailsFromHeadline(e.Uri)).ToList() ) )
.ToList();
Related
I have these example code:
private async Task<IEnumerable<long>> GetValidIds1(long[] ids)
{
var validIds = new List<long>();
var results = await Task.WhenAll(ids.Select(i => CheckValidIdAsync(i)))
.ConfigureAwait(false);
for (int i = 0; i < ids.Length; i++)
{
if (results[i])
{
validIds.Add(ids[i]);
}
}
return validIds;
}
private async Task<IEnumerable<long>> GetValidIds2(long[] ids)
{
var validIds = new ConcurrentBag<long>();
await Task.WhenAll(ids.Select(async i =>
{
var valid = await CheckValidIdAsync(i);
if (valid)
validIds.Add(i);
})).ConfigureAwait(false);
return validIds;
}
private async Task<bool> CheckValidIdAsync(long id);
I currently use GetValidIds1() but it has inconvenience of having to tie input ids to result using index at the end.
GetValidIds2() is what i want to write but there are a few concerns:
I have 'await' in select lambda expression. Because LINQ is lazy evaluation, I don't think it would block other CheckValidIdAsync() calls from starting but exactly who's context does it suspend? Per MSDN doc
The await operator suspends evaluation of the enclosing async method until the asynchronous operation represented by its operand completes.
So in this case, the enclosing async method is lambda expression itself so it doesn't affect other calls?
Is there a better way to process result of async method and collect output of that process in a list?
Another way to do it is to project each long ID to a Task<ValueTuple<long, bool>>, instead of projecting it to a Task<bool>. This way you'll be able to filter the results using pure LINQ:
private async Task<long[]> GetValidIds3(long[] ids)
{
IEnumerable<Task<(long Id, bool IsValid)>> tasks = ids
.Select(async id =>
{
bool isValid = await CheckValidIdAsync(id).ConfigureAwait(false);
return (id, isValid);
});
var results = await Task.WhenAll(tasks).ConfigureAwait(false);
return results
.Where(e => e.IsValid)
.Select(e => e.Id)
.ToArray();
}
The above GetValidIds3 is equivalent with the GetValidIds1 in your question. It returns the filtered IDs in the same order as the original ids. On the contrary the GetValidIds2 doesn't guarantee any order. If you have to use a concurrent collection, it's better to use a ConcurrentQueue<T> instead of a ConcurrentBag<T>, because the former preserves the insertion order. Even if the order is not important, preserving it makes the debugging easier.
I have a method that returns a object. In my parent function I have a list of IDs.
I would like to call the method for each ID I have and then have the objects added to a list. Right now I have written a loop that calls the method passing each ID and waits for the returned object and then goes to the next ID.
Can this be done in parallel? Any help here would be most helpful.
Something like this maybe:
List<int> ids = new List<int>();
List<object> result = new List<object>();
Parallel.ForEach(ids, (id, state, index) => {
result.Add(new { Id = id }); // You class instance here.
});
I think Task parallel libraries will help you
Task[] tasks = new Task[2];
tasks[0] = Task.Factory.StartNew(() => YourFunction());
tasks[1] = Task.Factory.StartNew(() => YourFunction());
Task.WaitAll(tasks);// here it will wait untill all the functions get completed
I want to combine the result of 2 tasks in one List collection.
Make sure that- I want to run both methods in parallel.
Code:
List<Employee> totalEmployees = new List<Employee>();
Method1:
public async Task<IEnumerable<Employee>> SearchEmployeeFromDb();
Method2:
public async Task<IEnumerable<Employee>> GetEmployeeFromService();
Now, I want to hold the result of these two methods in totalEmployees field, also these 2 method should run asynchronously.
While many answers are close, the cleanest and most efficient option is using Task.WhenAll combined with SelectMany:
async Task<IEnumerable<Employee>> Combine()
{
var results = await Task.WhenAll(SearchEmployeeFromDb(), GetEmployeeFromService());
return results.SelectMany(result => result);
}
This assumes that by parallel you mean concurrently. If you wish to run these operations with multiple threads from the beginning (including the synchronous parts of the async method) you need to also use Task.Run to offload work to a ThreadPool thread:
private async Task<IEnumerable<Employee>> Combine()
{
var results =
await Task.WhenAll(Task.Run(() => SearchEmployeeFromDb()), Task.Run(() => GetEmployeeFromService()));
return results.SelectMany(result => result);
}
Start both tasks
Use Task.WhenAll to wait for both tasks to finish
Use Enumerable.Concat to combine the results
var searchEmployeesTask = SearchEmployeeFromDb();
var getEmployeesTask = GetEmployeeFromService();
await Task.WhenAll(searchEmployeesTask, getEmployeesTask);
var totalEmployees = searchEmployeesTask.Result.Concat(getEmployeesTask.Result);
You can use Task.WhenAll to create a task which will return when all supplied tasks are complete
var result = await Task.WhenAll(SearchEmployeeFromDb(),GetEmployeeFromService());
var combined = result[0].Concat(result[1]);
Something like this should work:
var t1 = SearchEmployeeFromDb()
var t2 = GetEmployeeFromService()
await Task.WhenAll(t1, t2)
// Now use t1.Result and t2.Result to get `totalEmployees`
Use ConfigureAwait(false) to avoid deadlocking, define the tasks, execute and then await.
var fromDbTask = SearchEmployeeFromDb().ConfigureAwait(false);
var fromServiceTask = GetEmployeeFromService().ConfigureAwait(false);
var fromDbResult = await fromDbTask;
var totalEmployees = new List(fromDbResult);
var fromServiceResult = await fromServiceResult;
totalEmployees.AddRange(fromServiceResult);
... or use whichever way you want to merge the two lists.
I updated the solution, it was unneccessary to create the list and then append the first result. We wait for the first method to finish and then create the list.
I have the following code:
public OrderInfo GetPreviousOrder(int customerId)
{
using (var entities = new ContosoEntities())
{
var previousOrder = entities.Orders
.OrderByDescending(x => x.CreatedDate)
.FirstOrDefault(x => x.CustomerId == customerId);
var response = new OrderInfo
{
Customer = RetrieveCustomerInfo(customerId),
};
if (previousOrder != null)
{
response.FollowUps = Mapper.Map<IEnumerable<OrderFollowUp>, OrderFollowUpData[]>(previousOrder.FollowUps);
response.Appointments = Mapper.Map<IEnumerable<Appointment>, OrderAppointment[]>(previousOrder.Appointments);
response.DiscussionTopics = previousOrder.DiscussionTopics.Select(discussionTopic => discussionTopic.Topic).ToArray();
}
var actions = new Action[]
{
() => response.Invoices = RetriveInvoices(customerId, previousOrder),
() => response.Items = RetrieveItems(customerId, previousOrder)
};
Parallel.ForEach(actions, new ParallelOptions { MaxDegreeOfParallelism = 10 }, action => action());
return response;
}
}
In summary, this method gets the info of a customer's previous order. Most of the order info is stored in my project's database (accessed using Entity Framework), but a few things are pulled from another source (customer info, invoices and items).
Sometimes, when the RetriveInvoices method gets executed (and the customer had a previous order stored in our database), the value of the previousOrder variable passed is null. Likewise, some other times, it's the RetrieveItems method that gets a null value and some other times, the code behaves as (I) expected, meaning, the previousOrder variable actually has the info of the previous order passed to both methods. Both methods are being executed until their respective last line, there are no exceptions being thrown. And no, both methods only read the previousOrder variable.
Can somebody explain to me why I'm seeing this behavior?
public void SubmitMessagesToQueue_OneMessage_SubmitSuccessfully()
{
var messageServiceClientMock = new Mock<IMessageServiceClient>();
var queueableMessage = CreateSingleQueueableMessage();
var message = queueableMessage[0];
var xml = QueueableMessageAsXml(queueableMessage);
messageServiceClientMock.Setup(proxy => proxy.SubmitMessage(xml)).Verifiable();
//messageServiceClientMock.Setup(proxy => proxy.SubmitMessage(It.IsAny<XmlElement>())).Verifiable();
var serviceProxyFactoryStub = new Mock<IMessageServiceClientFactory>();
serviceProxyFactoryStub.Setup(proxyFactory => proxyFactory.CreateProxy()).Returns(essageServiceClientMock.Object);
var loggerStub = new Mock<ILogger>();
var client = new MessageClient(serviceProxyFactoryStub.Object, loggerStub.Object);
client.SubmitMessagesToQueue(new List<IMessageRequestDTO> {message});
//messageServiceClientMock.Verify(proxy => proxy.SubmitMessage(xml), Times.Once());
messageServiceClientMock.Verify();
}
I'm starting using Moq and struggling a bit.
I'm trying to verify that messageServiceClient is receiving the right parameter, which is an XmlElement, but I can't find any way to make it work. It works only when I don't check a particular value.
Any ideas?
Partial answer:
I've found a way to test that the xml sent to the proxy is correct, but I still don't think it's the right way to do it.
public void SubmitMessagesToQueue_OneMessage_SubmitSuccessfully()
{
var messageServiceClientMock = new Mock<IMessageServiceClient>();
messageServiceClientMock.Setup(proxy => proxy.SubmitMessage(It.IsAny<XmlElement>())).Verifiable();
var serviceProxyFactoryStub = new Mock<IMessageServiceClientFactory>();
serviceProxyFactoryStub.Setup(proxyFactory => proxyFactory.CreateProxy()).Returns(messageServiceClientMock.Object);
var loggerStub = new Mock<ILogger>();
var client = new MessageClient(serviceProxyFactoryStub.Object, loggerStub.Object);
var message = CreateMessage();
client.SubmitMessagesToQueue(new List<IMessageRequestDTO> {message});
messageServiceClientMock.Verify(proxy => proxy.SubmitMessage(It.Is<XmlElement>(xmlElement => XMLDeserializer<QueueableMessage>.Deserialize(xmlElement).Messages.Contains(message))), Times.Once());
}
By the way, how could I extract the expression from the Verify call?
If the verification logic is non-trivial, it will be messy to write a large lambda method (as your example shows). You could put all the test statements in a separate method, but I don't like to do this because it disrupts the flow of reading the test code.
Another option is to use a callback on the Setup call to store the value that was passed into the mocked method, and then write standard Assert methods to validate it. For example:
// Arrange
MyObject saveObject;
mock.Setup(c => c.Method(It.IsAny<int>(), It.IsAny<MyObject>()))
.Callback<int, MyObject>((i, obj) => saveObject = obj)
.Returns("xyzzy");
// Act
// ...
// Assert
// Verify Method was called once only
mock.Verify(c => c.Method(It.IsAny<int>(), It.IsAny<MyObject>()), Times.Once());
// Assert about saveObject
Assert.That(saveObject.TheProperty, Is.EqualTo(2));
I've been verifying calls in the same manner - I believe it is the right way to do it.
mockSomething.Verify(ms => ms.Method(
It.IsAny<int>(),
It.Is<MyObject>(mo => mo.Id == 5 && mo.description == "test")
), Times.Once());
If your lambda expression becomes unwieldy, you could create a function that takes MyObject as input and outputs true/false...
mockSomething.Verify(ms => ms.Method(
It.IsAny<int>(),
It.Is<MyObject>(mo => MyObjectFunc(mo))
), Times.Once());
private bool MyObjectFunc(MyObject myObject)
{
return myObject.Id == 5 && myObject.description == "test";
}
Also, be aware of a bug with Mock where the error message states that the method was called multiple times when it wasn't called at all. They might have fixed it by now - but if you see that message you might consider verifying that the method was actually called.
EDIT: Here is an example of calling verify multiple times for those scenarios where you want to verify that you call a function for each object in a list (for example).
foreach (var item in myList)
mockRepository.Verify(mr => mr.Update(
It.Is<MyObject>(i => i.Id == item.Id && i.LastUpdated == item.LastUpdated),
Times.Once());
Same approach for setup...
foreach (var item in myList) {
var stuff = ... // some result specific to the item
this.mockRepository
.Setup(mr => mr.GetStuff(item.itemId))
.Returns(stuff);
}
So each time GetStuff is called for that itemId, it will return stuff specific to that item. Alternatively, you could use a function that takes itemId as input and returns stuff.
this.mockRepository
.Setup(mr => mr.GetStuff(It.IsAny<int>()))
.Returns((int id) => SomeFunctionThatReturnsStuff(id));
One other method I saw on a blog some time back (Phil Haack perhaps?) had setup returning from some kind of dequeue object - each time the function was called it would pull an item from a queue.
A simpler way would be to do:
ObjectA.Verify(
a => a.Execute(
It.Is<Params>(p => p.Id == 7)
)
);
I believe that the problem in the fact that Moq will check for equality. And, since XmlElement does not override Equals, it's implementation will check for reference equality.
Can't you use a custom object, so you can override equals?
Had one of these as well, but the parameter of the action was an interface with no public properties. Ended up using It.Is() with a seperate method and within this method had to do some mocking of the interface
public interface IQuery
{
IQuery SetSomeFields(string info);
}
void DoSomeQuerying(Action<IQuery> queryThing);
mockedObject.Setup(m => m.DoSomeQuerying(It.Is<Action<IQuery>>(q => MyCheckingMethod(q)));
private bool MyCheckingMethod(Action<IQuery> queryAction)
{
var mockQuery = new Mock<IQuery>();
mockQuery.Setup(m => m.SetSomeFields(It.Is<string>(s => s.MeetsSomeCondition())
queryAction.Invoke(mockQuery.Object);
mockQuery.Verify(m => m.SetSomeFields(It.Is<string>(s => s.MeetsSomeCondition(), Times.Once)
return true
}