Develop Reference

Capture Slow Output from Method

I have a slow running utility method that logs one line of output at a time. I need to be able to output each of those lines and then read them from other locations in code. I have attempted using Tasks and Streams similar to the code below:
public static Task SlowOutput(Stream output)
{
Task result = new Task(() =>
{
using(StreamWriter sw = new StreamWriter(output))
{
for(var i = 0; i < int.MaxValue; i++)
{
sw.WriteLine(i.ToString());
System.Threading.Thread.Sleep(1000);
}
}
}
}
And then called like this:
MemoryStream ms = new MemoryStream();
var t = SlowOutput(ms);
using (var sr = new StreamReader(ms))
{
while (!t.IsCompleted)
{
Console.WriteLine(sr.ReadLine())
}
}
But of course, sr.ReadLine() is always empty because as soon as the method's sw.WriteLine() is called, it changes the position of the underlying stream to the end.
What I'm trying to do is pipe the output of the stream by maybe queueing up the characters that the method outputs and then consuming them from outside the method. Streams don't seem to be the way to go.
Is there a generally accepted way to do this?

What I would do is switch to a BlockingCollection<String>.
public static Task SlowOutput(BlockingCollection<string> output)
{
return Task.Run(() =>
{
for(var i = 0; i < int.MaxValue; i++)
{
output.Add(i);
System.Threading.Thread.Sleep(1000);
}
output.Complete​Adding();
}
}
consumed by
var bc = BlockingCollection<string>();
SlowOutput(bc);
foreach(var line in bc.GetConsumingEnumerable()) //Blocks till a item is added to the collection. Leaves the foreach loop after CompleteAdding() is called and there are no more items to be processed.
{
Console.WriteLine(line)
}

Threading and SqlFileStream. The process cannot access the file specified because it has been opened in another transaction

I am extracting content of the Files in SQL File Table. The following code works if I do not use Parallel.
I am getting the following exception, when reading sql file stream simultaneously (Parallel).
The process cannot access the file specified because it has been opened in another transaction.
TL;DR:
When reading a file from FileTable (using GET_FILESTREAM_TRANSACTION_CONTEXT) in a Parallel.ForEach I get the above exception.
Sample Code for you to try out:
https://gist.github.com/NerdPad/6d9b399f2f5f5e5c6519
Longer Version:
Fetch Attachments, and extract content:
var documents = new List<ExtractedContent>();
using (var ts = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled))
{
var attachments = await dao.GetAttachmentsAsync();
// Extract the content simultaneously
// documents = attachments.ToDbDocuments().ToList(); // This works
Parallel.ForEach(attachments, a => documents.Add(a.ToDbDocument())); // this doesn't
ts.Complete();
}
DAO Read File Table:
public async Task<IEnumerable<SearchAttachment>> GetAttachmentsAsync()
{
try
{
var commandStr = "....";
IEnumerable<SearchAttachment> attachments = null;
using (var connection = new SqlConnection(this.DatabaseContext.Database.Connection.ConnectionString))
using (var command = new SqlCommand(commandStr, connection))
{
connection.Open();
using (var reader = await command.ExecuteReaderAsync())
{
attachments = reader.ToSearchAttachments().ToList();
}
}
return attachments;
}
catch (System.Exception)
{
throw;
}
}
Create objects for each file:
The object contains a reference to the GET_FILESTREAM_TRANSACTION_CONTEXT
public static IEnumerable<SearchAttachment> ToSearchAttachments(this SqlDataReader reader)
{
if (!reader.HasRows)
{
yield break;
}
// Convert each row to SearchAttachment
while (reader.Read())
{
yield return new SearchAttachment
{
...
...
UNCPath = reader.To<string>(Constants.UNCPath),
ContentStream = reader.To<byte[]>(Constants.Stream) // GET_FILESTREAM_TRANSACTION_CONTEXT()
...
...
};
}
}
Read the file using SqlFileStream:
Exception is thrown here
public static ExtractedContent ToDbDocument(this SearchAttachment attachment)
{
// Read the file
// Exception is thrown here
using (var stream = new SqlFileStream(attachment.UNCPath, attachment.ContentStream, FileAccess.Read, FileOptions.SequentialScan, 4096))
{
...
// extract content from the file
}
....
}
Update 1:
According to this article it seems like it could be an Isolation level issue. Has anyone ever faced similar issue?

The transaction does not flow in to the Parallel.ForEach, you must manually bring the transaction in.
//Switched to a thread safe collection.
var documents = new ConcurrentQueue<ExtractedContent>();
using (var ts = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled))
{
var attachments = await dao.GetAttachmentsAsync();
//Grab a reference to the current transaction.
var transaction = Transaction.Current;
Parallel.ForEach(attachments, a =>
{
//Spawn a dependant clone of the transaction
using (var depTs = transaction.DependentClone(DependentCloneOption.RollbackIfNotComplete))
{
documents.Enqueue(a.ToDbDocument());
depTs.Complete();
}
});
ts.Complete();
}
I also switched from List<ExtractedContent> to ConcurrentQueue<ExtractedContent> because you are not allowed call .Add( on a list from multiple threads at the same time.

Returning a large collection serialized as JSON via MVC controller

I have a very large result set which I want to return to an Ajax call using JSON.
I started with creating a collection of objects and then serialize the collection but the collection creation would throw a System.OutOfMemoryException.
I've now tried to change the implementation to stream the JSON without having a collection but I still get the System.OutOfMemoryException.
Here my current code snippets.
using (var stream = new MemoryStream())
{
using (var streamWriter = new StreamWriter(stream))
{
using (var jsonWriter = new JsonTextWriter(streamWriter))
{
var serializer = new JsonSerializer();
serializer.Serialize(jsonWriter,new { pins = MakePins(model), missingLocations = 0 });
jsonWriter.Flush();
}
}
stream.Seek(0, SeekOrigin.Begin);
return new FileStreamResult(stream, "application/json");
The MakePins function looks like this:
var pinData = _geographyService.GetEnumerationQueryable()
.SelectMany(x => x.EnumeratedPersonRoleCollection)
.ApplyFilter(model).Where(x => x.EnumerationCentre.Location != null)
.AsNoTracking()
.AsEnumerable();
return pinData.Select(item => new MapPin
{
Id = item.EnumerationCentre.EnumerationCentreUid.ToString(),
Name = item.Person.FullName,
FillColour = GetMapPinColour(item, model),
Latitude = item.EnumerationCentre.Location.Latitude,
Longitude = item.EnumerationCentre.Location.Longitude,
Count = item.IssuedVoucherCollection.Count()
});
I've tried using a yield return instead of the select but the OutOfMemoryException is throw withing the Select function.
I've done a fair bit of googling but can't quite see what else I could try.

Your current solution still have the same problem, because just before return you collect and store all data in the memory stream
You can try something in the following fashion:
public ActionResult RobotsText() {
Response.ContentType = "application/json";
Response.Write("[");
foreach(var item in Items)
{
Response.Write(JsonSerializer.Serialize(item));
if ( /*not last*/)
{
Response.Write(",");
}
}
Response.Write("]");
return new EmptyResult();
}

The process cannot access the file because it is being used by another process. XML

I am calling below method in a loop with same xmlRequestPath and xmlResponsePath files. Two loop counts it executes fine in the 3rd iteration I am getting exception "The process cannot access the file because it is being used by another process.".
public static void UpdateBatchID(String xmlRequestPath, String xmlResponsePath)
{
String batchId = "";
XDocument requestDoc = null;
XDocument responseDoc = null;
lock (locker)
{
using (var sr = new StreamReader(xmlRequestPath))
{
requestDoc = XDocument.Load(sr);
var element = requestDoc.Root;
batchId = element.Attribute("BatchID").Value;
if (batchId.Length >= 16)
{
batchId = batchId.Remove(0, 16).Insert(0, DateTime.Now.ToString("yyyyMMddHHmmssff"));
}
else if (batchId != "") { batchId = DateTime.Now.ToString("yyyyMMddHHmmssff"); }
element.SetAttributeValue("BatchID", batchId);
}
using (var sw = new StreamWriter(xmlRequestPath))
{
requestDoc.Save(sw);
}
using (var sr = new StreamReader(xmlResponsePath))
{
responseDoc = XDocument.Load(sr);
var elementResponse = responseDoc.Root;
elementResponse.SetAttributeValue("BatchID", batchId);
}
using (var sw = new StreamWriter(xmlResponsePath))
{
responseDoc.Save(sw);
}
}
Thread.Sleep(500);
requestDoc = null;
responseDoc = null;
}
Exception is occurring at using (var sw = new StreamWriter(xmlResponsePath)) in above code.
Exception:
The process cannot access the file 'D:\Projects\ESELServer20130902\trunk\Testing\ESL Server Testing\ESLServerTesting\ESLServerTesting\TestData\Assign\Expected Response\Assign5kMACResponse.xml' because it is being used by another process.

Maybe at the third loop the stream is still being closed, so it tells you that it is non accessible. Try waiting a bit before calling it again in the loop, for example:
while (...)
{
UpdateBatchID(xmlRequestPath, xmlResponsePath);
System.Threading.Thread.Sleep(500);
}
Or, close explicitly the stream instead of leaving the work to the garbage collector:
var sr = new StreamReader(xmlResponsePath);
responseDoc = XDocument.Load(sr);
....
sr.Close();

Instead of using two streams, a Write and a Read stream, try using only a FileStream, since the problem might be that after loading the file the stream remains opened until the garbadge collector actives.
using (FileSteam f = new FileStream(xmlResponsePath))
{
responseDoc = XDocument.Load(sr);
var elementResponse = responseDoc.Root;
elementResponse.SetAttributeValue("BatchID", batchId);
responseDoc.Save(sw);
}

Task.WaitAll not waiting on other async methods

I'm asynchronously retrieving some rss articles with my Portable Class Library that uses the Microsoft.Bcl library (which doesn't have Task.WhenAll). Each article has a url to rss comments that I need to asynchronously retrieve as well.
The code below is my library. I call GetArticles() but it does not return any of the which creates a list of tasks that call GetComments() to asynchronously get the comments.
I've tried using Task.WaitAll in GetArticles to wait for the comments but it does not block the thread. Any help would be appreciated.
private const string ArticlesUri = "";
public async Task<List<ArticleBrief>> GetArticles()
{
var results = new List<ArticleBrief>();
try
{
var wfw = XNamespace.Get("http://wellformedweb.org/CommentAPI/");
var media = XNamespace.Get("http://search.yahoo.com/mrss/");
var dc = XNamespace.Get("http://purl.org/dc/elements/1.1/");
var t = await WebHttpRequestAsync(ArticlesUri);
StringReader stringReader = new StringReader(t);
using (var xmlReader = System.Xml.XmlReader.Create(stringReader))
{
var doc = System.Xml.Linq.XDocument.Load(xmlReader);
results = (from e in doc.Element("rss").Element("channel").Elements("item")
select
new ArticleBrief()
{
Title = e.Element("title").Value,
Description = e.Element("description").Value,
Published = Convert.ToDateTime(e.Element("pubDate").Value),
Url = e.Element("link").Value,
CommentUri = e.Element(wfw + "commentRss").Value,
ThumbnailUri = e.Element(media + "thumbnail").FirstAttribute.Value,
Categories = GetCategoryElements(e.Elements("category")),
Creator = e.Element(dc + "creator").Value
}).ToList();
}
var tasks = new Queue<Task>();
foreach (var result in results)
{
tasks.Enqueue(
Task.Factory.StartNew(async ()=>
{
result.Comments = await GetComments(result.CommentUri);
}
));
}
Task.WaitAll(tasks.ToArray());
}
catch (Exception ex)
{
// should do some other
// logging here. for now pass off
// exception to callback on UI
throw ex;
}
return results;
}
public async Task<List<Comment>> GetComments(string uri)
{
var results = new List<Comment>();
try
{
var wfw = XNamespace.Get("http://wellformedweb.org/CommentAPI/");
var media = XNamespace.Get("http://search.yahoo.com/mrss/");
var dc = XNamespace.Get("http://purl.org/dc/elements/1.1/");
var t = await WebHttpRequestAsync(uri);
StringReader stringReader = new StringReader(t);
using (var xmlReader = System.Xml.XmlReader.Create(stringReader))
{
var doc = System.Xml.Linq.XDocument.Load(xmlReader);
results = (from e in doc.Element("rss").Element("channel").Elements("item")
select
new Comment()
{
Description = e.Element("description").Value,
Published = Convert.ToDateTime(e.Element("pubDate").Value),
Url = e.Element("link").Value,
Creator = e.Element(dc + "creator").Value
}).ToList();
}
}
catch (Exception ex)
{
// should do some other
// logging here. for now pass off
// exception to callback on UI
throw ex;
}
return results;
}
private static async Task<string> WebHttpRequestAsync(string url)
{
//TODO: look into getting
var request = WebRequest.Create(url);
request.Method = "GET";
var response = await request.GetResponseAsync();
return ReadStreamFromResponse(response);
}
private static string ReadStreamFromResponse(WebResponse response)
{
using (Stream responseStream = response.GetResponseStream())
using (StreamReader sr = new StreamReader(responseStream))
{
string strContent = sr.ReadToEnd();
return strContent;
}
}
private List<string> GetCategoryElements(IEnumerable<XElement> categories)
{
var listOfCategories = new List<string>();
foreach (var category in categories)
{
listOfCategories.Add(category.Value);
}
return listOfCategories;
}
Updated Code from Solution, just added .UnWrap() on the Enqueue method:
var tasks = new Queue<Task>();
foreach (var result in results)
{
tasks.Enqueue(
Task.Factory.StartNew(async ()=>
{
result.Comments = await GetComments(result.CommentUri);
}
).Unwrap());
}
Task.WaitAll(tasks.ToArray());

It is waiting appropriately. The problem is that you are creating a Task which creates another task (i.e. StartNew is returning a Task<Task> and you are only waiting on the outer Task which completes rather quickly (it completes before the inner Task is complete)).
The questions will be:
Do you really want that inner task?
If yes, then you can use Task.Unwrap to get a proxy task that represents the completion of both the inner and outer Task and use that to Wait on.
If no, then you could remove the use of async/await in StartNew so that there is not an inner task (I think this would be prefered, it's not clear why you need the inner task).
Do you really need to do a synchronous Wait on an asynchronous Task? Read some of Stephen Cleary's blog: http://blog.stephencleary.com/2012/02/async-unit-tests-part-1-wrong-way.html
As an aside, if you are not using C# 5, then watch out for closing over the foreach variable result See
Has foreach's use of variables been changed in C# 5?, and
http://blogs.msdn.com/b/ericlippert/archive/2009/11/12/closing-over-the-loop-variable-considered-harmful.aspx)

In Microsoft.Bcl.Async we couldn't add any static methods to Task. However, you can find most of the methods on TaskEx, for example, TaskEx.WhenAll() does exist.