Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I still working on learning Xamarin.iOs, and for a little proof on concept, I'm using Monotouch.Dialog to create an employee directory. It works great on the iPhone simulator but when I execute it on the physical devices, 3 weird things happen:
Randomly (sometime on the simulator but more often on the device), I've a timeout exception calling HttpWebRequest.GetRequestStream() in an async method. However, I create a new HttpWebRequest each time the method is called and I think I properly close and dispose everything. Here's a snippet of the code :
var wr = HttpWebRequest.Create(url);
byte[] streamContent = System.Text.Encoding.UTF8.GetBytes(body);
Stream dataStream = wr.GetRequestStream(); //timeout on that statement !
dataStream.Write(streamContent, 0, streamContent.Length);
dataStream.Close();
using (var response = (await wr.GetResponseAsync().ConfigureAwait(false)))
{
if (response != null)
{
try
{
var webResponse = response as HttpWebResponse;
if (webResponse != null && webResponse.StatusCode == HttpStatusCode.OK)
{
using (StreamReader reader = new StreamReader(webResponse.GetResponseStream()))
{
var responseText = reader.ReadToEnd();
result = JsonConvert.DeserializeObject<T>(responseText);
reader.Close();
}
}
}
finally
{
if (response != null)
{
response.Close();
}
}
}
}
I load my data asynchronously but if I use the async / await keywords, my application failed to launch on the device because it looks like the UI thread is waiting for my load to complete at it takes too much time. I fixed this problem by using Task.Factory.StartNew instead but I was wondering why the behavior is different on the device and on the simulator.
My last problem is that when I receive the answer from my web service, I build a list of Element that I add to my section. I've many elements but not that much (about 700). It take about 2 seconds to refresh the screen in the simulator but more than 1 minute on the device (a fourth gen ipod touch). There's the code I use to create my elements and refresh the screen (it runs async so that's why I use InvokeOnMainThread()) :
private void updateEmployeeList(IEnumerable<EmployeSummary> list, Section section)
{
if (list != null && list.Any())
{
var q = list.Select(item =>
{
StyledStringElement newCell = new StyledStringElement(item.FullName) { Accessory = UITableViewCellAccessory.DisclosureIndicator };
newCell.Tapped +=
() => _detailScreenVM.ShowDetail(item);
return newCell;
}).ToList();
_logger.Log("Items to add : {0}", q.Count);
InvokeOnMainThread(() =>
{
_logger.Log("Starting updating screen");
section.AddAll(q);
_logger.Log("Ending updating screen.");
});
}
}
Here the result of the console at execution time :
Items to add : 690 at 15:29:57.896041
Starting updating screen at 15:29:57.903079
Ending updating screen at 15:31:03.548430
It's probably something I do wrong with the async programming model but I can't figure out what exactly.
Thanks
I guess the timeout is a System.Net.WebException: The operation has timed out at System.Net.HttpWebRequest.GetRequestStream(). If you look at the Response.StatusCode property of the exception, you'll certainly find that it's a 4xx http error. Which means the server and not the client is responsible for this. So fix this server side.
You have very little time (17 seconds) for your AppDelegate.FinishedLaunching to return. If you exceed this time, your app is killed. Starting a background thread is the way to go. The restriction might be slightly different on the simulator, as it's well known that debug code in the simulator is slower than running release code on the device.
I don't know what's wrong here and don't know the internals of MonoTouch.Dialog, but a way to fix it would be to use a UITableView so the rendering of the cell only happens when required.
Related
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 1 year ago.
Improve this question
I have a web application that makes industrial scheduling calculations, I'm trying to log events in Azure cosmos db table after every update that happens to the schedule without affecting the application performance (screen Loading time).
That means, I want to fire log method in the BACKGROUND and the end user will not feel it (no freeze or extra loading time) and without making the UI wait for this operation to be done.
I added the next C# lines of code just after finishing the whole calculations:
private List<JPIEventEntity> batch = new List<JPIEventEntity>();
private List<List<JPIEventEntity>> batchesList = new List<List<JPIEventEntity>>();
Thread newThread = new Thread(() => myJPIEventHandler.checkForJPIEventsToSend(customer, author, model));
newThread.Start();
/*
* check if there are any changes or updates in the calculations and log their events.
*/
internal void checkForJPIEventsToSend(JPIBaseCustomer customer, JPIBaseUser author, SchedulingModel afterModel)
{
myCustomer = customer;
myUser = author;
// Looking for deleted Jobs
foreach (Job beforeJob in myBeforeModel.Jobs)
{
if (!afterModel.Jobs.Any(x => x.Guid == beforeJob.Guid))
{
//Looking for deleted Tasks and Log the deletion
foreach (Operation beforeOp in beforeJob.Operations)
{
//if (!afterJob.Operations.Any(x => x.Guid == beforeOp.Guid))
logTaskEvent(EventType.Delete, beforeOp, "", "");
}
//Log Job Deletion
logJobEvent(EventType.Delete, beforeJob, "", "");
}
}
//Comparison
foreach (Job afterJob in afterModel.Jobs)
{
if (myBeforeModel.Jobs.Any(x => x.Guid == afterJob.Guid))
{
Job beforeJob = myBeforeModel.Jobs.First(x => x.Guid == afterJob.Guid);
if (beforeJob.Name != afterJob.Name)
logJobEvent(EventType.NameChanged, afterJob, beforeJob.Name, afterJob.Name);
if (beforeJob.ReleaseDate != afterJob.ReleaseDate)
logJobEvent(EventType.ReleaseDateChanged, afterJob, beforeJob.ReleaseDate, afterJob.ReleaseDate);
if (beforeJob.DueDate != afterJob.DueDate)
logJobEvent(EventType.DueDateChanged, afterJob, beforeJob.DueDate, afterJob.DueDate);
if (beforeJob.IsDueDateExceeded != afterJob.IsDueDateExceeded)
logJobEvent(EventType.DueDateExceededChanged, afterJob, beforeJob.IsDueDateExceeded.ToString(), afterJob.IsDueDateExceeded.ToString());
if (beforeJob.ProcessingState != afterJob.ProcessingState)
{
logJobEvent(EventType.StatusChanged, afterJob,
convertProcessingStateToStatus(beforeJob.ProcessingState.ToString()), convertProcessingStateToStatus(afterJob.ProcessingState.ToString()));
}
if (beforeJob.SequenceNumber != afterJob.SequenceNumber && afterJob.ProcessingState != JobProcessingState.Finished)
logJobEvent(EventType.SequenceNumberChanged, afterJob, beforeJob.SequenceNumber, afterJob.SequenceNumber);
if (beforeJob.CustomQuantity != afterJob.CustomQuantity)
logJobEvent(EventType.QuantityChanged, afterJob, beforeJob.CustomQuantity, afterJob.CustomQuantity);
DateTime? beforeStart = beforeJob.ProcessingStart != null ? beforeJob.ProcessingStart : beforeJob.PlannedStart;
DateTime? afterStart = afterJob.ProcessingStart != null ? afterJob.ProcessingStart : afterJob.PlannedStart;
if (beforeStart != afterStart)
logJobEvent(EventType.StartDateChanged, afterJob, beforeStart, afterStart);
DateTime? beforeEnd = beforeJob.ProcessingEnd != null ? beforeJob.ProcessingEnd : beforeJob.PlannedEnd;
DateTime? afterEnd = afterJob.ProcessingEnd != null ? afterJob.ProcessingEnd : afterJob.PlannedEnd;
if (beforeEnd != afterEnd)
logJobEvent(EventType.EndDateChanged, afterJob, beforeEnd, afterEnd);
TimeSpan? beforeBuffer = beforeJob.DueDate != null ? (beforeJob.DueDate - beforeEnd) : new TimeSpan(0L);
TimeSpan? afterBuffer = afterJob.DueDate != null ? (afterJob.DueDate - afterEnd) : new TimeSpan(0L);
if (beforeBuffer != afterBuffer)
logJobEvent(EventType.BufferChanged, afterJob, beforeBuffer, afterBuffer);
}
//Collect the Batches in one list of batches
CollectBatches();
//Log all the Batches
LogBatches(batchesList);
}
/*
* Collectes the events in one batch
*/
private void logJobEvent(EventType evtType, Job afterJob, string oldValue, string newValue)
{
var eventGuid = Guid.NewGuid();
JPIEventEntity evt = new JPIEventEntity();
evt.Value = newValue;
evt.PrevValue = oldValue;
evt.ObjectGuid = afterJob.Guid.ToString();
evt.PartitionKey = myCustomer.ID; //customer guid
evt.RowKey = eventGuid.ToString();
evt.EventType = evtType;
evt.CustomerName = myCustomer.Name;
evt.User = myUser.Email;
evt.ParentName = null;
evt.ObjectType = JOB;
evt.ObjectName = afterJob.Name;
evt.CreatedAt = DateTime.Now;
batch.Add(evt);
}
/*
* Collectes the Events lists in an enumerable of Batches (max capacity of a single batch insertion is 100).
*/
private void CollectBatches()
{
//batchesList = new List<List<JPIEventEntity>>();
if (batch.Count > 0)
{
int rest = batch.Count;
var nextBatch = new List<JPIEventEntity>();
if (batch.Count > MaxBatchSize) //MaxBatchSize = 100
{
foreach (var item in batch)
{
nextBatch.Add(item);
rest = rest - 1; //rest = rest - (MaxBatchSize * hundreds);
if (rest < MaxBatchSize && nextBatch.Count == (batch.Count % MaxBatchSize))
{
batchesList.Add(nextBatch);
}
else if (nextBatch.Count == MaxBatchSize)
{
batchesList.Add(nextBatch);
nextBatch = new List<JPIEventEntity>();
}
}
}
else
{
batchesList.Add(batch);
}
}
}
private void LogBatches(List<List<JPIEventEntity>> batchesList)
{
if (batchesList.Count > 0)
{
JPIEventHandler.LogBatches(batchesList);
}
}
/*
* Insert Events into database
*/
public static void LogBatches(List<List<JPIEventEntity>> batchesList)
{
foreach (var batch in batchesList)
{
var batchOperationObj = new TableBatchOperation();
//Iterating through each batch entities
foreach (var Event in batch)
{
batchOperationObj.InsertOrReplace(Event);
}
var res = table.ExecuteBatch(batchOperationObj);
}
}
Inside the 'checkForJPIEventsToSend' method, I'm checking if there's any changes or updates in the calculations and insert events (hundreds or even thousands of lines) into the cosmos db table as batches.
After putting the method in a separate thread (as shown above) I still have an EXTRA LOADING duration of 2 to 4 seconds after every operation, which is something critical and bad for us.
Am I using the multi-threading correctly?
Thank you in advance.
As I understand your situation you have a front end application such as a desktop app or a website that creates requests. For each request you
Perform some calculations
Write some data to storage (Cosmos DB)
It is unclear whether you must display some result to the front end after these steps are complete. Your options depend on this question.
Scenario 1: The front end is waiting for the results of the calculations or database changes
The front end requires some result from the calculations or database changes, so the user is forced to wait for this to complete. However you want to avoid freezing your front end whilst you perform the long running tasks.
The solution most developers reach for here is to perform the work in a background thread. Your main thread waits for this background thread to complete and return a result, at which point the main/UI thread will update the front end with the result. This is because the main thread is often the only thread allowed to update the front end.
How you offload the work to a background thread depends on the type of work load you have. If the work is mostly I/O such as File I/O, Network I/O (writing to Azure CosmosDB) then you want to use the Async methods of the Cosmos API and async/await.
See https://stackoverflow.com/a/18033198/6662878
If the work you are doing is mostly CPU based, then threads will only speed up the processing if the problem can be broken into parts and run in parallel across multiple background threads. If the problem cannot be broken down and parallelised then running the work on a single background thread has a small cost associated with thread switching, but in turn this frees up the main/UI thread whilst the CPU based work is in progress in the background.
See https://learn.microsoft.com/en-us/dotnet/standard/asynchronous-programming-patterns/consuming-the-task-based-asynchronous-pattern
You will need to think about how you handle exceptions that occur on background threads, and how the code you run in your background thread will respond to a request to stop processing.
See https://learn.microsoft.com/en-us/dotnet/standard/threading/canceling-threads-cooperatively
Caveat: if any thread is carrying out very CPU intensive work (such as compressing or encrypting large amounts of data, encoding audio or video etc) this can often cause the front end to freeze, stop responding, drop network requests etc. If you have some processor intensive work to complete you need to think about how the work is spread over CPU cores, or CPUs.
Scenario 2: The front end does not need to display any specific result for the request
In this scenario you have more flexibilty about how and when you perform your background work because you can simply respond to the front end request with an acknowledgement that the request is received and will be processed in the (near) future. For example a Web API may respond with a 201 ACCEPTED HTTP response code to signal this.
You now want to queue the requests and process them somewhere other than your main/UI thread. There are a number of options, background threads being one of them, though not the simplest. You may also consider using a framework like https://www.hangfire.io/.
Another popular approach is to create a completely separate service or microservice that is responsible for your picking up requests from a queue and performing the work.
See https://learn.microsoft.com/en-us/dotnet/architecture/microservices/architect-microservice-container-applications/asynchronous-message-based-communication
Multithreading should come with a big warning message. Sometimes it is unavoidable, but it is always difficult and troublesome to get right. The C# APIs have evolved over time and so there's a lot to learn and a lot of ground to cover. It is often seen as a quick option to convert an application to be multithreaded, though you should be wary of this. Although more complex architectures as discussed in the link above seem overly burdensome, they invariably force you to think through a number of issues that come up when you begin to split up your application into threads, or processes, or services.
I'm working in C#/Xamarin on the Android portion of a cross-platform PCL solution - I'm very new to Xamarin and mobile dev in general.
In a loop, I'm trying to query a REST service for some geoinformation (which works fine), deserialise the result into an object (also works fine) and then add said object to a list (works fine once). The problem I have is that every time the method getting the deserialised object returns, the collection for them has magically been emptied (Count is 0).
This is where the list lives:
List<Thingy> thingyList = new List<Thingy>();
//Setting some variables
//.
//.
using (HttpClient client = new HttpClient())
{
//Positive offset loop
for (double latUnderTest = Lat; latUnderTest <= latOffsetCoordsMax; latUnderTest += latOffset3M)
{
//Bound to our max coords
latUnderTest = latUnderTest > latOffsetCoordsMax ? latOffsetCoordsMax : latUnderTest;
for (double longUnderTest = Long; longUnderTest <= longOffsetCoordsMax; longUnderTest += longOffset3M)
{
//Bound to our max coords
longUnderTest = longUnderTest > longOffsetCoordsMax ? longOffsetCoordsMax : longUnderTest;
//Don't query areas we've already done
if (HasLocationBeenQueried(latUnderTest, longUnderTest))
{
continue;
}
Thingy thingy = await GetThingyForCoords(latUnderTest, longUnderTest, client);
if (thingy != null)
{
thingyList.Add(thingy);
AdjustQueriedArea(latUnderTest, longUnderTest);
}
}
}
//Negative offset loop, not reached for the purpose of this
//.
//.
return thingyList;
}
Yes, the loops are a bit disgusting but this was just meant to be a quick and dirty first run. Anyway, this is the method making the request:
public async Task<Thingy> GetThingyForCoords(double Lat, double Long, HttpClient Client)
{
try
{
using (HttpResponseMessage resp = await Client.GetAsync(aUrlIKnowWorks))
{
return resp.IsSuccessStatusCode ? JsonConvert.DeserializeObject<Thingy>(await resp.Content.ReadAsStringAsync()) : null;
}
}
catch (Exception e)
{
return null;
}
}
I'm not adding the same object every loop iteration or anything silly like that and the request isn't returning nulls every time so I'm at a loss - my guess is this is some weird threading-related issue stemming from using async on mobile, or some undocumented incompatibility between HttpClient and Xamarin, but I don't really know where to go from here debugging-wise.
Because you only ever call list.Add() there is no way the list can be emptying itself. Therefore you must be looking a different list.
The function must be being called more than once, creating a new list each time. When debugging it is not always obvious that you are at a different list on a different thread.
As an afterthought: Something which might help you detect this kind of issue in the future is to turn on "Show threads in source".
Then, when debugging you will see these icons on the left, hinting that there is a second thread currently waiting at that line of code.
Although, personally I find them a bit unclear.
I've searched through the questions here and all I see are simple theoretical BS. So here is my scenario : we have a new application and one spoiled consumer with their own older system. So in our system when an evaluation reaches a specific state, we are to transmit all the data to their service.
The first part is simple: get the data from the record, put it into their datacontracts, and send the data to them. But the second part is where it gets slippery because it requires sending all supporting documents. So in a real world case I have a Referral Document, an Assessment Document, and a Summary of Findings. So in my main code I'm just saying this :
if (client.ReferralDocument != null)
response = TransmitDocumentAsync(client.ReferralDocument);
if (client.Assessment != null)
response = TransmitDocumentAsync(client.Assessment);
if (cilent.Summary != null)
response = TransmitDocumentAsync(client.Summary);
Now the method called is asynchronous and it is simply
public static async Task<Response> TransmitDocumentAsync(byte[] document)
{
InterimResponse x = await proxy.InsertAttachmentAsync(document, identifier);
return new Task<Response>(new Response(InterimResponse);
}
So I am able to basically throw those documents 'over the wall' to be uploaded without waiting. But what I'm stuck on is how I handle the returned objects and how do I know which document it is tied to?
What I'm asking is what I need to add after the three calls to handle any errors returned as well as any other issues or exceptions that arise? Do I just do an await on return? Do I have 3 return objects (referalResponse, assessmentResponse, summaryResponse) and issue an await on each one? Am I overthinking this and just let things end without concern for the results? :)
If you want to await them one at a time:
if (client.ReferralDocument != null)
await TransmitDocumentAsync(client.ReferralDocument);
if (client.Assessment != null)
await TransmitDocumentAsync(client.Assessment);
if (cilent.Summary != null)
await TransmitDocumentAsync(client.Summary);
If you want to issue them all, and then await all the responses at the same time:
var responses = new List<Task>();
if (client.ReferralDocument != null)
responses.Add(TransmitDocumentAsync(client.ReferralDocument));
if (client.Assessment != null)
responses.Add(TransmitDocumentAsync(client.Assessment));
if (cilent.Summary != null)
responses.Add(TransmitDocumentAsync(client.Summary));
Response[] r = await Task.WhenAll(responses);
On a side note, your TransmitDocumentAsync is incorrect. It should not be constructing a new Task, only the new Response.
I currently have a console application written in C# for production server environment. It's quite simple, but I've been testing it for +- 3 months now on my own server (semi-production, but not a disaster if the program fails). I've been getting stack overflow errors every few weeks or so, though.
Of course, pasting my entire source code here would be quite a long piece, so I will try to explain it the best I can: the program is in an infinite while loop (this is probably the cause of the issue), and it checks every second a few small things and prints to the console every so often (if no activity, every 15min, otherwise up-to every second).
Now why and how can I fix the stack overflow errors? Being able to run it for a couple weeks without issue may seem like a lot, but it is obviously not in a server production environment. My guess is that it's the fact I'm using a while loop, but what alternatives do I have?
Edit: here's a portion of the code:
int timeLoop = 0;
while (true)
{
// If it has been +-10min since last read of messages file, read again
if (timeLoop > 599)
{
Console.WriteLine(DateTime.Now.ToString("HH:mm:ss") + " Reading messages...");
messagesFile = File.ReadAllText(#"messages.cfg");
messages = messagesFile.Split(Environment.NewLine.ToCharArray(), StringSplitOptions.RemoveEmptyEntries);
timeLoop = 0;
}
// For each message, check if time specified equals current time, if so say message globally.
foreach (string x in messages)
{
string[] messageThis = x.Split('~');
int typeLoop = 0;
if (messageThis[2] != "no-loop")
typeLoop = Convert.ToInt16(messageThis[2].Remove(0, 5));
DateTime checkDateTime = DateTime.ParseExact(messageThis[0], "HH:mm:ss", null);
if (typeLoop == 0)
{
if (checkDateTime.ToString("HH:mm:ss") == DateTime.Now.ToString("HH:mm:ss"))
publicMethods.sayGlobal(messageThis[1]);
}
else
{
DateTime originalDateTime = checkDateTime;
do
{
checkDateTime = checkDateTime.AddHours(typeLoop);
if (checkDateTime.ToString("HH:mm:ss") == DateTime.Now.ToString("HH:mm:ss"))
publicMethods.sayGlobal(messageThis[1]);
} while (checkDateTime.ToString("HH:mm:ss") != originalDateTime.ToString("HH:mm:ss"));
}
}
timeLoop++;
Thread.Sleep(1000);
}
Also, I forgot that I actually have this code on Github, which probably helps a lot. I know you shouldn't be using any links to code and have them here, so that's why I included the snippet above - but if you are interested in helping me out the repository is located here. Another note - I know that doing it like this is not very accurate on timing - but this is not much of an issue at the moment.
BattleEyeClient.Connect can call itself in some (failing) circumstances, and this method can be called by sayGlobal - so you probably want to change this code block (from line 98):
catch
{
if (disconnectionType == BattlEyeDisconnectionType.ConnectionLost)
{
Disconnect(BattlEyeDisconnectionType.ConnectionLost);
Connect();
return BattlEyeConnectionResult.ConnectionFailed;
}
else
{
OnConnect(loginCredentials, BattlEyeConnectionResult.ConnectionFailed);
return BattlEyeConnectionResult.ConnectionFailed;
}
}
Maybe keep track of how many reconnection attempts you make or transform this section of code so that it is a while loop rather than a recursive call during this failure mode.
Even worse, of course, is if that recursive Connect call succeeds, this catch block then returns BattlEyeConnectionResult.ConnectionFailed which it probably shouldn't.
I am working on a WebCrawler implementation but am facing a strange memory leak in ASP.NET Web API's HttpClient.
So the cut down version is here:
[UPDATE 2]
I found the problem and it is not HttpClient that is leaking. See my answer.
[UPDATE 1]
I have added dispose with no effect:
static void Main(string[] args)
{
int waiting = 0;
const int MaxWaiting = 100;
var httpClient = new HttpClient();
foreach (var link in File.ReadAllLines("links.txt"))
{
while (waiting>=MaxWaiting)
{
Thread.Sleep(1000);
Console.WriteLine("Waiting ...");
}
httpClient.GetAsync(link)
.ContinueWith(t =>
{
try
{
var httpResponseMessage = t.Result;
if (httpResponseMessage.IsSuccessStatusCode)
httpResponseMessage.Content.LoadIntoBufferAsync()
.ContinueWith(t2=>
{
if(t2.IsFaulted)
{
httpResponseMessage.Dispose();
Console.ForegroundColor = ConsoleColor.Magenta;
Console.WriteLine(t2.Exception);
}
else
{
httpResponseMessage.Content.
ReadAsStringAsync()
.ContinueWith(t3 =>
{
Interlocked.Decrement(ref waiting);
try
{
Console.ForegroundColor = ConsoleColor.White;
Console.WriteLine(httpResponseMessage.RequestMessage.RequestUri);
string s =
t3.Result;
}
catch (Exception ex3)
{
Console.ForegroundColor = ConsoleColor.Yellow;
Console.WriteLine(ex3);
}
httpResponseMessage.Dispose();
});
}
}
);
}
catch(Exception e)
{
Interlocked.Decrement(ref waiting);
Console.ForegroundColor = ConsoleColor.Red;
Console.WriteLine(e);
}
}
);
Interlocked.Increment(ref waiting);
}
Console.Read();
}
The file containing links is available here.
This results in constant rising of the memory. Memory analysis shows many bytes held possibly by the AsyncCallback. I have done many memory leak analysis before but this one seems to be at the HttpClient level.
I am using C# 4.0 so no async/await here so only TPL 4.0 is used.
The code above works but is not optimised and sometimes throws tantrum yet is enough to reproduce the effect. Point is I cannot find any point that could cause memory to be leaked.
OK, I got to the bottom of this. Thanks to #Tugberk, #Darrel and #youssef for spending time on this.
Basically the initial problem was I was spawning too many tasks. This started to take its toll so I had to cut back on this and have some state for making sure the number of concurrent tasks are limited. This is basically a big challenge for writing processes that have to use TPL to schedule the tasks. We can control threads in the thread pool but we also need to control the tasks we are creating so no level of async/await will help this.
I managed to reproduce the leak only a couple of times with this code - other times after growing it would just suddenly drop. I know that there was a revamp of GC in 4.5 so perhaps the issue here is that GC did not kick in enough although I have been looking at perf counters on GC generation 0, 1 and 2 collections.
So the take-away here is that re-using HttpClient does NOT cause memory leak.
I'm no good at defining memory issues but I gave it a try with the following code. It's in .NET 4.5 and uses async/await feature of C#, too. It seems to keep memory usage around 10 - 15 MB for the entire process (not sure if you see this a better memory usage though). But if you watch # Gen 0 Collections, # Gen 1 Collections and # Gen 2 Collections perf counters, they are pretty high with the below code.
If you remove the GC.Collect calls below, it goes back and forth between 30MB - 50MB for entire process. The interesting part is that when I run your code on my 4 core machine, I don't see abnormal memory usage by the process either. I have .NET 4.5 installed on my machine and if you don't, the problem might be related to CLR internals of .NET 4.0 and I am sure that TPL has improved a lot on .NET 4.5 based on resource usage.
class Program {
static void Main(string[] args) {
ServicePointManager.DefaultConnectionLimit = 500;
CrawlAsync().ContinueWith(task => Console.WriteLine("***DONE!"));
Console.ReadLine();
}
private static async Task CrawlAsync() {
int numberOfCores = Environment.ProcessorCount;
List<string> requestUris = File.ReadAllLines(#"C:\Users\Tugberk\Downloads\links.txt").ToList();
ConcurrentDictionary<int, Tuple<Task, HttpRequestMessage>> tasks = new ConcurrentDictionary<int, Tuple<Task, HttpRequestMessage>>();
List<HttpRequestMessage> requestsToDispose = new List<HttpRequestMessage>();
var httpClient = new HttpClient();
for (int i = 0; i < numberOfCores; i++) {
string requestUri = requestUris.First();
var requestMessage = new HttpRequestMessage(HttpMethod.Get, requestUri);
Task task = MakeCall(httpClient, requestMessage);
tasks.AddOrUpdate(task.Id, Tuple.Create(task, requestMessage), (index, t) => t);
requestUris.RemoveAt(0);
}
while (tasks.Values.Count > 0) {
Task task = await Task.WhenAny(tasks.Values.Select(x => x.Item1));
Tuple<Task, HttpRequestMessage> removedTask;
tasks.TryRemove(task.Id, out removedTask);
removedTask.Item1.Dispose();
removedTask.Item2.Dispose();
if (requestUris.Count > 0) {
var requestUri = requestUris.First();
var requestMessage = new HttpRequestMessage(HttpMethod.Get, requestUri);
Task newTask = MakeCall(httpClient, requestMessage);
tasks.AddOrUpdate(newTask.Id, Tuple.Create(newTask, requestMessage), (index, t) => t);
requestUris.RemoveAt(0);
}
GC.Collect(0);
GC.Collect(1);
GC.Collect(2);
}
httpClient.Dispose();
}
private static async Task MakeCall(HttpClient httpClient, HttpRequestMessage requestMessage) {
Console.WriteLine("**Starting new request for {0}!", requestMessage.RequestUri);
var response = await httpClient.SendAsync(requestMessage).ConfigureAwait(false);
Console.WriteLine("**Request is completed for {0}! Status Code: {1}", requestMessage.RequestUri, response.StatusCode);
using (response) {
if (response.IsSuccessStatusCode){
using (response.Content) {
Console.WriteLine("**Getting the HTML for {0}!", requestMessage.RequestUri);
string html = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
Console.WriteLine("**Got the HTML for {0}! Legth: {1}", requestMessage.RequestUri, html.Length);
}
}
else if (response.Content != null) {
response.Content.Dispose();
}
}
}
}
A recent reported "Memory Leak" in our QA environment taught us this:
Consider the TCP Stack
Don't assume the TCP Stack can do what is asked in the time "thought appropriate for the application". Sure we can spin off Tasks at will and we just love asych, but....
Watch the TCP Stack
Run NETSTAT when you think you have a memory leak. If you see residual sessions or half-baked states, you may want to rethink your design along the lines of HTTPClient reuse and limiting the amount of concurrent work being spun up. You also may need to consider using Load Balancing across multiple machines.
Half-baked sessions show up in NETSTAT with Fin-Waits 1 or 2 and Time-Waits or even RST-WAIT 1 and 2. Even "Established" sessions can be virtually dead just waiting for time-outs to fire.
The Stack and .NET are most likely not broken
Overloading the stack puts the machine to sleep. Recovery takes time and 99% of the time the stack will recover. Remember also that .NET will not release resources before their time and that no user has full control of GC.
If you kill the app and it takes 5 minutes for NETSTAT to settle down, that's a pretty good sign the system is overwhelmed. It's also a good show of how the stack is independent of the application.
The default HttpClient leaks when you use it as a short-lived object and create new HttpClients per request.
Here is a reproduction of this behavior.
As a workaround, I was able to keep using HttpClient as a short-lived object by using the following Nuget package instead of the built-in System.Net.Http assembly:
https://www.nuget.org/packages/HttpClient
Not sure what the origin of this package is, however, as soon as I referenced it the memory leak disappeared. Make sure that you remove the reference to the built-in .NET System.Net.Http library and use the Nuget package instead.