My problem is being unable or not knowing how to clear the memory being flooded by images (bitmaps) not being used anymore. The function's purpose is to change the background of the form to a new image every x amount of seconds.
The memory usage will inevitably overflow and it will crash. But even when I am changing to a different window, I run this.Close() and the memory usage is still constantly increasing.
Here is the function:
public async void WaitSomeTime(String[] favs, int time)
{
while (true)
{
var rnd = new Random();
favs = favs.OrderBy(item => rnd.Next()).ToArray();
foreach (string fav in favs)
{
await Task.Delay(time);
Image img = new Bitmap(fav);
this.pictureBoxBG.Image = img;
}
}
}
So far I've tried the Dispose method but to no avail, I don't completely understand it. I've tried the 'using' statement but that causes an error in Program.cs (entry point). I'm sure it's a simple fix but I'm out of ideas and GPT3 isn't helping very well, thanks in advance.
Using the Diagnostic Tools in Visual Studio we can get an idea of how your current code behaves with respect to process memory over a ~5-minute period. Updates are set to occur ~10 times a second to provide a test load. Even though the image files I'm using for testing aren't very big the effect on memory consumption is apparent.
One approach you could try that seems to keep this it in check is to explicitly dispose of images and then collect garbage. It seems reasonable to apply this after each complete pass through the list of favs but you get general drift and can apply it wherever you choose.
To be clear, the critical step is to Dispose the previous image and GC will take care of it from there. But if your goal is to keep the footprint as low as possible this "extraordinary measure" allows you to exercise more control.
public partial class MainForm : Form
{
public MainForm()
{
InitializeComponent();
pictureBoxBG.SizeMode = PictureBoxSizeMode.StretchImage;
string dir = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "Images");
string[] favs = Enumerable.Range(0, 10).Select(_=>Path.Combine(dir, $"image-{_}.png")).ToArray();
_task = WaitSomeTime(favs, 100);
}
Task _task;
Random rnd = new Random();
public async Task WaitSomeTime(String[] favs, int time)
{
while (true)
{
List<Image> trash= new List<Image>();
favs = favs.OrderBy(item => rnd.Next()).ToArray();
foreach (string fav in favs)
{
await Task.Delay(time);
if(pictureBoxBG.Image != null)
{
trash.Add(pictureBoxBG.Image);
}
Image img = new Bitmap(fav);
pictureBoxBG.Image = img;
}
for (int i = 0; i < trash.Count; i++)
{
trash[i].Dispose();
trash[i] = null;
}
// Requesting an expedited disposal of managed memory.
GC.Collect();
}
}
}
Related
I've notice in my test code that using the ConcurrentQueue<> somehow does not release resources after Dequeueing and eventually I run out of memory. Or the Garbage collection is not happening frequently enough.
Here is a snippet of the code. I know the ConcurrentQueue<> store references and yes, I do want create a new object each time so if the enqueueing is faster than dequeueing, memory will continue to rise. Also a screenshot of the memory usage. For testing, I sent through 5000 byte arrays with 500000 elements each.
There is a similar question asked:
ConcurrentQueue holds object's reference or value? "out of memory" exception
and everything mentioned in that post is what I experienced ... except that the memory won't release after dequeueing, even when the Queue is emptied.
I would appreciate any thoughts/insights to this.
ConcurrentQueue<byte[]> TestQueue = new ConcurrentQueue<byte[]>();
Task EnqTask = Task.Factory.StartNew(() =>
{
for (int i = 0; i < ObjCount; i++)
{
byte[] InData = new byte[ObjSize];
InData[0] = (byte)i; //used to show different array object
TestQueue.Enqueue(InData);
System.Threading.Thread.Sleep(20);
}
});
Task DeqTask = Task.Factory.StartNew(() =>
{
int Count = 0;
while (Count < ObjCount)
{
byte[] OutData;
if (TestQueue.TryDequeue(out OutData))
{
OutData[1] = 0xFF; //just do something with the data
Count++;
}
System.Threading.Thread.Sleep(40);
}
Picture of memory
I load data from my DB and when I push buttons to recieve different data I get "hill climbing, change max number of threads 5" in the log, and it makes the app slow when I try to gather the data.
Any idea how to resolve this? Or is this even making the app slower? It sure seems like something is spooky because it takes a few extra seconds when I load data on an android-device compared to an ios.
This is my code:
static public async Task<JObject> getContacts ()
{
var httpClientRequest = new HttpClient ();
try {
var result = await httpClientRequest.GetAsync ("http://address.com");
var resultString = await result.Content.ReadAsStringAsync ();
var jsonResult = JObject.Parse (resultString);
return jsonResult;
} catch {
return null;
}
}
And how I use it:
async void createData (object s, EventArgs a)
{
var getContacts = await parseAPI.getContacts ();
if (getContacts != null) {
listview.ItemsSource = null;
theList = new List <items> ();
foreach (var items in getContacts["results"]) {
theList.Add (new items () {
Name = items ["Name"].ToString (),
Number = items ["Number"].ToString ()
});
}
}
listview.ItemsSource = theList;
}
"Hill climbing" is a fairly common within the Mono runtime as the adaptative thread pool count is increasing (or decreasing) based upon current depends.
Personally I doubt that a thread count of 5 is causing any problems within your app. Seeing 30, 50, 100+ could/would be a problem as the thrashing of context switching can/will bring an app (and OS) to its knees.
In terms of OS "speed", the iOS simulator vs. an Android emulator is huge. Instrument and Perf test on actual devices.
The Mono Heuristic thread pool reference:
https://github.com/mono/mono/blob/master/mono/metadata/threadpool-ms.c
hill_climbing_change_thread_count (gint16 new_thread_count, ThreadPoolHeuristicStateTransition transition)
{
ThreadPoolHillClimbing *hc;
g_assert (threadpool);
hc = &threadpool->heuristic_hill_climbing;
mono_trace (G_LOG_LEVEL_INFO, MONO_TRACE_THREADPOOL, "[%p] hill climbing, change max number of threads %d", mono_native_thread_id_get (), new_thread_count);
hc->last_thread_count = new_thread_count;
hc->current_sample_interval = rand_next (&hc->random_interval_generator, hc->sample_interval_low, hc->sample_interval_high);
hc->elapsed_since_last_change = 0;
hc->completions_since_last_change = 0;
}
I will try to tell my problem in as simple words as possible.
In my UWP app, I am loading the data async wise on my Mainpage.xaml.cs`
public MainPage()
{
this.InitializeComponent();
LoadVideoLibrary();
}
private async void LoadVideoLibrary()
{
FoldersData = new List<FolderData>();
var folders = (await Windows.Storage.StorageLibrary.GetLibraryAsync
(Windows.Storage.KnownLibraryId.Videos)).Folders;
foreach (var folder in folders)
{
var files = (await folder.GetFilesAsync(Windows.Storage.Search.CommonFileQuery.OrderByDate)).ToList();
FoldersData.Add(new FolderData { files = files, foldername = folder.DisplayName, folderid = folder.FolderRelativeId });
}
}
so this is the code where I am loading up a List of FolderData objects.
There in my other page Library.xaml.cs I am using that data to load up my gridview with binding data.
protected override void OnNavigatedTo(NavigationEventArgs e)
{
try
{
LoadLibraryMenuGrid();
}
catch { }
}
private async void LoadLibraryMenuGrid()
{
MenuGridItems = new ObservableCollection<MenuItemModel>();
var data = MainPage.FoldersData;
foreach (var folder in data)
{
var image = new BitmapImage();
if (folder.files.Count == 0)
{
image.UriSource = new Uri("ms-appx:///Assets/StoreLogo.png");
}
else
{
for (int i = 0; i < folder.files.Count; i++)
{
var thumb = (await folder.files[i].GetThumbnailAsync(Windows.Storage.FileProperties.ThumbnailMode.VideosView));
if (thumb != null) { await image.SetSourceAsync(thumb); break; }
}
}
MenuGridItems.Add(new MenuItemModel
{
numberofvideos = folder.files.Count.ToString(),
folder = folder.foldername,
folderid = folder.folderid,
image = image
});
}
GridHeader = "Library";
}
the problem I am facing is that when i launch my application, wait for a few seconds and then i navigate to my library page, all data loads up properly.
but when i try to navigate to library page instantly after launching the app, it gives an exception that
"collection was modified so it cannot be iterated"
I used the breakpoint and i came to know that if i give it a few seconds the List Folder Data is already loaded properly asyncornously, but when i dnt give it a few seconds, that async method is on half way of loading the data so it causes exception, how can i handle this async situation? thanks
What you need is a way to wait for data to arrive. How you fit that in with the rest of the application (e.g. MVVM or not) is a different story, and not important right now. Don't overcomplicate things. For example, you only need an ObservableCollection if you expect the data to change while the user it looking at it.
Anyway, you need to wait. So how do you wait for that data to arrive?
Use a static class that can be reached from everywhere. In there put a method to get your data. Make sure it returns a task that you cache for future calls. For example:
internal class Data { /* whatever */ }
internal static class DataLoader
{
private static Task<Data> loaderTask;
public static Task<Data> LoadDataAsync(bool refresh = false)
{
if (refresh || loaderTask == null)
{
loaderTask = LoadDataCoreAsync();
}
return loaderTask;
}
private static async Task<Data> LoadDataCoreAsync()
{
// your actual logic goes here
}
}
With this, you can start the download as soon as you start the application.
await DataLoader.LoadDataAsync();
When you need the data in that other screen, just call that method again. It will not download the data again (unless you set refresh is true), but will simply wait for the work that you started earlier to finish, if it is not finished yet.
I get that you don't have enough experience.There are multiple issues and no solution the way you are loading the data.
What you need is a Service that can give you ObservableCollection of FolderData. I think MVVM might be out of bounds at this instance unless you are willing to spend a few hours on it. Though MVVM will make things lot easier in this instance.
The main issue at hand is this
You are using foreach to iterate the folders and the FolderData list. Foreach cannot continue if the underlying collection changes.
Firstly you need to start using a for loop as opposed to foreach. 2ndly add a state which denotes whether loading has finished or not. Finally use observable data source. In my early days I used to create static properties in App.xaml.cs and I used to use them to share / observe other data.
I have a CVS file with over 1 Million rows of data. I am planning to read them in parallel to improve efficiency. Can I do something like the following or is there a more efficient method?
namespace ParallelData
{
public partial class ParallelData : Form
{
public ParallelData()
{
InitializeComponent();
}
private static readonly char[] Separators = { ',', ' ' };
private static void ProcessFile()
{
var lines = File.ReadLines("BigData.csv");
var numbers = ProcessRawNumbers(lines);
var rowTotal = new List<double>();
var totalElements = 0;
foreach (var values in numbers)
{
var sumOfRow = values.Sum();
rowTotal.Add(sumOfRow);
totalElements += values.Count;
}
MessageBox.Show(totalElements.ToString());
}
private static List<List<double>> ProcessRawNumbers(IEnumerable<string> lines)
{
var numbers = new List<List<double>>();
/*System.Threading.Tasks.*/
Parallel.ForEach(lines, line =>
{
lock (numbers)
{
numbers.Add(ProcessLine(line));
}
});
return numbers;
}
private static List<double> ProcessLine(string line)
{
var list = new List<double>();
foreach (var s in line.Split(Separators, StringSplitOptions.RemoveEmptyEntries))
{
double i;
if (Double.TryParse(s, out i))
{
list.Add(i);
}
}
return list;
}
private void button2_Click(object sender, EventArgs e)
{
ProcessFile();
}
}
}
I'm not sure it's a good idea. Depending on your hardware, the CPU won't be a bottleneck, the disk read speed will.
Another point: if your storage hardware is a magnetic hard disk, then then disk read speed is strongly related to how the file is physically stored in the disk; if the file is not fragmented (i.e. all file chunks are sequentially stored on the disk), you'll have better performances if you read line by line sequentially.
One solution would be to read the whole file in one time (if you have enough memory space, for 1 million row it should be OK) using File.ReadAllLines, store all lines in a string array, then process (i.e. parse using string.Split...etc.) in your Parallel.Foreach, if the rows order is not important.
In general you should try to avoid having disk access on multiple threads. The disk is a bottleneck and will block, so might impact performance.
If the size of the lines in the file is not an issue, you should probably read the entire file in first, and then process in parallel.
If the file is too large to do that or it's not practical, then you could use BlockingCollection to load it. Use one thread to read the file and populate the BlockingCollection and then Parallel.ForEach to process the items in it. BlockingCollection allows you to specify the max size of the collection, so it will only read more lines from the file as what's already in the collection is processed and removed.
static void Main(string[] args)
{
string filename = #"c:\vs\temp\test.txt";
int maxEntries = 2;
var c = new BlockingCollection<String>(maxEntries);
var taskAdding = Task.Factory.StartNew(delegate
{
var lines = File.ReadLines(filename);
foreach (var line in lines)
{
c.Add(line); // when there are maxEntries items
// in the collection, this line
// and thread will block until
// the processing thread removes
// an item
}
c.CompleteAdding(); // this tells the collection there's
// nothing more to be added, so the
// enumerator in the other thread can
// end
});
while (c.Count < 1)
{
// this is here simply to give the adding thread time to
// spin up in this much simplified sample
}
Parallel.ForEach(c.GetConsumingEnumerable(), i =>
{
// NOTE: GetConsumingEnumerable() removes items from the
// collection as it enumerates over it, this frees up
// the space in the collection for the other thread
// to write more lines from the file
Console.WriteLine(i);
});
Console.ReadLine();
}
As with some of the others, though, I have to ask the question: Is this something you really need to try optimizing through parallelization, or would a single-threaded solution perform well enough? Multithreading adds a lot of complexity and it's sometimes not worth it.
What kind of performance are you seeing that you want to improve upon?
I checked those lines on my computer and it looks like using Parallel to read csv file without any cpu-expensive computation make no sense. It takes more time to run this in parallel than in one thread. Here are my result:
For code above:
2699ms 2712ms (Checked twice just to confirm results)
Then with:
private static IEnumerable<List<double>> ProcessRawNumbers2(IEnumerable<string> lines)
{
var numbers = new List<List<double>>();
foreach(var line in lines)
{
lock (numbers)
{
numbers.Add(ProcessLine(line));
}
}
return numbers;
}
Gives me: 2075ms 2106ms
So I think that if those numbers in csv does not require to be computed somehow (with some extensive calculation or so) in program and then stored in program, than it make no sense to use parallelism in such case as this add some overhead to it.
Environment : .net 4.0
I have a task that transforms XML files with a XSLT stylesheet, here is my code
public string TransformFileIntoTempFile(string xsltPath,
string xmlPath)
{
var transform = new MvpXslTransform();
transform.Load(xsltPath, new XsltSettings(true, false),
new XmlUrlResolver());
string tempPath = Path.GetTempFileName();
using (var writer = new StreamWriter(tempPath))
{
using (XmlReader reader = XmlReader.Create(xmlPath))
{
transform.Transform(new XmlInput(reader), null,
new XmlOutput(writer));
}
}
return tempPath;
}
I have X threads that can launch this task in parallel.
Sometimes my input file are about 300 MB, sometimes it's only a few MB.
My problem : I get OutOfMemoryException when my program try to transform some big XML files in the same time.
How can I avoid these OutOfMemoryEception ? My idea is to stop a thread before executing the task until there is enough available memory, but I don't know how to do that. Or there is some other solution (like putting my task in a distinct application).
Thanks
I don't recommend blocking a thread. In worst case, you'll just end up starving the task that could potentially free the memory you needed, leading to deadlock or very bad performance in general.
Instead, I suggest you keep a work queue with priorities. Get the tasks from the Queue scheduled fairly across a thread pool. Make sure no thread ever blocks on a wait operation, instead repost the task to the queue (with a lower priority).
So what you'd do (e.g. on receiving an OutOfMemory exception), is post the same job/task onto the queue and terminate the current task, freeing up the thread for another task.
A simplistic approach is to use LIFO which ensures that a task posted to the queue will have 'lower priority' than any other jobs already on that queue.
Since .NET Framework 4 we have API to work with good old Memory-Mapped Files feature which is available many years within from Win32API, so now you can use it from the .NET Managed Code.
For your task better fit "Persisted memory-mapped files" option,
MSDN:
Persisted files are memory-mapped files that are associated with a
source file on a disk. When the last process has finished working with
the file, the data is saved to the source file on the disk. These
memory-mapped files are suitable for working with extremely large
source files.
On the page of MemoryMappedFile.CreateFromFile() method description you can find a nice example describing how to create a memory mapped Views for the extremely large file.
EDIT: Update regarding considerable notes in comments
Just found method MemoryMappedFile.CreateViewStream() which creates a stream of type MemoryMappedViewStream which is inherited from a System.IO.Stream.
I believe you can create an instance of XmlReader from this stream and then instantiate your custom implementation of the XslTransform using this reader/stream.
EDIT2: remi bourgarel (OP) already tested this approach and looks like this particular XslTransform implementation (I wonder whether ANY would) wont work with MM-View stream in way which was supposed
The main problem is that you are loading the entire Xml file. If you were to just transform-as-you-read the out of memory problem should not normally appear.
That being said I found a MS support article which suggests how it can be done:
http://support.microsoft.com/kb/300934
Disclaimer: I did not test this so if you use it and it works please let us know.
You could consider using a queue to throttle how many concurrent transforms are being done based on some sort of artificial memory boundary e.g. file size. Something like the following could be used.
This sort of throttling strategy can be combined with maximum number of concurrent files being processed to ensure your disk is not being thrashed too much.
NB I have not included necessary try\catch\finally around execution to ensure that exceptions are propogated to calling thread and Waithandles are always released. I could go into further detail here.
public static class QueuedXmlTransform
{
private const int MaxBatchSizeMB = 300;
private const double MB = (1024 * 1024);
private static readonly object SyncObj = new object();
private static readonly TaskQueue Tasks = new TaskQueue();
private static readonly Action Join = () => { };
private static double _CurrentBatchSizeMb;
public static string Transform(string xsltPath, string xmlPath)
{
string tempPath = Path.GetTempFileName();
using (AutoResetEvent transformedEvent = new AutoResetEvent(false))
{
Action transformTask = () =>
{
MvpXslTransform transform = new MvpXslTransform();
transform.Load(xsltPath, new XsltSettings(true, false),
new XmlUrlResolver());
using (StreamWriter writer = new StreamWriter(tempPath))
using (XmlReader reader = XmlReader.Create(xmlPath))
{
transform.Transform(new XmlInput(reader), null,
new XmlOutput(writer));
}
transformedEvent.Set();
};
double fileSizeMb = new FileInfo(xmlPath).Length / MB;
lock (SyncObj)
{
if ((_CurrentBatchSizeMb += fileSizeMb) > MaxBatchSizeMB)
{
_CurrentBatchSizeMb = fileSizeMb;
Tasks.Queue(isParallel: false, task: Join);
}
Tasks.Queue(isParallel: true, task: transformTask);
}
transformedEvent.WaitOne();
}
return tempPath;
}
private class TaskQueue
{
private readonly object _syncObj = new object();
private readonly Queue<QTask> _tasks = new Queue<QTask>();
private int _runningTaskCount;
public void Queue(bool isParallel, Action task)
{
lock (_syncObj)
{
_tasks.Enqueue(new QTask { IsParallel = isParallel, Task = task });
}
ProcessTaskQueue();
}
private void ProcessTaskQueue()
{
lock (_syncObj)
{
if (_runningTaskCount != 0) return;
while (_tasks.Count > 0 && _tasks.Peek().IsParallel)
{
QTask parallelTask = _tasks.Dequeue();
QueueUserWorkItem(parallelTask);
}
if (_tasks.Count > 0 && _runningTaskCount == 0)
{
QTask serialTask = _tasks.Dequeue();
QueueUserWorkItem(serialTask);
}
}
}
private void QueueUserWorkItem(QTask qTask)
{
Action completionTask = () =>
{
qTask.Task();
OnTaskCompleted();
};
_runningTaskCount++;
ThreadPool.QueueUserWorkItem(_ => completionTask());
}
private void OnTaskCompleted()
{
lock (_syncObj)
{
if (--_runningTaskCount == 0)
{
ProcessTaskQueue();
}
}
}
private class QTask
{
public Action Task { get; set; }
public bool IsParallel { get; set; }
}
}
}
Update
Fixed bug in maintaining batch size when rolling over to next batch window:
_CurrentBatchSizeMb = fileSizeMb;