Develop Reference

Fastest way to read files in a multi-processing environment? C#

I have the following challenge:
I have a Azure Cloud Worker Role with many instances. Every minute, each instance spins up about 20-30 threads. In each thread, it needs to read some metadata about how to process the thread from 3 objects. The objects/data reside in a remote RavenDb and even though RavenDb is very fast at retrieving the objects via HTTP, it is still under a considerable load from 30+ workers that are hitting it 3 times per thread per minute (about 45 requests/sec). Most of the time (like 99.999%) the data in RavenDb does not change.
I've decided to implement local storage caching. First, I read a tiny record which indicates if the metadata has changed (it changes VERY rarely), and then I read from local file storage instead of RavenDb, if local storage has the object cached. I'm using File.ReadAllText()
This approach appears to be bogging the machine down and procesing slows down considerably. I'm guessing the disks on "Small" Worker Roles are not fast enough.
Is there anyway, I can have OS help me out and cache those files? Perhaps there is an alternative to caching of this data?
I'm looking at about ~1000 files of varying sizes ranging from 100k to 10mb in size stored on each Cloud Role instance

Not a straight answer, but three possible options:
Use the built-in RavenDB caching mechanism
My initial guess is that your caching mechanism is actually hurting performance. The RavenDB client has caching built-in (see here for how to fine-tune it: https://ravendb.net/docs/article-page/3.5/csharp/client-api/how-to/setup-aggressive-caching)
The problem you have is that the cache is local to each server. If server A downloaded a file before, server B will still have to fetch it if it happens to process that file the next time.
One possible option you could implement is divide the workload. For example:
Server A => fetch files that start with A-D
Server B => fetch files that start with E-H
Server C => ...
This would ensure that you optimize the cache on each server.
Get a bigger machine
If you still want to employ your own caching mechanism, there are two things that I imagine could be the bottleneck:
Disk access
Deserialization of the JSON
For these issues, the only thing I can imagine would be to get bigger resources:
If it's the disk, use premium storage with SSD's.
If it's deserialization, get VM's with a bigger CPU
Cache files in RAM
Alternatively, instead of writing the files to disk, store them in memory and get a VM with more RAM. You shouldn't need THAT much RAM, since 1000 files * 10MB is still just 1 GB. Doing this would eliminate disk access and deserialization.
But ultimately, it's probably best to first measure where the bottleneck is and see if it can be mitigated by using RavenDB's built-in caching mechanism.

Understanding memory leaks in ASP.net using RedGate Memory Profiler

I am running a large ASP.net 4.0 website. It uses a popular .Net content management system, has thousands of content items, hundreds of concurrent users - is basically a heavy website.
Over the course of 1 day the memory usage of the IIS7 worker process can rise to 8-10GB. The server has 16GB installed and is currently set to recycle the app pool once per day.
I am getting pressured to reduce memory usage. Much of the memory usage is due to caching of large strings of data - but the cache interval is only set to 5-10 minutes - so these strings should eventually expire from memory.
However after running RedGate Memory Profiler I can see what I think are memory leaks. I have filtered my Instance List results by objects that are "kept in memory exclusively by Disposed Objects" ( I read on the RedGate forum that this is how you find memory leaks ). This gave me a long list of strings that are being held in memory.
For each string I use Instance Retention Graph to see what holds it in memory. The System.string objects seem to have been cached at some point by System.Web.Caching.CacheDependency. If I follow the graph all the way up it goes through various other classes including System.Collections.Specialized.ListDictionary until it reaches System.Web.FileMonitor. This makes some sense as the strings are paths to a file (images / PDFs / etc).
It seems that the CMS is caching paths to files, but these cached objects are then "leaked". Over time this builds up and eats up RAM.
Sorry this is long winded... Is there a way for me to stop these memory leaks? Or to clear them down without resorting to recycling the app pool? Can I find what class / code is doing the caching to see if I can fix the leak?

It sounds like the very common problem of stuff being left in memory as part of session state. If that's the case your only options are 1. don't put so much stuff in each user's session, 2. Set the session lifetime to something shorter (the default is 20 minutes, I think), and 3. periodically recycle the app pool.
As part of 1. I found that there are "good ways" and "bad ways" of presenting data in a data grid control. You may want to check that you are copying only the data you need and not accidentally maintaining references to the entire datagrid.

Starting a java virtual machine with -Xms8M parameter fails in .NET Compact Framework (C#)

I'm looking for a solution for about 1 1/2 days now and just can't get to the point. I tried to start a *.lnk file in PocketPC 2003 out of our C# application. This *.lnk file contains a link to evm.exe which is a JVM for PocketPC. Argument passed is (besides others) -Xms8M which tells the JVM to reserve at least 8MB of memory.
If directly started from Windows Explorer there's no problem.
Now I created a process in C# pointing to the *.lnk file. When I try to start it the JVM console opens and brings up one of two errors: "EVM execution history too large" or "failed to initialize heap (Phase 1)" (or something like that).
If I delete the mentioned parameter the application comes up with no problem.
Because of this behaviour I assume that there is too few memory assigned to the newly created process. Is this realistic? And if: is there a way to assign more memory to the newly created process? Or am I completely wrong and have to go some other way (if any available)?
Edit:
--CodeSnippet--
this.myStartProcess = new Process { StartInfo = { FileName = appName },EnableRaisingEvents = true };
this.myStartProcess.Start()--CodeSnippet--
Edit 2:
After doing some more research it turned out that the real problem is that there are very limited resources available, eaten up by my launcher application (which is about 1.8 MB in total after starting) over time.
To improve things I started to study how the garbage collector works in Windows Mobile and so used two techniques to bring up the virtual machine.
First one is to reduce the memory taken by my own application by sending it to the background (SendToBack() method of the form) and waiting for the garbage collector to finish (GC.WaitForPendingFinalizers()).
After that I'm looking for 9 MB of free space in program memory before trying to bring the VM up. If there isn't enough space I try to shift the needed memory from storage memory to program memory.
This two techniques improved things a lot!
There's still a problem to my launcher application. The allocated bytes (strings and boxed objects to be concrete) increase over time when my launcher application is in front... It's about 30 kb in 10 minutes. After 24 hours the device will be rebooted automatically. At the moment I assume the launcher will be in front for about 10 minutes total during that period. Nevertheless it's not good to have memory leaks. Anyone got an idea how to chase this down?
Thanks in advance
Best regards
Marcel

It looks you have two reasons for which this could happen:
The default value provided to MinWorkingSet and MaxWorkingSet Properties are not satisfactory for your requirements. From http://msdn.microsoft.com/en-us/library/system.diagnostics.process.maxworkingset.aspx
The working set of a process is the set of memory pages currently
visible to the process in physical RAM memory. These pages are
resident and available for an application to use without triggering a
page fault.
The working set includes both shared and private data. The shared data
includes the pages that contain all the instructions that your
application executes, including the pages in your .dll files and the
system.dll files. As the working set size increases, memory demand
increases.
A process has minimum and maximum working set sizes. Each time a
process resource is created, the system reserves an amount of memory
equal to the minimum working set size for the process. The virtual
memory manager attempts to keep at least the minimum amount of memory
resident when the process is active, but it never keeps more than the
maximum size.
The system sets the default working set sizes. You can modify these
sizes using the MaxWorkingSet and MinWorkingSet members. However,
setting these values does not guarantee that the memory will be
reserved or resident.
It is effectively impossible to reserve the memory you are requiring for the JVM on your machine, because the way the OS manages memory ( which I would find really suprising because every modern os has virtual memory support)

MongoDB slow writes causes socket time out exception

I am having performance issues with MongoDB.
Running on:
MongoDB 2.0.1
Windows 2008 R2
12 GB RAM
2 TB HDD (5400 rpm)
I've written a daemon which removes and inserts records async. Each hour most of the collections are cleared and they'll get new inserted data (10-12 million deletes and 10-12 million inserts). The daemon uses ~60-80 of the CPU while inserting the data (due calculating 1+ million knapsack problems). When I fire up the daemon it can do it's job about 1-2 mins till it crashes due a socket time out (writing data to the MongoDB server).
When I look in the logs I see it takes about 30 seconds to remove data in the collection. It seems it has something to do with the CPU load and memory usage.., because when I run the daemon on a different PC everything goes fine.
Is there any optimization possible or I am just bound to using a separate PC for running the daemon (or pick another document store)?
UPDATE 11/13/2011 18:44 GMT+1
Still having problems.. I've made some modifications to my daemon. I've decreased the concurrent number of writes. However the daemon still crashes when the memory is getting full (11.8GB of 12GB) and receives more load (loading data into the frontend). It crashes due a long insert/remove of MongoDB(30 seconds). The crash of the daemon is because of MongoDB is responding slow (socket time out exception). Ofcourse there should be try/catch statements to catch such exceptions, but it should not happen in the first place. I'm looking for a solution to solve this issue instead of working around it.
Total storage size is: 8,1 GB
Index size is: 2,1 GB
I guess the problem lies in that the working set + indexes are too large to store in memory and MongoDB needs to access the HDD (which is slow 5400 rpm).. However why would this be a problem? Aren't there other strategies to store the collections (e.g. in seperate files instead of large chunks of 2GB). If an Relational database can read/write data in an acceptable amount of time from the disk, why can't MongoDB?
UPDATE 11/15/2011 00:04 GMT+1
Log file to illustrate the issue:
00:02:46 [conn3] insert bargains.auction-history-eu-bloodhoof-horde 421ms
00:02:47 [conn6] insert bargains.auction-history-eu-blackhand-horde 1357ms
00:02:48 [conn3] insert bargains.auction-history-eu-bloodhoof-alliance 577ms
00:02:48 [conn6] insert bargains.auction-history-eu-blackhand-alliance 499ms
00:02:49 [conn4] remove bargains.crafts-eu-agamaggan-horde 34881ms
00:02:49 [conn5] remove bargains.crafts-eu-aggramar-horde 3135ms
00:02:49 [conn5] insert bargains.crafts-eu-aggramar-horde 234ms
00:02:50 [conn2] remove bargains.auctions-eu-aerie-peak-horde 36223ms
00:02:52 [conn5] remove bargains.auctions-eu-aegwynn-horde 1700ms
UPDATE 11/18/2011 10:41 GMT+1
After posting this issue in the mongodb usergroup we found out that "drop" wasn't issued. Drop is much faster then a full remove of all records.
I am using official mongodb-csharp-driver. I issued this command collection.Drop();. However It didn't work, so for the time being I used this:
public void Clear()
{
if (collection.Exists())
{
var command = new CommandDocument {
{ "drop", collectionName }
};
collection.Database.RunCommand(command);
}
}
The daemon is quite stable now, yet I have to find out why the collection.Drop() method doesn't work as it supposed to, since the driver uses the native drop command aswell.

Some optimizations may be possible:
Make sure your mongodb is not running in verbose mode, this will ensure minimal logging and hence minimal I/O . Else it writes every operation to a log file.
If possible by application logic, convert your inserts to bulk inserts.Bulk insert is supported in most mongodb drivers.
http://www.mongodb.org/display/DOCS/Inserting#Inserting-Bulkinserts
Instead of one remove operation per record, try to remove in bulk.
eg. collect "_id" of 1000 documents, then fire a remove query using $in operator.
You will have 1000 times less queries to mongoDb.
If you are removing/inserting for same document to refresh data, try considering an update instead.
What kind of deamon are you running ? If you can share more info on that,it may be possible to optimize that too to reduce CPU load.

It could be totally unrelated, but there was an issue in 2.0.0 that had to do with CPU consumption. after upgrade to 2.0.0 mongo starts consuming all cpu resources locking the system, complains of memory leak

Unless I have misunderstood, your application is crashing, not mongod. Have you tried to remove MongoDB from the picture and replacing writes to MongoDB with perhaps writes to the file system?
Maybe this will bring light to some other issue inside your application that is not related specifically to MongoDB.

I had something similar happen with SQL Server 2008 on Windows Server 2008 R2. For me, it ended up being the network card. The NIC was set to auto-sense the connection speed which was leading to occasional dropped/lost packets which was leading to the socket timeout problems. To test you can ping the box from your local workstation and kick off your process to load the Windows 2008 R2 server. If it is this problem eventually you'll start to see the timeouts on your ping command
ping yourWin2008R2Server -n 1000
The solution ended up being to explicitly set the NIC connection speed
Manage Computer > Device Manager > Network Adapters > Properties and then depending on the nic you'll have either a link speed setting tab or have to go into another menu. You'll want to set this to exactly the speed of the network it is connected to. In my DEV environment it ended up being 100Mbps Half duplex.
These types of problems, as you know, can be a real pain to track down!
Best to you in figuring it out.

The daemon is stable now, After posting this issue in the mongodb usergroup we found out that "drop" wasn't issued. Drop is much faster then a full remove of all records.

How to resolve Out of memory exception in WP7 using dictionary?

I build an application in WP7 where i require to load around 20000 hard coded data({'a',"XYZ"},{'b',"mno"},....)on which i have to perform the search. So i trying to do this by creating a dictionary making 'a' as key and value as "XYZ". As soon as my dictionary gets filled it gives Out of memory exception.
How can i solve this problem considering that i m building WP7 application?
Or Is there some way other than using dictionary?

Whenever you are loading so much data onto a phone, you're doing it wrong. Firstly, the bandwidth issue is going to kill your app. Second, the memory issue has already killed your app. Thirdly, the CPU issue is going to kill your app. The conclusion is, your user ends up killing your app.
Recommended solution: find a way to categorize the data so that not all of it must download to the phone. Do your processing on the server where it belongs (not on a phone).
If you insist on processing so much data on the phone, first try to manage the download size. Remember you're talking about a mobile phone here, and not everywhere has max 3G speeds. Try to compress the data structure as much as possible (e.g. using a tree to store common prefixes). Also try to zip up the data before downloading.
Then count your per-object memory usage aggressively. Putting in 20,000 strings can easily consume a lot of memory. You'd want to reduce the size of per-object memory usage as possible. In your example, you are just putting strings in there, so I can't guess how you'd be using up the tens of MB allowable on a WP7 app. However, if you are putting not just strings, but large objects, count the bytes.
Also, manage fragementation aggressively. The last thing you'll want to do is to new Dictionary() then dict.Add(x,y); in a for-loop. When the dictionary's internal table space runs full, it gets allocated to a new place, and the entire dictionary copied to the new place, wasting the original space. You end up having lots of fragmented memory space. Do a new Dictionary(20000) or something to reserve the space first in one go.

Instead of storing it in memory as a Dictionary you can store it in a Database(wp7sqlite) and fetch only the data required.In this way you can store whatever amount of data.
Edit
No nothing is required in extra from user end.you can create the database using sqlite manager,attach this to the project.Copy DB to Isolated storage on first usage.and you can access the DB whenever you want .Check this Link DB helper.This link uses sqlitewindowsphone instead of WP7Sqlite.I prefer wp7sqlite Since i got an error using sqlitewindowsphone.