I need to tweak some variables (only in a development setting) without having to restart IIS or anything (so I assume Web.Config is the wrong place to put them). Where is the easiest place to put about 500 config settings that have to be read for every request and written to, like I said, while IIS is running?
EDIT: Like I said, this is only for some Q&D development so I don't care about performance in any way. A database is a bit of overkill (and is probably more work than I want to deal with), I want something fast (like Settings), that I don't have to worry about parsing and can read from and write to. If I do XML, where do I write the file to so I don't have to spend time messing around with permissions?
In a database?
500 Config Settings to be read for every request? I'd put them in a database so they can be indexed and cached. A separate XML or data file would also most likely be cached in memory by the web server, but still wouldn't provide the performance an indexed database table could. But it depends on how you are accessing the settings.
You can just make your own "config" file. Just don't name it .config. Then you can read it just like a text file and set all your properties. Just have to either implement your own file monitoring class or something to know the file has changed so you can update your code.
With that many configuration options a database system, with some well thought caching is going to most likely be the best idea overall!
You have to be sure to consider the impacts of loading/storing them on all requests as well, as even with small sized values, that can be a big amount of overhead. SO caching is going to be very important.
I know you said you don't want a database, but with 500 settings, it just seems like the best solution.
That said, if you really don't want a database, you could always dump them into an xml file stored locally and just read/write when needed.
Related
We have an application on the web that must allow the user to upload files with zip codes, these files are .csv's files. Any user will be able to upload the file from their computer, the issue is that the file may contain thousands of records. Right now i am getting the file, making sure it has the right headers but I am pushing the records one by one into the database.
I am using c# asp.net, is there a better way to do this?, more efficient from the code?. We cant use any external importers or data importers or tools like sql server business intelligence. How can I do this?, i was reading something about putting it in memory and then push it to the database?. Any urls, examples or suggestions would be much appreciated.
Regards
Firstly, I'm pretty sure that what you are asking is actually "How do you process a large file and insert the processed data into the database?".
Now assuming I am correct I would say the question is akin to 'how long is a piece of string?'. The reality is that an implementation for processing large files into a database is highly specific to your requirements.
However, at the simplest end of the spectrum you could simply upload the file straight into a table (or folder) and create a windows service that runs every x minutes, traverses through the table, picks each file and processes your data using bulk inserts and the prepare method (which may give you some performance benefits).
Alternatively you could look at something like MSMQ (Microsoft Message Queuing) and save any uploaded files direct to a queue which is then completely independent of your application and can be processed at any point in time along with easily scaled out.
At the end of the day though, honestly I don't think anyone here can give you a 'correct' answer to your question cause there really isn't one and you'll only be able to find improvements to your implementation by experimentation.
if this contains up to a million record, best to do this is to create a service to manage the inserting of records into the database to avoid timeout and prevent the web iis stress.
if you make it a windows service you can notify the service to process the zip files in certain directory where it was uploaded.
also, i would suggest to use bulk insert for more faster database transactions.
if there are validation you can probably stage the data into a different database and validate the data then push to the final database.
Since these records are in the same table and would then not be related to each other, Parallel.ForEach may be a valid answer here. Assuming you have a static method (may not necessarily need to be static) that inserts an individual record into the db, you can run Parallel.ForEach loop over an array where each index of the array represents a line of the CSV.
This assumes that uploading the large file to the server isn't the initial issue. If that is also part of the issue I would reccomend zipping the file and then using something like SharpZipLib to unzip it once it is uploaded. Since text compresses very well this may be the biggest boon to performance from the user's perspective.
I have created a website where a non-trivial number of users will be reading from and/or writing to a single XML file at the exact same time.
I am not asking for suggestions on an alternative to the current setup I just want to know if I was wrong with that approach or correct. If it was wrong to do it that way, please explain to me why that is wrong?
Reads are sort-of fine. You load the file into RAM, and it's plenty fast. The only problem is stale data, but since we're just talking about reads at this point it's okay.
Writes are the real problem. No one runs a serious web site using xml as the data store where there's potentially more than one writer (or at very most a handful) in the same file. This is the problem a relational database (or even nosql database) was made to solve. Look into sql server express edition instead (it's free).
I am working on an application that reads and makes edits to an xml file but using the XDocument class Load() and Save() methods. If another application, or another instance of my application is running then one document could potentially overwrite changes that the other has made if it's in memory xdocument is not continually updating (reLoading). The simultaneous running programs would never edit the same section of the xml file at the same time. What is the best way to solve this problem? Should I just do a Load right before I make every change or is there a more elegant approach?
The best solution would be not to use XML.
Use a (small) database.
Using any kind of text file in a multi-user situation is difficult enough and the fact that it is to different sections can only be used profitable if you have fixed-length and (therefore) fixed-position records (lines). XML does not deliver this.
Doing a load-before-edit will only make problems appear less often. There always is the chance that changes will be lost, you will have race-conditions at the filesystem level. To make it work you have to use a scheme with locking files. You will need an extra file for this.
You are talking about multiple processes being able to modify the file. If you want to keep the file instead of putting that data into a common store (MS SQL) you will need to implement a mutex to make sure only one application can access file at any moment of time.
I am creating an RSS reader as a hobby project, and at the point where the user is adding his own URL's.
I was thinking of two things.
A plaintext file where each url is a single line
SQLite where i can have unique ID's and descriptions following the URL
Is the SQLite idea to much of an overhead or is there a better way to do things like this?
What about as an OPML file? It's XML, so if you needed to store more data then the OPML specification supplies, you can always add your own namespace.
Additionally, importing and exporting from other RSS readers is all done via OPML. Often there is library support for it. If you're interested in having users switch then you have to support OPML. Thansk to jamesh for bringing that point up.
Why not XML?
If you're dealing with RSS anyway you mayaswell :)
Do you plan just to store URLs? Or you plan to add data like last_fetch_time or so?
If it's just a simple URL list that your program will read line-by-line and download data, store it in a file or even better in some serialized object written to a file.
If you plan to extend it, add comments/time of last fetch, etc, I'd go for SQLite, it's not that much overhead.
If it's a single user application that only has one instance, SQLite might be overkill.
You've got a few options as I see it:
SQLite / Database layer. Increases the dependencies your code needs to run. But allows concurrent access
Roll your own text parser. Complexity increases as you want to save more data and you're re-inventing the wheel. Less dependency and initially, while your data is simple, it's trivial for a novice user of your application to edit.
Use XML. It's well formed & defined and text editable. Could be overkill for storing just a URL though.
Use something like pickle to serialize your objects and save them to disk. Changes to your data structure means "upgrading" the pickle files. Not very intuitive to edit for a novice user, but extremely easy to implement.
I'd go with the XML text file option. You can use the XSD tool built into Visual Studio to create a DataTable out of the XML data, and it easily serializes back into the file when needed.
The other caveat is that I'm sure you're going to want the end user to be able to categorize their RSS feeds and be able to potentially search/sort them, and having that kind of datatable style will help with this.
You'll get easy file storage and access, the benefit of a "database" structure, but not quite the overhead of SQLite.
I plan to be storing all my config settings in my application's app.config section (using the ConfigurationManager.AppSettings class). As the user changes settings using the app's UI (clicking checkboxes, choosing radio buttons, etc.), I plan to be writing those changes out to the AppSettings. At the same time, while the program is running I plan to be accessing the AppSettings constantly from a process that will be constantly processing data. Changes to settings via the UI need to affect the data processing in real-time, which is why the process will be accessing the AppSettings constantly.
Is this a good idea with regard to performance? Using AppSettings is supposed to be "the right way" to store and access configuration settings when writing .Net apps, but I worry that this method wasn't intended for a constant load (at least in terms of settings being constantly read).
If anyone has experience with this, I would greatly appreciate the input.
Update: I should probably clarify a few points.
This is not a web application, so connecting a database to the application might be overkill simply for storing configuration settings. This is a Windows Forms application.
According to the MSDN documention, the ConfigurationManager is for storing not just application level settings, but user settings as well. (Especially important if, for instance, the application is installed as a partial-trust application.)
Update 2: I accepted lomaxx's answer because Properties does indeed look like a good solution, without having to add any additional layers to my application (such as a database). When using Properties, it already does all the caching that others suggested. This means any changes and subsequent reads are all done in memory, making it extremely fast. Properties only writes the changes to disk when you explicitly tell it to. This means I can make changes to the config settings on-the-fly at run time and then only do a final save out to disk when the program exits.
Just to verify it would actually be able to handle the load I need, I did some testing on my laptop and was able to do 750,000 reads and 7,500 writes per second using Properties. That is so far above and beyond what my application will ever even come close to needing that I feel quite safe in using Properties without impacting performance.
since you're using a winforms app, if it's in .net 2.0 there's actually a user settings system (called Properties) that is designed for this purpose. This article on MSDN has a pretty good introduction into this
If you're still worried about performance then take a look at SQL Compact Edition which is similar to SQLite but is the Microsoft offering which I've found plays very nicely with winforms and there's even the ability to make it work with Linq
Check out SQLite, it seems like a good option for this particular scenario.
Dylan,
Don't use the application config file for this purpose, use a SQL DB (SQLite, MySQL, MSSQL, whatever) because you'll have to worry less about concurrency issues during reads and writes to the config file.
You'll also have better flexibility in the type of data you want to store. The appSettings section is just a key/value list which you may outgrow as time passes and as the app matures. You could use custom config sections but then you're into a new problem area when it comes to the design.
The appSettings isn't really meant for what you are trying to do.
When your .NET application starts, it reads in the app.config file, and caches its contents in memory. For that reason, after you write to the app.config file, you'll have to somehow force the runtime to re-parse the app.config file so it can cache the settings again. This is unnecessary
The best approach would be to use a database to store your configuration settings.
Barring the use of a database, you could easily setup an external XML configuration file. When your application starts, you could cache its contents in a NameValueCollection object or HashTable object. As you change/add settings, you would do it to that cached copy. When your application shuts down, or at an appropriate time interval, you can write the cache contents back out to file.
Someone correct me if I'm wrong, but I don't think that AppSettings is typically meant to be used for these type of configuration settings. Normally you would only put in settings that remain fairly static (database connection strings, file paths, etc.). If you want to store customizable user settings, it would be better to create a separate preferences file, or ideally store those settings in a database.
I would not use config files for storing user data. Use a db.
Could I ask why you're not saving the user's settings in a database?
Generally, I save application settings that are changed very infrequently in the appSettings section (the default email address error logs are sent to, the number of minutes after which you are automatically logged out, etc.) The scope of this really is at the application, not at the user, and is generally used for deployment settings.
one thing I would look at doing is caching the appsettings on a read, then flushing the settings from the cache on the write which should minimize the amount of actual load the server has to deal with for processing the appSettings.
Also, if possible, look at breaking the appSettings up into configSections so you can read write and cache related settings.
Having said all that, I would seriously consider looking at storing these values in a database as you seem to actually be storing user preferences, and not application settings.