In my application programming experience, I have always worked with a SQL Server (or Access) database on the back end that stores application data. I'm now looking at some business requirements that work with data that would fit well in a relational database, but they require it to be stored in a single, portable, custom file that the application will create, and load from. I know it's a very common concept for an application to save off a single file or document that it can later load and continue to work on, but I'm not sure how to achieve this with complex data. Encrypting xml comes to mind, but that would be very slow to work with or potentially eat up a lot of memory if I had to load it all back into objects first. What are some options?
I recommend you use a SQLLite or Firebird embedded database. There are other options as well. They support single-file usage and will give you a clear upgrade path for future versions of you schema (upgrade SQL scripts).
I did not understand how encryption plays into this.
When running in a .NET environment I think that SQL Server Compact is worth looking into. It is basically a mini SQL Server that doesn't have to be installed and configured as a service, but instead is an dll that you reference. You can use normal data access tools like linq-to-sql and entity framework.
SQLite comes to my mind. Its a single file based DB. Here is a link to convert SQL server databases to SQLite. Also check out Using SQLite in your C# Application
Related
I'm doing some research for a new project and I'm trying to determine if it's possible (and advisable) to load a SQLite database into memory, perform CRUD operations against it, and persist it back out.
I've seen many examples of utilizing SQLite databases (in memory) for unit testing, and in all of those examples, the data is just trashed in the end - this is NOT what I wish to do.
I'm going to likely use Microsoft SQL Server to manage the overall site data and act as a storage engine (users and credentials and their associated SQLite databases, etc).
When a user selects a SQLite database in the UI, I would like to load it into memory on the server, allow the user to operate against it, and then persist it back to the storage engine (SQL Server) without needing to save a .db file to the filesystem.
I'm comfortable with aspects of EF Core + SQL Server and SQLite (against the filesystem). But what's new to me is the idea of operating against a SQLite database in memory.
So my questions are:
Is this possible with EF Core?
If so, how would I configure my SQLite DbContext class to accomplish this?
Are there any major downsides to this?
Thank you
Is this possible with EF Core?
Yes. Why not? SQLite essentially treats in-memory database the same as any other. There are particular considerations, but you essentially insert and query data in the same way.
If so, how would I configure my SQLite DbContext class to accomplish this?
Once again, refer to documentation. You would need to provide a specific connection string. (Not to discourage SO questions, but if you're going to research and test this you really should research information available on sqlite.org. It is has great, thorough documentation--at least compared to many open-source projects... sometimes a bit scattered, but still accessible.)
Perhaps more complicated than specifying an appropriate connection string is actually loading an existing database file into memory. The default, basic behavior is to only create an empty database in memory. There are multiple ways to load the data, and this question has some useful answers.
Are there any major downsides to this?
You have apparently already identified some of the downsides, but probably no more than any project which needs to merge/synchronize databases. There is no short answer to that question and it is much too broad for Stack Overflow.
You specifically mention syncing data to an SQL Server without saving the data to a disk file. You will certainly have to perform a series of queries form sqlite, massage the data into a corresponding update statement for SQL Server, then execute that on the server. Perhaps there are third party tools to do that same thing for file-based databases, but I suspect that you'd end up performing the same operation with a disc file anyway.
I'm currently working on a project that will be taking in a large amount of data over a set period of time. I currently do not have access to any database engine on the network my software will be deployed on. As such, I'm working with an XML dataset, and employing an MVC type framework. I chose XML originally because it can mimic to a decent degree the relational design of a database. I store the data in a singleton and my win-forms access the data via objects in my singleton and the win-forms do not have direct knowledge of the underlying data structure.
I'm using .NET 3.5, as the PC's my software will be deployed on are XP computers and not internet connected, and therefore, do not have access to download .NET 4.0.
I'm accessing the XML using LINQ to XML objects and storing the element/attribute values into objects and collections that can be accessed through properties and methods in my data access layer.
My main concern is that as the XML file gets larger, the overhead could become overly cumbersome. Other than setting up a database server, would there be a better data storage system to use that would be better than XML? I need my data to follow a relational format for what I'm attempting to accomplish. I've already looked into using delimited/csv formats and those do not satisfy my project requirements.
EDIT: This datasource cannot be embedded into the application, and there will be multiple applications accessing the datasource. There are 2 applications that will be used, a "host" application that will be managing the data and a "client" application which will be used for data collection. The "client" application will have multiple instances over multiple workstations on the LAN.
SQL Server Express LocalDB is an embedded version of SQL server, but requires only a file on the machine running it. This might be a good option, will be smaller, faster, and easier to work with than XML, will have identical syntax and similar performance to a real SQL server and you can use all of the built in .Net SQL libraries and features without installing a third party data store.
You can also have multiple apps connect to the same DB file, and still get ACID features of SQL server.
http://www.microsoft.com/sqlserver/en/us/editions/2012-editions/express.aspx
http://blogs.msdn.com/b/sqlexpress/archive/2011/07/12/introducing-localdb-a-better-sql-express.aspx
If file size is your primary concern, you could use compression on the files to keep them smaller on disk. .NET 3.5 has some compression algorithms built in. XML is very repetitive text, which compresses very well. If you are free to break your data up into many files, that many also help keep the sizes down.
JSON is lighter-weight than XML. As for whether you can use it in a relational fashion , that would depend on exactly what you are trying to do. Some relational uses of JSON have been covered here:
Using JSon like a Relational SQL Database (Javascript)
I'm currently developing app that requires storage of lots of information from different controls. I've never done this before, and after reading couple things on this topic got a little confused. Do I need to download SQL server 2008 and work with it in order to store data from WinForm? If yes, then what is the service-side item for? Can I use for storage?I don't need to import data from database(at least not for now), I just need stuff to save somewhere, and I would like to know where exactly. Thank you!
You can choose to store your data in a serverless database (SQL CE, SQLite for example) if you don't need fancy database stuff like stored procedures, weird indexes etc. Both of the above mentioned technologies are compatible with Entity Framework, hence you can use code first approach (you can write the classes and the database would be generated on the fly).
here you can read more about it.
http://www.codeproject.com/Articles/318010/Entity-Framework-Code-First-Let-s-Try-It
And here's SQLite assembly for .Net
http://system.data.sqlite.org/index.html/doc/trunk/www/index.wiki
Good luck
You don't have to store anything in a database, you could for example serialize whatever you need to store into isolated storage
I have read one line from your question " SQL server 2008 and work with it in order to store data from WinForm"; Mean you want to save some data and that will be coming from data entry from winform.. RIght?
If that is the case you can adopt any of the RDBMS software that suites your finance + capacity needs and other perameters.
You can surely go for SQL server 2008 or Oracle or MySql or (List will go one).
What is the nature of your data?
Just because it's big does not necessarily mean a full-blown DBMS (or even an embedded database) is the best fit. Have you considered simply serializing the data to plain files?
But if your data is "relational" and you need to do the things such as querying, integrity enforcement, transactions, concurrent access by multiple clients (etc..), then you should definitely consider a database approach, including MS SQL Server, Oracle, IBM DB2, Sybase etc...
There are good open-source DBMSes as well: PostgreSQL, Firebird, MySQL...
All other applications I've written have been network/web apps where I've used an SQL database to store data and that has made sense to me.
If I'm creating an application that does not need to be networked ever, is there a standard way to store this data permanently? A text file is possible, but doesn't give me the benefits of querying an SQL server nor is it very secure. Is there something similar to an SQL server that I can initiate and save on starting up my program?
Perhaps there is some structure I've never come across?
From what I've read I might be able to do something as mentioned above like with SQLite. Does that make sense for large and distributed applications?
Thanks in advance for any clarifications on how to design these types of programs.
Edit: to clarify what #TomTom was saying, it is not a large amount of data like he is suggesting. I would be surprised if it ever got over several gigs of data. My point in saying large was that it seemed unreasonable to create some sort of a data structure that I could save into a text file, load up/search through to grab my data compared to using an SQL-like database.
Reading through the answers apparently SQLite or something similar is reasonable to use for desktop applications. I'll continue looking into it and probably use it to track data for my next project.
You can use an embedded database - this can be a SQL database, but does not have to be.
For windows, look at SQLite, SQL Server Compact and RavenDB (for a non SQL, document database).
You could still use SQL database, but locally. Try SQLite.
Other option to use Windows built-in database engine which name is Esent. Fast, but not really convenient to use its Api.
How do i create a db file in C#? a friend told me it was in the toolbox and not to use sqlite. I dont see anything that could be it, nor what it is called. google didnt help:(
Could it be...
SQL Server Compact Edition – A lightweight, in-process database engine designed to run on devices and desktops and is geared toward local data storage. Compact Edition includes a subset of SQL Server 2005 data types and shares common elements of the Transact-SQL (T-SQL) language with the data service engines.
There is no file-based database provider built in to c# or the .NET Framework. There are of course pre-existing connectors for using SQL Server (which includes SQL Express), but if you need a fully functional RDBMS that is file-based, you need to use something like SQLite or Firebird (also a fan of VistaDB, not free or open source but VERY solid and pretty affordable).
On the Add New Item menu: "Service-based Database" or "Sql Server Database" if it is an asp.net application. I am sure your friend meant it as "create a sql express db file in Visual Studio".
That said, if you wanted to fill an empty database, with tables that correspond to a c# model, you could create a linq2sql model, and use its CreateDatabase to do that for you :)
You might want to check this http://quickstarts.asp.net/QuickStartv20/aspnet/doc/data/vwd.aspx (visual web developer link, but it applies).
For a full/in-depth explanation of how SQL express can be used with a semi file based approach check and its limitations:
http://www.databasejournal.com/features/mssql/article.php/3704171/SQL-Server-2005-Express-Edition---Part-8---XCopy-Deployment.htm
Perhaps you might try Microsoft LocalDB.
It is file based but uses a low level SQLExpress Installation to host it.
If you are using Visual Studio or Web Developer Express, there are indeed ways to easily create a MS SQLExpress database. Just go to Add New Item... and it should be one of the available file types.
Keep in mind you have to have installed either Microsoft SQL Express Edition (free, as in beer!) or Microsoft SQL (very un-free!, in all senses). If you haven't done this, you don't get the option of creating a database file so easily. If you have got it yet, you can get it here.
As other answerers have mentioned, strictly speaking this is NOT a C# feature. MS SQL and it's derivatives, are database applications, much like Oracle, MySQL, or PostgreSQL. It's just that Microsoft Visual Studio makes using the Microsoft database product very easy by default. Differentiating between C#, Visual Studio, and any database programs will probably get you better answers, faster, no matter where you ask. :)
Every database has a file system in some binary format more than likely custom and uses a cache to control the flow of the database(s) lifetime.
If you create a database system, you will need some type of cache because you only want to read from the file if the cache has already released it.
If you have 1000 clients tapping into the same db, you certainly don't want to read/write to the file for each client request, so you want to manage a queue of clients and run it against the cache so that the cache knows not to release the db after its time span for lifetime is reached put to rather update the time span, therefore, not having to reload the file, if disposed and queued again, until the queue referencing the db object is empty.
Creating a well designed cache it used by all rdbms's so that duplicate objects are not created and files are not reloaded if not need be.
You can use
FileDB - A C# database to store files
http://filedb.codeplex.com
There is a MVC Example in the source that allows you to upload files and also has drag and drop support.
It then saves it into just one file in a location that you specified.
Like this:
private string pathDB = #"C:\CMS-MVC\Parts\FileManager\filedb-19055\trunk\MvcTest\Data\MvcData.dat";
That one file will store all of your files in that one "container".
You are mistaken. Databases are not developed in C#. Databases are built using a database system such as Oracle, MS SQL Server, MySQL, and numerous others.
Once you build a database using one of the above providers, you can then perform actions on the database using your programming language of choice (in your case C#) to get data out of and put data into it.