I'm working on an application which will be updating multiple databases (SQL Server 2008 and Oracle 11g). TransactionScope seemed like the logical way to ensure updates were always committed correctly, but it seems that installing MSDTC is not going to be an option. In the future, it's also possible this application could be using data sources which don't support distributed transactions.
I've spent many hours trying to come up with another solution but nothing seems like it will work. All searches point back to TransactionScope and distributed transactions.
The application is written in C#, using the Entity Framework. Anyone have any suggestions, which won't require being escalated to distributed transactions? Here's a list of ideas I've had which have gone nowhere.
+TransactionScope: Can't use MSDTC. Future data sources may not support distributed transactions.
+Manually track and rollback transactions: I haven't found a good way to do this within Entity Framework.
+Queue/log failures so they can be re-committed by another process: Can't come up with a good way to store the failed commits generically. Also need to make sure the re-commit doesn't overwrite newer data.
#ThinkJet. That related link is an interesting opinion. In my case a small failure, like what is described, would not be a huge deal. We currently have other stuff in place which tries to keep all these systems in sync (not always successfully). If one or two transactions did fail it should be picked up by these processes.
After reading through these comments, I might try to have this library write the data to it's own database. Then, sync those changes to the other sources so that the other applications can see the changes. It would cause a slight delay in some updates but even that would be better than what we have now.
Related
I'm doing some research for a new project and I'm trying to determine if it's possible (and advisable) to load a SQLite database into memory, perform CRUD operations against it, and persist it back out.
I've seen many examples of utilizing SQLite databases (in memory) for unit testing, and in all of those examples, the data is just trashed in the end - this is NOT what I wish to do.
I'm going to likely use Microsoft SQL Server to manage the overall site data and act as a storage engine (users and credentials and their associated SQLite databases, etc).
When a user selects a SQLite database in the UI, I would like to load it into memory on the server, allow the user to operate against it, and then persist it back to the storage engine (SQL Server) without needing to save a .db file to the filesystem.
I'm comfortable with aspects of EF Core + SQL Server and SQLite (against the filesystem). But what's new to me is the idea of operating against a SQLite database in memory.
So my questions are:
Is this possible with EF Core?
If so, how would I configure my SQLite DbContext class to accomplish this?
Are there any major downsides to this?
Thank you
Is this possible with EF Core?
Yes. Why not? SQLite essentially treats in-memory database the same as any other. There are particular considerations, but you essentially insert and query data in the same way.
If so, how would I configure my SQLite DbContext class to accomplish this?
Once again, refer to documentation. You would need to provide a specific connection string. (Not to discourage SO questions, but if you're going to research and test this you really should research information available on sqlite.org. It is has great, thorough documentation--at least compared to many open-source projects... sometimes a bit scattered, but still accessible.)
Perhaps more complicated than specifying an appropriate connection string is actually loading an existing database file into memory. The default, basic behavior is to only create an empty database in memory. There are multiple ways to load the data, and this question has some useful answers.
Are there any major downsides to this?
You have apparently already identified some of the downsides, but probably no more than any project which needs to merge/synchronize databases. There is no short answer to that question and it is much too broad for Stack Overflow.
You specifically mention syncing data to an SQL Server without saving the data to a disk file. You will certainly have to perform a series of queries form sqlite, massage the data into a corresponding update statement for SQL Server, then execute that on the server. Perhaps there are third party tools to do that same thing for file-based databases, but I suspect that you'd end up performing the same operation with a disc file anyway.
Folks - apologies for rehashing this topic as I see even here on Stack, there are so many questions on the topic already.
But I find myself in an interesting place and I'm hoping you can help.
High level Question: can SQL SERVER have the leeway to decide that a view should be wrapped in a ISOLATION LEVEL SNAPSHOT?
I know that sounds like a crazy question but I'm trying to exhaust all avenues per an issue I'm encountering.
I'm working with an application that runs 35 queries to retrieve data from another database via Link Server. The queries are simple selects against one table respectively. All DB operations are carried out against SQL SERVER, and retrieval code is ADO.NET/c#, etc.
34 of the queries work flawlessly - but there's this one bad apple, and for it, I get the transaction isolation level snapshot issue.
I've also tested data retrieval outside of the application and when I implement the below snippet on the "problem" query, I also get the issue:
using (var trans = conn.BeginTransaction(IsolationLevel.Snapshot))
However, when I do NOT implement it on said query, all is well.
I've also tested this against the other queries - with and without "Shapshot" - and my results are predictable... With "Shapshot" in place, no queries process... When not implemented, all queries process...
My results suggest that the application is responsible for changing up the data retrieval strategy.
Per their knowledge base, I found this: Locking is handled by the database level (MS SQL Server/Oracle) and not by "us". Generally, locking is row level but the optimizer may choose something different
Unfortunately I don't have native access to the boiler-plate application code responsible for data retrieval. I suspect that this particular query/table has one or more key words - either in the column or query/table naming - that trigger the application to use an alternate retrieval strategy. Per the developer forums, I've asked of this is the case and I'm awaiting a reply...
Anyway back to their mention of the optimizer may choose something different- their optimizer, or perhaps the database optimizer? Can SQL SERVER be set up to make a "judgement call" ? Is the statement unclear or do I just not enough of SQL SERVER and its capabilities?
I know it seems like a crazy question but I really want to knock out all possible avenues here.
Thank you kindly for suspending your disbelieve and humoring that crazy post :)
Apparently objects with the word "valuation" (perhaps because of the sensitive nature implied) cause the application to build the transaction. Once I changed the view name, the data returned to the client successfully.
So yes the application was/is the issue.
I started off a few days ago thinking that I could write a simple toolkit for locking (restricting access to) and unlocking resources in our application. So for example by opening a project it would get locked to that user in some way, then when the next person came to open it they would not be able to modify it. Or an operation needs to write to certain files and read from certain files, so it locks them (in some metadata record in the database) as such before performing the operation.
Now that I've done a fair bit of design it is becoming apparent that there are actually quite a lot of little gotchas: I'm now in a situation where to describe the functionality I need I have 5 different types of lock defined (owner, write, read-only, delete, no-delete) and need to specify the relationships between them in the context of being applied to the same object and parent-child hierarchies.
That will translate into a fair amount code to maintain and testing and given this I'm wondering if there are any free .NET libraries available that do this sort of thing out of the box (I'm off on Christmas hols next week and I think it would be easier for one of the other devs to plug in a library rather than finish off my tooklit and plug it into the application).
We're using C# .NET and SQL Server 2008 R2.
UPDATE:
By resources I mean files referenced by metadata in a database.
chances are you're over-designing. DBs are meant to be concurrently accessible and use Transactions for critically interdependent operations.
You should reconsider your data flow and allow multiple users to work together.
For example if a user downloads the parts of a project they are working on and then commits them all at once. Some merging may be required of course. Depending on your task you could also use a version control system (SharpSVN) instead of or along with DB
You are describing a pessimistic locking strategy; this is best handled by enlisting your code into the database transaction (aka distributed transactions), so your code can commit/rollback the transaction, and hence release any locks. This can be achieved using COM+ Transactions. However, it may be better to use an optimistic locking stragegy instead; see here for a more detailed answer.
I wonder what you are using for updating a client database when your program is patched?
Let's take a look at this scenario:
You have a desktop application (.net, entity framework) which is using sql server compact database.
You release a new version of your application which is using extended database.
The user downloads a patch with modified files
How do you update the database?
I wonder how you are doing this process. I have some conception but I think more experienced people can give me better and tried solutions or advice.
You need a migration framework.
There are existing OSS libraries like FluentMigrator
project page
wiki
long "Getting started" blogpost
Entity Framework Code First will also get its own migration framework, but it's still in beta:
Code First Migrations: Beta 1 Released
Code First Migrations: Beta 1 ‘No-Magic’ Walkthrough
Code First Migrations: Beta 1 ‘With-Magic’ Walkthrough (Automatic Migrations)
You need to provide explicitly or hidden in your code DB upgrade mechanism, and - thus implement something like DB versioning chain
There are a couple of aspects to it.
First is versioning. You need some way of tying teh version of teeh db to the version of the program, could be something as simple as table with a version number in it. You need to check it on executing the application as well.
One fun scenario is you 'update' application and db successfully, and then for some operational reason the customer restores a previous version of the db, or if you are on a frequent patch cycle, do you have to do each patch in order or can thay catch up. Do you want to deal with application only or database only upgrades differently?
There's no one right way for this, you have to look at what sort of changes you make, and what level of complexity you are prepared to maintain in order to cope with everything that could go wrong.
A couple a of things worth looking at.
Two databases, one for static 'read-only' data, and one for more dynamic stuff. Upgrading the static data, can then simply be a restore from a resource within the upgrade package.
The other is how much can you do with meta-data, stored in db tables. For instance a version based xsd to describe your objects instead of a concrete class. That's goes in your read only db, now you've updated code and application with a restore and possibly some transforms.
Lots of ways to go, just remember
'users' will always find some way of making you look like an eejit, by doing something you never thought they would.
The more complex you make the system, the more chance of the above.
And last but not least, don't take short cuts on data version conversions, if you lose data integrity, everything else you do will be wasted.
I have a WCF application that at present is using XML based file storage to store data that gets used to generate reports. Besides this processing decisions are made based on information stored in these XML files.
I'm now hitting volumes of around 30 000 text files. This is incredibly taxing, and the application at times comes to a grinding halt.
I've always wanted to swop out the XML DAL in favor of an RDBMS, but project managers simply won't allow it. But they would be willing to look at a serverless solution for example SQLLite. I am really tempted to just dive right in and start using it as a replacement DAL (Data Access Layer).
I would need no more than around 20 tables in the whole solution, and I would expect to get no more than around 20 000 - 100 000 transactions a day, however this is extreme, the real volumes would be less than this in most cases.
Update
I am not expecting a great deal of simultaneous connections, when I say transactions, I essentially mean 1 or 2 clients that make calls and execute against the database in order. At times there might be a possibility of external clients making quick calls to the DB. But the bulk of DB connections will be done by my WCF service, which is a back end scheduled task, not serving 100's of people across an organization.
Another good point is that I only need to retain data for 90 days, so the DB shouldn't grow too big.
My main concerns are:
How reliable is SQLLite? What if the DB File gets corrupted, will I loose all processing Data. How easy is the DB to back up? Will it handle my volumes? And lastly how well does the .net provider work (located here: http://sourceforge.net/projects/sqlite-dotnet2/).
If you have any experience with SQLLite, please post your experiences so I can make aan informed decision to switch or not.
Thanks in advance...
SQLite is as reliable as your OS and hardware.
Its transactional rate is similar to SQL server, and often faster because it's all in process.
The .NET ADO provider works great.
To back up the DB, stop the service and copy the file. If the journal file is present copy it too.
EDIT: SQLite uses UTF-8 by default so with the ADO-NET provider you should be able to avoid losing accents (just so long as you follow the typical XML in string rules).
You could consider Microsoft's Sql Compact Edition.
It's like sqlite, in terms of being a single file embedded database, but has better integration with the .net framework :)
SQLite seems reliable, and even with Microsoft's one, don't expect to receive much support in case of a corrupted database.
Given your transaction volume I'd say the fact that the DB itself is a single monolithic file with only file system locking available could be a problem.
There is no row based locking as far as I know.
I used SQLite with the .Net provider without problems in a monouser enviroment, except for one concern: accents, wich don't showed correcly. The backup is quite simply: the SQLite database is an plain text file. Simply copy it.
I use Sqlite for storing XML config data and have had no problems with it. I use the System.Data.Sqlite provider: http://sqlite.phxsoftware.com/. It's solid and has a good support forum. It also includes a LINQ provider. It also integrates with VS 2008 so you can use Server Explorer to query tables. The examples and documentation also show how to use parameterized commands and transactions for increased performance.
The release candidate for LinqPad now supports Sqlite: http://www.linqpad.net/Beta.aspx.
Sqlite stores everything in a single file, which can be backed up like any other binary file.
Sqlite only supports file-level locking, but shouldn't present a performance problem since it doesn't sound like you'll have a large number of simultaneous transactions.
Unicode shouldn't be a problem. This link in the forum addresses an area where someone was trying to read unicode characters with an incompatible utility http://sqlite.phxsoftware.com/forums/t/954.aspx.
This site shows how to do case-insenitive UTF8 comparisons using System.Data.Sqlite via a custom collator, with Russian characters as an example: http://www.codeproject.com/KB/database/SQLiteUTF8CIComparison.aspx.