Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I created an application in C# using Winforms which has daily transaction of 2000 rows of data per day. I'm using SQL Server 2012 but I'm trying to use SQLite because of his fame and most people refer this
So, can you give me some ideas which one is better for my needs?
Thanks
SQLite integrates with your .NET application better than SQL server
SQLite is generally a lot faster than SQL Server.
However, SQLite only supports a single writer at a time (meaning the execution of an individual transaction). SQLite locks the entire database when it needs a lock (either read or write) and only one writer can hold a write lock at a time. Due to its speed this actually isn't a problem for low to moderate size applications, but if you have a higher volume of writes (hundreds per second) then it could become a bottleneck. There are a number of possible solutions like separating the database data into different databases and caching the writes to a queue and writing them asynchronously. However, if your application is likely to run into these usage requirements and hasn't already been written for SQLite, then it's best to use something else like SQL Server that has finer grained locking.
SQLite is a nice fast database to use in standalone applications. There's dozens of GUI's around to create the schema you want and interfaces for pretty much any language you would want (C#/C/C++/Java/Python/Perl). It's also cross platform and is suitable for Windows, Linux, Mac, Android, iOS and many other operating systems.
Here are some advantages for SQLite:
Perfomance
In many cases at least 2-3 times faster than MySQL/PostgreSQL.
No socket and/or TCP/IP overhead - SQLite runs in the same process as your application.
Functionality
Sub-selects, Triggers, Transactions, Views.
Up to 281 TB of data storage.
Small memory footprint.
Self-contained: no external dependencies.
Atomic commit and rollback protect data integrity.
Easily movable database.
Security
Each user has their own completely independent database(s).
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
I am having trouble phrasing this correctly but is it possible to have a desktop based application where all of the work is done on the server (specifically with the database) instead of the client? The use case is that for an internal application where our server is located, the application works great. However, remote workers that have to use the application really struggle with latency issues over our VPN. Our current solution is to just make it into a web application but the problem with that is we feel we will lose efficiency in the application. Ideally we would create a desktop app in something along the lines of WPF but with all of the work (database connection/calls) not done on the client. It doesn't necessarily have to be Microsoft but that is what we are trying to go with. Does anyone have any insight into how this could be done? Thanks in advance!
#Dan's comment: well it's really hard to diagnose without seeing the application itself. Since your application is connecting directly to the Database server, the question would be: Is the db server returning large datasets that are being processed by your desktop application; if yes, then moving that logic to a server application that connects to the database to perform the processing and return a result would reduce network latency if that result is smaller than the source data it processed.
However this quickly spirals into other questions. What data processing is being done? Depending on what is happening with the data would change your server side architecture. If it's CPU intensive than it will be important that your server side application can horizontally scale with demand.
All that being said, this is only posted as an answer because it wouldn't fit in a comment. It's not really an answer, and your question really requires a good architect to sit with you and look at the needs of your users and application to address this fairly.
UPDATED
So based on your comments I can't be sure if this is better or worse in your scenario, but you could try VDI solutions or application streaming solutions. The great thing is that despite whether it works out or not you can test an application streaming solution with no changes to your application. However depending on your network and security requirements the real work would be getting your systems connected with the application streaming service.
You could try something like https://aws.amazon.com/appstream2/ AWS APPSTREAM, and see if this is any faster for your users. Above and beyond that you would probably need to get another set of eyes on your solution and architecture to help you redesign/rearchitect the application to work within the constraints you are dealing with.
Talk to your VPN software vendor to see if you can better scale their solution.
Configure your VPN to use a split tunnel connection so that all of your users internet traffic doesn't route over your VPN gateway unless that's a requirement for you.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
We have a customer which connects to our servers through Satellite. However, they have a major concern that the facilities in which they want to run our applications, at many times have connection issues. So, they want to run our applications locally (on specific PC's). Data from the host system would feed to the local machines continuously. If the connection is lost, the PC still have enough data to conduct it's business until the connection is restored. At that time, the data changes from the PC are reflected back to the host system and visa versa.
I guess we would be considering some type of replication (this is all new to me). This has many questions but here are the main ones.
If we replicate, then they need a copy of SQL Server on each PC. We are talking about 60 sites which would be very expensive due to licensing. Also, other support costs.
Is it better to always run replication or only in the event that the connection was lost?
How does the local system get in sync with the hosted system?
Just looking for a better/less expensive solution.
The way I see it, there are two ways to for it (depending on your requirements.
If you think the problem will not persist you can use the circuit breaker pattern:
https://learn.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker
Handle faults that might take a variable amount of time to recover from, when connecting to a remote service or resource. This can improve the stability and resiliency of an application.
If you need to retry indefinitely and you can't afford to lose data then you will need a custom solution.
On a totally local environment you could go with either a local database like sql lite, where you can store items and retry if not successful, or store the calls in Microsoft
Queue. Then you call build a service that reads the database or the queue and retries.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
Since on a server side application the work is done by the server, and since it also needs to serve other request, I would like to know if there are any real benefits of using Parallel processing in server side applications? The way I see it, I think it is usually bad to use parallel-processing? focusing the CPU power only on part of the problem, other requests cannot get server?
If there are advantages, I guess they should be considered only when specific conditions are meat. So, what are some good guidelines of when to use the Parallel class in server applications ?
You are balancing two concerns: Fast response for a given user, and supporting all users that wish to connect to the server in a given time period.
Before considering parallelism for faster computation for a given user, consider whether precomputation and caching allow you to meet your performance requirements. Perform hotspot analysis and see if there are opportunities to optimize existing code.
If your deployment hardware is a given, observe the CPU load during peak times. If the CPU is busy (rule of thumb 70%+ utilization), parallel computing will be detrimental to both concerns. If the CPU isn't heavily loaded, you might improve response time for a given user without affecting the number of users the server can handle at once (benchmark to be sure).
If you aren't meeting your single-user performance targets and have exhausted options to precalculate and cache (and have analyzed performance hotspots and don't see opportunities to optimize), you can always parallelize workloads that lend themselves to parallel computation if you're willing to upgrade your server(s) as needed so that during peak periods you don't over-tax the CPU.
As with most performance-related questions: it depends on a lot of factors. Things like:
do you tend to have a lot of requests hitting your server at the same time?
how much of a typical request's turnaround time is spent waiting on I/O, as opposed to actually exercising the CPU?
are you able to have multiple instances of your server code sitting behind a load balancer?
how well does the operation you're looking at get optimized by parallelizing?
how important is it for the operation you're performing to return an answer to the user faster than it would without parallelism?
In my experience, most of the time for typical request is spent waiting for things like database queries and REST API calls to complete, or loading files from a disk. These are not CPU-intensive operations, and inasmuch as they can be made concurrent that can usually be done by simply orchestrating async Tasks in a concurrent manner, not necessarily using parallel threads.
Also in my experience, most attempts to use the TPL to improve performance of an algorithm end up yielding only marginal performance improvements, whereas other approaches (like using more appropriate data structures, caching, etc.) often yield improvements of orders of magnitude.
And of course if your application isn't going too slow for your needs in the first place then any optimization would count as premature optimization, which you want to avoid.
But if you for some reason find yourself doing a CPU-intensive operation that responds well to parallelism, in a part of your code that absolutely must perform faster than it currently does, then parallel processing is a good choice.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
Currently we were working on a cloud migration project where we made the following changes:
SQL Server 2014 to Azure SQL PaaS
Redis cache[Windows porting] to Azure Redis PaaS
Static files from shared drives to Azure File Service
Implemented Transient Fault Handling for database interactions.
HttpSessionState changed from SQL Server to Custom[Redis PaaS]
The Application had two web applications that use the same database:
One built in Classic model of dot net coding with web-forms.
The other application built using dot net MVC4.
After we moved the applications from existing Rackspace environment[2 servers each with 4GB RAM] to Azure and ran a load test and received the following results:
The MVC4 application is fractionally faster.
The Web-Form application started performing poorly, with same load,
response time increased from 0.46 seconds to 45.8 seconds.
The memory usage is same, database utilization is around 30%-40% and the CPU utilization in nearly 100%(all the web-servers) at 1100 concurrent users(at Rackspace, it served 4500 concurrent users).
We tested the application 2 D5 azure server VMs with RAM being higher and CPU being faster.
Can anyone highlight how such drastic performance drop(one application performing almost same, other one performing almost 100 times slower) is possible?
NB: One observation, the CPU utilization stays at 100% even after 30mins of stopping the load-test. Then it drops quickly.
I will second the notion (emphatically) that you invest as much time and energy as you can in profiling your application to identify bottlenecks. Run profiles on-premises and in Azure and compare, if possible.
Your application clearly has many moving parts and a reasonably large surface area... that's no crime, but it does mean that its hard to pinpoint the issue(s) you're having without some visibility into runtime behavior. The issue could lie in your Redis caching, in the static file handling, or in the session state loading/unloading/interaction cycle. Or it could be elsewhere. There's no magic answer here... you need data.
That said... I've consulted on several Azure migration projects and my experience tells me that one area to look closer at is the interaction between your ASP.NET Web Forms code and SQL. Overly-chatty applications (ones that average multiple SQL calls per HTTP request) and/or ones that issue expensive queries that perform lots of logic on the database or return large result sets, tend to exhibit poor performance in public clouds like Azure, where code and data may not be co-located, "noisy neighbor" problems can impact database performance, etc. These issues aren't unique to Web Forms applications or Azure, but they tend to be exacerbated in older, legacy applications that were written with an assumption of code and data being physically close. Since you don't control (completely) where your code and data live in Azure relative to each other, issues that may be masked in an on-prem scenario can surface when moving to the cloud.
Some specifics to consider:
take a close look at the use of data binding in your Web Forms app... in practice it tends to encourage expensive queries and transfer of large result sets from database to application, something you might sometimes get away with on-premises but not in the cloud
take another look at your SQL configuration... you don't mention what tier you're using (Basic, Standard, Premium) but this choice can have a big impact on your overall app performance (and budget!). If it turns out (for example) that your Web Forms app does issue expensive queries, then use of a higher tier may help
Azure SQL DB tiers
familiarize yourself with the notion of "cloud native" vs. "cloud capable" applications... generally speaking, just because you can find a way to run an application in the cloud doesn't mean its ideally suited to do so. From your description it sounds like you've made some effort to utilize some cloud-native services, so that's a good start. But if I had to guess (since we can't see your code) I'd think that you might need some additional refactoring in your Web Forms app to make it more efficient and better able to run in an environment you don't have 100% control over.
More on cloud-native
dated but still relevant advice on Azure migration
If you can give us more details on where you see bottlenecks, we can offer more specific advice.
Best of luck!
There is some loop in the code that cause 100% CPU.
When the problem occurs, take a dump from (the kudu). Analyze it in windbg by
1) list threads cpu time with !runaway
2) check the callstack of the threads, specifically the greatest cpu consumer with
~*e!clrstack and with ~*kb
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
I'm designing an accounting application with more than 400 tables in SQL Server.
About 10% of those tables are operational tables, and the others are used for decoding and reference information.
For example, Invoice tables (Master and details) use about 10 table to decode information like buyer, item , marketer and ... .
I want to know is it acceptable to cache decode tables in asp.net cache and do not query them from SQL Server (I know that changes to cache items should commit on SQL Server too). And use cache items for decoding?
I think it makes it so much faster than regular applications.
Maybe all of them together(Cache tables) are about 500 MB after some years because they don't change frequently.
If you've got the RAM then it's fine to use 500 MB.
However unless you have a performance problem now then caching will only cause problems. Don't fix problems that you haven't encountered, design for performance and optimize only when you have problems - because otherwise the optimization can cause more problems that it solves.
So I would advise that usually it is better to ensure that your queries are optimized and well structured, you have the correct indexes on the tables and that you issue a minimum amount of queries.
Although 500MB isn't a lot of data to cache, with all due respect, usually SQL Server will do a better job of caching than you can - providing that you use it correctly.
Using a cache will always improve performance; at a cost of higher implementation complexity.
For static data that never changes a cache is useful; but it still needs to be loaded and shared between threads which in itself can present challenges.
For data that rarely changes it becomes much more complex simply because it could have changed. If a single application (process) is the only updater of a cache then it isn't as difficult, but still not a simple task.
I have spent months optimizing a offline batch processing system (where the code has complete control of the database for a period of 12 hours). Part of the optimisation is to use various caches and data reprojections. All of the caches are readonly. Memory usage is around the 10gb mark during execution, database is around 170gb, 60 million records.
Even with the caching there has been considerable changes to the underlying schema to improve efficiency. The readonly caches are to eliminate reading during processing; to allow multi threaded processing and to improve the insert performance.
Processing rate has gone from 6 items processed per second 20 months ago to around 6000 items per second (yesterday) - but there is a genuine need for this optimization as the number of items to process has risen from 100,000 to 8 million in the same period.
If you don't have a need then don't optimize.