Develop Reference

SQL server - realtime and historical dashboard reading form tables

I have APPS Service hosted on azure using Azure SQL Database with around 15 tables:
- assistances
- users
- eventLogs
etc.
Currently I have around 150k records, and on daily based my DB is receiving around 2000 new assistances with new users related. On my app I have a cron, which is making a lot operations every 1 mins to all tables (updating, inserting etc).
Right now my aim is to create some nice dashboard, which will display data for admins (like states of assistances, number of assistances delay etc) - basicly reading data from those tables. It should give as well possibility to filter by dates (from - to date) - so in worst case scenario few users can generate report for month (aprox. 60k records) in the same time. I'm afraid doing it directly on my prod database, due to fact, that I've already cron going on with a lot operations, so I'm worry about locking.
My ideas are:
- sql database warehouse -> the biggest problem is the cost of it.
- replication to second DB, which will be used for querying data for dashboard. - I'm not convince about this solution.
- replication to noSQL database (pushing only important information) and use it for source of dashboard. - I don't have experience with such solution so far.
Do you have maybe some suggestion what will be the best?

In the end, I've used Geo-replication option from Azure, which is using snapshot isolation, so it's great! Even MS Azure recommend to use this geo-replication database as second DB used for read-only operations! I've tested and working great :)

You can use Azure automation to schedule those tasks that run every minute, instead of doing that from the application. You can know more about Azure automation here.
Instead of using Geo-replication consider using SQL Azure Data Sync. Make your primary database a “hub” database and use a replica for reporting. You can learn more about SQL Data Sync here.
You can also use Power BI to create your dashboards as explained here.
Hope this helps.

How to update huge sql data in asp .net application using c#

I am doing web application using c# .net and sql server 2008 as back end. Where application read data from excel and insert into sql table. For this mechanism I have used SQLBulkCopy function which work very well. Sql table has 50 fields from which system_error and mannual_error are two fields. After inserting records in 48 columns I need to re-ckeck all this records and update above mentioned two columns by specific errors e.g. Name filed have number, qty Not specified etc. For this I have to check each column by fetching in datatable and using for loop.
Its work very well when record numbers are 1000 to 5000. But it took huge time say 50 minutes when records are around 100,000 or more than this.
Initially I have used simple SQL Update Query then I had used stored procedure but both requires same time.
How to increase the performance of application? What are other ways when dealing with huge data to update? Do suggestions.

I hope this is why people use mongodb and no SQL systems. You can update huge data setsby optimizing your query. Read more here:
http://www.sqlservergeeks.com/blogs/AhmadOsama/personal/450/sql-server-optimizing-update-queries-for-large-data-volumes
Also check:Best practices for inserting/updating large amount of data in SQL Server 2008

One thing to consider is that iterating over a database table row by row, rather than performing set based update operations would incur a significant performance hit.
If you are in fact performing set based updates on your data and still have significant performance problems you should look at the execution plan of your queries so that you can workout where and why they are performing so badly.

Reading Huge Data Using SQL Stored Procedure and C# (SQL Server 2005)

We have a requirement to pull huge data from SQL Server 2005 database for reporting purpose. Our stored procedure is returning more than 15,000 rows.
When I call the procedure from the application (MVC 4.0) the request is timing out!!! (May be because of the data size)
Is there is any best practice to read such a huge data from SQL Server 2005 database using
MVC 4.0 Application???

You're seeing a timeout because your SQL query takes a long time to finish. This is not due to the size of the result (15,000 records is not a huge amount of data), but because the query runs inefficiently.
Maybe you're missing a couple of indices, maybe the stored procedure is written the wrong way - it's impossible to know from here. Try optimizing your query or database (if you have a DBA available, they can help. If not, the Management Studio can have some tips for you).
If you can't optimize the query or the database, you're left with increasing the time out, as others suggested.

Even i faced the same problem, but i was about to render more than 1,48,000 records. So the solution for this is using multithreading. You will be having one method which fetches the data from database, call that particular method in a seperate thread. Your data will be loaded in less than 5 seconds. Multithreading has been introduced only to manipulate large number of data without lagging performance.

First Que is why you are not using Dataset and Data source view in the Reporting(If its reporting in SQL server).
If its not Reporting Services and you only want to use C# code then try to make some helper function for it.
see here for the timeout option
http://forums.asp.net/t/1040377.aspx
and also here for optimising the code and SP
enter link description here

Here are couple tips on how you can use to optimize this:
Optimize query – see if you can optimize your query in some way. Add indices to your tables, check where statements and such.. I can’t really give you any specific recommendations w/o seeing the query and knowing the schema. See what others have already suggested on this topic.
Limit the amount of data stored procedure is returning – my guess is that MVC app doesn’t really need all 15k rows but a lot more. Check out this post: efficient way to implement paging . This will not speed up the query so much but it will make the app more efficient.

Reduce the number of rows

In my application I have a SQL Server 2008 table Employee Swipedaily_Tbl with 11 columns
where the employee daily swipes are inserted.
And I have about 8000 employees in my company. This means there will be at least 16000 rows created daily..
I am planing to delete all the rows at the end of a month and save them to another table in order to increase performance...... or back up the previous month data as dmb file from by application itself
As I am a new to SQL Server and DBA, can anyone suggest whether there is a better idea?
Can I create a dump file from the application?

Either by using Partitioning Table so inserting new data in huge volume database table won't effect its performance or using Script to backup data monthly wise using SQL Job and delete from existing one but if you are using Identity column you might need some changes in script to avoid conflict in old and new data.

Create an identical table
Create a SQL script to copy all the data older than a given date
(say today's date) to that table and delete from your table
Configure a SQL agent job to execute that script on the 1st of every
month
However, with proper indexing, you should be OK to reatian the data in your original table itself for a much longer period - 365 day x 8000 employees x 2 swipes = 5.84 million records, not too much for SQL server to handle.
Raj

You can create another table identical to Swipedaily_Tbl(11 columns) with additional one column that would tell when specific record was inserted in the backup table. You can then create a script that would backup the data older than one month and delete that data from the orignal table. You can then create a batch or a console application that could be scheduled to run at the end of month.
Hope this help.
Thanks.

It would depend on your requirements for the "old" data.
Personally, I would strongly consider using table partitioning.
See: http://technet.microsoft.com/en-us/library/dd578580(v=sql.100).aspx
Keep all records in table; this will make queries that look at current and historic data simultaneously simpler and potentially cheaper.

As all too often, it depends. Native partitioning requires the Enterprise Edition of SQL Server, however there are ways around it (although not very clean), like this.
If you do have the Enterprise Edition of SQL Server, I would take a serious look at partitioning (well linked in some of the other answers here), however I wouldn't split on a monthly basis, maybe a quarterly or semi-annual basis, as at two swipes per day is less than half a million rows per month, and a 1.5-3 mil. row table isn't that much for SQL server to handle.
If you are experiencing performance issues at this point in time with maybe a few months of data, have you reviewed the most frequent queries hitting the table and ensured that they're using indexes?

Multiple MDF FIles VS Single DataBase(SQL Server)

I'm working on a web2 project that i would like has thousands of rows per day by users.
for handling this size of data i designed database like this:
one .mdf and .ldf file as Minor DataBase and 1 Major DB to save and query the User Accounts and DataBase Files addresses.
i have worked several months for this plan and now i can manage it easily.
i want to know if it is good idea to handle huge size of Independent datas ?
witch has better performance in your opinion ? opening connection of many small .mdf files or just a huge dataBase.
afterwards i'll divide the mdf Repository in several computers.
all of them are handled by C# and linq (.net4)
// Later Descriptions
i built this plan and it works fine.
for example: opening every small mdf file takes 1sec time and query it in 0.0sec. it makes static time for every connection but in single Database for 50rows system must find them in for instance 200,000 rows and takes about 4-5sec in my system with simple select query with Primary key.
for other instance i want to get a row between 500,000 rows to bind page content and select 50 Comments between 2milmions row, and get count of votes of every comment, view count in day, week, month and total. count of likes, answer of comments and get more datas from 2-3 other tables, this querys are heavy and take more time than small slave database.
i think a good design and processes must work easy for system.
the only problem is that small slave databases with sql server files takes more physical size about 3MB per DataBase.

There is no reason to split something that could/should exist as a single database in to multiple independent parts.
There are already mechanisms to partition a single logical database across multiple files: Files and Filegroups Architecture as well as to partition large tables (A few thousand rows per day doesn't really qualify as a large table).

"Thousands of rows per day" should be pocket change for Sql Server.
First, I voted up Alex K answer. File groups will get you to where you want to be most likely. Partitioned tables may be overkill, and is only available in Enterprise version and is not for the light hearted.
What I will add is:
http://www.google.com/#q=glenn+berry+dmv&bav=on.2,or.r_gc.r_pw.&fp=73d2ceaabb6b01bf&hl=en
You need to tweak your indexes. In the good vs. better vs. best category, Glenn Berry's DMV queries are "better". Those queries will help you fix the majority of issues.
In the "best" category is pain staking looking at each stored procedure, and looking at the execution plan and trying out different things. This is what a good dba is able to provide.
Here are some "basics" on file setup considerations. Pay attention the TEMP database setup.
http://technet.microsoft.com/en-us/library/cc966534.aspx

its difficult to manage small MDF file u have to go with SQL server and SQL server database provide 10GB data storage per one database os its easy