Hey everybody, got an interesting question I think. I've got a Silverlight3 application which is querying a SQL Server 2005 database asynchronously. However, some of the data sets the queries return are massive and so I'm looking into data pagination.
Goals:
1) Just in time data calls - I only want to query for page 3's data when the user clicks to go to page 3.
2) I want a slider to control which page I'm on - SliderControl is SL3 with its movement binded to a stored procedure call (is my initial guess as to an approach).
3) Read ahead data as a label for the slider. So the slider will say "Page 1 of 50" or perhaps "Gant - Hart". Some sort of indication as to where you are in the data without actually querying for all the data until the user drops the slider into position.
I'm almost positive that I need to do some sort of a query to get the total number of rows and some sort of bookmark data that the query will eventually return. Otherwise I don't know how to split up the slider nor would I be able to do page labels (The gant to hart stuff).
Anyone have experience with this kind of thing? Thanks.
There are usually at least 2 separate lookups involved here, the first to get the pager information and a second to retrieve the data appropriate to the current page.
The pager (slider) needs to know the total number results that would be returned if you weren't doing paging. This is needed to show the pretty "Total # of Results" indicator for your use and to calculate how many total pages you've got. In your case you're going to want to format a nice visual display of the pages for selection (determined by your desired records/page value) in your slider. That's your first query; it should return just a single scalar int or long and you can calculate what you need for your slider display.
The second results include the actual records to be displayed for a given page.
Step 1 of 2 for a page of data is to run a query to filter and order your results by a primary key present within the data you're returning, throwing the key values in a temp table with an AUTO_INCREMENT/IDENTITY field or getting the row number over for the results of a derived table. (Summary: 2 fields, 1 for sequence/1 for primary key to join in step 2).
Step 2 of 2 is to join the key values with your tables which contain data, ordering by the sequence determined in Step 1, selecting only rows numbered (desired start row) to (desired end row) as determined by the page number and page size you've selected.
A stored procedure is strongly advised to keep the data on the server during this operation while you're having to examine more data than you really care to bring back to optimize bandwidth.
Related
In a method that takes the parameters int pageNum and int pageSize I am trying to return data from dynamics based on that specific page and size.
I am using a QueryExpression and I set the exp.PageInfo page number and page size to achieve this which works fine until page 51 with a page size of 100 which produces the error "Paging cookie is required when trying to retrieve a set of records on any high pages"
This brings me to near duplicate question territory (such as Dynamics CRM - How to get the second page of a fetchXml query beyond the 5000 elements? ) but the claims "you do not need the paging cookie" are just not correct at all it seems, there is nothing I can do for results beyond 5k that do not produce that error.
I am now paging over -the entire result set- (which allows me to get the PagingCookie from the previous results to pass to the next page request) and then returning the data I want from that set, but that is super slow. I have made it faster by dynamically altering the query in the paging loop so it only returns columns if the current data is in the requested page range which has shaved about 30secs off the query, but still very slow for such a large data set.
So, is there A) some thing that will enable me to get these high results without a paging cookie? Is this a QueryExpression limitation for example? or B) a faster way of handling this problem than iterating over all results until the page I want?
Thanks.
Unfortunately there's no way to "fast forward" to high pages in your query result.
You'll have to use the paging cookie just the way you are doing right now.
If you knew the last and next record from a previous query you could try creating a high-page-cookie to start with.
As #Aron suggest in his answer, the only improvement might be gained from sorting and/or filtering/partitioning the data (by createdon, etc.).
Filtering the data set to return fewer pages might be something to investigate. Or, if the desired page is closer to the end of the data set, reverse the sort order.
I'm using jquery datatables to display a grid which uses webapi to retrieve it's data. The webapi uses linq to query a mssql database and it neatly uses filtering, sorting and skip/take to assemble it's query on a well-indexed table containing about a million records (and growing). A common scenario.
And it performs really well. The browser has to wait about 50 ms for the response (while paginating for example) to return.
However, after I took a look with a profiling tool I noticed about 25 ms to be used just selecting the total rowcount of the table. Which I want to know because I want the datatable to display something like: "displaying row 1 to 10 of 45.000 filtered out of 1.000.000" needing the total count.
I don't actually need to know the precise total count (it's just informative) every trip from the server so I perhaps could keep the value server side and refresh it every second in a different task without it interfering with the data retrieval of datatables. I would just return the 'close enough' value of the total row count.
Is there a solid mechanism for that? I've tried to put the total rowcount in a static used by multiple users during multiple callbacks and every time it was requested a async task was fired to refresh it.
That feels icky however, sharing the static and having a different thread update it doesn't feel all that stable to me. I've looked at SqlDependency to push the recordcount every time it changes from my data to my domain model but that doesn't seem to support SELECT COUNT(Id) FROM TABLE scenarios.
Any thoughts?
You could use one of the system tables if possible. You could ping this every minute and stick it in the cache. This article has two that it claims are sufficient options:
--The way the SQL management studio counts rows (look at table properties, storage
--, row count). Very fast, but still an approximate number of rows.
SELECT CAST(p.rows AS float)
FROM sys.tables AS tbl
INNER JOIN sys.indexes AS idx ON idx.object_id = tbl.object_id and idx.index_id < 2
INNER JOIN sys.partitions AS p ON p.object_id=CAST(tbl.object_id AS int)
AND p.index_id=idx.index_id
WHERE ((tbl.name=N'Transactions'
AND SCHEMA_NAME(tbl.schema_id)='dbo'))
or
--Quick (although not as fast as method 2) operation and equally important, reliable.
SELECT SUM (row_count)
FROM sys.dm_db_partition_stats
WHERE object_id=OBJECT_ID('Transactions')
AND (index_id=0 or index_id=1);
Have you considered taking the count when a query is performed and then echoing the value out to your clients via SignalR?
Basically, when the LINQ call returns get a .Count() and hand off the value to a background thread to let SignalR notify the clients of the update, at the same time you return the data to the requesting client.
SignalR will activate a javascript function in all of the client pages, where you can then take the passed in value and display it somewhere on the page.
http://www.asp.net/signalr
After a little bit of advise.
We are producing a standard list of data with filters.
This filtered list can have 3 alternative views which represent the data in different manor
my question is:
should the stored procedure return the data in the format that is required for each view for the front end?
or should the business layer re-format the data?
I have always believed it best to get in and out of the database as fast as possible, and allow the business layer to handle formatting.
Thanks in advance for any advise
UPDATE:-
by formating,
1 view contain all data withing a scrollable table
2 view will group data withing headings, (click on a heading to expand to view)
3 view will group data into date groups and display dates as table row headers (7 day intivials) and the data count will be displayed within those intivals
I have always believed it best to get in and out of the database as
fast as possible, and allow the business layer to handle formatting.
Short answer: Correct, you have answered your own question.
The database returns data to your application, how it is presented is up to the application. What happens when you want to supply the same data to another application? Do you want to have to write procedures every time you want the data in a different format? Of course not.
1: view contain all data withing a scrollable table
You cannot return a scrolling window from a procedure. Anyway, how would the database know how much data will need to be scrolled? It does not know the size of window/viewport you are using.
2: view will group data withing headings, (click on a heading to expand to view)
This requires user interaction which is solely the domain of the presentation layer.
3: view will group data into date groups and display dates as table
row headers (7 day intivials) and the data count will be displayed
within those intivals
You can return multiple result sets from a procedure, to give the results and the counts, but again this is more easily handled in the application.
In general you should use the presentation layer to format your data output. You can have the stored procedure return the raw data set and then set up, for example 3 different XSLT files for the 3 different presentation types. This way you can always do DataTable.ToXML and apply the XSLT template to this output to produce the format that the user needs.
I have an web application that when you go onto a page a ajax call is made to get all the data from the database. It stores the data into a Global variable and as the user scrolls down the page it loads more into the page. Now I would like to be able to call the database and grab lets say 20 items from the database as they scroll down, instead of getting them all on the page load. Is there a good way of doing this? Thanks for any help.
here is what I would like to do
1.Page load (get first 20 items)
2.As they scroll down near bottom(Ajax call to get the next 20 and append it to table, list etc)
mysql help for the correct query.
MySQL supports the LIMIT keyword on selects, e.g.:
SELECT foo,bar
FROM charlie
WHERE criterion = value
LIMIT 20
It accepts up to two arguments. If you supply one, it's the number of rows (max) to return. If you supply two, the first is an offset (skip) value, and the second is a limit. E.g. (from the docs):
SELECT * FROM tbl LIMIT 5,10; # Retrieve rows 6-15
Combine LIMIT with ORDER BY and barring inserts, you have paging.
Yes, the LIMIT option on the SELECT will do the trick. You'll obviously also need to keep track of the current position, and use the offset argument with LIMIT to make sure that you keep moving forward in the set.
LIMIT [offset], limit
So, after your query for the first 20, you'd need to store that position (20), and then use that as the offset:
LIMIT 20, 20
Then you'll need to increase the offset so that your limit is:
LIMIT 40, 20
I have an operation (That I can't change) that starts threads that make calls to our Oracle database to see if a certain hotel(s) has availability on a certain date.
If a date/hotel combination has availability, that thread returns information about the date/hotel in the form of a DataTable that is merged into a Main DataTable of results. Yes, I know ... I inherited this.
So I am trying to re-write this operation. I still must query Oracle in threads to get the availability information, but I want to display the data as it is returned (in chunks of 5, 10? I'm flexible), instead of having the user sit in front of the screen for up to 4 minutes before a complete result is spat out into a GridView.
How do I do this directly from an .aspx page so I can make a web service call and populate a grid (JqGrid?) with the results?
If I haven't provided enough information or described what I am trying to achieve, please let me know and I will elaborate.
Oracle provides a field on each row called "rowid"
(http://www.adp-gmbh.ch/ora/concepts/rowid.html)
The first time you send the query, send in the int (x) to define what the highest rownumber you want is. Have the service return the total number of rows and the first x rows.
Then, the 2nd time you send the query, get the next x rows, rinse and repeat.
Basically, you need to send an ajax query for rows x through y each time until you have them all loaded.
I would recommend paging as well, since users typically don't want to see hundreds of results at a time.