In a method that takes the parameters int pageNum and int pageSize I am trying to return data from dynamics based on that specific page and size.
I am using a QueryExpression and I set the exp.PageInfo page number and page size to achieve this which works fine until page 51 with a page size of 100 which produces the error "Paging cookie is required when trying to retrieve a set of records on any high pages"
This brings me to near duplicate question territory (such as Dynamics CRM - How to get the second page of a fetchXml query beyond the 5000 elements? ) but the claims "you do not need the paging cookie" are just not correct at all it seems, there is nothing I can do for results beyond 5k that do not produce that error.
I am now paging over -the entire result set- (which allows me to get the PagingCookie from the previous results to pass to the next page request) and then returning the data I want from that set, but that is super slow. I have made it faster by dynamically altering the query in the paging loop so it only returns columns if the current data is in the requested page range which has shaved about 30secs off the query, but still very slow for such a large data set.
So, is there A) some thing that will enable me to get these high results without a paging cookie? Is this a QueryExpression limitation for example? or B) a faster way of handling this problem than iterating over all results until the page I want?
Thanks.
Unfortunately there's no way to "fast forward" to high pages in your query result.
You'll have to use the paging cookie just the way you are doing right now.
If you knew the last and next record from a previous query you could try creating a high-page-cookie to start with.
As #Aron suggest in his answer, the only improvement might be gained from sorting and/or filtering/partitioning the data (by createdon, etc.).
Filtering the data set to return fewer pages might be something to investigate. Or, if the desired page is closer to the end of the data set, reverse the sort order.
Related
I have an endpoint that returns a list of "Order" objects for a commerce app. Each order can have a number of items attached, each with its own cost; because there is no guarantee that the items' individual cost will not change until the order is placed, the Total Cost for each order is being calculated on the server and then appended to the returned results.
I am now trying to add pagination to this endpoint, and it is my understanding that pagination should always be left to the database when possible. I also get that in order for pagination to work properly, the results must be ordered in the same manner for every page.
You can probably guess where I'm having an issue, how can I implement pagination here when I can only get the element I'd like to order by AFTER I receive my results.
I could simply retrieve all the results in the query and perform the pagination on the server using Skip() and Take(), but that seems incredibly inefficient and probably not the best solution. Any advice on what I could do, besides disabling ordering by that specific property entirely?
By using row_number() you can assign SerialNo to all rows
and you can take input from-to and can take data as
SerialNo between from and to
We have a set of catch all search pages we're creating in our ASP.NET application. We have an initial search page, a SERP, and then a single item details page. All 3 pages have a search bar with initial criteria, more criteria, and advanced criteria choices.
When we put all of our criteria together, in addition to the main search box we have 20 different criteria parameters (from price, to price, sale item, date created, etc.) and then three collections of parameter IDs. These collections are from a list of the Manufacturers, Product Lines, and Categories our users can search from. So we have this fixed set of 20 fields and then 3 collections that could have a manufacturer or two, or could hold a collection of 100 Guids for the lines whose checkboxes they selected and want to search through.
In our old system we had a single form solution and we just posted back and submitted everything to our business object, passing it into a method that returned the results. In this new form we need to submit the results from page to page and persist this criteria. We're trying to figure out the best way to persist the data, when I say best I mean most efficient.
Querystring - This isn't going to work with large collections of Guid values for the 3 collections.
Session - We would create a criteria object and store it in the Session. As they move from page to page we can pull it out. At our peak we probable have 200-300 people using the server concurrently and the search is our most used form. I'm worried about performance with all those session variables.
Database - We were thinking of serializing and stashing this criteria object into the database (SQL Server 2k5) and the users would always have a current Search or last Search in the database. This eliminates some of the web server load from the Session solution but I'm worried this object load, serialization, db round trip, and unload is going to slow the forms down and affect user experience.
I'm looking for advice on which method is going to work most efficiently for us or if there is an accepted best practice or pattern I've overlooked.
With HTML5 you can use localStorage and sessionStorage, which makes the client keep the information in their browser.
http://www.w3schools.com/html/html5_webstorage.asp
Here's a problem I experience (simplified example):
Let's say I have several tables:
One customer can have mamy products and a product can have multiple features.
On my asp.net front-end I have a grid with customer info:
something like this:
Name Address
John 222 1st st
Mark 111 2nd st
What I need is an ability to filter customers by feature. So, I have a dropdown list of available features that are connected to a customer.
What I currently do:
1. I return DataTable of Customers from stored procedure. I store it in viewstate
2. I return DataTable of features connected to customers from stored procedure. I store it in viewstate
3. On filter selected, I run stored procedure again with new feature_id filter where I do joins again to only show customers that have selected feature.
My problem: It is very slow.
I think that possible solutions would be:
1. On page load return ALL data in one viewstate variable. So basically three lists of nested objects. This will make my page load slow.
2. Perform async loazing in some smart way. How?
Any better solutions?
Edit:
this is a simplified example, so I also need to filter customer by property that is connected through 6 tables to table Customer.
The way I deal with these scenarios is by passing in Xml to SQL and then running a join against that. So Xml would look something like:
<Features><Feat Id="2" /><Feat Id="5" /><feat Id="8" /></Features>
Then you can pass that Xml into SQL (depending on what version of SQL there are different ways), but in the newer version's its a lot easier than it used to be:
http://www.codeproject.com/Articles/20847/Passing-Arrays-in-SQL-Parameters-using-XML-Data-Ty
Also, don't put any of that in ViewState; there's really no reason for that.
Storing an entire list of customers in ViewState is going to be hideously slow; storing all information for all customers in ViewState is going to be worse, unless your entire customer base is very very small, like about 30 records.
For a start, why are you loading all the customers into ViewState? If you have any significant number of customers, load the data a page at a time. That will at least reduce the amount of data flowing over the wire and might speed up your stored procedure as well.
In your position, I would focus on optimizing the data retrieval first (including minimizing the amount you return), and then worry about faster ways to store and display it. If you're up against unusual constraints that prevent this (very slow database; no profiling tools; not allowed to change stored procedures) than please let us know.
Solution 1: Include whatever criteria you need to filter on in your query, only return and render the requested records. No need to use viewstate.
Solution 2: Retrieve some reasonable page limit of customers, filter on the browser with javascript. Allow easy navigation to the next page.
I have an operation (That I can't change) that starts threads that make calls to our Oracle database to see if a certain hotel(s) has availability on a certain date.
If a date/hotel combination has availability, that thread returns information about the date/hotel in the form of a DataTable that is merged into a Main DataTable of results. Yes, I know ... I inherited this.
So I am trying to re-write this operation. I still must query Oracle in threads to get the availability information, but I want to display the data as it is returned (in chunks of 5, 10? I'm flexible), instead of having the user sit in front of the screen for up to 4 minutes before a complete result is spat out into a GridView.
How do I do this directly from an .aspx page so I can make a web service call and populate a grid (JqGrid?) with the results?
If I haven't provided enough information or described what I am trying to achieve, please let me know and I will elaborate.
Oracle provides a field on each row called "rowid"
(http://www.adp-gmbh.ch/ora/concepts/rowid.html)
The first time you send the query, send in the int (x) to define what the highest rownumber you want is. Have the service return the total number of rows and the first x rows.
Then, the 2nd time you send the query, get the next x rows, rinse and repeat.
Basically, you need to send an ajax query for rows x through y each time until you have them all loaded.
I would recommend paging as well, since users typically don't want to see hundreds of results at a time.
Hey everybody, got an interesting question I think. I've got a Silverlight3 application which is querying a SQL Server 2005 database asynchronously. However, some of the data sets the queries return are massive and so I'm looking into data pagination.
Goals:
1) Just in time data calls - I only want to query for page 3's data when the user clicks to go to page 3.
2) I want a slider to control which page I'm on - SliderControl is SL3 with its movement binded to a stored procedure call (is my initial guess as to an approach).
3) Read ahead data as a label for the slider. So the slider will say "Page 1 of 50" or perhaps "Gant - Hart". Some sort of indication as to where you are in the data without actually querying for all the data until the user drops the slider into position.
I'm almost positive that I need to do some sort of a query to get the total number of rows and some sort of bookmark data that the query will eventually return. Otherwise I don't know how to split up the slider nor would I be able to do page labels (The gant to hart stuff).
Anyone have experience with this kind of thing? Thanks.
There are usually at least 2 separate lookups involved here, the first to get the pager information and a second to retrieve the data appropriate to the current page.
The pager (slider) needs to know the total number results that would be returned if you weren't doing paging. This is needed to show the pretty "Total # of Results" indicator for your use and to calculate how many total pages you've got. In your case you're going to want to format a nice visual display of the pages for selection (determined by your desired records/page value) in your slider. That's your first query; it should return just a single scalar int or long and you can calculate what you need for your slider display.
The second results include the actual records to be displayed for a given page.
Step 1 of 2 for a page of data is to run a query to filter and order your results by a primary key present within the data you're returning, throwing the key values in a temp table with an AUTO_INCREMENT/IDENTITY field or getting the row number over for the results of a derived table. (Summary: 2 fields, 1 for sequence/1 for primary key to join in step 2).
Step 2 of 2 is to join the key values with your tables which contain data, ordering by the sequence determined in Step 1, selecting only rows numbered (desired start row) to (desired end row) as determined by the page number and page size you've selected.
A stored procedure is strongly advised to keep the data on the server during this operation while you're having to examine more data than you really care to bring back to optimize bandwidth.