I have an web application that when you go onto a page a ajax call is made to get all the data from the database. It stores the data into a Global variable and as the user scrolls down the page it loads more into the page. Now I would like to be able to call the database and grab lets say 20 items from the database as they scroll down, instead of getting them all on the page load. Is there a good way of doing this? Thanks for any help.
here is what I would like to do
1.Page load (get first 20 items)
2.As they scroll down near bottom(Ajax call to get the next 20 and append it to table, list etc)
mysql help for the correct query.
MySQL supports the LIMIT keyword on selects, e.g.:
SELECT foo,bar
FROM charlie
WHERE criterion = value
LIMIT 20
It accepts up to two arguments. If you supply one, it's the number of rows (max) to return. If you supply two, the first is an offset (skip) value, and the second is a limit. E.g. (from the docs):
SELECT * FROM tbl LIMIT 5,10; # Retrieve rows 6-15
Combine LIMIT with ORDER BY and barring inserts, you have paging.
Yes, the LIMIT option on the SELECT will do the trick. You'll obviously also need to keep track of the current position, and use the offset argument with LIMIT to make sure that you keep moving forward in the set.
LIMIT [offset], limit
So, after your query for the first 20, you'd need to store that position (20), and then use that as the offset:
LIMIT 20, 20
Then you'll need to increase the offset so that your limit is:
LIMIT 40, 20
Related
I have a query that gets executed on an SQLite database table from a WPF application; this query can return a large amount of records (from about 80000 up to about 1 million) which I down-sample before displaying it.
It takes about 10 seconds to return 700000 records, can it be optimized in some way?
The WHERE clause filters the records by a date time column:
(WHERE CollectedOn > #startTime AND CollectedOn < #endTime)
and I'm selecting all of the 18 columns of the table.
Does the number of columns influence the executing time of the query?
Thanks for the comments. I should point out a few more things:
The data I collect needs to be displayed in a chart; since I want to display only 600 points my algorithm picks one point every 600 from those 700000 records. Can this be achieved in a single query?
These are some of the things I would consider:
Further narrowing down the number of returned records (you said that you down-sample before displaying... can you down-sample within the database or even the WHERE clause)?
Do you really need all the records at once? Maybe paging would help (see LIMIT and OFFSET)
You could try to use an index to speed up your query. Use EXPLAIN to find out what your query does exactly... afterwards you can optimize joins and selections (also use indices for joins).
Narrowing down the attributes is always a good thing to do (instead of just returning all columns), but at least for simple queries (no subselects), it will have less influence than selecting the right rows using the WHERE clause. Also search for "selection" and "projection" about this issue.
I created a new table and a new sequence, I have two C# web services trying to insert records into this table using same query utilizing mySequence.nextval (and yes I checked it many times, they both use mySequence.nextval).
The two web services are inserting rows to the table, but the mySequence.nextval is returning numbers out of sequence
Here is how the records were created, showing PrimaryKey which gets its value from mySequence.nextval
1 21 22 23 2 3 24 25 4 27 28 5
So far no duplicates but why is mySequence.nextval jumping back and forth? and should I worry about it
Update:
The sequence is created with cache_size = 20
I will wager that your database is running RAC (Real Application Clusters). Assuming that is the case and that you create the sequence with all the default settings, that's the expected behavior.
The default setting is to cache 20 values. Each node in the RAC cluster, by default, will have a separate cache. Assuming that you have a cluster with two nodes A and B, the first time a nextval is requested on A, A will cache values 1-20 and return a value of 1. If the next request for a nextval is made on B, B will cache values 21-40 and return a value of 21. From there, the value you get will depend on the node that your connection happens to be running on.
Generally, this shouldn't be a problem. Sequences generate unique numbers. The numbers generally need not be consecutive. If you really need values to be returned sequentially because you are doing something like ordering by the sequence-generated value to determine the "first" or "last" row, you can use the ORDER clause when you create the sequence to force values to be returned in order. That has a negative performance implication in a RAC database, however, because it increases the amount of communication that needs to go on between the nodes to synchronize the values being returned. If you need to determine the "first" or "last" row, it's generally better to add a date or a timestamp column to the table and order by that rather than assuming that the primary key is generated sequentially.
From the docs...
Sequence numbers are generated independently of tables, so the same sequence can be used for one or for multiple tables. It is possible that individual sequence numbers will appear to be skipped, because they were generated and used in a transaction that ultimately rolled back. Additionally, a single user may not realize that other users are drawing from the same sequence.
http://docs.oracle.com/cd/B28359_01/server.111/b28286/statements_6015.htm#SQLRF01314
I have a simple tool for searching in a given db. The user can provide numerous conditions and my tool puts the sql query together based on that. However I want to prevent the query to be executed in case it returns too many records. For e.g. in case the user leaves all the filters blank then the query would pull all the records from the db which would take tens of minutes. Of course it's not necessary for any of my users. So I want some limitation.
I was thinking about running a count() sql query with the same conditions before each 'real' query, but that takes too much time.
Is there any option to measure the records 'during' the query and stop it if a certain amount is being reached? Throwing some exception asking the user to refine the search.
I use this approach:
State that you want to fetch AT MOST 100 rows. Construct your query so it returns at most 101 rows (with TOP N or the more generic ANSI way by filtering on row_number). Then you can easily detect whether there is more. You can act accordingly, in my case, show a 'read more'.
You could run a test query to search the database with the user defined options and only return the id field of the returned results, this would be very quick and also allow you to test the count().
Then if all is ok then you can run the full query to return all of their results.
Following on from the answer above, if you are working with large amounts of data, select top N, with the fast query option.
E.g.
SELECT TOP 101 [ColumnName]
FROM [Table]
OPTION (FAST 101)
This depends on your application and how you want it to work.
If your only wanting to displaying data on a table and setting a maximum size to your query is enough. You can use TOP in your select statement.
SELECT TOP N [ColumnName]
But considering you said a count takes too much time then I think your concerned about handling a very large data set and maybe manipulating it not necessarily just getting a limited set of data from the query.
Then one method is to break apart the job into chunks and run across the job size so grab the first N rows then the next N rows and repeat until there is no more values to be returned. You can also have record keeping for rollbacks and checkpoints to ensure data integrity.
Similar questions maybe:
query to limit records returned by sql query based on size of data
How to iterate through a large SQL result set with multiple related tables
I have an operation (That I can't change) that starts threads that make calls to our Oracle database to see if a certain hotel(s) has availability on a certain date.
If a date/hotel combination has availability, that thread returns information about the date/hotel in the form of a DataTable that is merged into a Main DataTable of results. Yes, I know ... I inherited this.
So I am trying to re-write this operation. I still must query Oracle in threads to get the availability information, but I want to display the data as it is returned (in chunks of 5, 10? I'm flexible), instead of having the user sit in front of the screen for up to 4 minutes before a complete result is spat out into a GridView.
How do I do this directly from an .aspx page so I can make a web service call and populate a grid (JqGrid?) with the results?
If I haven't provided enough information or described what I am trying to achieve, please let me know and I will elaborate.
Oracle provides a field on each row called "rowid"
(http://www.adp-gmbh.ch/ora/concepts/rowid.html)
The first time you send the query, send in the int (x) to define what the highest rownumber you want is. Have the service return the total number of rows and the first x rows.
Then, the 2nd time you send the query, get the next x rows, rinse and repeat.
Basically, you need to send an ajax query for rows x through y each time until you have them all loaded.
I would recommend paging as well, since users typically don't want to see hundreds of results at a time.
Hey everybody, got an interesting question I think. I've got a Silverlight3 application which is querying a SQL Server 2005 database asynchronously. However, some of the data sets the queries return are massive and so I'm looking into data pagination.
Goals:
1) Just in time data calls - I only want to query for page 3's data when the user clicks to go to page 3.
2) I want a slider to control which page I'm on - SliderControl is SL3 with its movement binded to a stored procedure call (is my initial guess as to an approach).
3) Read ahead data as a label for the slider. So the slider will say "Page 1 of 50" or perhaps "Gant - Hart". Some sort of indication as to where you are in the data without actually querying for all the data until the user drops the slider into position.
I'm almost positive that I need to do some sort of a query to get the total number of rows and some sort of bookmark data that the query will eventually return. Otherwise I don't know how to split up the slider nor would I be able to do page labels (The gant to hart stuff).
Anyone have experience with this kind of thing? Thanks.
There are usually at least 2 separate lookups involved here, the first to get the pager information and a second to retrieve the data appropriate to the current page.
The pager (slider) needs to know the total number results that would be returned if you weren't doing paging. This is needed to show the pretty "Total # of Results" indicator for your use and to calculate how many total pages you've got. In your case you're going to want to format a nice visual display of the pages for selection (determined by your desired records/page value) in your slider. That's your first query; it should return just a single scalar int or long and you can calculate what you need for your slider display.
The second results include the actual records to be displayed for a given page.
Step 1 of 2 for a page of data is to run a query to filter and order your results by a primary key present within the data you're returning, throwing the key values in a temp table with an AUTO_INCREMENT/IDENTITY field or getting the row number over for the results of a derived table. (Summary: 2 fields, 1 for sequence/1 for primary key to join in step 2).
Step 2 of 2 is to join the key values with your tables which contain data, ordering by the sequence determined in Step 1, selecting only rows numbered (desired start row) to (desired end row) as determined by the page number and page size you've selected.
A stored procedure is strongly advised to keep the data on the server during this operation while you're having to examine more data than you really care to bring back to optimize bandwidth.