Related
What SQL can be used to list the tables, and the rows within those tables in an SQLite database file - once I have attached it with the ATTACH command on the SQLite 3 command line tool?
There are a few steps to see the tables in an SQLite database:
List the tables in your database:
.tables
List how the table looks:
.schema tablename
Print the entire table:
SELECT * FROM tablename;
List all of the available SQLite prompt commands:
.help
The .tables, and .schema "helper" functions don't look into ATTACHed databases: they just query the SQLITE_MASTER table for the "main" database. Consequently, if you used
ATTACH some_file.db AS my_db;
then you need to do
SELECT name FROM my_db.sqlite_master WHERE type='table';
Note that temporary tables don't show up with .tables either: you have to list sqlite_temp_master for that:
SELECT name FROM sqlite_temp_master WHERE type='table';
It appears you need to go through the sqlite_master table, like this:
SELECT * FROM dbname.sqlite_master WHERE type='table';
And then manually go through each table with a SELECT or similar to look at the rows.
The .DUMP and .SCHEMA commands doesn't appear to see the database at all.
To show all tables, use
SELECT name FROM sqlite_master WHERE type = "table"
To show all rows, I guess you can iterate through all tables and just do a SELECT * on each one. But maybe a DUMP is what you're after?
Use .help to check for available commands.
.table
This command would show all tables under your current database.
There is a command available for this on the SQLite command line:
.tables ?PATTERN? List names of tables matching a LIKE pattern
Which converts to the following SQL:
SELECT name FROM sqlite_master
WHERE type IN ('table','view') AND name NOT LIKE 'sqlite_%'
UNION ALL
SELECT name FROM sqlite_temp_master
WHERE type IN ('table','view')
ORDER BY 1
To list the tables you can also do:
SELECT name FROM sqlite_master
WHERE type='table';
I use this query to get it:
SELECT name FROM sqlite_master WHERE type='table'
And to use in iOS:
NSString *aStrQuery=[NSString stringWithFormat:#"SELECT name FROM sqlite_master WHERE type='table'"];
Try PRAGMA table_info(table-name);
http://www.sqlite.org/pragma.html#schema
According to the documentation, the equivalent of MySQL's SHOW TABLES; is:
The ".tables" command is similar to setting list mode then executing
the following query:
SELECT name FROM sqlite_master
WHERE type IN ('table','view') AND name NOT LIKE 'sqlite_%'
UNION ALL
SELECT name FROM sqlite_temp_master
WHERE type IN ('table','view')
ORDER BY 1;
However, if you are checking if a single table exists (or to get its details), see LuizGeron's answer.
As of the latest versions of SQLite 3 you can issue:
.fullschema
to see all of your create statements.
The easiest way to do this is to open the database directly and use the .dump command, rather than attaching it after invoking the SQLite 3 shell tool.
So... (assume your OS command line prompt is $) instead of $sqlite3:
sqlite3> ATTACH database.sqlite as "attached"
From your OS command line, open the database directly:
$sqlite3 database.sqlite
sqlite3> .dump
Via a union all, combine all tables into one list.
select name
from sqlite_master
where type='table'
union all
select name
from sqlite_temp_master
where type='table'
Use:
import sqlite3
TABLE_LIST_QUERY = "SELECT * FROM sqlite_master where type='table'"
Use .da to see all databases - one is called 'main'.
Tables of this database can be seen by:
SELECT distinct tbl_name from sqlite_master order by 1;
The attached databases need prefixes you chose with AS in the statement ATTACH, e.g., aa (, bb, cc...) so:
SELECT distinct tbl_name from **aa.sqlite_master** order by 1;
Note that here you get the views as well. To exclude these add:
where type = 'table'
before ' order'
Since nobody has mentioned about the official reference of SQLite, I think it may be useful to refer to it under this heading:
https://www.sqlite.org/cli.html
You can manipulate your database using the commands described in this link. Besides, if you are using Windows OS and do not know where the command shell is, that is in the SQLite's site:
https://www.sqlite.org/download.html
After downloading it, click sqlite3.exe file to initialize the SQLite command shell. When it is initialized, by default this SQLite session is using an in-memory database, not a file on disk, and so all changes will be lost when the session exits. To use a persistent disk file as the database, enter the ".open ex1.db" command immediately after the terminal window starts up.
The example above causes the database file named "ex1.db" to be opened and used, and created if it does not previously exist. You might want to use a full pathname to ensure that the file is in the directory that you think it is in. Use forward-slashes as the directory separator character. In other words use "c:/work/ex1.db", not "c:\work\ex1.db".
To see all tables in the database you have previously chosen, type the command .tables as it is said in the above link.
If you work in Windows, I think it might be useful to move this sqlite.exe file to same folder with the other Python files. In this way, the Python file writes to and the SQLite shell reads from .db files are in the same path.
The ".schema" commando will list available tables and their rows, by showing you the statement used to create said tables:
sqlite> create table_a (id int, a int, b int);
sqlite> .schema table_a
CREATE TABLE table_a (id int, a int, b int);
I am issuing an SQL query in my visual studio application to search a database. I am having trouble joining two tables into one query search. I am looking to combine the First and Last name of an employee and match it up to a phone number. In one table I have an abbreviated name, and on the other table I have the full name.
Example
Code
cmdTxt.Append("SELECT partner.staffid, staff.Forename,staff.surname FROM tblpartner LEFT JOIN tblstaff ON staff.StaffID = partner.staffid ORDER BY staff.forename , staff.surname ");
I would Like to have a field that shows the Forename , surename, and phone number in one query search
You are using "partner" and "staff" as table aliases, but you aren't assigning those aliases to the table names. If you want to use those instead of the full table names, you need:
SELECT partner.staffid, staff.Forename,staff.surname, partner.phone
FROM tblpartner partner
LEFT JOIN tblstaff staff ON staff.StaffID = partner.staffid
ORDER BY staff.forename , staff.surname
As general advice, get your query working outside of your string literal; enter it into SQL Server Management Studio or something similar. Had you done that, it should have clearly showed you what the problem was. Then when you have your query set how you want, you can copy it into your Command Text in your Visual Studio application.
I am trying to update my mysql table "upproj" using Visual Studio 2010 (c#).
I want to execute the following query :
string Query = " update tlog.upproj set summ=(select sum(grade) from tlog.upproj group by ams having count(*) =2); ";
But I get error
"You can't specify target table 'upproj' for update in FROM clause".
When I execute the previous query on Mysql Query browser I don't get any error, but my project needs to execute this query from visual studio.
Is there any way to fix that?
As the error hints at, you can't specify the table you're updating in the UPDATE statement's FROM clause (in the sub-query); the same is true if you were to try to use it in a WHERE clause.
You can accomplish the task of "using the table you're updating" if you use a JOIN though.
Based on the schema inferred by your existing query, the ams field is unique to a set grouping and you can, therefore, add that as part of your clause.
Try the following (untested):
UPDATE
tlog.upproj AS u1
JOIN (SELECT ams, SUM(grade) AS grade_sum FROM tlog.upproj GROUP BY ams HAVING COUNT(*) = 2) AS u2
ON u1.ams = u2.ams
SET
summ = u2.grade_sum;
This should update each record for ams with the total sum of grade for that same ams. If a specific ams doesn't have 2 entries, it isn't updated.
Why does the Entity Framework generate nested SQL queries?
I have this code
var db = new Context();
var result = db.Network.Where(x => x.ServerID == serverId)
.OrderBy(x=> x.StartTime)
.Take(limit);
Which generates this! (Note the double select statement)
SELECT
`Project1`.`Id`,
`Project1`.`ServerID`,
`Project1`.`EventId`,
`Project1`.`StartTime`
FROM (SELECT
`Extent1`.`Id`,
`Extent1`.`ServerID`,
`Extent1`.`EventId`,
`Extent1`.`StartTime`
FROM `Networkes` AS `Extent1`
WHERE `Extent1`.`ServerID` = #p__linq__0) AS `Project1`
ORDER BY
`Project1`.`StartTime` DESC LIMIT 5
What should I change so that it results in one select statement? I'm using MySQL and Entity Framework with Code First.
Update
I have the same result regardless of the type of the parameter passed to the OrderBy() method.
Update 2: Timed
Total Time (hh:mm:ss.ms) 05:34:13.000
Average Time (hh:mm:ss.ms) 25:42.000
Max Time (hh:mm:ss.ms) 51:54.000
Count 13
First Seen Nov 6, 12 19:48:19
Last Seen Nov 6, 12 20:40:22
Raw query:
SELECT `Project?`.`Id`, `Project?`.`ServerID`, `Project?`.`EventId`, `Project?`.`StartTime` FROM (SELECT `Extent?`.`Id`, `Extent?`.`ServerID`, `Extent?`.`EventId`, `Extent?`.`StartTime`, FROM `Network` AS `Extent?` WHERE `Extent?`.`ServerID` = ?) AS `Project?` ORDER BY `Project?`.`Starttime` DESC LIMIT ?
I used a program to take snapshots from the current process in MySQL.
Other queries were executed at the same time, but when I change it to just one SELECT statement, it NEVER goes over one second. Maybe I have something else that's going on; I'm asking 'cause I'm not so into DBs...
Update 3: The explain statement
The Entity Framework generated
'1', 'PRIMARY', '<derived2>', 'ALL', NULL, NULL, NULL, NULL, '46', 'Using filesort'
'2', 'DERIVED', 'Extent?', 'ref', 'serveridneventid,serverid', 'serveridneventid', '109', '', '45', 'Using where'
One liner
'1', 'SIMPLE', 'network', 'ref', 'serveridneventid,serverid', 'serveridneventid', '109', 'const', '45', 'Using where; Using filesort'
This is from my QA environment, so the timing I pasted above is not related to the rowcount explain statements. I think that there are about 500,000 records that match one server ID.
Solution
I switched from MySQL to SQL Server. I don't want to end up completely rewriting the application layer.
It's the easiest way to build the query logically from the expression tree. Usually the performance will not be an issue. If you are having performance issues you can try something like this to get the entities back:
var results = db.ExecuteStoreQuery<Network>(
"SELECT Id, ServerID, EventId, StartTime FROM Network WHERE ServerID = #ID",
serverId);
results = results.OrderBy(x=> x.StartTime).Take(limit);
My initial impression was that doing it this way would actually be more efficient, although in testing against a MSSQL server, I got <1 second responses regardless.
With a single select statement, it sorts all the records (Order By), and then filters them to the set you want to see (Where), and then takes the top 5 (Limit 5 or, for me, Top 5). On a large table, the sort takes a significant portion of the time. With a nested statement, it first filters the records down to a subset, and only then does the expensive sort operation on it.
Edit: I did test this, but I realized I had an error in my test which invalidated it. Test results removed.
Why does Entity Framework produce a nested query? The simple answer is because Entity Framework breaks your query expression down into an expression tree and then uses that expression tree to build your query. A tree naturally generates nested query expressions (i.e. a child node generates a query and a parent node generates a query on that query).
Why doesn't Entity Framework simplify the query down and write it as you would? The simple answer is because there is a limited amount of work that can go into the query generation engine, and while it's better now than it was in earlier versions it's not perfect and probably never will be.
All that said there should be no significant speed difference between the query you would write by hand and the query EF generated in this case. The database is clever enough to generate an execution plan that applies the WHERE clause first in either case.
If you want to get the EF to generate the query without the subselect, use a constant within the query, not a variable.
I have previously created my own .Where and all other LINQ methods that first traverse the expression tree and convert all variables, method calls etc. into Expression.Constant. It was done just because of this issue in Entity Framework...
I just stumbled upon this post because I suffer from the same problem. I already spend days tracking this down and it it is just a poor query generation in mysql.
I already filed a bug at mysql.com http://bugs.mysql.com/bug.php?id=75272
To summarize the problem:
This simple query
context.products
.Include(x => x.category)
.Take(10)
.ToList();
gets translated into
SELECT
`Limit1`.`C1`,
`Limit1`.`id`,
`Limit1`.`name`,
`Limit1`.`category_id`,
`Limit1`.`id1`,
`Limit1`.`name1`
FROM (SELECT
`Extent1`.`id`,
`Extent1`.`name`,
`Extent1`.`category_id`,
`Extent2`.`id` AS `id1`,
`Extent2`.`name` AS `name1`,
1 AS `C1`
FROM `products` AS `Extent1` INNER JOIN `categories` AS `Extent2` ON `Extent1`.`category_id` = `Extent2`.`id` LIMIT 10) AS `Limit1`
and performs pretty well. Anyway, the outer query is pretty much useless. Now If I add an OrderBy
context.products
.Include(x => x.category)
.OrderBy(x => x.id)
.Take(10)
.ToList();
the query changes to
SELECT
`Project1`.`C1`,
`Project1`.`id`,
`Project1`.`name`,
`Project1`.`category_id`,
`Project1`.`id1`,
`Project1`.`name1`
FROM (SELECT
`Extent1`.`id`,
`Extent1`.`name`,
`Extent1`.`category_id`,
`Extent2`.`id` AS `id1`,
`Extent2`.`name` AS `name1`,
1 AS `C1`
FROM `products` AS `Extent1` INNER JOIN `categories` AS `Extent2` ON `Extent1`.`category_id` = `Extent2`.`id`) AS `Project1`
ORDER BY
`Project1`.`id` ASC LIMIT 10
Which is bad because the order by is in the outer query. Theat means MySQL has to pull every record in order to perform an orderby which results in using filesort
I verified that SQL Server (Comapact at least) does not generate nested queries for the same code
SELECT TOP (10)
[Extent1].[id] AS [id],
[Extent1].[name] AS [name],
[Extent1].[category_id] AS [category_id],
[Extent2].[id] AS [id1],
[Extent2].[name] AS [name1],
FROM [products] AS [Extent1]
LEFT OUTER JOIN [categories] AS [Extent2] ON [Extent1].[category_id] = [Extent2].[id]
ORDER BY [Extent1].[id] ASC
Actually the queries generated by Entity Framework are few ugly, less than LINQ 2 SQL but still ugly.
However, very probably you database engine will make the desired execution plan, and the query will run smoothly.
I am receiving a large list of current account numbers daily, and storing them in a database. My task is to find added and released accounts from each file. Right now, I have 4 SQL tables, (AccountsCurrent, AccountsNew, AccountsAdded, AccountsRemoved). When I receive a file, I am adding it entirely to AccountsNew. Then running the below queries to find which we added and removed.
INSERT AccountsAdded(AccountNum, Name) SELECT AccountNum, Name FROM AccountsNew WHERE AccountNumber not in (SELECT AccountNum FROM AccountsCurrent)
INSERT AccountsRemoved(AccountNum, Name) SELECT AccountNum, Name FROM AccountsCurrent WHERE AccountNumber not in (SELECT AccountNum FROM AccountsNew)
TRUNCATE TABLE AccountsCurrent
INSERT AccountsCurrent(AccountNum, Name) SELECT AccountNum, Name FROM AccountsNew
TRUNCATE TABLE AccountsNew
Right now, I am differencing about 250,000 accounts, but this is going to keep growing. Is this the best method, do you have any other ideas?
EDIT:
This is an MSSQL 2000 database. I'm using c# to process the file.
The only data I am focused on is the accounts that were added and removed between the last and current files. The AccountsCurrent, is only used to determine what accounts were added or removed.
To be honest, I think that I'd follow something like your approach. One thing is that you could remove the truncate, do a rename of the "new" to "current" and re-create "new".
Sounds like a history/audit process that might be better done using triggers. Have a separate history table that captures changes (e.g., timestamp, operation, who performed the change, etc.)
New and deleted accounts are easy to understand. "Current" accounts implies that there's an intermediate state between being new and deleted. I don't see any difference between "new" and "added".
I wouldn't have four tables. I'd have a STATUS table that would have the different possible states, and ACCOUNTS or the HISTORY table would have a foreign key to it.
Using IN clauses on long lists can be slow.
If the tables are indexed, using a LEFT JOIN can prove to be faster...
INSERT INTO [table] (
[fields]
)
SELECT
[fields]
FROM
[table1]
LEFT JOIN
[table2]
ON [join condition]
WHERE
[table2].[id] IS NULL
This assumes 1:1 relationships and not 1:many. If you have 1:many you can do any of...
1. SELECT DISTINCT
2. Use a GROUP BY clause
3. Use a different query, see below...
INSERT INTO [table] (
[fields]
)
SELECT
[fields]
FROM
[table1]
WHERE
EXISTS (SELECT * FROM [table2] WHERE [condition to match tables 1 and 2])
-- # This is quick provided that all fields to match the two tables are
-- # indexed in both tables. Should then be much faster than the IN clause.
You could also subtract the intersection to get the differences in one table.
If the initial file is ordered in a sensible and consistent way (big IF!), it would run considerably faster as a C# program which logically compared the files.