Develop Reference

SQL/C# - Apply a function to columns within a SQL query

Is there a way to parse a given SQL SELECT query and wrap each column with a function call e.g. dbo.Foo(column_name) prior to running the SQL query?
We have looked into using a regular expression type 'replace' on the column names, however, we cannot seem to account for all the ways in which a SQL query can be written.
An example of the SQL query would be;
SELECT
[ColumnA]
, [ColumnB]
, [ColumnC] AS [Column C]
, CAST([ColumnD] AS VARCHAR(11)) AS [Bar]
, DATEPART([yyyy], GETDATE()) - DATEPART([yyyy], [ColumnD]) AS [Diff]
, [ColumnE]
FROM [MyTable]
WHERE LEN([ColumnE]) > 0
ORDER BY
[ColumnA]
, DATEPART([yyyy], [ColumnD]) - DATEPART([yyyy], GETDATE());
The result we require would be;
SELECT
[dbo].[Foo]([ColumnA])
, [dbo].[Foo]([ColumnB])
, [dbo].[Foo]([ColumnC]) AS [Column C]
, CAST([dbo].[Foo]([ColumnD]) AS VARCHAR(11)) AS [Bar]
, DATEPART([yyyy], GETDATE()) - DATEPART([yyyy], [dbo].[Foo]([ColumnD])) AS [Diff]
, [dbo].[Foo]([ColumnE])
FROM [MyTable]
WHERE LEN([dbo].[Foo]([ColumnE])) > 0
ORDER BY
[dbo].[Foo]([ColumnA])
, DATEPART([yyyy], [dbo].[Foo]([ColumnD])) - DATEPART([yyyy], GETDATE());
Any or all of the above columns might need the function called on them (including columns used in the WHERE and ORDER BY) which is why we require a query wide solution.
We have many pre-written queries like the above which need to be updated, which is why a manual update will be difficult.
The above example shows that some result columns might be calculated and some have simply been renamed. Most are also made up with joins and some contain case statements which I have left out for the purpose of this example.
Another scenario which would need to be accounted for is table name aliasing e.g. SELECT t1.ColumnA, t2.ColumnF etc.
Either a SQL or C# solution for solving this problem would be ideal.

Instead of replacing each occurrence of every column, you can replace the statement...
FROM MyTable
...with a subselect that includes all existing columns with the function call:
FROM (
SELECT dbo.Foo(ColumnA) AS ColumnA, dbo.Foo(ColumnB) AS ColumnB,
dbo.Foo(ColumnC) AS ColumnC --etc.
FROM MyTable
) AS MyTable
The rest of the query can remain unchanged. In case of table aliasing, you simply replace AS Table1 with AS t1.
Another option you should consider is to create views in your database that would be essentially the subselect. Combined with a naming convention, you can easily replace the occurrences in your FROM (and JOIN) statements with the view name:
FROM MyTable_Foo AS t1
If you want to replace all queries that you'll ever use, consider renaming the tables and creating views that are named like the old tables.
On a more general note: You should reconsider your approach to the underlying problem, since what you are doing here takes away much of the power of SQL. The worst thing here is that once you call the function on all columns, you will not be able to use the indices on those columns, which could mean a serious hit on DB performance.

Using SQLite ADO.Net Provider Need to Put GUID Directly Into Query's Where Clause Without Use of a Parameter

Is there anyway to just have the GUID be part of the SQL Query itself without using a parameter?
Let me explain exactly what I am trying to do and why. I am working with an existing application that uses an ADO.Net to create and connect to a SQLite database. The current structure of the database and the method for querying it are horrendous. I am in the midst of a redesign of how it works fundamentally. However, that is not something that can be accomplished quickly. While that redesign is being completed, I have a situation that needs a band-aid solution. As a heads up, the reasoning behind the initial implementation of the code is obscure and appears to have been built by someone who had little database knowledge. Refactoring the entire database to not need this situation is the ultimate solution, but for now I'm just looking to work within the existing structure.
The design of the database relies on GUID's to uniquely identify rows. To perform filtered retrieval of the data the system dynamically builds a command that has an IN clause listing the GUIDs that should be retrieved. Right now the GUIDs are inserted into the query by using a parameter with type GUID, so the query will look like
SELECT * FROM data_table WHERE guid_col IN( ?, ?, ?, ?)
The problem comes in when I need to retrieve a relatively large amount of information. SQLite has a limitation of a 1000 parameters in a single query. If I need to pass in more than a 1000 GUIDs for the retrieval the query will just break. When building the above string it loops over a list of GUID's to insert the question marks and create the paramters. My band-aid fix to the problem was going to be to have the GUID value directly inserted into where the question marks are currently and forego having parameters in the query. After all, its kind of using parameters for a purpose they don't need to be used for.
The problem with this 'solution' is that I can't seem to get the GUID to match the data in the column i.e. the query is always returning null. I understand that GUIDs are not a native type to SQLite and underneath it is actually being represented as a BLOB (yes I am sure we are using the BLOB and not the string representation). Yet, I've been unable to get the query to execute correctly.
I've tried all the following so far:
I've tried calling ToString() on the GUID so the query looks like
SELECT * FROM data_table WHERE guid_col IN
( 'b5080d4e-37c3-4286-9c3a-413e8c367f36', 'aa0ff789-3ce9-4552-9840-5ed4d73c1e2c')
I've tried calling ToString("N") on the GUID so the query looks like
SELECT * FROM data_table WHERE guid_col IN
( 'b5080d4e37c342869c3a413e8c367f36', 'aa0ff7893ce9455298405ed4d73c1e2c')
I've tried calling ToString("B") on the GUID so the query looks like
SELECT * FROM data_table WHERE guid_col IN
( '{b5080d4e-37c3-4286-9c3a-413e8c367f36}',
'{aa0ff789-3ce9-4552-9840-5ed4d73c1e2c}')
I've tried calling ToByteArray() on the GUID and putting the result into the query by adding each byte to the string calling ToString("X") on each byte so the query looks like
SELECT * FROM data_table WHERE guid_col IN
( '4ED8B5C33786429C3A413E8C367F36', '89F7FAAE93C524598405ED4D73C1E2C')
In reading the SQLite documentation I read the following "BLOB literals are string literals containing hexadecimal data and preceded by a single "x" or "X" character. If I try to apply this to my query so it looks like
SELECT * FROM data_table WHERE guid_col IN
( x'4ED8B5C33786429C3A413E8C367F36', x'89F7FAAE93C524598405ED4D73C1E2C')
I get an error saying that "x" is not a recognized symbol.
Is it possible to get the GUID into the query string without the use of the parameter?

I suggest that you use a temporary table for something like this. For example...
PRAGMA temp_store = MEMORY;
CREATE TEMP TABLE tbl_guids (guid_col text);
INSERT INTO tbl_guids VALUES ('b5080d4e-37c3-4286-9c3a-413e8c367f36');
INSERT INTO tbl_guids VALUES ('aa0ff789-3ce9-4552-9840-5ed4d73c1e2c');
... more inserts ...
SELECT * FROM data_table WHERE guid_col IN ( SELECT guid_col FROM tbl_guids);
DROP TABLE tbl_guids;
Make sure to wrap a transaction around all of the INSERT INTO statements. This will help with performance by a ton. Also SQLite allows for temp tables to exist entirely in memory by setting PRAGMA temp_store = MEMORY
If you have a multiple users accessing the table at the same time sharing the same connection, you will need to create some uniqueness with the temp table such as adding a random number to the table name such as tbl_guids_9087.

You can pass GUID's in as strings if you set "BinaryGuid=False" in the connection string.
If this is done, "ToString()" should work fine.

How to perform a count on a arbitrary query (possibly containing a order by)

I have been tasked with updating our internal framework we use in-house. One of the things the framework does is you pass it a query and it will return the number of rows the query has in it (The framework makes heavy use of DataReaders so we need the total before hand for UI things).
The query that the count needs to be done on can be different from project to project (SOL-injection is not a issue, the query is not from user input, just hard coded in from another programmer when they use the framework for their project.) and I was told that just having the programmers write a second query for the count is unacceptable.
Currently the solution is to do the following (I did not write this, I was just told to fix it).
//executes query and returns record count
public static int RecordCount(string SqlQuery, string ConnectionString, bool SuppressError = false)
{
//SplitLeft is just myString.Substring(0, myString.IndexOf(pattern)) with some error checking. and InStr is just a wrapper for IndexOf.
//remove order by clause (breaks count(*))
if (Str.InStr(0, SqlQuery.ToLower(), " order by ") > -1)
SqlQuery = Str.SplitLeft(SqlQuery.ToLower(), " order by ");
try
{
//execute query
using (SqlConnection cnSqlConnect = OpenConnection(ConnectionString, SuppressError))
using (SqlCommand SqlCmd = new SqlCommand("select count(*) from (" + SqlQuery + ") as a", cnSqlConnect))
{
SqlCmd.CommandTimeout = 120;
return (Int32)SqlCmd.ExecuteScalar();
}
}
catch (Exception ex)
{
if (SuppressError == false)
MessageBox.Show(ex.Message, "Sql.RecordCount()");
return -1;
}
}
However it breaks on queries like (again, not my query, I just need to make it work)
select [ClientID], [Date], [Balance]
from [Ledger]
where Seq = (select top 1 Seq
from [Ledger] as l
where l.[ClientID] = [Ledger].[ClientID]
order by [Date] desc, Seq desc)
and Balance <> 0)
as it will removes everything after the order by and breaks the query. I thought I may go from simple string matching to a more complicated parser, but before I do that I wanted to ask if there is a better way.
UPDATE: The order by clause is dropped because if you include it using my method or a CTE you will get the error The ORDER BY clause is invalid in views, inline functions, derived tables, subqueries, and common table expressions, unless TOP or FOR XML is also specified.
Some more details: This framework is used for writing conversion applications. We write apps to pull data from a clients old database and move it in to our database format when a customer buys our CRM software. Often we are working with source tables that are poorly written and can be several Gigs in size. We do not have the resources to hold the whole table in memory so we use a DataReader to pull the data out so everything is not in memory at once. However a requirement is a progress bar with the total number of records to be processed. This RecordCount function is used to figure the max of the progress bar. It works fairly well, the only snag is if the programmer writing the conversion needs to order the data output, having a order by clause in the outer most query breaks count(*)
Partial Solution: I came up with this while trying to figure it out, it will not work 100% of the time but I think it will be better than the current solution
If I find a order by clause, I then check to see if the first thing in the query is a select (and no Top following) I replace that beginning text with select top 100 percent. It works better but I am not posting this as a solution as I am hoping for a universal solution.

Assuming you aren't going to see anything but fairly ordinary select statements, I don't think you don't need a full-on SQL parser to do what you want. You can reasonably make the assumption that you've got syntactically valid SQL. You need to build a tokenizer (lexical analyzer), though.
The lexical analysis needed for Transact SQL is pretty simple. The token list consist of (off the top of my head, since it's been a while since I had to do this):
whitespace
two types of comments:
---style comments
/.../`-style comments
three types of quoted literals:
string literals (e.g., `'my string literal'), and
two flavors of quoting reserved words for use as column or object names:
ANSI/ISO style, using double quotes (e.g., "table")
Transact-SQL style, using square-brackets (e.g., [table])
hex literals (e.g., 0x01A2F)
numeric literals (e.g. 757, -3218, 5.4 or -7.6E-32, 5.0m , $5.3201 etc.)
words, reserved or not: a unicode letter, underscore (''), 'at'-sign ('#') or hash ('#'), followed by zero or more of unicode letters, decimal digits, underscore ('') or the at-, dollar- or hash- signs ('#', '$' or '#').
operators, including parentheses.
It can pretty much all be done with regular expressions. If you were using Perl, you'd be done in a day, easy. It'll probably take a bit longer in C#, though.
I would probably treat comments as whitespace and collapse multiple sequences of whitespace and comment into a single whitespace token as it facilitates the recognition of constructs such as order by.
The reason you don't need a parser is that you don't really care very much about the parse tree. What you do care about is nested parentheses. So...
Once you've gotten a lexical analyzer that emits a stream of tokens, all you need to do is eat and discard tokens counting open/closing parentheses until you see a 'from' keyword at parenthetical depth 0.
Write select count(*) into your StringBuilder.
Start appending tokens (including the from) into the StringBuilder until you see an 'order by' at parenthetical depth 0. You'll need to build a certain amount of look-ahead into your lexer to do this (which see my earlier note regarding the collapsing of sequences of whitespace and/or comments into a single whitespace token.)
At this point, you should be pretty much done. Execute the query.
NOTES
Parameterized queries likely won't work.
Recursive queries, with a CTE and a with clause will probably get broken.
This will discard anything past the ORDER BY clause: if the query uses query hint, a FOR clause, or COMPUTE/COMPUTE BY, your results will likely differ from the original query (especially with any compute clauses, since those break up the queries result sets).
Bare UNION queries will get broken, since something like
select c1,c2 from t1
UNION select c1,c2 from t2
will get turned into
select count(*) from t1
UNION select c1,c2 from t2
All this is completely untested, just my thoughts based on oddball stuff I've had to do over the years.

Instead of modifying the existing clauses of the query - how about inserting a new clause, the INTO clause.
SELECT *
INTO #MyCountTable -- new clause to create a temp table with these records.
FROM TheTable
SELECT ##RowCount
-- or maybe this:
--SELECT COUNT(*) FROM #MyCountTable
DROP TABLE #MyCountTable
TSql query modification seems to be an eternal struggle to be the lastest thing that happens.

would you post a answer of how to do this "the right way" using IQueryable
Suppose you had some arbitrary query:
IQueryable<Ledger> query = myDataContext.Ledgers
.Where(ledger => ledger.Seq ==
myDataContext.Ledgers
.Where(ledger2 => ledger2.ClientId == ledger.ClientId)
.OrderByDescending(ledger2 => ledger2.Date)
.ThenByDescending(ledger2 => ledger2.Seq)
.Take(1).SingleOrDefault().Seq
)
.Where(ledger => ledger.Balance != 0);
Then you just get the Count of the rows, no need for any custom method or query manipulation.
int theCount = query.Count();
//demystifying the extension method:
//int theCount = System.Linq.Queryable.Count(query);
LinqToSql will include your desire for a count into the query text.

I guess you want to drop the order by clause to improve the performance. The general case is quite complex and you will need full sql parser to drop the ordering clause.
Also, did you check the comparative performance of
select count(id) from ....
v/s
select count(*) from (select id, a+b from ....)
The problem is that the a+b will need to be evaluated in latter, essentially executing query twice.
If you want a progress bar because the retrieval itself is slow then this is completely counter-productive, because you will spend almost the same amount of time estimating the count.
And if the application is complex enough that the data can change between the two query execution then you don't even know how reliable the count is.
So: the real answer is that you cannot get a count on arbitrary query in efficient way. For a non-efficient way, if your resultset is rewindable, then go the end of resultset, figure out the row count and then go back to the first row.

What if rather than try to re-build your query, you do something like:
WITH MyQuery AS (
select [ClientID], [Date], [Balance]
from [Ledger]
where Seq = (select top 1 Seq
from [Ledger] as l
where l.[ClientID] = [Ledger].[ClientID]
order by [Date] desc, Seq desc)
and Balance <> 0)
)
SELECT COUNT(*) From MyQuery;
Note I haven't tested this on SQL Server 2005 but it should work.
Update:
We've confirmed SQL Server 2005 does not support an ORDER BY clause within a CTE. This does, however, work with Oracle and perhaps other databases.

I wouldn't edit or try to parse the SQL at all, but you may have to use an EVIL CURSOR (don't worry, we won't explicitly iterate through anything). Here, I would simply pass your ad-hoc SQL to a proc which runs it as a cursor, and returns the number of rows in the cursor. There may be some optimizations available, but I've kept it simple, and this should work for any valid select statement (even CTEs) that you pass to it. No need to code and debug your own T-SQL lexer or anything.
create proc GetCountFromSelect (
#SQL nvarchar(max)
)
as
begin
set nocount on
exec ('declare CountCursor insensitive cursor for ' + #SQL + ' for read only')
open CountCursor
select ##cursor_rows as RecordCount
close CountCursor
deallocate CountCursor
end
go
exec GetCountFromSelect '// Your SQL here'
go

C# problem querying dbase file; problem using WHERE clause

I'm querying a dbase .dbf file with odbc from within my c# code and am having a problem using a 'where' clause in my query. I can retrieve and read the records fine if I just 'select * from FILE.DBF', and every example I see on web pages as I search for an answer show just that much syntax. I've tried multiple ways of constructing the select statement with a 'where' and so far they all fail. So, I'm wondering whether I just can NOT use a 'where' clause in a query against a dbase file, or whether I simply haven't hit on the correct syntax yet.
I've tried:
select * from FILE.DBF where GROUP = 21;
select * from FILE.DBF where GROUP = '21';
select * from FILE.DBF where GROUP = "21";
The result of all of these is the error: ERROR [42000] [Microsoft][ODBC dBase Driver] Syntax error in WHERE clause.
Any help will be appreciated.

Try surrounding the word GROUP with brackets ... as in ..
select * from FILE.DBF where [GROUP] = 21;
GROUP is a SQL keyword and it's most likely causing some issues.

GROUP is a keyword used for SQL itself. Try running the same query but with a different 'where' clause, by substituting 'Group' with another field instead (and a different condition too, naturally). If the query works, then 'GROUP' is being mixed up with the SQL syntax for GROUP BY, and thus you might need to use brackets or some other character to enclose the field name.

Modifying sql query,dont want to use join operation

The following query uses join operation, i want the same query to execute without using join operation. Is it possible to do so? If yes than how can i do it?
select jname, jcode from heardt inner join judge ON heardt.jud1 = jcode
The reason i am trying to do this is because i am using odbc connection for mysql and join operations are not getting executed at all as web page is loading for infinitely long and not giving any output. That is the reason why i want to try without using join operation

I don't know your rationale, I find JOINS much easier to read but you can replace them by joining (no pun intented) the tables in the where clause.
select jname
, jcode
from heardt
, judge
where heardt.jud1 = judge.jcode

There is no additional filter on that query. It might cause the query to return many rows. This could cause a slowdown, depending on the number of records in your table.
You should consider limiting the number of returned records from the query.
Something else you need to check, if there is an index on the JCode field

Select jname, jud1 from heardt where not jud1 is null
EDIT: Ok, this was quick. So: Why do you need the 'join'?

The query Select jname, jud1 from heardt where not jud1 is null shows that jud1 has a value, but not that that value is valid. The join or where validates the relationship between the tables.
If your query takes a very long time to execute with the join in place it is most likely that you do not have correct indexes on the tables causing a table scan to take place instead of and indexed search.

i am using odbc connection for mysql and join operations are not getting executed as web page is loading for infinitely long and not giving any output. That is the reason why i want to try without using join operation
That's probably not because your JOIN is not getting executed, but because your JOIN query is taking too long. And that's probably because you don't have the correct index defined (an index, preferably a clustered index, on judge.jcode).
If the join is still taking too long after adding such an index you could consider precaching the query with a table or indexed view (latter however not supported in MySQL).

If you are able to run it SQL Manager you should be able to run it on the ODBC Connection, if not there is something wrong with the way you are instantiating that connection in C#.
Can you post the c# code you are using so we can make a better judged answer for you.

As lieven pointed out I think his solution is a good one
select jname
, jcode
from heardt
, judge
where heardt.jud1 = judge.jcode
But you should create indexes in the fields you are joining, therefore the result will be provided much quickly so add
Create index a1 on heardt(jud1);
Create index a2 on judge(jcode);
I think this is the better possible option