My company uses raw, untyped DataSets filled via Stored Procedures exclusively. I have been tasked with finding a way to retrieve very large result sets (paging) and ways to get Lazy Loading functionality (at least I think this is lazy loading; I'm still learning that stuff to be honest) so we aren't pulling back tens of thousands of rows in one batch and hogging server resources.
I personally am not that familiar with DataSets as I avoid them whenever possible, and I would rather get rid of them entirely here, but saying "Change everything to use LINQ/EF" isn't going to be a valid answer since there's no business value to management (and it would take too long to redo things, so the idea would be shot down immediately).
Are there some resources I can look into to get this same kind of functionality but using standard untyped DataSets?
EDIT: Also, I need a solution that can work with dynamically created SQL that does not use a stored procedure.
All you need to do is to modify your stored procedure to page the result set. This of course will also mean that you'll have to pass as parameters certain criteria such as page number etc. Assuming you're using SQL Server 05 or newer, take a look at the following:
http://www.codeproject.com/KB/database/PagingResults.aspx
You'll need to implement paging inside your stored procedures. I assume you're using Sql Server, so here's a link:
http://www.davidhayden.com/blog/dave/archive/2005/12/30/2652.aspx
Note that this has nothing to do with DataSets per se. Presumably, your code generates a DataSet from a stored procedure call. If you rewrite your procs to do paging, your code will then generate a DataSet that contains only the requested page's records.
You could use the DataSet returned by your original proc to implement paging, by caching the DataSet and returning only selected rows to the client (or more accurately, using only selected rows of the DataSet to generate the client HTML), but this is a super-duper, really bad idea.
I had the same problem with asp.net 2.0 website, there is no "lazy-loading" solution to this. In order to paginate the data-sets I am using 2 sprocs that will help me wrap the paging functionality on every select I am doing.
CREATE PROCEDURE [dbo].[Generic_Counting]
#tables VARCHAR(MAX),
#filter VARCHAR(MAX) = '1=1'
AS
BEGIN
SET NOCOUNT ON;
DECLARE #strQuery VARCHAR(8000)
SET #strQuery = ' SELECT COUNT(*) FROM '+ #tables +'
WHERE '+ #filter
execute (#strQuery)
IF ##ERROR<>0
BEGIN
--error on generic count
SET NOCOUNT OFF
RETURN 10067
END
SET NOCOUNT OFF
RETURN 0
END
GO
CREATE PROCEDURE [dbo].[Generic_Paging]
#tables VARCHAR(1000),
#pk VARCHAR(100),
#pageNumber INT = 1,
#pageSize INT = 10,
#fields VARCHAR(MAX) = '*',
#filter VARCHAR(MAX) = '1=1',
#orderBy VARCHAR(MAX) = NULL
AS
BEGIN
SET NOCOUNT ON;
DECLARE #strQuery VARCHAR(8000)
DECLARE #strMinRecord VARCHAR(12);
DECLARE #strMaxRecord VARCHAR(12);
SET #strMinRecord = CONVERT(VARCHAR(12),((#pageNumber -1)*#pageSize + 1))
SET #strMaxRecord = CONVERT(VARCHAR(12), (#pageNumber * #pageSize))
-- Use ROW_NUMBER function
SET #strQuery ='
WITH Generic_CTE As
(
SELECT ''RowNumber'' = ROW_NUMBER() OVER(ORDER BY ' +
ISNULL(#orderBy,#pk) +'),' +
#fields +
' FROM ' + #tables +
' WHERE ('+ #filter +')
)
SELECT ' + #fields + '
FROM Generic_CTE
WHERE RowNumber BETWEEN ' + #strMinRecord +' AND '+ #strMaxRecord
--print #strQuery
execute (#strQuery)
IF ##ERROR<>0
BEGIN
--error on generic paging
SET NOCOUNT OFF
RETURN 10066
END
SET NOCOUNT OFF
RETURN 0
END
GO
You could take a look at the Value List Handler pattern, designed to be used where "the client requires a list of items ... for presentation. The number of items in the list is unknown and can be quite large in many instances."
The examples (in the link above and here) are for Java but should translate to asp.net fairly readily.
Related
I have an application where I want to produce historical charts from our database.
These charts can be many different parameters that are stored in our table, with a coupled datetime.
The stored procedure would contain something like:
CREATE PROCEDURE [dbo].[getLog]
-- Add the parameters for the stored procedure here
#getIdEntity INT = 0 ,
#columnName VARCHAR(100),
#startDate DATETIME2,
#endDate DATETIME2
AS
SELECT logs.[DateTime],
logs.#column1 as Property
from dbo.LogTable logs
WHERE logs.[IdEntity] = #IdEntity AND logs.[DateTime] >= #startDate
AND logs.[DateTime] <= #endDate
AND logs.column1 IS NOT NULL;'
This does not work, inserting a column as a parameter is not possible.
The answer was, of course, dynamic SQL, which obviously leads to a SQL injection problem.
the procedure then becomes:
ALTER PROCEDURE [dbo].[getLog]
-- Add the parameters for the stored procedure here
#getIdEntity INT = 0 ,
#columnName VARCHAR(100),
#startDate DATETIME2,
#endDate DATETIME2
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
-- Insert statements for procedure here
DECLARE
#dynamicSql NVARCHAR(500),
#timespan INT
SET #dynamicSql = '
SELECT [DateTime],
[' +#columnName+'] as Property from dbo.EntityTransactionLogEntry etl
WHERE etl.[IdEntity] = '+ CAST(#getIdEntity AS VARCHAR(10))
+ ' AND etl.[DateTime] >= '''+ (CONVERT (VARCHAR(50),#startDate,121)) +''''
+ ' AND etl.[DateTime] <= '''+ (CONVERT (VARCHAR(50),#endDate,121)) +''''
+ ' AND etl.[' +#columnName+'] IS NOT NULL;
EXEC (#dynamicSql)'
Which is obviously still vulnerable to SQL inject.
Is simply adding:
IF NOT EXISTS (SELECT * FROM
INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'EntityTransactionLogEntry'
AND COLUMN_NAME = #columnName)
BEGIN
PRINT 'Bad Input'
RETURN
END
(I also limit maximum time span that is query-able, but I'm not worried about that here)
before the Dynamic SQL portion sufficient to protect this from injection?
It is designed to fail in an invalid column name is supplied.
If yes, then presumably creating a view with only column names I approve of querying would be allowed to limit the columns that can be used?
I am specifically worried about injection, but I think I am safe, this will only be available from our API.
Is simply adding [a check in INFORMATION_SCHEMA] before the Dynamic SQL portion sufficient to protect this from injection?
By definition, no. What's happening is still SQL injection.
What you're asking, technically, is whether it's safe from malicious SQL injection.
Your checking INFORMATION_SCHEMA is a reasonable protection. It's a form of whitelisting, which is the technique we must use for things that can't be replaced by query parameters.
If there were a column in EntityTransactionLogEntry whose name could "break" the delimited identifier syntax, you'd be in trouble, but I assume you have control over the column names in your own table, and you wouldn't do that.
You should restrict your INFORMATION_SCHEMA query to your own TABLE_SCHEMA. As your query is currently written, it would be satisfied if any table catalogued in I_S has that column name.
Also, I'm not sure why you are not using query parameters in your terms for IdEntity and DateTime, where the values can be replaced by query parameters. You should do that.
Welcome Flippie
Tipically you can do some like
parameters..
.. paramterx varchar(100) = null,
paramtery varchar(100) = null
select *
from entitytransactionlog
where
logs.[IdEntity] = #IdEntity AND logs.[DateTime] >= #startDate
AND logs.[DateTime] <= #endDate
and (columnvalue = parameterx or parameterx is null)
and (columnvalue2 = parametery or parametery is null)
so sp is invoked like
exec sp parameterx = 'value'
the validation "OR" in bracket let every values if value in parameter is not defined
I have written a stored procedure to insert values into a table where the primary key will be auto incremented. when I try to import this in Visual Studio 2013, In function Imports when I select "Get Column Information" it says "The selected procedure or function" returns no columns.
I read about it many articles and also included SET FMTONLY OFF in my code but it still does not work. Amateur in ASP.Net and C#. So can anyone explain to me What to do in a clear manner
USE [DB_Name]
GO
/****** Object: StoredProcedure [dbo].[usp_makePost] Script Date: 04-04-2015 19:16:04 ******/
SET FMTONLY OFF
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE procedure [dbo].[usp_makePost]
#FK_strUser_Id varchar(11),
#strPost_Title varchar(100),
#strPost_Content varchar(1000),
#dTime_of_post datetime,
#iUp_Vote int,
#iDown_Vote int,
#FK_strRootword_Id varchar(11)
as
begin
DECLARE #PK_strPost_Id VARCHAR(11);
DECLARE #PreFix VARCHAR(10) = 'POST';
DECLARE #Id INT;
SELECT #Id = ISNULL(COUNT(PK_strPost_Id),0) + 1 FROM Tbl_Post
SELECT #PK_strPost_Id = #PreFix + RIGHT('0000' + CAST(#Id AS VARCHAR(7)), 7)
insert into Tbl_Name values(#PK_strPost_Id,#FK_strUser_Id,#strPost_Title,#strPost_Content,#dTime_of_post,#iUp_Vote,#iDown_Vote,#FK_strRootword_Id)
end
Your stored procedure doesn't do any data retrieve operation (ie, any SELECT). It just does an INSERT plus some variable manipulation. Those SELECTs out there only assign variables too, so nothing really produces any kind of result set.
Therefore client programs are completely right in that there are no columns or any kind of output from this procedure. Maybe you intended to add some sort of return table?
Think of the stored procedure as a data source for your front end application. Now for it to have data, it has to end with a SELECT clause, because only then can it have data. Clearly your application is expecting data. So without going into much details,
either you need to tell your application to stop expecting data.
or, modify the procedure so that it starts giving data.
Probably you would need to add a SELECT * FROM Tbl_Name in the end of stored proc or something similar.
All, I have a dynamic SQL Query that I am executing from a C# application. The problem query is an INSERT statement, which is run from within a C# loop, being executed sequentially on many databases to create a single data warehouse [database]. I have run this code one 100+ databases in a single batch without problem; however, I have just come across one specific database where the query
DECLARE #DbName NVARCHAR(128);
SET #DbName = (SELECT TOP 1 [DbName]
FROM [IPACostAdmin]..[TmpSpecialOptions]);
DECLARE #FilterSql NVARCHAR(MAX);
SET #FilterSql = (SELECT TOP 1 [AdditionalSQL]
FROM [IPACostAdmin]..[TmpSpecialOptions]);
DECLARE #SQL NVARCHAR(MAX);
DECLARE #SQL1 NVARCHAR(MAX);
DECLARE #SQL2 NVARCHAR(MAX);
SET #SQL1 =
'INSERT INTO [' + #DbName + ']..[Episode] WITH(TABLOCK)
([EstabID],..., [InclFlag]) ';
SET #SQL2 =
'SELECT
[EstabID],..., [InclFlag]
FROM [B1A] ' + #FilterSql + ';';
SET #SQL = #SQL1 + #SQL2;
EXEC sp_executesql #SQL;
Goes from taking roughly three seconds for an insert of 20,000-30,000 records to 40+ minutes! Now, after long deliberation and experiments, I have just worked out the fix for this; it is to use
EXEC sp_executesql #SQL WITH RECOMPILE;
This brings it back down to < 2s for the insert.
This SQL is executed from the application once for each database in the batch, the current execution of this statement should be totally separate from the preceding ones as far as the server is concerned (as I understand it), but it is not; it seems SQL is cashing the dynamic SQL in this case.
I would like to know what is happening here for this single site? Where will I need to ensure I use the RECOMPILE option in future to prevent such issues?
Thanks for your time.
_Note. I appreciate that this recompiles the query, but I am baffelled as to why the server is using the same execution plan in the first place. each time this query is run it is against a different database using a different Initial Catalog using a different SqlConnection.
when you do RECOMPILE, sql server will generate each time new execution plan and execute it. other wise it will try to use an existing execution plan stored in the procedure cache, which may be wrong for the current query as in dynamic query, conditions and parameters get changed each time it executes..
Here is my current implementation of a stored procedure which returns Order status for a given Order ID. There are two situations,
there is matched Order ID and I will retrieve the related status,
there is no matched Order ID (i.e. non-existing Order ID).
My confusion is, how to implement the two functions elegantly/efficiently in one stored procedure, so that I return matched Order ID for situation 1 and also indicate client no matched Order ID in situation 2?
I am using VSTS 2008 + C# + ADO.Net + .Net 3.5 as client, and using SQL Server 2008 as server.
CREATE PROCEDURE [dbo].[GetStatus]
#ID [nvarchar](256),
#Status [int] output
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
-- Insert statements for procedure here
SELECT #Status = [Status]
FROM [dbo].[OrderStatus]
WHERE (#ID = [ID]);
END
thanks in advance,
George
why are you using output parameter.
you just need to take your stored procedure result in dataset of the data access layer.
just check that if (dataset != null) then take value else return appropriate message to your business layer.
There are multiple approaches you can take:
Keep everything as is and in your .NET code, if the #status value returned is DBNull, then it will indicate situation 2, otherwise situation 1.
Add a RETURN statement to the SPROC and use
Dim returnValue As New SqlParameter("#RETURN_VALUE", SqlDbType.Int)
returnValue.Direction = ParameterDirection.ReturnValue
Cmd.Parameters.Add(returnValue)
in your .NET code to explicitly identify what the SPROC returned and take action accordingly.
As an additional tip, use a SET instead of SELECT when assigning the value to #Status variable in the SPROC. This will guarantee that you get a NULL back if there is no match found. So,
`
-- Insert statements for procedure here
SET #Status = SELECT [Status]
FROM [dbo].[OrderStatus]
WHERE (#ID = [ID]);`
You can use the "if statements" inside the stored procedures. the web site at bottom gives you some tips.
http://translate.google.com.br/translate?u=http%3A%2F%2Fmail.firebase.com.br%2Fpipermail%2Flista_firebase.com.br%2F2005-November%2F021883.html&sl=pt&tl=en&hl=pt-BR&ie=UTF-8
I'm using Linq-to-SQL with a SQL Server backend (of course) as an ORM for a project. I need to get the result set from a stored procedure that returns from a dynamically-created table. Here's what the proc looks like:
CREATE procedure [RetailAdmin].[TitleSearch] (
#isbn varchar(50), #author varchar(50),
#title varchar(50))
as
declare #L_isbn varchar(50)
declare #l_author varchar(50)
declare #l_title varchar(50)
declare #sql nvarchar(4000)
set #L_isbn = rtrim(ltrim(#isbn))
set #l_author = rtrim(ltrim(#author))
set #l_title = rtrim(ltrim(#title))
CREATE TABLE #mytemp(
[storeid] int not NULL,
[Author] [varchar](100) NULL,
[Title] [varchar](400) NULL,
[ISBN] [varchar](50) NULL,
[Imprint] [varchar](255) NULL,
[Edition] [varchar](255) NULL,
[Copyright] [varchar](100) NULL,
[stockonhand] [int] NULL
)
set #sql = 'select a.storeid, Author,Title, thirteendigitisbn ISBN,
Imprint,Edition,Copyright ,b.stockonhand from ods.items a join ods.inventory b on
a.itemkey = b.itemkey where b.stockonhand <> 0 '
if len(#l_author) > 0
set #sql = #sql + ' and author like ''%'+#L_author+'%'''
if len(#l_title) > 0
set #sql = #sql + ' and title like ''%'+#l_title+'%'''
if len(#L_isbn) > 0
set #sql = #sql + ' and thirteendigitisbn like ''%'+#L_isbn+'%'''
print #sql
if len(#l_author) <> 0 or len(#l_title) <> 0 or len(#L_isbn) <> 0
begin
insert into #mytemp
EXECUTE sp_executesql #sql
end
select * from #mytemp
drop table #mytemp
I didn't write this procedure, but may be able to influence a change if there's a really serious problem.
My present problem is that when I add this procedure to my model, the designer generates this function:
[Function(Name="RetailAdmin.TitleSearch")]
public int TitleSearch([Parameter(DbType="VarChar(50)")] string isbn,
[Parameter(DbType="VarChar(50)")] string author,
[Parameter(DbType="VarChar(50)")] string title)
{
IExecuteResult result = this.ExecuteMethodCall(this,
((MethodInfo)(MethodInfo.GetCurrentMethod())), isbn, author, title);
return ((int)(result.ReturnValue));
}
which doesn't look anything like the result set I get when I run the proc manually:
Can anybody tell me what's going wrong here?
This is basically the same problem as this question but due to the poor phrasing from the OP it was never really answered.
Thanks Marc for your reply. I will see about making the changes you suggested.
The problem was the temp table. Linq to Sql just doesn't know what to do with them. This was particularly difficult to diagnose, because Visual Studio caches information about stored procs, so when it initially failed to find a result set it set the return as a default integer type and didn't update when I made changes to the stored proc. Getting VS to recognize a change requires you to:
Delete proc from the dbml
delete the server connection from Server Explorer
save the dbml to force a recompile
close the project and restart VS
recreate the server connection and import the proc
You might not have to do every one of those steps, but that's what worked for me. What you need to do, if you must use a temp table, is to create a barebones proc that simply returns the correct schema, and then alter it to do what you want after you've imported it into the OR Designer.
First - IMPORTANT - your SQL is vulnerable to injection; the inner command should be parameterized:
if len(#l_author) > 0
set #sql = #sql + ' and author like ''%''+#author+''%'''
EXECUTE sp_executesql #sql, N'#author varchar(100)', #L_author
This passes the value of #L_author in as the #author parameter in the dynamic command - preventing injection attacks.
Second - you don't really need the temp table. It isn't doing anything for you... you just INSERT and SELECT. Perhaps just EXEC and let the results flow to the caller naturally?
In other circumstances a table-variable would be more appropriate, but this doesn't work with INSERT/EXEC.
Are the columns the same for every call? If so, either write the dbml manually, or use a temp SP (just with "WHERE 1=0" or something) so that the SET FMT_ONLY ON can work.
If not (different columns per usage), then there isn't an easy answer. Perhaps use regular ADO.NET in this case (ExecuteReader/IDataReader - and perhaps even DataTable.Fill).
Of course, you could let LINQ take the strain... (C#):
...
if(!string.IsNullOrEmpty(author)) {
query = query.Where(row => row.Author.Contains(author));
}
...
etc
There's no real easy way to do this. I've had the same problem in the past. I think the issue is that Linq to Sql has no way of "figuring out" which type will be returned since you're building up the SELECT statement at execution time. What I did to get around this, was in the stored proc, I did just a select and selected all the columns that I possibly needed. Then, I had Linq to Sql generate the function based on that. Then, I went back to SQL and changed the stored proc back to the way it's supposed to be. The trick here is not to regenerate your DBML.