sp_executesql Causes Undesired Implicit Conversion - c#

I've been suffering from this for a while now. In SQL Server Profiler, I confirmed that using SqlDataAdapter.Fill produces a sp_executesql command. Sometimes, the query takes too long to execute, the cause used to be one of the following two cases:
All of my text database fields are varchar. We know that sp_executesql only accepts unicode types (nvarchar, nchar). Specifying SqlDbType.Varchar for the parameter type in C# sometimes causes a big performance problem (for certain queries) because of the fact that an implicit conversion takes place in order to convert the varchar parameter to the nvarchar needed for sp_executesql. Aparently, that conversion was taking place on every row scan.
So I specified SqlDbType.NVarChar instead and the problem is solved temporarily. One day, I ended up having the same performance problem (for a different query) for the same cause but in another way. An implicit conversion was taking place to convert all the varchar values in the specified database column to nvarchar. Reverting the type to SqlDbType.VarChar fixed the problem for that case.
I know the problem is not due to Parameter Sniffing because I tried using sp_updatestats to refresh the cached plans and using OPTION(RECOMPILE) to make sure a new plan gets generated. But that did not solve the problem.
I cannot wrap my head around this and I wish to avoid using sp_executesql altogether if possible. Any idea how to avoid the undesired implicit conversions?
EDIT: It's worth noting that the queries that are generated are dynamic (coming from a query builder feature). Also, it's worth noting that the problem only showed when a "contain" operand (in sql: like %%) was used.
EDIT: This good article emphasizes the same problem. However, he author sadly declares that there is no solution.. yet, hopefully.

Related

Cannot insert sdo_geometry with more than 500 vertices

I have the following table
CREATE TABLE MYTABLE (MYID VARCHAR2(5), MYGEOM MDSYS.SDO_GEOMETRY );
AND the sql statement below:
INSERT INTO MYTABLE (MYID,MYGEOM) VALUES
( 255, SDO_GEOMETRY(2003, 2554, NULL, SDO_ELEM_INFO_ARRAY(1,1003,1),
SDO_ORDINATE_ARRAY(-34.921816571,-8.00119170599993,
...,-34.921816571,-8.00119170599993)));
Even after read several articles about possible solutions, I couldn't find out how to insert this sdo_geometry object.
The Oracle complains with this message:
ORA-00939 - "too many arguments for funcion"
I know that it's not possible to insert more then 999 values at once.
I tried stored procedure solutions, but I'm not Oracle expert, and maybe I missed something.
Could someone give me an example of code in c# or plsql ( or the both ) with or without stored procedure, to insert that row?
I'm using Oracle 11g, OracleDotNetProvider v 12.1.400 on VS2015 AND my source of spatial data comes from an external json ( so, no database-to-database ) and I can only use solutions using this provider, without datafiles or direct database handling.
I'm using SQLDeveloper to test the queries.
Please, don't point me articles if you are not sure that works with this row/value
I finally found an effective solution. Here: Constructing large sdo_geometry objects in Sql Developer and SqlPlus. Pls-00306 Error
The limitation you see is old. It is based on the idea that no-one would ever write a function that would have more than 1000 parameters (actually 999 input parameters and 1 return value).
However with the advent of multi-valued attributes (VARRAYs) and objects, this is no longer true. In particular for spatial types, the SDO_ORDINATE attribute is really an object type (implemented as a VARRAY) and the reference to SDO_ORDINATE is the constructor of that object type. Its input can be an array (if used in some programming language) or a list of numbers, each one being considered a parameter to a function - hence the limit to 999 numbers).
That happens only if you hard-code the numbers in your SQL statement. But that is a bad practice generally. The better practice is to use bind variables, and object types are no exception. The proper way is to construct an array with the coordinates you want to insert and pass those to the insert statement. Or construct the entire SDO_GEOMETRY object as a bind variable.
And of course, the very idea of constructing a complex geometry entirely manually by hardcoding the coordinates is absurd. That shape will either be loaded from a file (and a loading tool will take care of that), or capture by someone drawing a shape over a map - and then your GIS/capture tool will pass the coordinates to your application for insertion into your database.
In other words, that limitation to 999 attributes / numbers is rarely seen in real life. When it does, it reflects misunderstandings on how those things work.

Stored Procedure sometimes returns short, sometimes returns int

I'm working with a legacy codebase and need to call a stored procedure that I'm not allowed to modify. This stored procedure returns a row or multiple rows of validation data.
Example of result set (two columns, code and text):
0 "success"
OR
3 "short error"
4 "detailed error"
In the procedure itself, the message is selected simply as:
Select 0 as code, 'success' as text
Problem:
I'm using Entity Framework to map the result of this stored procedure to a custom class:
public class ValidationResult
{
public int code { get; set; }
public string text { get; set; }
}
The call itself:
var result = context.Database.SqlQuery<ValidationResult>(#"old_sproc").ToList();
I've written some integration tests, and have noticed that when the procedure returns the success message, the 0 comes across as a short. When it returns a non-zero message, it comes across as an int. I assumed that setting code as an int, the short would fit in. Unfortunately, I get the following exception for my success test:
The specified cast from a materialized 'System.Int16' type to the 'System.Int32' type is not valid.
When I switch code to a short to make my success test pass, my failure test fails with the following exception:
The specified cast from a materialized 'System.Int32' type to the 'System.Int16' type is not valid.
ADO.NET is an answer
One solution is to fall back to ADO.NET's SqlDataReader object, so I have that as a fallback solution. I'm wondering if there is something I can do on the EF side to get this working, though.
(This is a follow-up to my previous answer. It is only relevant for sql-server-2012 and later.)
Short answer:
var sql = "EXECUTE old_sproc WITH RESULT SETS ((code INT, text VARCHAR(MAX)))";
var result = context.Database.SqlQuery<ValidationResult(sql).ToList();
Approach taken in this answer:
This answer will follow in your footsteps and use SqlQuery to execute your stored procedure. (Why not an altogether different approach? Because there might not be any alternative. I'll go into this further below.)
Let's start with an observation about your current code:
var result = context.Database.SqlQuery<ValidationResult>(#"old_sproc").ToList();
The query text "old_sproc" is really abbreviated T-SQL for "EXECUTE old_sproc". I am mentioning this because it's easy to think that SqlQuery somehow treats the name of a stored procedure specially; but no, this is actually a regular T-SQL statement.
In this answer, we will modify your current SQL only a tiny bit.
Implicit type conversions with the WITH RESULT SETS clause:
So let's stay with what you're already doing: EXECUTE the stored procedure via SqlQuery. Starting with SQL Server 2012, the EXECUTE statement supports an optional clause called WITH RESULT SETS that allows you to specify what result sets you expect to get back. SQL Server will attempt to perform implicit type conversions if the actual result sets do not match that specification.
In your case, you might do this:
var sql = "EXECUTE old_sproc WITH RESULT SETS ((code INT, text VARCHAR(MAX)))";
var result = context.Database.SqlQuery<ValidationResult(sql).ToList();
The added clause states that you expect to get back one result set having a code INT and a text VARCHAR(MAX) column. The important bit is code INT: If the stored procedure happens to produce SMALLINT values for code, SQL Server will perform the conversion to INT for you.
Implicit conversions could take you even further: For example, you could specify code as VARCHAR(…) or even NUMERIC(…) (and change your C# properties to string or decimal, respectively).
If you're using Entity Framework's SqlQuery method, it's unlikely to get any neater than that.
For quick reference, here are some quotes from the linked-to MSDN reference page:
"The actual result set being returned during execution can differ from the result defined using the WITH RESULT SETS clause in one of the following ways: number of result sets, number of columns, column name, nullability, and data type."
"If the data types differ, an implicit conversion to the defined data type is performed."
Do I have to write a SQL query? Isn't there another (more ORM) way?
None that I am aware of.
Entity Framework has been evolving in a "Code First" direction in the recent past (it's at version 6 at this time of writing), and that trend is likely to continue.
The book "Programming Entity Framework Code First" by Julie Lerman & Rowan Miller (published in 2012 by O'Reilly) has a short chapter "Working with Stored Procedures", which contains two code examples; both of which use SqlQuery to map a stored procedure's result set.
I guess that if these two EF experts do not show another way of mapping stored procedures, then perhaps EF currently does not offer any alternative to SqlQuery.
(P.S.: Admittedly the OP's main problem is not stored procedures per se; it's making EF perform an automatic type conversion. Even then, I am not aware of another way than the one shown here.)
If you can't alter the stored procedure itself, you could create a wrapper stored procedure which alters the data in some way, and have EF call that.
Not ideal of course, but may be an option.
(Note: If you're working with SQL Server 2012 or later, see my follow-up answer, which shows a much shorter, neater way of doing the same thing described here.)
Here's a solution that stays in EF land and does not require any database schema changes.
Since you can pass any valid SQL to the SqlQuery method, nothing stops you from passing it a multi-statement script that:
DECLAREs a temporary table;
EXECUTEs the stored procedure and INSERTs its result into the temporary table;
SELECTs the final result from that temporary table.
The last step is where you can apply any further post-processing, such as a type conversion.
const string sql = #"DECLARE #temp TABLE ([code] INT, [text] VARCHAR(MAX));
INSERT INTO #temp EXECUTE [old_sproc];
SELECT CONVERT(INT, [code]) AS [code], [text] FROM #temp;";
// ^^^^^^^^^^^^^ ^^^^^^^^^^^
// this conversion might not actually be necessary
// since #temp.code is already declared INT, i.e.
// SQL Server might already have coerced SMALLINT
// values to INT values during the INSERT.
var result = context.Database.SqlQuery<ValidationResult>(sql).ToList();
In the entity framework data modeler page (Model Browser), either change the functional mapping to a specific int which works for the ValidationResult class or create a new functional mapping result class which has the appropriate int and use that as the resulting DTO class.
I leave this process a touch vague because I do not have access to the actual database; instead I provide the process to either create a new functional mapping or modify an existing one. Trial and error will help you overcome the incorrect functional mapping.
Another trick to have EF generate the right information is temporarily drop the stored proc and have a new one return a stub select such as:
select 1 AS Code , 'Text' as text
RETURN ##ROWCOUNT
The reasoning for this is that sometimes EF can't determine what the stored procedure ultimately returns. If that is the case, temporarily creating the stub return and generating EF from it provides a clear picture for the mappings. Then returning the sproc to its original code after an update sometimes does the trick.
Ignore the int/short. the text is always the same for the same number right? get just the text. have a switch case. Yes its a hack but unless you can fix the root of the problem (and you say you are not allowed) then you should go with the hack that will take the least amount of time to create and will not cause problems down the road for the next person maintaining the code. if this stored proc is legacy it will not have any new kinds of results in the future. and this solution together with a nice comment solves this and lets you go back to creating value somewhere else.
Cast the static message code to an int:
Select cast(0 as int) as code, 'success' as text
This ensures the literal returned is consistent with the int returned by the other query. Leave the ValidationResult.code declared as an int.
Note: I know I missed the part in the question about the SP can't be modified, but given that this makes the answer quite complicated, I'm leaving this here for others who may have the same problem, but are able to solve it much more easily by modifying the SP. This does work if you have a return type inconsistency in the SP and modifying is an option.
There is a workaround you could use if you don't find a better solution. Let it be an int. It will work for all error codes. If you get an exception you know the result was a success so you can add a try/catch for that specific exception. It's not pretty and depending on how much this runs it might impact performance.
Another idea, have you tried changing the type of code to object?

Which datatype and which insertion parameter for large data

Here a field in my data records could pass the limit of 8000 chars of nvarchar, and looking for a quite larger Data-Type, e.g about 9000 chars, Any ideas ?
At first I was using NvarChar(8000), after finding some could pass this boundary I used NText
to see what will happen next, with Entity Framework seems it could do the job as it's expected without defining any Insert statement and Data Adapter, During the programming the system changed to data Adapter and I should do the job with a Insert command, Now the parameter defined is look like this :
cmdIns.Parameters.Add("#story", SqlDbType.NText, 16, "Story")
it seems that the limitation of 16 will be increased automatically while using EF is used but not with the Data Adapter(And it just insert 16 chars of the data),
really don't know (can't remember) Is the test with EF passed even the items larger than 8000 ?
If so, I'm curious about the reason.
The situation is deciding the proper Data-Type and it's equivalent working parameter to be used on insertion point of this large data field.
Note : Here SQL Server CE is Used
Edit :
Sorry, I had to go at that time,
The Data-type which should be used is NTEXT with no alternative here
but defining the **insertion Statement and parameter** is a bit hassle,
unfortunately none of the suggested methods could do the desired job similar to the piece which I gave.
without defining the length it will give errors (run-time) !
And Using AddWithValue couldn't use a the DataAdapter and do the insertion in bulk.
Maybe I have to place it in another question, but this is a piece of this question, and a working answer here could be the complete one.
Any ideas ?
If I understood your question correctly you should be fine doing something like this, omitting the size as it isn't necessary:
cmdIns.Parameters.Add( new SqlParameter( "story", SqlDbType.NText )
{
Value = yourVariable;
} );
Use AddWithValue whenever you want to add a parameter by specifying its name and value. Like this command.Parameters.AddWithValue("#story", story);

Difference with Parameters.Add and Parameters.AddWithValue

Basically Commands has Parameters and parameters has functions like Add, AddWithValue, and etc. In all tutorials i've seen, i usually noticed that they are using Add instead of AddWithValue.
.Parameters.Add("#ID", SqlDbType.Int)
vs
.Parameters.AddWithValue("#ID", 1)
Is there a reason NOT to use AddWithValue? I'd prefer to use that over
Parameters.Add("#ID", SqlDbType.Int, 4).Value = 1
since it saves my coding time. So which is better to use? Which is safe to use? Does it improves performance?
With Add() method you may restrict user input by specifying type and length of data - especially for varchar columns.
.Parameters.Add("#name",SqlDbType.VarChar,30).Value=varName;
In case of AddWithValue() (implicit conversion of value) method, it sends nvarchar value to the database.
I believe there are also some cons to using AddWithValue which affect the SQL Cache Excection Plan, see the Parameter Length section here
Using AddWithValue() adds parameter with length of current value. If the length of your parameter value varies often this means new plan is generated every time. This makes your queries run slower(additional time for parsing, compiling) and also causes higher server load.
I'd use the AddWithValue for normal cases. And use Add(name, dbtype... only when your column type is different from how .net converts the CLR type.

How do I keep my stored procedure inputs from being silently truncated?

We use the standard System.Data classes, DbConnection and DbCommand, to connect to SQL Server from C#, and we have many stored procedures that take VARCHAR or NVARCHAR parameters as input. We found that neither SQL Server nor our C# application throws any kind of error or warning when a string longer than maximum length of a parameter is passed in as the value to that parameter. Instead, the value is silently truncated to the maximum length of the parameter.
So, for example, if the stored procedure input is of type VARCHAR(10) and we pass in 'U R PRETTY STUPID', the stored procedure receives the input as 'U R PRETTY', which is very nice but totally not what we meant to say.
What I've done in the past to detect these truncations, and what others have likewise suggested, is to make the parameter input length one character larger than required, and then check if the length of the input is equal to that new max length. So in the above example my input would become VARCHAR(11) and I would check for input of length 11. Any input of length 11 or more would be caught by this check. This works, but feels wrong. Ideally, the data access layer would detect these problems automatically.
Is there a better way to detect that the provided stored procedure input is longer than allowed? Shouldn't DbCommand already be aware of the input length limits?
Also, as a matter of curiosity, what is responsible for silently truncating our inputs?
Use VARCHAR(8000), NVARCHAR(4000) or even N/VARCHAR(MAX), for all the variables and parameters. This way you do not need to worry about truncation when assigning #variables and #parameters. Truncation may occur at actual data write (insert or update) but that is not silent, is going to trigger a hard error and you'll find out about it. You also get the added benefit of the stored procedure code not having to be changed with schema changes (change column length, code is still valid). And you also get better plan cache behavior from using consistent parameter lengths, see How Data Access Code Affects Database Performance.
Be aware that there is a slight performance hit for using MAX types for #variables/#parameters, see Performance comparison of varchar(max) vs. varchar(N).

Categories

Resources