Develop Reference

SQL Server adds empty spaces to value (char) when trying to set null

I am pulling data from SQL Server to my c# project. I have some textboxes that update from page-to-page. I have one textbox in particular that is set up to only accept two characters, which is setup in the database as char(2). If I were to delete those two characters and click my button to update the database and go to the next page, it stores two empty spaces. I need it to just be empty with no spaces. In my other textboxes, this issue does not occur. The database allows the data to be null. I am able to manually enter "null" in the database, but I need it to be done when erasing the two chars and updating it.

A column declared as CHAR(2) may contain one of the following:
2 characters
NULL.
A column declared as CHAR(2) may not contain any of the following:
0 characters
1 character
3 characters
etc.
When you try to store anything other than either 2 characters or NULL in the column, your database will troll you in the name of some ill-conceived notion of convenience: instead of generating an error, it will store something other than what you gave it to store.
(Amusingly enough, receiving an error when doing something wrong is, and historically has been, regarded as an inconvenience by a surprisingly large portion of programmers. But that's okay, that's how we get stackoverflow questions to answer.)
Specifically, your database will pad the value you are storing with spaces, to match the length of the column. So, if you try to store just one character, it will add one space. If you try to store zero characters, it will add two spaces.
Possible Solutions:
If you have the freedom to change the type of the column:
Declare it as VARCHAR instead of CHAR(2), so that it will contain exactly what you store in it.
If you do not have the freedom to change the type of the column:
You have to always be manually checking whether you are about to store an empty string into it, and if so, store NULL instead.
Note about Oracle
The Oracle RDBMS before version 11g (and perhaps also in more recent versions, I am not sure, if someone knows, please leave a comment) will do that last conversion for you: if you try to store an empty string, it will store NULL instead. This is extremely treacherous due to the following reasons:
It is yet one more example of the database system trolling you by storing something very different from what you gave it to store.
They apply the same rule to all types of character columns, even VARCHAR, which means that you cannot have an empty string even in columns that could accommodate one; you store either NULL or an empty string, you always get NULL back.
This behavior is completely different from the behavior of any other RDBMS.
The behavior is fixed, there is no way to configure Oracle to quit doing that.

Use SQL to return a JSON string

This is a "best practice" question. We are having internal discussions on this topic and want to get input from a wider audience.
I need to store my data in a traditional MS SQL Server table with normal columns and rows. I sometimes need to return a DataTable to my web application, and other times I need to return a JSON string.
Currently, I return the table to the middle layer and parse it into a JSON string. This seems to work well for the most part, but does occasionally take a while on large datasets (parsing the data, not returning the table).
I am considering revising the stored procedures to selectively return a DataTable or a JSON string. I would simply add a #isJson bit parameter to the SP.
If the user wanted the string instead of the table the SP would execute a query like this:
DECLARE #result varchar(MAX)
SELECT #result = COALESCE(#results ',', '') + '{id:"' + colId + '",name:"' + colName + '"}'
FROM MyTable
SELECT #result
This produces something like the following:
{id:"1342",name:"row1"},{id:"3424",name:"row2"}
Of course, the user can also get the table by passing false to the #isJson parameter.
I want to be clear that the data storage isn't affected, nor are any of the existing views and other processes. This is a change to ONLY the results of some stored procedures.
My questions are:
Has anyone tried this in a large application? If so, what was the result?
What issues have you seen/would you expect with this approach?
Is there a better faster way to go from table to JSON in SQL Server other than modifying the stored procedure in this way or parsing the string in the middle tier?

I personally think the best place for this kind of string manipulation is in program code in a fully expressive language that has functions and can be compiled. Doing this in T-SQL is not good. Program code can have fast functions that do proper escaping.
Let's think about things a bit:
When you deploy new versions of the parts and pieces of your application, where is the best place for this functionality to be?
If you have to restore your database (and all its stored procedures) will that negatively affect anything? If you are deploying a new version of your web front end, will the JSON conversion being tied into the database cause problems?
How will you escape characters properly? Are you sending any dates through? What format will date strings be in and how will they get converted to actual Date objects on the other end (if that is needed)?
How will you unit test it (and with automated tests!) to prove it is working correctly? How will you regression test it?
SQL Server UDFs can be very slow. Are you content to use a slow function, or for speed hack into your SQL code things like Replace(Replace(Replace(Replace(Value, '\', '\\'), '"', '\"'), '''', '\'''), Char(13), '\n')? What about Unicode, \u and \x escaping? How about splitting '</script>' into '<' + '/script>'? (Maybe that doesn't apply, but maybe it does, depending on how you use your JSON.) Is your T-SQL procedure going to do all this, and be reusable for different recordsets, or will you rewrite it each time into each SP that you need to return JSON?
You may only have one SP that needs to return JSON. For now. Some day, you might have more. Then if you find a bug, you have to fix it in two places. Or five. Or more.
It may seem like you are making things more complicated by having the middle layer do the translation, but I promise you it is going to be better in the long run. What if your product scales out and starts going massively parallel—you can always throw more web servers at it cheaply, but you can't so easily fix database server resource saturation! So don't make the DB do more work than it should. It is a data access layer, not a presentation layer. Make it do the minimum amount of work possible. Write code for everything else. You will be glad you did.
Speed Tips for String Handling in a Web Application
Make sure your web string concatenation code doesn't suffer from Schlemiel the Painter's Algorithm. Either directly write to the output buffer as JSON is generated (Response.Write), or use a proper StringBuilder object, or write the parts of the JSON to an array and Join() it later. Don't do plain vanilla concatenation to a longer and longer string over and over.
Dereference objects as little as possible. I don't know your server-side language, but if it happens to be ASP Classic, don't use field names--either get a reference to each field in a variable or at the very least use integer field indexes. Dereferencing a field based on its name inside a loop is (much) worse performance.
Use pre-built libraries. Don't roll your own when you can use a tried and true library. Performance should be equal or better to your own and (most importantly) it will be tested and correct.
If you're going to spend the time doing this, make it abstract enough to handle converting any recordset, not just the one you have now.
Use compiled code. You can always get the fastest code when it is compiled, not interpreted. If you identify that the JSON-conversion routines are truly the bottleneck (and you MUST prove this for real, do not guess) then get the code into something that is compiled.
Reduce string lengths. This is not a big one, but if at all possible use one-letter json names instead of many-letter. For a giant recordset this will add up to savings on both ends.
Ensure it is GZipped. This is not so much a server-side improvement, but I couldn't mention JSON performance without being complete.
Passing Dates in JSON
What I recommend is to use a separate JSON schema (itself in JSON, defining the structure of the virtual recordset to follow). This schema can be sent as a header to the "recordset" to follow, or it can be already loaded in the page (included in the base javascript files) so it doesn't have to be sent each time. Then, in your JSON parse callback (or post-callback on the final resultant object) look in the schema for the current column and do conversions as necessary. You might consider using ISO format since in ECMAScript 5 strict mode there is supposed to be better date support and your code can be simplified without having to change the data format (and a simple object detect can let you use this code for any browser that supports it):
Date
Dates are now capable of both parsing and outputting ISO-formatted dates.
The Date constructor now attempts to parse the date as if it was ISO-formatted, first, then moves on to the other inputs that it accepts.
Additionally, date objects now have a new .toISOString() method that outputs the date in an ISO format.
var date = new Date("2009-05-21T16:06:05.000Z");
print( date.toISOString() );
// 2009-05-21T16:06:05.000Z

I wouldn't do that way you are doing (contatenating)
You can try creating a CLR SQL function that uses JSON.net and returns a varchar.
See here how to create SQL CLR Functions:
http://msdn.microsoft.com/en-us/library/w2kae45k(v=vs.80).aspx
Something like this (untested code)
[Microsoft.SqlServer.Server.SqlFunction]
public static SqlString MyFunctionName(int id) {
// Put your code here (maybe find the object you want to serialize using the id passed?)
using (var cn = new SqlConnection("context connection=true") ) {
//get your data into an object
var myObject = new {Name = "My Name"};
return new SqlString(Newtonsoft.Json.JsonConvert.SerializeObject(myObject));
}
}

When using table parameters in LoadRunner, how do I select values from individual columns?

In LoadRunner, given a parameter table stored in a file MyTable.dat and a VUGEN script written in C#:
FirstHeader,SecondHeader,ThirdHeader
1A,1B,1C
2A,2B,2C
3A,3B,3C
I can use lr.eval_string("{MyTable}"); to return a whole row:
1A,1B,1C
I can use lr.next_row("MyTable.dat"); to advance to the next row
2A,2B,2C
However, it's not clear how to select an individual column.
The function reference for scripts written in C states that you can use lr_paramarr_idx for parameter arrays - but that doesn't appear to be available in C# & it doesn't make clear if a table row counts as a parameter array.
HP VUGen version 9.52.0.0.

Define individual parameters assigned to the different columns with your defined separator. If you have commas within your data, then use a different data separator, such as a tab (tsv format file) or I commonly use a pipe '|' symbol. If you don't have individual parameters set up and assigned to the individual columns then you will need to grab the whole row and break it apart yourself.
See lr.next_row() and lr.advance_param(). You may be using one where with the parameters explicitly defined you will want to use the other. lr.advance_param() would be the more common use, keeping in mind that when you iterate you are going to pick up some of this advancement on a natural basis, depending upon the definition of your parameters.
Given your questions you will want to take a look at two sections of the LoadRunner documentation, (1) the documentation on the parameterization engine for LoadRunner and (2) the section in the VUGEN manual dealing with advanced concepts and building virtual users in Visual Studio (there is some reinforcement on the parameterization concepts here).

This is a bad answer:
private string[] GetRowCells(string parameter)
{
string row = lr.eval_string("{" + parameter + "}");
return row.Split(',');
}
This is bad because:
If LoadRunner provides the facility for table parameters, there must be the capability for querying individual columns.
The above doesn't take account of columns that may include comma in their body:
For example, the following won't be parsed correctly:
FirstHeader,SecondHeader
"1,A","1,B"
"2,A","2,B"
"3,A","3,B"

Just use the column name that you want to work with.
lr_eval_string("{FirstHeader}");

How can I trim string property datatypes from database char datatypes in Fluent NHibernate

I have a SQL database and a Oracle database with the same schema.
Therefore I want to use my model classes for the two databases and all I will do is change the database connection string in the Fluent NHibernate configuration.
I have some database char data types for columns but in my model classes I have set them as string however when they are returned from queries they have padded white space.
How do I return them trimmed without causing problems when I query the database using these colums as they will need to match the fixed length specification.

You can create an implementation of NHibernate.UserTypes.IUserType that will trim the whitespace when fetched from the database and re-pad it when going back to the database.
In your fluent mapping, you just add .CustomType<T> where T is your IUserType implementation.
This article is helpful for properly implementing IUserType. Don't get hung up on methods like Assemble, Disassemble, DeepCopy, Replace -- it doesn't look like you'll ever hit those, even. You're most concerned with NullSafeGet, in which you'll trim, and NullSafeSet in which you'll re-pad.
update
Upon further consideration, I'm not sure you'll actually need to re-pad the value when inserting to the database -- the db itself will enforce the fixed length of the column.
In response to the link you provided in your comment, I think that implementation pretty much gets you there, but I do think you might need to modify it a bit. For one thing, you may not want to trim both leading and trailing whitespace, as Trim() will do. For another, in your override of Equals, you'll want
value
to equal
value

The char data type in SQL databases is padded with whitespace.
If you can modify the database, you can create a view that trims the columns. But a trimmed value won't match the untrimmed value. (In a SQL query, that is.) TRIM() is a standard SQL function, but not all platforms support it. I suppose you might be able to cast it to VARCHAR(); I'm not sure I've ever tried that.
If I were in your shoes, I think I'd cache literally the value the database contains. (That is, I'd store it in a variable of some kind.) I might trim it for display to the user, but any interaction between my application code and the database would be done using the actual values in the database.

How YOU parse user input?

Consider the following scenario:
http://www.yourdomain.com/Default.aspx?p=2
Now we ofcourse want to check if the querystring parameter p doesnt contain errors.
I now have this setup:
1) Check if p exists
2) Filter out html from p's value
3) htmlencode p's value
4) check if p is integer
5) check if p's integer exists in db
This is how I usual do it, though step 5 is ofcourse a performance hit.
Kind regards,
Mark

My view: Generally a querystring parameter of this kind isn't really "entered" by users but is submitted as a link. So over-complex slow validation isn't really necessary.
So I would just pass this through to the persistence / data layer and handle any errors that come back as a regular 404 Not Found or 500 Internal Server Error depending on the kind of system I'm working with.

If your intent is to use the parameter to retrieve something from the database, why filter out html or encode it? It's not like you're going to store it in the database, or display it on the front end. Just immediately throw it to the DAL if it exists. You're DAL should be smart enough to tell you if it failed to retrieve a record with that ID, or if the ID couldn't be parsed, etc..

If you are going to convert the input to an integer anyway, then steps 2 and 3 are not needed - just use int.TryParse to see what you have. I would encode and test the input for html only if you are expecting a string which you will use in a dynamic sql statement, or will be displaying on your site

What about:
int p = 0;
if(!Int32.TryParse(Request.QueryString["p"], out p))
throw new ArgumentOutOfRangeException("p");

Quite simple. For most data types (integers, decimals, doubles, dates and booleans) there is a very strict format. If the value does not parse under the strict format, it's an error.
Strings sometimes have a strict format, like an email address or a phone number. Those can be validated with a simple regexp. If it conforms, use it, otherwise it's an error.
Most of the time however strings will simply need to be persisted to the DB and later displayed again. In that case no processing is needed, aside from escaping when inserting into DB (unnecessary as well if you used parametrized queries)k, and HTML-encoding when rendering to the display.
This way any and all data is validated, and there is no risk of any injections whatsoever.
The rare exception of a loose format for a string is, well... rare. I can't think of any right now. For that you can afford some more extensive parsing and processing.
Added: Oh, yes, checking whether IDs (or other values) are valid in respect to a DB. You're doing it right, but think if you always need it. Quite often you can put the check into some other query that you have to do anyway. Like when you select data based on the ID, you don't need to explicitly check that it exists - just be ready that your query can return no data.
Sometimes you don't need to use the value at all, then you can simply ignore it.
But, of course, there are other times, like when inserting/updating data, that you indeed need to explicitly check whether the data exists and is valid in the current context.