Develop Reference

SQL bulkCopy writes empty string instead NULL

I have function that inserts new values into database:
public async Task BulkAdd(IDataReader data)
{
if (Connection.State == ConnectionState.Broken || Connection.State == ConnectionState.Closed)
{
await Connection.OpenAsync();
}
using (SqlBulkCopy bulk = new SqlBulkCopy(Connection))
{
bulk.DestinationTableName = GetTableName();
bulk.BatchSize = BATCH_SIZE;
bulk.BulkCopyTimeout = 0; // for infinity write 0
bulk.EnableStreaming = true;
await bulk.WriteToServerAsync(data);
}
}
insert strings are generated in order and look like:
,,11111,,7,,620,7 11111,04/15/2013 00:00:00,false,Bulgaria, and then are converted to CsvDataReader:
var csvStreamReader = MapDataExtraWithHeaders(reader, clientId, dataExtraHeadersMap, delimiter, uploadDataId, dateFormat);
using (var csv = new CsvReader(csvStreamReader))
{
csv.Configuration.BadDataFound = null;
csv.Configuration.Delimiter = delimiter;
// ADDED
csv.Configuration.TypeConverterCache.AddConverter<string>(new EmptyAsNullConverter());
var dataReader = new CsvDataReader(csv);
csv.ReadHeader();
if (!HeadersValid(csv.Context.HeaderRecord, DataHeaders))
throw new CvtException(CVTExceptionCode.Import.InvalidHeaders);
await _transactionsDataRepository.BulkAdd(dataReader);
}
I added null constraints like:
alter table [dbo].[Extra] add constraint [DF_Custom] default (null) for [Custom]
however when I look into what was added I see that instead of NULL, empty string was added. How can that be fixed?

You can write a custom converter to convert empty to null :
public class EmptyAsNullConverter : CsvHelper.TypeConversion.StringConverter
{
public override object ConvertFromString(string text, IReaderRow row, MemberMapData memberMapData)
{
if (string.IsNullOrEmpty(text))
return null;
else
return base.ConvertFromString(text, row, memberMapData);
}
}
Credit to brandonw
Example of use :
static void Main(string[] args)
{
var text =
"a,b,c,e,f" + Environment.NewLine +
"a,,c,e,f" + Environment.NewLine +
"a,b,c,,f";
using (var csv = new CsvReader(new StringReader(text), CultureInfo.InvariantCulture))
{
csv.Configuration.TypeConverterCache.AddConverter<string>(new EmptyAsNullConverter());
while(csv.Read())
{
for(int i = 0; i < 5; i++)
{
Console.Write($"{csv.GetField<string>(i) ?? "null"}\t");
}
Console.WriteLine();
}
}
Console.Read();
}
Output :
a b c e f
a null c e f
a b c null f

you can bind CSV to class and write a converter for it.
public class EmptyStringConverter : ConverterBase
{
public override object StringToField(string sourceString)
{
if (String.IsNullOrWhiteSpace(sourceString))
return null;
return sourceString;
}
}
[FieldConverter(typeof(EmptyStringConverter))]
public string MyStrField;

C# method that returns a SQL string without passing any variables

I am trying to create a method that runs a query then reads the 2 values it returns then place them in the global variables so I can access them in another page. My question is what method should I use because I have two variables to set. Typically I pass the variables that I will be using but in this case I'm not. This main seem simple but I can't think of a way to get these values. I am not sure how to look this problem up to research it either. I have included the code below what I have attempted so far. Thank you for you help.
public string getTotals3()
{
WorkerData workerData = new WorkerData();
StringBuilder sqlString = new StringBuilder();
sqlString.Append("SELECT DISTINCT DataWin8Data, DataWin7Data ");
sqlString.Append("FROM Data ");
sqlString.Append("WHERE Number = 4");
SqlDataReader reader = null;
SqlConnection dbConn = App_Code.DBHelper.getConnection();
try
{
reader = App_Code.DBHelper.executeQuery(dbConn, sqlString.ToString(), null);
if (reader != null)
{
while (reader.Read())
{
workerData.TotalCases4 = reader["DataWin8Data"] != DBNull.Value ? reader["DataWin8Data"].ToString() : string.Empty;
workerData.TotalPercentage4 = reader["DataWin7Data"] != DBNull.Value ? reader["DataWin7Data"].ToString() : string.Empty;
}
}
else
throw new Exception("No records returned");
}
catch (Exception ex)
{
throw ex;
}
finally
{
if (dbConn != null)
{
try { dbConn.Close(); dbConn.Dispose(); }
catch { }
}
if (reader != null)
{
try { reader.Close(); reader.Dispose(); }
catch { }
}
}
return workerData.ToString();
}

Don't use global variables. Return the values out of the method. The calling code should be in charge of placing those values wherever it needs. I recommend reading about Dependency Inversion Principle.
public WorkerData GetWorkerData()
{
...
using (SqlDataReader reader = ...)
{
if (reader.Read())
{
return new WorkerData
{
TotalCases4 = reader["DataWin8Data"] != DBNull.Value ? reader["DataWin8Data"].ToString() : string.Empty,
TotalCases3 = workerData.TotalPercentage4 = reader["DataWin7Data"] != DBNull.Value ? reader["DataWin7Data"].ToString() : string.Empty;
}
}
}
throw new ApplicationException("Could not retrieve worker data.");
}
From your calling class, simply do whatever you want with the return value:
WorkerData workerData = someClass.GetWorkerData();

The out parameter 'recordsReturned' must be assigned to before control leaves the current method ObjectCache

I'm stuck on trying to work out how to solve the error above, how can i assign the value to recordsReturned which is an output parameter from the database and add it to ObjectCache.
Before the data is cached I can get the value, but I cannot add it to the cache, i have tried out and ref, but with no luck.
Hopefully someone with more experience than me can help with my problem.
I've added my code below:
public class RetrieveTravelGuideForSearchResults : IRetrieveTravelGuideForSearchResults
{
private readonly string _dbConn;
public RetrieveTravelGuideForSearchResults()
{
_dbConn = ConfigurationManager.ConnectionStrings["ADO_DBConn"].ConnectionString;
}
public IEnumerable<DisplayTravelGuideForSearchResults> DisplayTravelGuideSearchResults(string id,out int recordsReturned)
{
string cacheData = "DisplayTravelGuideSearchResults" + RegexHelpers.RegexRemoveAllInvalidCharactersAndSpace(id.ToLower());
ObjectCache travelGuideCache = MemoryCache.Default;
var objectInCache = travelGuideCache.Get(cacheData) as IEnumerable<DisplayTravelGuideForSearchResults>;
//recordsReturned = 0;
if (objectInCache != null)
{
recordsReturned = 0; //How can I get value from output parameter and add it here
return objectInCache;
}
const string spName = "dbo.spFTSTravelGuide";
//recordsReturned = 0;
using (var cn = new SqlConnection(_dbConn))
{
using (var cmd = new SqlCommand(spName, cn))
{
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.Add(new SqlParameter("#SearchPhrase", SqlDbType.VarChar, 100));
cmd.Parameters["#SearchPhrase"].Value = id;
cmd.Parameters.Add(new SqlParameter("#RecordsReturn", SqlDbType.Int));
cmd.Parameters["#RecordsReturn"].Direction = ParameterDirection.Output;
var data = new List<DisplayTravelGuideForSearchResults>();
try
{
cn.Open();
using (var rdr = cmd.ExecuteReader(CommandBehavior.Default))
{
if (rdr.HasRows)
{
while (rdr.Read())
{
data.Add(new DisplayTravelGuideForSearchResults
{
Country = (string)rdr["TravelGuideCountry"],
Description = RegexHelpers.RegexRemoveHtmlTags((string)rdr["TravelGuideDescription"])
});
}
}
}
var policy = new CacheItemPolicy { AbsoluteExpiration = DateTime.Now.AddMinutes(15) };
travelGuideCache.Add(cacheData, data, policy);
recordsReturned = cmd.Parameters["#RecordsReturn"].Value as int? ?? 0;
return data;
}
catch (SqlException ex)
{
throw new ApplicationException(ex.InnerException.ToString());
}
catch (Exception ex)
{
throw new ApplicationException(ex.InnerException.ToString());
}
}
}
}
}

have got the answer to my problem, should have done
recordsReturned = cmd.Parameters["#RecordsReturn"].Value as int? ?? 0;
travelGuideCache.Add("recordsReturned", recordsReturned, policy);
and then
if (objectInCache != null)
{
recordsReturned = travelGuideCache.Get("recordsReturned") as int? ?? 0;
return objectInCache;
}
This now get the output parameter value and allows me to cache it.

How to read an enumerable without a loop without losing its scope?

I'm facing a pretty weird construct. The Foo type returned in an IEnumerable loses its data as soon as the enumeration ends. This means that I can't do a enumeration.First() because the data would be lost right away.
A loop over it works, but since I know it will contain only a single element that would be weird.
int Test(out int something)
IEnumerable<Foo> enumeration = ...
for (var foo in enumeration) {
something = foo.GetSomething ();
return foo.GetAnInt ();
}
something = 42;
return 0;
}
Another way I though of is abusing a Linq Select, but that's just as horrible.
Is there a way to work around this limitation? Fixing the root cause is obviously superior, but difficult in this case.
Edit: It's an IEnumerable<IDataRecord> that is yield returned from a transactioned SQL data reader.
public IEnumerable<IDataRecord> ExecuteReader (SqlCommand cmd)
{
using (var con = GetConnection()) {
con.Open ();
using (var tr = con.BeginTransaction ()) {
cmd.Connection = con;
var reader = cmd.ExecuteReader ();
while (reader.Read ()) {
yield return reader;
}
tr.Commit ();
}
}
}

The problem is that your ExecuteReader method does simply return the SqlDataReader itself (which implements IDataRecord), instead of returning a block of data. So when you do this:
var list = ExecuteReader(...).ToList();
In that case all elements of the list will be the same SqlDataReader instance, but after the ToList has been executed, the reader has been closed. I'm a bit surprised that you don't get an ObjectDisposedException.
For this to work, you need to return a copy of the data in the IDataRecord. You think you can iterate the elements in the data record. An other option is to change the ExecuteReader to the following:
public IEnumerable<T> ExecuteReader<T>(SqlCommand cmd,
Func<IDataRecord, T> recordCreator)
{
using (var con = GetConnection()) {
con.Open ();
using (var tr = con.BeginTransaction()) {
cmd.Connection = con;
var reader = cmd.ExecuteReader();
while (reader.Read()) {
yield return recordCreator(reader);
}
tr.Commit();
}
}
}
This way you can do the following:
var list = ExecuteReader(command, record => new
{
Item1 = record.GetInt("id"),
Item2 = record.GetString("name")
});
Note: I'm not sure why you need a transaction for this anyway.

How about
int Test(out int something)
{
IEnumerable<Foo> enumeration = ...
var values = enumeration
.Select(foo => new
{
something = foo.GetSomething(),
anInt = foo.GetAnInt()
})
.FirstOrDefault();
if (values != null)
{
something = values.something;
return values.anInt;
}
else
{
something = 42;
return 0;
}
}
GetSomething and GetAnInt are called while inside the enumeration.

Another idea could be to convert the result type of the method from IEnumerable to IEnumerator. That way, scope control is much easier, and returning single results does not require any (fake) loop, too.
Edit: I think I found a way to refactor the whole issue. This circumvents the initial problem by using new disposable class that contains the logic formerly found in the method. It's very readable and less code even.
public TransactedConnection GetConnection (string text)
{
return new TransactedConnection (_ConnectionString, text);
}
public class TransactedConnection : IDisposable
{
private readonly SQLiteCommand _Cmd;
private readonly SQLiteConnection _Con;
private readonly SQLiteTransaction _Tr;
public TransactedConnection (string connection, string text)
{
_Cmd = new SQLiteCommand (text);
_Con = new SQLiteConnection (connection);
_Con.Open ();
_Cmd.Connection = _Con;
_Tr = _Con.BeginTransaction ();
}
public void Dispose ()
{
_Tr.Commit ();
_Tr.Dispose ();
_Con.Dispose ();
_Cmd.Dispose ();
}
public SQLiteParameterCollection Parameters
{
get
{
return _Cmd.Parameters;
}
}
public int ExecuteNonQuery ()
{
return _Cmd.ExecuteNonQuery ();
}
public object ExecuteScalar ()
{
return _Cmd.ExecuteScalar ();
}
public SQLiteDataReader ExecuteReader ()
{
return _Cmd.ExecuteReader ();
}
}
public void Test (string match)
{
var text = "SELECT * FROM Data WHERE foo=#bar;";
using (var cmd = GetConnection (text)) {
cmd.Parameters.Add ("#bar", DbType.String).Value = match;
using (var reader = cmd.ExecuteReader ()) {
while (reader.Read ()) {
Console.WriteLine (reader["foo"]);
}
}
}
}

Using an ObjectDataSource with a GridView in a dynamic scenario

I have a search page that is tasked with searching 3.5 million records for individuals based on their name, customer ID, address, etc. The queries range from complex to simple.
Currently, this code relies on a SqlDataSource and a GridView. When a user types a serach term in and presses enter, the TextBoxChanged even runs a Search(term, type) function that changes the query that the SqlDataSource uses, adds the parameters, and rebinds the GridView.
It works well, but I've become obsessed with rewriting the code more efficiently. I want the paging to be done by SQL Server instead of the inefficiencies of a SqlDataSource in DataSet mode.
Enter the ObjectDataSource. Caveat: I have never used one before today.
I have spent the better part of the day putting together this class:
using System;
using System.Data;
using System.Data.SqlClient;
using System.Configuration;
using System.Text;
using System.Web;
using System.Web.Security;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Web.UI.WebControls.WebParts;
using System.Web.UI.HtmlControls;
/// <summary>
/// Summary description for MultiSearchData
/// </summary>
public class MultiSearchData
{
private string _connectionString = string.Empty;
private string _sortColumns = string.Empty;
private string _selectQuery = string.Empty;
private int _lastUpdate;
private int _lastRowCountUpdate;
private int _lastRowCount;
private SqlParameterCollection _sqlParams;
public MultiSearchData()
{
}
private void UpdateDate()
{
_lastUpdate = (int)(DateTime.UtcNow - new DateTime(1970, 1, 1)).TotalSeconds;
}
private string ReplaceFirst(string text, string search, string replace)
{
int pos = text.IndexOf(search);
if (pos < 0)
{
return text;
}
return text.Substring(0, pos) + replace + text.Substring(pos + search.Length);
}
public string SortColumns
{
get { return _sortColumns; }
set { _sortColumns = value; }
}
public SqlParameterCollection SqlParams
{
get { return _sqlParams; }
set { _sqlParams = value; }
}
public string ConnectionString
{
get { return _connectionString; }
set { _connectionString = value; }
}
public string SelectQuery
{
get { return _selectQuery; }
set
{
if (value != _selectQuery)
{
_selectQuery = value;
UpdateDate();
}
}
}
public DataTable GetFullDataTable()
{
return GetDataTable(AssembleSelectSql());
}
public DataTable GetPagedDataTable(int startRow, int pageSize, string sortColumns)
{
if (sortColumns.Length > 0)
_sortColumns = sortColumns;
return GetDataTable(AssemblePagedSelectSql(startRow, pageSize));
}
public int GetRowCount()
{
if (_lastRowCountUpdate == _lastUpdate)
{
return _lastRowCount;
}
else
{
string strCountQuery = _selectQuery.Remove(7, _selectQuery.IndexOf("FROM") - 7);
strCountQuery = strCountQuery.Replace("SELECT FROM", "SELECT COUNT(*) FROM");
using (SqlConnection conn = new SqlConnection(_connectionString))
{
conn.Open();
using (SqlCommand cmd = new SqlCommand(strCountQuery, conn))
{
if (_sqlParams.Count > 0)
{
foreach (SqlParameter param in _sqlParams)
{
cmd.Parameters.Add(param);
}
}
_lastRowCountUpdate = _lastUpdate;
_lastRowCount = (int)cmd.ExecuteScalar();
return _lastRowCount;
}
}
}
}
public DataTable GetDataTable(string sql)
{
DataTable dt = new DataTable();
using (SqlConnection conn = new SqlConnection(_connectionString))
{
using (SqlCommand GetCommand = new SqlCommand(sql, conn))
{
conn.Open();
if (_sqlParams.Count > 0)
{
foreach (SqlParameter param in _sqlParams)
{
GetCommand.Parameters.Add(param);
}
}
using (SqlDataReader dr = GetCommand.ExecuteReader())
{
dt.Load(dr);
conn.Close();
return dt;
}
}
}
}
private string AssembleSelectSql()
{
StringBuilder sql = new StringBuilder();
sql.Append(_selectQuery);
return sql.ToString();
}
private string AssemblePagedSelectSql(int startRow, int pageSize)
{
StringBuilder sql = new StringBuilder();
string originalQuery = ReplaceFirst(_selectQuery, "FROM", ", ROW_NUMBER() OVER (ORDER BY " + _sortColumns + ") AS ResultSetRowNumber FROM");
sql.Append("SELECT * FROM (");
sql.Append(originalQuery);
sql.Append(") AS PagedResults");
sql.AppendFormat(" WHERE ResultSetRowNumber > {0} AND ResultSetRowNumber <= {1}", startRow.ToString(), (startRow + pageSize).ToString());
return sql.ToString();
}
}
I don't know if it's pretty. It works. I give it a query in the ObjectCreating method:
protected void dataMultiSearchData_ObjectCreating(object sender, ObjectDataSourceEventArgs e)
{
MultiSearchData info;
info = Cache["MultiSearchDataObject"] as MultiSearchData;
if (null == info)
{
info = new MultiSearchData();
}
info.SortColumns = "filteredcontact.fullname";
info.ConnectionString = "Data Source=SERVER;Initial Catalog=TheDatabase;Integrated Security=sspi;Connection Timeout=60";
info.SelectQuery = #"SELECT filteredcontact.contactid,
filteredcontact.new_libertyid,
filteredcontact.fullname,
'' AS line1,
filteredcontact.emailaddress1,
filteredcontact.telephone1,
filteredcontact.birthdateutc AS birthdate,
filteredcontact.gendercodename
FROM filteredcontact
WHERE fullname LIKE 'Griffin%' AND filteredcontact.statecode = 0";
e.ObjectInstance = info;
}
protected void dataMultiSearchData_ObjectDisposing(object sender, ObjectDataSourceDisposingEventArgs e)
{
MultiSearchData info = e.ObjectInstance as MultiSearchData;
MultiSearchData temp = Cache["MultiSearchDataObject"] as MultiSearchData;
if (null == temp)
{
Cache.Insert("MultiSearchDataObject", info);
}
e.Cancel = true;
}
Once the class has the query, it wraps it in paging friendly SQL and we're off to the races. I've implemented caching so that it can skip some expensive queries. Etc.
My problem is, this completely breaks my pretty little Search(term, type) world. Having ot set the query in the ObjectCreating method is completely harshing my vibe.
I've been trying to think of a better way to do this all day, but I keep ending up with a really messy...do it all in ObjectCreating model that just turns my stomach.
How would you do this? How can I keep the efficiency of this new method whilst have the organizational simplicity of my former model?
Am I being too OCD?

I determined that it can't be done. Furthermore, after benchmarking this class I found it performed no better than a SqlDataSource but was much more difficult to maintain.
Thus I abandoned this project. I hope someone finds this code useful at some point though.