Develop Reference

Unpivot a table in C#

I'm building a function in C# to unpivot a complex table in a CSV file and insert it into a SQL table. The file looks something like this:
| 1/5/2018 | 1/5/2018 | 1/6/2018 | 1/6/2018...
City: | min: | max: | min: | max:
Boston(KBOS) | 1 | 10 | 5 | 12
My goal is to unpivot it like so:
airport_code | localtime | MinTemp | MaxTemp
KBOS | 1/5/2018 | 1 | 10
KBOS | 1/6/2018 | 5 | 12
My strategy is:
Store the first row of dates and the second row of headers into arrays
Use a CSV parser to read each following line and loop through each field
If the date that corresponds to the current field is same as the previous one, it's belongs in the same row. Put the data into the appropriate field.
Since there are only two temperature fields for each row, this row is complete can now be inserted.
Otherwise, start a new row and put the data into the appropriate field.
However, I'm running into a problem: Once insertRow is populated and inserted, I can't overwrite it or null all the fields and use it again - that throws an error that row has already been inserted. I can't move the declaration of insertRow inside the for loop because I need to preserve the data through multiple iterations to completely fill out the row. So instead I tried to declare it outside the loop but only initialize it inside the loop, something like:
if(insertRow == null)
{
insertRow = MyDataSet.tblForecast.NewtblForecastRow();
}
But that throws a "use of unassigned local variable" error. Any ideas about how I can preserve insertRow on some iterations and dispose of it on others? Or, any suggestions about a better way to do what I'm looking for? The relevant portion of the code is below:
using (TextFieldParser csvParser = new TextFieldParser(FileName))
{
csvParser.SetDelimiters(new string[] { "," });
csvParser.ReadLine(); //Skip top line
string[] dateList = csvParser.ReadFields();//Get dates from second line.
string[] fieldNames = csvParser.ReadFields();//Get headers from third line
//Read through file
while (!csvParser.EndOfData)
{
DataSet1.tblForecastRow insertRow = MyDataSet.tblForecast.NewtblForecastRow();
string[] currRec = csvParser.ReadFields();
//Get airport code
string airportCode = currRec[0].Substring(currRec[0].LastIndexOf("(") + 1, 4);
//Unpivot record
DateTime currDate = DateTime.Parse("1/1/1900");//initialize
DateTime prevDate;
for (int i = 1; i<fieldNames.Length; i++) //skip first col
{
prevDate = currDate;//previous date is the prior current date
DateTime.TryParse(dateList[i], out currDate);//set new current date
int val;
int.TryParse(currRec[i], out val);
switch (fieldNames[i].ToLower())
{
case "min:":
insertRow["MinTemp"] = val;
break;
case "max:":
insertRow["MaxTemp"] = val;
break;
}
if (currDate == prevDate)//if same date, at end of row, insert
{
insertRow["airport_code"] = airportCode;
insertRow["localTime"] = currDate;
insertRow["Forecasted_date"] = DateTime.Today;
MyDataSet.tblForecast.AddtblForecastRow(insertRow);
ForecastTableAdapter.Update(MyDataSet.tblForecast);
}
}
}
}

You create a new row when you finished handling the current one. And you already know where that is:
if (currDate == prevDate)//if same date, at end of row, insert
{
insertRow["airport_code"] = airportCode;
insertRow["localTime"] = currDate;
insertRow["Forecasted_date"] = DateTime.Today;
// we're storing insertRow
MyDataSet.tblForecast.AddtblForecastRow(insertRow);
// now it gets saved (man that is often)
ForecastTableAdapter.Update(MyDataSet.tblForecast);
// OKAY, let's create the new insertRow instance
insertRow = MyDataSet.tblForecast.NewtblForecastRow();
// and now on the next time we end up in this if
// the row we just created will be inserted
}
Your initial Row can be created outside the loop:
// first row creation
DataSet1.tblForecastRow insertRow = MyDataSet.tblForecast.NewtblForecastRow();
//Read through file
while (!csvParser.EndOfData)
{
// line moved out of the while loop
string[] currRec = csvParser.ReadFields();

Cassandra C# Driver - SELECT Statement Execution Slow

I'm running a 3-node Cassandra 3.0.0 cluster running on AWS EC2 i3.large instances and I've been playing around with using the C# driver for Cassandra. Executing the following query (which is very simple) takes approximately 300 ms (to scan the single partition and return the top 100 rows).
var rs = session.Execute("SELECT col1, col6, col7 FROM breadcrumbs WHERE col1='samplepk' LIMIT 100;");
My data model is:
Column 1 = a 13-character string
Column 2 = a 23-character string
Column 3 = a date/time timestamp
Column 4 = a 4-digit integer
Column 5 = a 3 digit integer
Column 6 = a latitude value
Column 7 = a longitude value
Column 8 = a 15-digit double
Column 9 = a 15-digit double
I defined my primary key as Col1, col2.
My C# driver code is as follows:
Cluster cluster = Cluster.Builder().AddContactPoint(~~~~~IP Here~~~~).Build();
ISession session = cluster.Connect(~~~keyspacename~~~);
long ticks = DateTime.Now.Ticks;
var rs = session.Execute("SELECT col2, col6, col7 FROM breadcrumbs WHERE partitionkey=~targetkey~ LIMIT 100;");
Console.WriteLine((DateTime.Now.Ticks - ticks)/Math.Pow(10,4)+" ms");
Console.ReadKey();
Is that abnormally slow or are my expectations too high? If it is slow, does anyone have any ideas about what's causing it?
If I forgot to provide any pertinent details, please leave a comment :) .
Thanks in advance.

Retrieving statistics for stored procedure, SelectRows is always 0

I'm trying to collect statistics on some SQL queries.
I'm using RetrieveStatistics() method of SqlDbConnection class to get statistics and ExecuteReader() method of SqlCommand to run query.
RetrieveStatistics() method returns dictionary filled with statistics on the query executed.
When I'm running regular query, SelectRows propety of dictionary contains actual number of rows returned by query. But when I'm running stored procedure, SelectRows is always zero, although reader definitely contains rows.
I call ResetStatistics() before each query and StatisticsEnabled is set to true.
Here's my Powershell code:
### Stored procedure
$cn = New-Object system.data.sqlclient.sqlconnection
$cn.ConnectionString = "Data Source=localhost;Initial Catalog=XXXX;Integrated Security=SSPI"
$cn.StatisticsEnabled = $true
$cmd = $cn.CreateCommand()
$cmd.CommandText = "[dbo].[spGetXXXX]"
$cmd.CommandType = "StoredProcedure"
$cn.Open()
$cmd.ExecuteReader()
# several rows returned
$cn.RetrieveStatistics()
Name Value
---- -----
BytesReceived 300
SumResultSets 1
ExecutionTime 5
Transactions 0
BuffersReceived 1
IduRows 0
ServerRoundtrips 1
PreparedExecs 0
BytesSent 132
SelectCount 1
CursorOpens 0
ConnectionTime 51299
Prepares 0
SelectRows 0
UnpreparedExecs 1
NetworkServerTime 3
BuffersSent 1
IduCount 0
### Regular SQL query
$cn2 = New-Object system.data.sqlclient.sqlconnection
$cn2.ConnectionString = "Data Source=localhost;Initial Catalog=XXXX;Integrated Security=SSPI"
$cn2.StatisticsEnabled = $true
$cmd2 = $cn2.CreateCommand()
$cmd2.CommandText = "SELECT * FROM XXXX"
$cn2.Open()
$cmd2.ExecuteReader()
#rows returned
$cn2.RetrieveStatistics()
Name Value
---- -----
BytesReceived 12357
SumResultSets 1
ExecutionTime 12
Transactions 0
BuffersReceived 2
IduRows 0
ServerRoundtrips 1
PreparedExecs 0
BytesSent 98
SelectCount 1
CursorOpens 0
ConnectionTime 11407
Prepares 0
SelectRows 112
UnpreparedExecs 1
NetworkServerTime 0
BuffersSent 1
IduCount 0

The difference between the two queries is that after stored procedure call reader stays open, and row number statistics is not updated.
So the correct code would be like:
$connectionn.ResetStatistics()
$reader = $command.ExecuteReader()
$reader.Close() # !
$connection.RetrieveStatistics()

Convert a file full of "INSERT INTO xxx VALUES" in to something Bulk Insert can parse

This is a followup to my first question "Porting “SQL” export to T-SQL".
I am working with a 3rd party program that I have no control over and I can not change. This program will export it's internal database in to a set of .sql each one with a format of:
INSERT INTO [ExampleDB] ( [IntField] , [VarcharField], [BinaryField])
VALUES
(1 , 'Some Text' , 0x123456),
(2 , 'B' , NULL),
--(SNIP, it does this for 1000 records)
(999, 'E' , null);
(1000 , 'F' , null);
INSERT INTO [ExampleDB] ( [IntField] , [VarcharField] , BinaryField)
VALUES
(1001 , 'asdg', null),
(1002 , 'asdf' , 0xdeadbeef),
(1003 , 'dfghdfhg' , null),
(1004 , 'sfdhsdhdshd' , null),
--(SNIP 1000 more lines)
This pattern continues till the .sql file has reached a file size set during the export, the export files are grouped by EXPORT_PATH\%Table_Name%\Export#.sql Where the # is a counter starting at 1.
Currently I have about 1.3GB data and I have it exporting in 1MB chunks (1407 files across 26 tables, All but 5 tables only have one file, the largest table has 207 files).
Right now I just have a simple C# program that reads each file in to ram then calls ExecuteNonQuery. The issue is I am averaging 60 sec/file which means it will take about 23 hrs for it to do the entire export.
I assume if I some how could format the files to be loaded with a BULK INSERT instead of a INSERT INTO it could go much faster. Is there any easy way to do this or do I have to write some kind of Find & Replace and keep my fingers crossed that it does not fail on some corner case and blow up my data.
Any other suggestions on how to speed up the insert into would also be appreciated.
UPDATE:
I ended up going with the parse and do a SqlBulkCopy method. It went from 1 file/min. to 1 file/sec.

Well, here is my "solution" for helping convert the data into a DataTable or otherwise (run it in LINQPad):
var i = "(null, 1 , 'Some''\n Text' , 0x123.456)";
var pat = #",?\s*(?:(?<n>null)|(?<w>[\w.]+)|'(?<s>.*)'(?!'))";
Regex.Matches(i, pat,
RegexOptions.IgnoreCase | RegexOptions.Singleline).Dump();
The match should be run once per value group (e.g. (a,b,etc)). Parsing of the results (e.g. conversion) is left to the caller and I have not tested it [much]. I would recommend creating the correctly-typed DataTable first -- although it may be possible to pass everything "as a string" to the database? -- and then use the information in the columns to help with the extraction process (possibly using type converters). For the captures: n is null, w is word (e.g. number), s is string.
Happy coding.

Apparently your data is always wrapped in parentheses and starts with a left parenthesis. You might want to use this rule to split(RemoveEmptyEntries) each of those lines and load it into a DataTable. Then you can use SqlBulkCopy to copy all at once into the database.
This approach would not necessarily be fail-safe, but it would be certainly faster.
Edit: Here's the way how you could get the schema for every table:
private static DataTable extractSchemaTable(IEnumerable<String> lines)
{
DataTable schema = null;
var insertLine = lines.SkipWhile(l => !l.StartsWith("INSERT INTO [")).Take(1).First();
var startIndex = insertLine.IndexOf("INSERT INTO [") + "INSERT INTO [".Length;
var endIndex = insertLine.IndexOf("]", startIndex);
var tableName = insertLine.Substring(startIndex, endIndex - startIndex);
using (var con = new SqlConnection("CONNECTION"))
{
using (var schemaCommand = new SqlCommand("SELECT * FROM " tableName, con))
{
con.Open();
using (var reader = schemaCommand.ExecuteReader(CommandBehavior.SchemaOnly))
{
schema = reader.GetSchemaTable();
}
}
}
return schema;
}
Then you simply need to iterate each line in the file, check if it starts with ( and split that line by Split(new[] { ',' }, StringSplitOptions.RemoveEmptyEntries). Then you could add the resulting array into the created schema-table.
Something like this:
var allLines = System.IO.File.ReadAllLines(path);
DataTable result = extractSchemaTable(allLines);
for (int i = 0; i < allLines.Length; i++)
{
String line = allLines[i];
if (line.StartsWith("("))
{
String data = line.Substring(1, line.Length - (line.Length - line.LastIndexOf(")")) - 1);
var fields = data.Split(new[] { ',' }, StringSplitOptions.RemoveEmptyEntries);
// you might need to parse it to correct DataColumn.DataType
result.Rows.Add(fields);
}
}

Inserting/Updating rows with some columns and preserve values of others

A little background information: I have a table called table_a, which has 12 columns. I want to insert or update rows with values for 6 of these columns, while I don't wanna lose the data in the other 6 columns. And I wanna do this with a parameterized query in C#.
field1 is Unique.
> SELECT * FROM table_a;
+----+--------+--------+---+---------+---------+
| Id | field1*| field2 |...|field11 | field12 |
+----+--------+--------+---+---------+---------+
| 1 | AA | BB |...| KK | LL |
| 2 | AA | BB |...| KK | LL |
| 3 | AA | BB |...| KK | LL |
| 4 | AA | BB |...| KK | LL |
+----+--------+--------+---+---------+---------+
The Problem is, my first thought was to use REPLACE INTO, unfortunately this will delete the 6 not touched values:
> REPLACE INTO table_a (field1, ..., field6) VALUES ('AA', ...);
> REPLACE INTO table_a (field1, ..., field6) VALUES ('AB', ...);
+----+--------+--------+---+---------+---------+
| Id | field1*| field2 |...| field11 | field12 |
+----+--------+--------+---+---------+---------+
| 1 | AA | BB |...| NULL | NULL |
| 2 | AB | BB |...| NULL | NULL |
| 3 | AC | BB |...| KK | LL |
| 4 | AD | BB |...| KK | LL |
+----+--------+--------+---+---------+---------+
My second thought was to use INSERT INTO ... ON DUPLICATE KEY UPDATE, but then I'd have to bind the parameters a second time, the first time in the INSERT part and the second time in the UPDATE part, like this:
INSERT INTO table_a (field1, ..., field6)
VALUES(?, ..., ?)
ON DUPLICATE KEY UPDATE
field1 = ?, ..., field6 = ?;
That would preserve my data, but I have to bind the parameters twice.
The third option would be to create another two queries and use the SELECT and INSERT INTO/UPDATE pattern.
So, my question is, how do I do this the smart way?

Your second option sounds like a winner for single row updates.
Your third option is good if you insert/update many rows at once (as it will not matter much that you have two queries then - providing each does only what it is supposed to do).
UPDATE:
Digging through documentation one finds that you can bind once if you wish - you can refer to the originally bound values with VALUES()
UPDATE2:
Well, actually you can not get to the bound values with VALUES(column), so instead two suggestions that actually might help:
did you check about using named parameters (then you would not need to bound them twice)?
did you consider stored procedures?

I think you've listed all the available options, along with the pros/cons of each. As for the third option, you would probably want to wrap your two queries in a transaction to ensure that the operation remains atomic.

Hi let's say you want to modify field2 to field6.
why wouldn't you do:
replace into table_a select field1,new_value2,...,new_value6,field7,...,field12 from table_a where field1=filter_field1;
You put the new values and you get the others value by querying the table you're updating.