Develop Reference

Sum formated times from Datatable into an textBox

I am trying to sum up values out of an datatable created out of an database query.
The times in MySQL DB are formatted like HH:mm:ss (ss is always 00)
I already had this running i just was unhappy with the result as for example 25 h as result resulted in 1.01:00:00 or 1 day and 1 hour when i wanted to receive 25 h as it is saved in the database, changing data type in mysql is not an option for me, as i have a bunch of calc's that need those as data type time.
Right now the code is not longer able to show the result in a textBox as I have more a less an error:
"Invalid use of aggregate function Sum () and Type: String."
As far as I understood I can not sum strings, just time and numbers, however I could do it before formatting time right when it gets pulled out of the db.
I did search in google for a good solution for a couple of days now, but the only options I see are thinks that I already tried around on my own (of cause tested again just more errors)
However here is the required code snippet:
MySqlCommand cmddttoday = conn.CreateCommand();
cmddttoday.CommandType = CommandType.Text;
cmddttoday.CommandText = "SELECT day, month, date_format (start, '%H:%i') start, date_format (stop, '%H:%i') stop, date_format (total, '%H:%i') total FROM arbeitszeiten WHERE username = '" + textBoxusername.Text + "' and day = '" + today + "' and month = '" + month + "' and year = '" + thisyear + "'";
cmddttoday.ExecuteScalar();
DataTable dttimetoday = new DataTable();
MySqlDataAdapter datadatoday = new MySqlDataAdapter(cmddttoday);
datadatoday.Fill(dttimetoday);
textBoxtimetoday.Text = dttimetoday.Compute("SUM(total)", "").ToString();
dttimetoday.Columns["day"].ColumnName = "Tag";
dttimetoday.Columns["month"].ColumnName = "Monat";
dttimetoday.Columns["start"].ColumnName = "Arbeit aufgenommen";
dttimetoday.Columns["stop"].ColumnName = "Arbeit abgelegt";
dttimetoday.Columns["total"].ColumnName = "Zeit gesammt";
Keep in mind, yes it does not work like this but without the format on total in the query it shows like explained before as 1.01:00:00
I would be very happy if we found a solution for this meanwhile annoying problem.
Thanks for your burning braincells ;)

I think using TIMESTAMPDIFF would be way easier to calculate the timedifference in minutes directly. From there you can use SUM to calculate the total of minutes. To get the hours you can then simply divide by 60. This would look like this:
SELECT SUM(TIMESTAMPDIFF(Minute,start, end)/60) from arbeitszeiten
Check out this Fiddle: Link
EDIT:
If thats the case, you have to use linq. The error you get is because the datatable column "total" is of the datatype string. Since you cant Sum a string this wont work. You cant convert withing the SUM function either since the Compute method requests a number inside of the brackets.
With linq you can do it like this:
textBoxtimetoday.Text = dttimetoday.AsEnumerable().Sum(r => Convert.ToDecimal(r.Field<string>(4))).ToString();
but you have to replace date_format (total, '%H:%i') total with date_format (total, '%H.%i') total

Cassandra C# Driver - SELECT Statement Execution Slow

I'm running a 3-node Cassandra 3.0.0 cluster running on AWS EC2 i3.large instances and I've been playing around with using the C# driver for Cassandra. Executing the following query (which is very simple) takes approximately 300 ms (to scan the single partition and return the top 100 rows).
var rs = session.Execute("SELECT col1, col6, col7 FROM breadcrumbs WHERE col1='samplepk' LIMIT 100;");
My data model is:
Column 1 = a 13-character string
Column 2 = a 23-character string
Column 3 = a date/time timestamp
Column 4 = a 4-digit integer
Column 5 = a 3 digit integer
Column 6 = a latitude value
Column 7 = a longitude value
Column 8 = a 15-digit double
Column 9 = a 15-digit double
I defined my primary key as Col1, col2.
My C# driver code is as follows:
Cluster cluster = Cluster.Builder().AddContactPoint(~~~~~IP Here~~~~).Build();
ISession session = cluster.Connect(~~~keyspacename~~~);
long ticks = DateTime.Now.Ticks;
var rs = session.Execute("SELECT col2, col6, col7 FROM breadcrumbs WHERE partitionkey=~targetkey~ LIMIT 100;");
Console.WriteLine((DateTime.Now.Ticks - ticks)/Math.Pow(10,4)+" ms");
Console.ReadKey();
Is that abnormally slow or are my expectations too high? If it is slow, does anyone have any ideas about what's causing it?
If I forgot to provide any pertinent details, please leave a comment :) .
Thanks in advance.

Convert a file full of "INSERT INTO xxx VALUES" in to something Bulk Insert can parse

This is a followup to my first question "Porting “SQL” export to T-SQL".
I am working with a 3rd party program that I have no control over and I can not change. This program will export it's internal database in to a set of .sql each one with a format of:
INSERT INTO [ExampleDB] ( [IntField] , [VarcharField], [BinaryField])
VALUES
(1 , 'Some Text' , 0x123456),
(2 , 'B' , NULL),
--(SNIP, it does this for 1000 records)
(999, 'E' , null);
(1000 , 'F' , null);
INSERT INTO [ExampleDB] ( [IntField] , [VarcharField] , BinaryField)
VALUES
(1001 , 'asdg', null),
(1002 , 'asdf' , 0xdeadbeef),
(1003 , 'dfghdfhg' , null),
(1004 , 'sfdhsdhdshd' , null),
--(SNIP 1000 more lines)
This pattern continues till the .sql file has reached a file size set during the export, the export files are grouped by EXPORT_PATH\%Table_Name%\Export#.sql Where the # is a counter starting at 1.
Currently I have about 1.3GB data and I have it exporting in 1MB chunks (1407 files across 26 tables, All but 5 tables only have one file, the largest table has 207 files).
Right now I just have a simple C# program that reads each file in to ram then calls ExecuteNonQuery. The issue is I am averaging 60 sec/file which means it will take about 23 hrs for it to do the entire export.
I assume if I some how could format the files to be loaded with a BULK INSERT instead of a INSERT INTO it could go much faster. Is there any easy way to do this or do I have to write some kind of Find & Replace and keep my fingers crossed that it does not fail on some corner case and blow up my data.
Any other suggestions on how to speed up the insert into would also be appreciated.
UPDATE:
I ended up going with the parse and do a SqlBulkCopy method. It went from 1 file/min. to 1 file/sec.

Well, here is my "solution" for helping convert the data into a DataTable or otherwise (run it in LINQPad):
var i = "(null, 1 , 'Some''\n Text' , 0x123.456)";
var pat = #",?\s*(?:(?<n>null)|(?<w>[\w.]+)|'(?<s>.*)'(?!'))";
Regex.Matches(i, pat,
RegexOptions.IgnoreCase | RegexOptions.Singleline).Dump();
The match should be run once per value group (e.g. (a,b,etc)). Parsing of the results (e.g. conversion) is left to the caller and I have not tested it [much]. I would recommend creating the correctly-typed DataTable first -- although it may be possible to pass everything "as a string" to the database? -- and then use the information in the columns to help with the extraction process (possibly using type converters). For the captures: n is null, w is word (e.g. number), s is string.
Happy coding.

Apparently your data is always wrapped in parentheses and starts with a left parenthesis. You might want to use this rule to split(RemoveEmptyEntries) each of those lines and load it into a DataTable. Then you can use SqlBulkCopy to copy all at once into the database.
This approach would not necessarily be fail-safe, but it would be certainly faster.
Edit: Here's the way how you could get the schema for every table:
private static DataTable extractSchemaTable(IEnumerable<String> lines)
{
DataTable schema = null;
var insertLine = lines.SkipWhile(l => !l.StartsWith("INSERT INTO [")).Take(1).First();
var startIndex = insertLine.IndexOf("INSERT INTO [") + "INSERT INTO [".Length;
var endIndex = insertLine.IndexOf("]", startIndex);
var tableName = insertLine.Substring(startIndex, endIndex - startIndex);
using (var con = new SqlConnection("CONNECTION"))
{
using (var schemaCommand = new SqlCommand("SELECT * FROM " tableName, con))
{
con.Open();
using (var reader = schemaCommand.ExecuteReader(CommandBehavior.SchemaOnly))
{
schema = reader.GetSchemaTable();
}
}
}
return schema;
}
Then you simply need to iterate each line in the file, check if it starts with ( and split that line by Split(new[] { ',' }, StringSplitOptions.RemoveEmptyEntries). Then you could add the resulting array into the created schema-table.
Something like this:
var allLines = System.IO.File.ReadAllLines(path);
DataTable result = extractSchemaTable(allLines);
for (int i = 0; i < allLines.Length; i++)
{
String line = allLines[i];
if (line.StartsWith("("))
{
String data = line.Substring(1, line.Length - (line.Length - line.LastIndexOf(")")) - 1);
var fields = data.Split(new[] { ',' }, StringSplitOptions.RemoveEmptyEntries);
// you might need to parse it to correct DataColumn.DataType
result.Rows.Add(fields);
}
}

Issue with data table Select statement

The following VB line, where _DSversionInfo is a DataSet, returns no rows:
_DSversionInfo.Tables("VersionInfo").Select("FileID=88")
but inspection shows that the table contains rows with FileID's of 92, 93, 94, 90, 88, 89, 215, 216. The table columns are all of type string.
Further investigation showed that using the ID of 88, 215 and 216 will only return rows if the number is quoted.
ie _DSversionInfo.Tables("VersionInfo").Select("FileID='88'")
All other rows work regardless of whether the number is quoted or not.
Anyone got an explanation of why this would happen for some numbers but not others? I understand that the numbers should be quoted just not why some work and others don't?
I discovered this in some VB.NET code but (despite my initial finger pointing) don't think it is VB.NET specific.

According to the MSDN documentation on building expressions, strings should always be quoted. Failing to do so produces some bizarro unpredictable behavior... You should quote your number strings to get predictable and proper behavior like the documentation says.
I've encounted what you're describing in the past, and kinda tried to figure it out - here, pop open your favorite .NET editor and try the following:
Create a DataTable, and into a string column 'Stuff' of that DataSet, insert rows in the following order: "6", "74", "710", and Select with the filter expression "Stuff = 710". You will get 1 row back. Now, change the first row into any number greater than 7 - suddenly, you get 0 rows back.
As long as the numbers are ordered in proper descending order using string ordering logic (i.e., 7 comes after 599) the unquoted query appears to work.
My guess is that this is a limitation of how DataSet filter expressions are parsed, and it wasn't meant to work this way...
The Code:
// Unquoted filter string bizzareness.
var table = new DataTable();
table.Columns.Add(new DataColumn("NumbersAsString", typeof(String)));
var row1 = table.NewRow(); row1["NumbersAsString"] = "9"; table.Rows.Add(row1); // Change to '66
var row2 = table.NewRow(); row2["NumbersAsString"] = "74"; table.Rows.Add(row2);
var row4 = table.NewRow(); row4["NumbersAsString"] = "90"; table.Rows.Add(row4);
var row3 = table.NewRow(); row3["NumbersAsString"] = "710"; table.Rows.Add(row3);
var results = table.Select("NumbersAsString = 710"); // Returns 0 rows.
var results2 = table.Select("NumbersAsString = 74"); // Throws exception "Min (1) must be less than or equal to max (-1) in a Range object." at System.Data.Select.GetBinaryFilteredRecords()
Conclusion: Based on the exception text in that last case, there appears to be some wierd casting going on inside filter expressions that is not guaranteed to be safe. Explicitely putting single quotes around the value for which you're querying avoids this problem by letting .NET know that this is a literal.

DataTable builds an index on the columns to make Select() queries fast. That index is sorted by value, then it uses a binary search to select the range of records that matches the query expression.
So the records will be sorted like this 215,216,88,89,90,92,93,94. A binary search is done treating them as integer (as per our filter expression) cannot locate certain records because, it is designed to only search properly sorted collections.
It indexes the data as string and Binary search searches as number. See the below explanation.
string[] strArr = new string[] { "115", "118", "66", "77", "80", "81", "82" };
int[] intArr = new int[] { 215, 216, 88, 89, 90, 92, 93, 94 };
int i88 = Array.BinarySearch(intArr, 88); //returns -ve index
int i89 = Array.BinarySearch(intArr, 89); //returns +ve index
This should be a bug in the framework.

this error usually comes due to invalid data table column type in which you are going to search
i got this error when i was using colConsultDate instead of Convert(colConsultDate, 'System.DateTime')
because colConsultDate was a data table column of type string which i must have to convert into System.DateTime therefor your search query should be like
string query = "Convert(colConsultDate, 'System.DateTime') >= #" + sdateDevFrom.ToString("MM/dd/yy") + "# AND Convert(colConsultDate, 'System.DateTime') <= #" + sdateDevTo.ToString("MM/dd/yy") + "#";
DataRow[] dr = yourDataTable.Select(query);
if (dr.Length > 0)
{
nextDataTabel = dr.CopyToDataTable();
}

#Val Akkapeddi just wanna add things to your answer.
if you do something like this it would be benefited specially when you have to use comparison operators. because you put quotes around 74 it will be treated as string. please see yourself by actually writing code. Comparison operators
(decimal is just for reference you can add your desired datatype instead.)
var results2 = table.Select("Convert(NumbersAsString , 'System.Decimal') = 74.0")

Help with linq to sql insert error

I have a database table with all columns to allow nulls for testing purposes. Between all of my columns I have int, varchar or bit datatypes. When I try to submit the form I get the following error message:
Value was either too large or too small for an Int32.
Here is the code:
using (storeDataContext db = new storeDataContext())
{
db.Dealerssses.InsertOnSubmit(new Dealersss
{
AppFName = txtFName.Text,
AppLName = txtLName.Text,
AppTitle = ddlTitles.SelectedItem.Text,
AppPhone = Convert.ToInt32(txtPhone.Text),
AppEmail = txtEmail.Text,
AppAddress = txtAddress.Text,
AppCity = txtCity.Text,
AppState = txtState.Text,
AppZip = Convert.ToInt32(txtZip.Text),
BusName = txtBusName.Text,
BusCA = Convert.ToInt32(txtBusResale.Text),
BusContact = txtBusContact.Text,
BusDBA = txtDBA.Text,
BusEIN = Convert.ToInt32(txtBusEIN.Text),
BusEmail = txtBusEmail.Text,
BusFax = Convert.ToInt32(txtBusFax.Text),
BusMonth = ddlMonthStart.SelectedItem.Text,
BusNumEmployees = Convert.ToInt32(txtBusEmployees.Text),
BusPhone = Convert.ToInt32(txtBusPhone.Text),
BusYear = int.Parse(txtYearStart.Text),
Active = false
});
db.SubmitChanges();
};

Int32.MaxValue is 2,147,483,647, which is only 10 digits long.
Your values are too large for an Int32.
Your Phone, Fax, ZIP, and EIN fields should be strings (NVARCHARs), not numbers.

I'm betting it's the value for your phone number fields. Int32 is a 4 byte integer with a max value of 2147483647. Most phone numbers will overflow that.

My guess is it is on your phone #'s Int32's have a value of:
-2,147,483,648 to 2,147,483,647
So if you have the phone # of 517111111 (5,171,111,111), you are too large.
You should use varchar/char for phone numbers.