This may seem strange at first but I hope it makes sense! I have a list<string> of mysql insert statements that I iterate through and execute on a one by one basis, this works ok but what I am trying to achieve is a performance boost.
I would like to pass all the values in a single query for example instead of
INSERT INTO tb1(field1,field2,field3) VALUES (1,2,3);
INSERT INTO tb1(field1,field2,field3) VALUES (4,5,6);
INSERT INTO tb1(field1,field2,field3) VALUES (7,8,9);
I would like to do
INSERT INTO tb1(field1,field2,field3) VALUES (1,2,3), (4,5,6), (7,8,9);
so basically I need to build a long query string keeping the first query as a starting point and appending the bracket section from values to the end for whatever the count of objects in the list may be - is this possible?
This should work. I'm using regex to find the columns and values and string.Substring to find the table-name. Then i put all together with string.Format and string.Join and a simple Linq query:
IEnumerable<string> inserts = // add your insert-strings here;
string firstInsert = inserts.First();
int tableIndex = firstInsert.IndexOf("INSERT INTO ") + "INSERT INTO ".Length;
string table = firstInsert.Substring(
tableIndex, firstInsert.IndexOf("(", tableIndex) - tableIndex);
var regex = new System.Text.RegularExpressions.Regex(#"\(([^)]+)\)",System.Text.RegularExpressions.RegexOptions.Compiled);
string columns = regex.Matches(firstInsert)[0].Value;
IEnumerable<string> values = inserts.Select(sql => regex.Matches(sql)[1].Value);
string insertAll = string.Format("INSERT INTO {0}{1} VALUES {2};"
, table
, columns
, string.Join(",", values));
DEMO with your sample data.
Just in case you might be inserting to different tables (or sets of columns within tables),
var perTableQueries = separateQueries
.Select(s => s.Split(new[] {" VALUES "}, StringSplitOptions.None))
.Where(a => a.Length == 2)
.GroupBy(a => a[0], StringComparer.InvariantCultureIgnoreCase)
.Select(g => String.Format("{0} VALUES {1};",
g.Key,
String.Join(",", g.Select(a => a[1].TrimEnd(';')).ToArray())));
will make a single insert per table/column set (if your SQL is consistent).
That said, if you're making a separate request to the database for each line, your biggest performance saving might just be
var singleRequest = String.Join("", perTableQueries.ToArray());
Related
I need to select rows from with a string of IDs that are not the PK and int value x,
and datatable.select() takes to much time. Is there faster way to select them.
this is my code:
String a = "Id1, id2, ...";
meldunng.Select("columnname1 = " + 2 " AND columnname2 IN (" + a + ")")
If my question is confusing or I have worded it wrong please tell me.
Thanks in andvance
You could use a HashSet<string> to improve performance and LINQ for the query:
var ids = new HashSet<string>(new[] {"Id1", "id2", "id3"}, StringComparer.OrdinalIgnoreCase); // i guess you want to ignore the case
var matchingRows = meldunng.AsEnumerable()
.Where(row => row.Field<int>("columnname1") == 2 && ids.Contains(row.Field<string>("columnname2")));
Note that this is just the LINQ query and you need to "materialize" it to something, for example a DataRow[]:
DataRow[] resultRows = matchingRows.ToArray();
I'm curious how I can compare two data tables in C#. I have two data tables, data table one contains FirstName and LastName, data table 2 has Field1, Field2, First_Name, and Last_Name.
I want to find records that exist in data table 1 that do not exist in data table 2. Anyone ever done this before? Any help would be appreciated. Thank you!
Using LINQ would be most natural, but you will need to convert away from the DataTable to use Except.
var In_dt1_only = dt1.AsEnumerable().Select(r => new { first = r.Field<string>("First"), last = r.Field<string>("Last")}).Except(dt2.AsEnumerable().Select(r => new { first = r.Field<string>("First"), last = r.Field<string>("Last")}));
If you need the original DataRows, you can use a Where instead:
var datarows_in_dt1_only = dt1.AsEnumerable().Where(dr1 => !dt2.AsEnumerable().Any(dr2 => dr1.Field<string>("First") == dr2.Field<string>("First") && dr1.Field<string>("Last") == dr2.Field<string>("Last")));
I have an array or string:
private static string[] dataNames = new string[] {"value1", "value2".... };
I have table in my SQL database with a column of varchar type. I want to check which values from the array of string exists in that column.
I tried this:
public static void testProducts() {
string query = "select * from my table"
var dataTable = from row in dt.AsEnumerable()
where String.Equals(row.Field<string>("columnName"), dataNames[0], StringComparison.OrdinalIgnoreCase)
select new {
Name = row.Field<string> ("columnName")
};
foreach(var oneName in dataTable){
Console.WriteLine(oneName.Name);
}
}
that code is not the actual code, I am just trying to show you the important part
That code as you see check according to dataNames[index]
It works fine, but I have to run that code 56 times because the array has 56 elements and in each time I change the index
is there a faster way please?
the Comparison is case insensitive
First, you should not filter records in memory but in the datatabase.
But if you already have a DataTable and you need to find rows where one of it's fields is in your string[], you can use Linq-To-DataTable.
For example Enumerable.Contains:
var matchingRows = dt.AsEnumerable()
.Where(row => dataNames.Contains(row.Field<string>("columnName"), StringComparer.OrdinalIgnoreCase));
foreach(DataRow row in matchingRows)
Console.WriteLine(row.Field<string>("columnName"));
Here is a more efficient (but less readable) approach using Enumerable.Join:
var matchingRows = dt.AsEnumerable().Join(dataNames,
row => row.Field<string>("columnName"),
name => name,
(row, name) => row,
StringComparer.OrdinalIgnoreCase);
try to use contains should return all value that you need
var data = from row in dt.AsEnumerable()
where dataNames.Contains(row.Field<string>("columnName"))
select new
{
Name = row.Field<string>("columnName")
};
Passing a list of values is surprisingly difficult. Passing a table-valued parameter requires creating a T-SQL data type on the server. You can pass an XML document containing the parameters and decode that using SQL Server's convoluted XML syntax.
Below is a relatively simple alternative that works for up to a thousand values. The goal is to to build an in query:
select col1 from YourTable where col1 in ('val1', 'val2', ...)
In C#, you should probably use parameters:
select col1 from YourTable where col1 in (#par1, #par2, ...)
Which you can pass like:
var com = yourConnection.CreateCommand();
com.CommandText = #"select col1 from YourTable where col1 in (";
for (var i=0; i< dataNames.Length; i++)
{
var parName = string.Format("par{0}", i+1);
com.Parameters.AddWithValue(parName, dataNames[i]);
com.CommandText += parName;
if (i+1 != dataNames.Length)
com.CommandText += ", ";
}
com.CommandText += ");";
var existingValues = new List<string>();
using (var reader = com.ExecuteReader())
{
while (read.Read())
existingValues.Add(read["col1"]);
}
Given the complexity of this solution I'd go for Max' or Tim's answer. You could consider this answer if the table is very large and you can't copy it into memory.
Sorry I don't have a lot of relevant code here, but I did a similar thing quite some time ago, so I will try to explain.
Essentially I had a long list of item IDs that I needed to return to the client, which then told the server which ones it wanted loaded at any particular time. The original query passed the values as a comma separated set of strings (they were actually GUIDs). Problem was that once the number of entries hit 100, there was a noticeable lag to the user, once it got to 1000 possible entries, the query took a minute and a half, and when we went to 10,000, lets just say you could boil the kettle and drink your tea/coffee before it came back.
The answer was to stick the values to check directly into a temporary table, where one row of the table represented one value to check against. The temporary table was keyed against the user who performed the search, so this meant other users searches wouldn't become corrupted with each other, and when the user logged out, then we knew which values in the search table could be removed.
Depending on where this data comes from will depend on the best way for you to load the reference table. But once it is there, then your new query will look something like:-
SELECT Count(t.*), rt.dataName
FROM table t
RIGHT JOIN referenceTable rt ON tr.dataName = t.columnName
WHERE rt.userRef = #UserIdValue
GROUP BY tr.dataName
The RIGHT JOIN here should give you a value for each of your reference table values, including 0 if the value did not appear in your table. If you don't care which one don't appear, then changing it to an INNER JOIN will eliminate the zeros.
The WHERE clause is to ensure that your search only returns the unique items that you are looking for at the moment - the design should consider that concurrent access will someday occur here (even if it doesn't at the moment), so writing something in to protect it is advisable.
I am getting data like 200k records from the database and store it in a linq result with a ColumnName EMAIL. Now,I want to show all emails from the linq result and adding them to a TextBox by separating with a ,.
Actually,I have prepared DataTable with that linq result and have combined all row data with the code :
var dataLists = (from xx in VDC.SURVEY_EMAIL_LIST
where xx.EMAIL_GROUP_ID == ListGroupID
select xx).ToList();
DataTable DtDataLists = LINQToDataTable(dataLists);
EmailIDS = string.Join(",", DtDataLists.AsEnumerable().Select(x => x["EMAILID"].ToString()).ToArray());
But,for preparing DataTable,it is taking a long time.
So,I thought of preparing the string EmailIDS directly from the linq result.
Can anyone help me?
This code should work for you but I'm not sure that it'll be much faster:
string.Join(",", dataLists.Select(x => x.EMAILID));
I am trying to query names in one table and then use that result to pull the master records into a DataGridView. So what I need to do is get the names from the interest table that are like what is put into text boxes then use those results to pull the data from the CaseSelector table and set the bindingsource filter to those results. Why can't I seem to set results to the caseSelectorBindingSourceFilter
var results = from CaseSelector in db.CaseSelectors
from Interest in db.Interests
where SqlMethods.Like(Interest.First, txtFirst.Text) && SqlMethods.Like(Interest.Last, txtLast.Text)
select CaseSelector;
caseSelectorBindingSource.Filter = ("CaseNumberKey =" + results);
You can find examples for LINQ join queries here:
http://code.msdn.microsoft.com/101-LINQ-Samples-3fb9811b
I don't know your DB schema but you're looking for something along the lines of:
from c in db.Cases
join i in db.Interest on i.CaseNumberKey equals c.CaseNumberKey
select c
What you've just retrieved is a SQL Set of the valid caseNumberKeys.
Your casefilter should be something like
caseSelectorBindingSource.Filter = "CaseNumberKey in " + interestsresultsAsSQLSET;
You'll have to iterate over the interestsresults and convert it to the string representation of a SQLSET. If you're feeling lucky, you could just try to .toString() it and see what happens.
string interestsResultsAsSQLSet = "(";
//write most of them
for(int i=0; i<interestsresults.size() - 1; i++) {
interestsResultAsSQLSet.append(interestsresults[i] + ",");
//write the last one, with right paren
interestsresults.append(interestsresults[interestsresults.size() -1] + ")");
I write Java all day, ignore my basic language errors. :P