I wrote a program and it's working fine. But the thing is, It's taking much time to process.
I'm having more then millions data in a data table. So really its very hard to wait till the process complete. I hear something about LinQ method. But i'm not familiar in that. Can any one help me to speed up my code.
foreach (var ch in lsCh)
{
foreach (var dt in lsDt)
{
foreach (var cs in lsCs)
{
DataRow[] result = Dtsrc.Select("Channel = '" + ch.ToString() + "' AND Date = '" + dt.ToString() + "' AND CPSTime = '" + cs + "'");
if (result.Length > 0)
{
//Some calculation
}
Dtsrc.AcceptChanges();
}
}
}
Since the types of the DataColumns and the lists are not clear you might have to change them accordingly. But maybe this shows you a more efficient way anyway:
I would use HashSet<T> instead of List<T>. If you can't change the type in general you could create instances by passing the lists to the constructor:
var setCh = new HashSet<int>(lsCh);
var setDt = new HashSet<DateTime>(lsDt);
var setCs = new HashSet<DateTime>(lsCs);
Now this query should be much more efficient:
var rowsToProcess = Dtsrc.AsEnumerable()
.Where(r => setCh.Contains(r.Field<int>("Channel")) && setDt.Contains(r.Field<DateTime>("Date")) && setCs.Contains(r.Field<DateTime>("CPSTime")));
foreach (DataRow row in rowsToProcess)
{
//Some calculation
}
Related
I'm new to Dapper, and writing a query that will pull from a provided schema and table, along with using dynamic ordering and filtering.
Dapper make dynamic parameters very simple, however, I'm not sure how to do this with tables in the order by and where clauses. Here's my method below, and I see the issues with SQL injection:
public GridData GetGridData(string schema, string table, TableDataParameters tableDataParameters)
{
using (var dbConnection = VarConnection)
{
dbConnection.Open();
if (!this.TableExists(dbConnection, schema, table))
{
throw new ItemNotFoundException($"Could not locate table {schema}.{table}.");
}
string orderyByClause = string.Join(",", tableDataParameters.SortModel.Select(s => $"[{s.ColId}] {(s.Sort.ToLower() == "asc" ? "asc" : "desc")}"));
var parameters = new DynamicParameters();
string whereClause;
if (tableDataParameters.FilterModel == null || !tableDataParameters.FilterModel.Any())
{
whereClause = "1=1";
}
else
{
whereClause = string.Join(" AND ", tableDataParameters.FilterModel.Select((fm, i) =>
{
string whereParam = $"whereParam{i}";
parameters.Add(whereParam, fm.Filter);
if (fm.Operation == "startsWith")
{
return $"[{fm.Column}] LIKE #{whereParam} + '%'";
}
throw new InvalidOperationException($"Unsupported filter operation '{fm.Operation}'");
}));
}
var query = $"SELECT COUNT(1) [total] " +
$"FROM [{schema}].[{table}] " +
$"WHERE {whereClause} " +
$"SELECT * " +
$"FROM [{schema}].[{table}] " +
$"WHERE {whereClause} " +
$"ORDER BY {orderyByClause} " +
$"OFFSET {tableDataParameters.StartIndex.Value} ROWS " +
$"FETCH NEXT {tableDataParameters.StopIndex.Value - tableDataParameters.StartIndex.Value} ROWS ONLY";
int total = 0;
using (var reader = dbConnection.ExecuteReader(query, parameters))
{
// First batch, it's the count
if (reader.Read())
{
total = reader.GetInt32(0);
}
var gridColumns = new List<GridColumn>();
var gridRows = new List<string[]>();
if (reader.NextResult() && reader.Read())
{
for (int i = 0; i < reader.FieldCount; i++)
{
string key = reader.GetName(i);
gridColumns.Add(new GridColumn(key, key, null, ""));
}
var items = new object[reader.FieldCount];
reader.GetValues(items);
gridRows.Add(items.Select(i => i.ToString()).ToArray());
}
while (reader.Read())
{
var items = new object[reader.FieldCount];
reader.GetValues(items);
gridRows.Add(items.Select(i => i.ToString()).ToArray());
}
return new GridData(tableDataParameters.StartIndex.Value, tableDataParameters.StopIndex.Value, total, gridRows.Count(), gridColumns.ToArray(), gridRows.ToArray());
}
}
}
Should I use something like DbCommandBuilder.QuoteIdentifier, https://msdn.microsoft.com/en-us/library/system.data.common.dbcommandbuilder.quoteidentifier(v=vs.110).aspx
in this case? That doesn't seem like it would help so much here.
Thanks!
Dynamic parameters is an oxymoron! Dapper makes parameters easy, but you can't paramaterize table and column names. This is a restriction of SQL, not dapper. If you really want to do this, you have to use dynamic sql and string methods, and you're on your own as regards SQL injection.
You will be happier and live longer if you don't do this. It's just a bad road. You're not adding much value, and you're potentially introducing a load of problems and limitations.
It looks like you're writing an app to browse a database. Good tools already exist for this!
I want to make the LINQ query in this method faster:
public string GeneraCodiceListaEventi(DateTime data)
{
string codice = String.Empty;
string[] range1 = new string[] { "08:00", "08:30", "09:00", "09:30", "10:00", "10:30", "11:00", "11:30", "12:00", "12:30", "13:00", "13:30", "14:00", "14:30", "15:00", "15:30", "16:00", "16:30", "17:00", "17:30", "18:00", "18:30", "19:00", "19:30", "20:00" };
string[] range2 = new string[] { "08:29", "08:30", "09:29", "09:59", "10:29", "10:59", "11:29", "11:59", "12:29", "12:59", "13:29", "13:59", "14:29", "14:59", "15:29", "15:59", "16:29", "16:59", "17:29", "17:59", "18:29", "18:59", "19:29", "19:59", "20:29" };
using (DatabaseDataContext contestoDB = new DatabaseDataContext())
{
contestoDB.ObjectTrackingEnabled = false;
for(int i=0; i<25; i++)
{
var eventi = (from db in contestoDB.Eventi
where db.DataPrenotazione.Date == data.Date && (db.DataPrenotazione.TimeOfDay >= TimeSpan.Parse(range1[i]) && db.DataPrenotazione.TimeOfDay <= TimeSpan.Parse(range2[i]))
select new
{
ID = db.ID,
IDCliente = db.IDCliente,
Note = db.Note,
Ore = db.DataPrenotazione.ToShortTimeString()
});
if (eventi.Any())
{
codice += "<li><span class='ora'>" + range1[i] + "</span><input type='checkbox' id='item-" + GetNumItem(range1[i]) + "'/><label for='item-" + GetNumItem(range1[i]) + "'>Espandi</label><ul>";
foreach (var e in eventi)
{
codice += "<li class='app'> " + e.Ore + " - " + GetNominativoClienteDaID(e.IDCliente) + CheckNota(e.Note);
}
codice += "</ul></li>";
}
else
{
codice += "<li><span class='ora'>" + range1[i] + "</span>" + noapp + "</li>";
}
}
}
return codice;
}
In this function I'll build a string to send to html using ajax and show in the browser. But how can I make the query faster? Is there an alternative?
i can give you at least 2 suggestion to get this method execute faster:
1) Instead of using 'string codice = String.Empty;' use a StringBuilder instead, like this: 'StringBuilder longString = new StringBuilder();'
You might need to add System.Text to the using references on top of your code file. StringBuilder is much faster then using a common string because of many reason you can read up here:
String vs. StringBuilder
2) Instead of using 2 arrays of string and then parsing those string EACH TIME You loop into Timespan objects, you should instead create 2 arrays of TIMESPANS. At the moment, you are converting strings to timespan 2 times (2 array) * 25 times (your loop), so, those are 50 conversion you don't need to do.
This is how you could slightly optimize your code.
Then, to optimize the db access, you should do only 1 query with all the results, and then building the html splitting the results through code.
More queries = more time
Other than 3dd's point, the speed of this query has almost nothing to do with your C# code, and everything to do with the database.
If Eventi and DataPrenotazione are both large and not correctly indexed, then the query will run slowly. It is best explored in SQL, and analysed with whatever tools your database provides for understanding query performance.
I've got a problem and no idea how to solve it in SQL server 2012 ,
I have table with +5,000,000 row record ( ID , Link ),
but when I queryselect N row shows error "Request timed out."
Code:
public void _Read()
{
LinQDataContext _DB = new LinQDataContext();
var _img = _DB.HotelImages.Where(o => o.ID_Hotel == Hotel_DA.ID_Hotel);
string _imghtml = "";
foreach (var item in _img)
{
_imghtml = _imghtml + "<a href=''><img src='" + item.Url_Image + "' alt=''></a>";
}
}
A covering index combined with fetching only the required columns (as already suggested) will result in the best performance improvement:
Index:
CREATE INDEX IX_ID_Hotel ON YourTable(ID_Hotel)
INCLUDE (Url_Image)
You should also make sure you context is disposed. And i suggest you to use a string builder instead of concatenanting strings.
string html;
var builder = new StringBuilder();
using (LinQDataContext _DB = new LinQDataContext())
{
var urls = _DB.HotelImages.Where(o => o.ID_Hotel == Hotel_DA.ID_Hotel)
.Select(x => x.Url_Image);
foreach (var url in urls)
{
builder.Append("<a href=''><img src='")
.Append(url)
.Append("'")
.Append("alt=''></a>");
}
}
html = builder.ToString();
Currently you are fetching complete HotelImage objects. Maybe they also contain blobs? You should narrow down the data you fetch from the database by selecting only Url_Image, because that's the only property you use:
var _img = _DB.HotelImages.Where(o => o.ID_Hotel == Hotel_DA.ID_Hotel)
.Select(x => x.Url_Image);
And then
_imghtml = _imghtml + "<a href=''><img src='" + item + "' alt=''></a>";
I have this method that should take an unknown amount of id's.
I got this method almost done but it isnt secure yet for obvious reasons, i know i could write my own method to strip the parameters but i would be more comfortable by using some build in method for this.
Here is the method
public static List<LocationModel> FetchCitiesByAreas(IEnumerable<string> areas)
{
using (var db = new BoligEnt())
{
var sqlQuery = new StringBuilder();
var first = true;
sqlQuery.Append("SELECT DISTINCT a.city AS City, a.zip AS Zip ");
sqlQuery.Append("FROM zip_city AS a ");
sqlQuery.Append("WHERE country = 1 ");
foreach (var d in areas)
{
if (first)
{
sqlQuery.Append("AND a.area_id = '" + d + "'");
first = false;
}
else
{
sqlQuery.Append("OR a.area_id = '" + d + "'");
}
}
return db.Database.SqlQuery<LocationModel>(sqlQuery.ToString()).ToList();
}
}
i know it have this function built in but as i stated earlier i dont know the exact amount of ids that will come in
db.Database.SqlQuery<LocationModel>("SELECT * FROM table WHERE id = #p0 ;", id).ToList();
Thanks
While I completely agree with paqogomez, in that you should just use LINQ to do the query, the .SqlQuery has the ability to take a parameter array. You could change your statement to look like this:
var sqlQuery = new StringBuilder();
sqlQuery.Append("SELECT DISTINCT a.city AS City, a.zip AS Zip ");
sqlQuery.Append("FROM zip_city AS a ");
sqlQuery.Append("WHERE country = 1 ");
for (int i = 0; i < areas.Count; i++)
{
if (i == 0)
{
sqlQuery.Append("AND (a.area_id = #p" + i.ToString());
}
else
{
sqlQuery.Append(" OR a.area_id = #p" + i.ToString());
}
}
sqlQuery.Append(")");
var results = db.Database.SqlQuery<LocationModel>(sqlQuery.ToString(), areas.ToArray()).ToList();
I added the missing parenthesis needed to your query to correctly filter out the OR results as well. I've also taken the assumption that areas is something like a List, or at least something you can easily get the count from.
Why dont you just use Linq?
var locations = (from zip in db.zip_city
where areas.Contains(zip.area_id) && zip.Country == 1
select new LocationModel{
City = zip.City,
Zip = zip.Zip
})
.Distinct()
.ToList();
If you still want to parameterize your query, you need to use EntityCommand
Also note that your query will fail because you havent put parenthesis around your OR statements.
I suggest structuring your sql like this:
string sqlQuery =
#"SELECT DISTINCT a.city AS City, a.zip AS Zip
FROM zip_city AS a
WHERE country = 1 AND (1=0 "
for (int i = 0; i < areas.Count; i++)
{
sqlQuery.Append("OR a.area_id = #d" + i.ToString() + " ");
}
sqlQuery.Append(")");
So i have been learning Linq to SQL recently and I got it to query out record, but for the like of me, I couldn't figure out how to change the table's field. I really appreciate your help guys:
using (var db = new DataClasses1DataContext())
{
var ns = from nod in db.Nodes
where nod.Name == "MFMN"
select nod;
int pop = db.Nodes.Count();
foreach (var n in ns)
{
Console.WriteLine(n.NodeID + " " + n.lkpNodeType + " " + n.Name + " " + n.DeploymentID);
}
}
Just create an anonymous type like:
var ns = from nod in db.Nodes
where nod.Name == "MFMN"
select new
{
Identification = nod.NodeID,
//all other fields
};
and then:
foreach (var n in ns)
{
Console.WriteLine(n.Identification..... //rest of the fields
}
But, you have to specify all your fields that you need in your code. It is also useful when you only want to select specific columns and not all of the columns.