Develop Reference

How to reduce time taken to read 40000 records in SQL DB hosted in a different region

I have a SP which fetches multiple result sets.
while iterating through the result set, one of the result gives 40000 records in result set.
Process of converting the result set to List , it takes couple of minutes to process the list, which is not desirable.
Is there a way to improve the way we can read data quicker, faster?
var testEquipments = dataReader.GetList<TestEquipmentData>().ToList();
public static IEnumerable<T> GetList<T>(this DbDataReader reader) where T : new()
{
var properties = typeof(T).GetProperties();
var modelProperties = new List<string>();
var columnList = reader.GetSchemaTable().Select().Select(r => r.ItemArray[0].ToString());
while (reader.Read())
{
var element = Activator.CreateInstance<T>();
IDictionary<string, string> databaseMappings = DBColumn<T>();
string columnName;
foreach (var f in properties)
{
if (!columnList.Contains(f.Name) && !databaseMappings.ContainsKey(f.Name))
{
continue;
}
columnName = databaseMappings.ContainsKey(f.Name) ? databaseMappings[f.Name] : f.Name;
var o = (object)reader[columnName];
if (o.GetType() != typeof(DBNull))
{
f.SetValue(element, ChangeType(o, f.PropertyType), null);
}
}
yield return element;
}
}

evaluate columnList once instead of reevaluating it for every row
don't create databaseMappings for every row
don't use reflection
don't use linq
the first two are most important.
3. and 4. are ok as long as you do it once only and not in the loop.
hint: maybe check out dapper or something like that.
also ... don't select() twice to get the columnList?

Convert dynamically shaped query results into IEnumerabale or IQueryable

I am attempting to take the results of a Stored Procedure that that contains dynamic columns, different shapes of data, and convert that into an IEnumerable or IQueryable.
In the middleware, I am using EF and since I have never worked with dynamic queries, I am at a loss. I looked at the DataTable.ToSchemaTable() to see if that could convert to anything useful and hit a wall. Recently, I started looking into dynamics, however, I still can't seem to figure out how to convert to IEnumerable.
The clientside needs to have data in the form of {"FieldName":"FieldValue"}, and the result set needs to be pre-processed via an IEnumerable prior to being converted to JSON.

Here is the code I am working with that solved the issue.
public static class DataTableExtensions
{
public static IEnumerable<dynamic> AsDynamicEnumerable(this DataTable table)
{
if (table == null)
{
yield break;
}
foreach (DataRow row in table.Rows)
{
IDictionary<string, object> dRow = new ExpandoObject();
foreach (DataColumn column in table.Columns)
{
var value = row[column.ColumnName];
dRow[column.ColumnName] = Convert.IsDBNull(value) ? null : value;
}
yield return dRow;
}
}
}

Revisiting IEnumerable to DataTable extension method -- Issue with strings

I am working on a "ToDataTable()" that I can use on IEnumerable. Lots of examples of that, such as here: Convert IEnumerable to DataTable. DataTables are desirable for me because I am coming from a Foxpro realm where we are used to cursors. So, I dig DataTables because they are relatively simple, easy to work with, and can house decent column meta-data. And I've already got a bunch of code that can display them nicely (with sort and filter grid headers), export to Excel, etc.
Someone made a really good comment on the thread I linked to above: "This doesn't work if the type is string as the properties are chars and length but the get value then tries to put the full string into the char column."
Problem is, certain things fit the extension method call (like a one-dimensional array of strings) but then throw errors when the conversion takes place. I wrote an extension method that adds IsValueType() to the mix, but the problem is that returns false for strings. Finally, I added two kludges: a parameter to force the item in the sequence to be treated as a value, and to treat the item in the sequence as a value if it is found to be of type "String".
Is there a better way of doing this, or are strings just an odd man out due to their unique breed of typing and reflection results? The following code works with every IEnumerable I can think of, and it works with one-dimensional arrays for all DataTable column types except for string (well, it works for strings with the hard-coded kludge, but would error out otherwise). It's not a big deal to hard-code the string scenario, I just wish there was a more elegant way to do this. Any thoughts?
public static DataTable ToDataTable<T>(this IEnumerable<T> items, string tableName = "", bool treatItemAsValue = false)
{
// We want a single extension method that can take in an enumerable sequence (such as a LINQ query)
// and return the result as a DataTable. We want this to be a one stop shop for converting
// various objects into DataTable format, as DataTables are a nice parallel to Foxpro cursors.
if (items == null) { return null; }
Type itemType = typeof(T);
bool typeIsNullable = itemType.IsGenericType && typeof(T).GetGenericTypeDefinition().Equals(typeof(Nullable<>));
string itemTypeName = "";
bool typeIsValue = false;
Type itemUnderlyingType = itemType;
if (typeIsNullable)
{
// Type of enumerable item is nullable, so we need to find its base type.
itemUnderlyingType = Nullable.GetUnderlyingType(itemType);
}
typeIsValue = itemUnderlyingType.IsValueType;
itemTypeName = itemUnderlyingType.Name;
DataTable dt = new DataTable();
DataColumn col = null;
if ((treatItemAsValue) || (itemTypeName == "String"))
{
// We have been asked to treat the item in the sequence as a value, of the items
// in the sequence are strings. Strings are NOT considered a value type in regards
// to IsValueType(), but when item values are assessed, it will be the value
// of the string that tries to pull in.
typeIsValue = true;
}
if (itemTypeName == "DataRow")
{
// Special case. If our enumerable type is DataRow, then we can utilize a more appropriate
// (built-in) extension method to convert enumerable DataRows to a DataTable.
dt = ((IEnumerable<DataRow>)items).CopyToDataTable();
}
else
{
// We must have an enumerable sequence/collection of some other type, possibly anonymous.
// Get properties of the enumerable to add as columns to the data table.
if (typeIsValue)
{
// Our enumerable items are of a value type (e.g. integers in a one-dimensional array).
col = dt.Columns.Add();
col.AllowDBNull = typeIsNullable;
col.ColumnName = itemTypeName;
col.DataType = itemUnderlyingType;
// Now walk through the enumeration and add rows to our data table (single values).
foreach (var item in items)
{
dt.Rows.Add(item);
}
}
else
{
// The type should be something we can walk through the properties of in order to
// generate properly named and typed columns of our DataTable.
PropertyInfo[] props = typeof(T).GetProperties(BindingFlags.Public | BindingFlags.Instance);
foreach (var prop in props)
{
Type propType = prop.PropertyType;
// Is it a nullable type? Get the underlying type.
if (propType.IsGenericType && propType.GetGenericTypeDefinition().Equals(typeof(Nullable<>)))
{
propType = new NullableConverter(propType).UnderlyingType;
}
dt.Columns.Add(prop.Name, propType);
}
// Now walk through the enumeration and add rows to our data table.
foreach (var item in items)
{
var values = new object[props.Length];
for (int i = 0; i < props.Length; i++)
{
values[i] = props[i].GetValue(item, null);
}
dt.Rows.Add(values);
}
}
}
// Give the DataTable a reasonable name.
if (tableName.Length == 0)
{
if (typeof(T).IsAnonymous())
{
// Anonymous types have really goofy names, so there is no use using that as table name.
tableName = "Anonymous";
}
else
{
// This is NOT an anonymous type, so we can use the type name as table name.
tableName = typeof(T).Name;
}
}
return dt;
}

Well, no responses except a comment that pretty much missed the point, so I'll just post what I ended up with. This final version also accounts for items with no properties (such as "Object" itself), and does some better handing of null items/values:
public static DataTable ToDataTable<T>(this IEnumerable<T> items, string tableName = "", bool treatItemAsValue = false)
{
// We want a single extension method that can take in an enumerable sequence (such as a LINQ query)
// and return the result as a DataTable. We want this to be a one stop shop for converting
// various objects into DataTable format, as DataTables are a nice parallel to Foxpro cursors.
if (items == null) { return null; }
Type itemType = typeof(T);
bool typeIsNullable = itemType.IsGenericType && typeof(T).GetGenericTypeDefinition().Equals(typeof(Nullable<>));
string itemTypeName = "";
bool typeIsValue = false;
Type itemUnderlyingType = itemType;
if (typeIsNullable)
{
// Type of enumerable item is nullable, so we need to find its base type.
itemUnderlyingType = Nullable.GetUnderlyingType(itemType);
}
typeIsValue = itemUnderlyingType.IsValueType;
itemTypeName = itemUnderlyingType.Name;
DataTable dt = new DataTable();
DataColumn col = null;
PropertyInfo[] props = typeof(T).GetProperties(BindingFlags.Public | BindingFlags.Instance);
if ((treatItemAsValue) || (itemTypeName == "String") || (props.Length == 0))
{
// We have been asked to treat the item in the sequence as a value, or the items
// in the sequence is a string which cannot be "flattened" properly by analyzing properties.
// OR, the type has no properties to put on display, so we should just use the item directly.
// (like the base "Object" type).
typeIsValue = true;
}
if (itemTypeName == "DataRow")
{
// Special case. If our enumerable type is DataRow, then we can utilize a more appropriate
// (built-in) extension method to convert enumerable DataRows to a DataTable.
dt = ((IEnumerable<DataRow>)items).CopyToDataTable();
}
else
{
// We must have an enumerable sequence/collection of some other type, possibly anonymous.
// Get properties of the enumerable to add as columns to the data table.
if (typeIsValue)
{
// Our enumerable items are of a value type (e.g. integers in a one-dimensional array).
col = dt.Columns.Add();
// Whether or not the type is nullable, the value might be null (e.g. for type "Object").
col.AllowDBNull = true;
col.ColumnName = itemTypeName;
col.DataType = itemUnderlyingType;
// Now walk through the enumeration and add rows to our data table (single values).
foreach (var item in items)
{
dt.Rows.Add(item);
}
}
else
{
// The type should be something we can walk through the properties of in order to
// generate properly named and typed columns of our DataTable.
foreach (var prop in props)
{
Type propType = prop.PropertyType;
// Is it a nullable type? Get the underlying type.
if (propType.IsGenericType && propType.GetGenericTypeDefinition().Equals(typeof(Nullable<>)))
{
propType = new NullableConverter(propType).UnderlyingType;
}
dt.Columns.Add(prop.Name, propType);
}
// Now walk through the enumeration and add rows to our data table.
foreach (var item in items)
{
if (item != null)
{
// Can only add an item as a row if it is not null.
var values = new object[props.Length];
for (int i = 0; i < props.Length; i++)
{
values[i] = props[i].GetValue(item, null);
}
dt.Rows.Add(values);
}
}
}
}
// Give the DataTable a reasonable name.
if (tableName.Length == 0)
{
if (typeof(T).IsAnonymous())
{
// Anonymous types have really goofy names, so there is no use using that as table name.
tableName = "Anonymous";
}
else
{
// This is NOT an anonymous type, so we can use the type name as table name.
tableName = typeof(T).Name;
}
}
return dt;
}
Hope someone finds this useful...

How to efficiently create a list of objects inside of a generic method?

So, I have an application which lies on a database. So far, the results of my queries all went into a DataTable object like this:
DataTable data = new DataTable();
data.Load(someQuery.ExecuteReader());
Now, I want to load my data into a list of a strongly typed objects. Something like this:
List<MyClass> data = someQuery.Load<MyClass>();
However, my first take on writing that method ended up running almost three times slower than DataTable.Load(IDataReader) method. Basically, I have user GetConstructor(null).Invoke(null) to create and object and I have used PropertyInfo.SetValue(reader.GetValue()) to fill it with data.
Is there a better way to do this?
The method used:
public List<T> LoadData<T>(DbCommand query)
{
Type t = typeof(T);
List<T> list = new List<T>();
using (IDataReader reader = query.ExecuteReader())
{
while (reader.Read())
{
T newObject = (T)t.GetConstructor(null).Invoke(null);
for (int ct = 0; ct < reader.FieldCount; ct++)
{
PropertyInfo prop = t.GetProperty(reader.GetName(ct));
if (prop != null)
prop.SetValue(newObject, reader.GetValue(ct), null);
}
list.Add(newObject);
}
}
return list;
}

To do this efficiently requires metaprogramming. You can use libraries to help. For example, "FastMember" includes a TypeAccessor which provides fast access to instance creation and member-access by name. However, this example is also basically exactly how "dapper" works, so you could just use dapper:
int id = ...
var data = connection.Query<Order>(
"select * from Orders where CustomerId = #id",
new { id }).ToList();
You can also open up the "dapper" code to see what it does.

You can execute your query using linQ and get the Generic List and then if you want to conver it to DataTable then use the following Code, it may help you.
public DataTable ListToDataTable<T>(IEnumerable<T> list)
{
PropertyDescriptorCollection properties =
TypeDescriptor.GetProperties(typeof(T));
DataTable table = new DataTable();
foreach (PropertyDescriptor prop in properties)
table.Columns.Add(prop.Name, Nullable.GetUnderlyingType(prop.PropertyType) ?? prop.PropertyType);
foreach (T item in list)
{
DataRow row = table.NewRow();
foreach (PropertyDescriptor prop in properties)
row[prop.Name] = prop.GetValue(item) ?? DBNull.Value;
table.Rows.Add(row);
}
return table;
}
it will work for any strongly type class. Please check the time it takes to execute.
Thanks,

Alternatives to using property.setvalue() for efficiency purposes

What alternatives are there to property.setvalue()? I've read that it is very slow. I'm using it to map IDataReader to POCO objects.
This is a truncated version of the code. Everything here is very new to me. I know there a lot of frameworks that accomplish this task. However, we can't use them.
public class DbAutoMapper<T>
{
public IEnumerable<T> MapToList(IDataReader reader)
{
var list = new List<T>();
while (reader.Read())
{
var obj = Activator.CreateInstance<T>();
foreach (PropertyInfo prop in obj.GetType().GetProperties())
{
foreach (var attribute in prop.GetCustomAttributes(true))
{
prop.SetValue(obj, value, null);
}
}
list.Add(obj);
}
return list;
}
}

Firstly: why are you repeating the reflection for every attribute, when you don't use the attribute?
Second: assuming you intended to map this by name, column-to-property (which isn't what the code currently does), consider a tool like dapper, which does all this for you, including cached high-performance reflection-emit. It'll also handle the command itself for you. For example:
string region = "North";
var customers = conn.Query<Customer>(
#"select * from Customers where Region = #region",
new { region } // full parameterization, the easy way
).ToList();
If you need more control, consider FastMember, which provides fast member-access (again, reflection-emit), but without being specific to data access:
var accessor = TypeAccessor.Create(typeof(T));
string propName = // something known only at runtime
while( /* some loop of data */ ) {
var obj = new T();
foreach(var col in cols) {
string propName = // ...
object cellValue = // ...
accessor[obj, propName] = cellValue;
}
yield return obj;
}

A few approaches come to mind...
Skip Reflection
public class DbAutoMapper<T> where T : IInitFromReader, new()
{
public IEnumerable<T> MapToList(IDataReader reader)
{
var list = new List<T>();
while (reader.Read())
{
IInitFromReader obj = new T;
obj.InitFromReader(reader);
list.Add(obj);
}
return list;
}
}
Then you'll have to implement the InitFromReader in each of your entitiy objects. Obviously, this skips the benefits of reflection (less code).
Code Generation
Maintaining this code for (InitFromReader) is painful, so you could opt to generate it. This in many ways gives you the best of both worlds:
You don't have to maintain (by hand) a lot of code
You don't take the performance hit of reflection.