Is this an efficient way of bulk inserting using Dapper?
Also, is this more efficient than creating a stored procedure and passing models to it?
Is this an efficient way of bulk inserting using Dapper?
Also, is this more efficient than creating a stored procedure and passing models to it?
category = new Category
{
Name = "category",
Description = "description",
Created = null,
LastModified = null,
CategoryPictures = new CategoryPicture[]
{
new CategoryPicture
{
CategoryId = 3,
PictureId = 2,
Picture = new Picture
{
Url = "newUrl"
}
},
new CategoryPicture
{
CategoryId = 3,
PictureId = 2,
Picture = new Picture
{
Url = "url"
}
}
}
};
string sql = #"INSERT INTO Categories(Name, Description, Created, LastModified)
VALUES(#Name, #Description, #Created, #LastModified)";
await conn.ExecuteAsync(sql, new
{
category.Name,
category.Description,
category.Created,
category.LastModified
});
string catPicInsert = #"INSERT INTO CategoryPictures(fk_CategoryId, fk_PictureId)
VALUES(#CategoryId, #PictureId)";
await conn.ExecuteAsync(catPicInsert, category.CategoryPictures);
string PicInsert = #"INSERT INTO Pictures(Url)
VALUES(#Url)";
await conn.ExecuteAsync(PicInsert, category.CategoryPictures.Select(x => x.Picture).ToList());
It won't be hugely slow, but it won't be anywhere near as fast as a bulk copy. Options, assuming SQL Server:
it is possible to use TVPs with Dapper, but the only convenient way to do this is by packing your input data into a DataTable; there are examples of TVP usage in the Dapper repo, or I can knock one out, but they're inconvenient because you need to declare the parameter type at the server
you can use SqlBulkCopy to throw data into the database independent of Dapper; FastMember has ObjectReader that can construct an IDataReader over a typed sequence, suitable for use withSqlBulkCopy
If you're not using SQL Server, you'll need to look at vendor-specific options for your RDBMS.
I've changed it up into using multiple sprocs for each table.
public async Task<bool> CreateCategoryAsync(Category category)
{
category = new Category
{
Name = "category",
Description = "description",
Created = null,
LastModified = null,
CategoryPictures = new CategoryPicture[]
{
new CategoryPicture
{
CategoryId = 1,
PictureId = 2,
Picture = new Picture
{
Url = "newUrl"
}
},
new CategoryPicture
{
CategoryId = 2,
PictureId = 2,
Picture = new Picture
{
Url = "url"
}
}
}
};
string sql = #"EXEC Categories_Insert #Categories;
EXEC CategoryPictures_Insert #CategoryPictures;
EXEC Pictures_Insert #Pictures";
var spParams = new DynamicParameters(new
{
Categories = category.ToDataTable().AsTableValuedParameter("CategoriesType"),
CategoryPictures = category.CategoryPictures.ListToDataTable()
.AsTableValuedParameter("CategoryPictureType"),
Pictures = category.CategoryPictures.Select(x => x.Picture)
.ListToDataTable()
.AsTableValuedParameter("PicturesType")
});
using (var conn = SqlConnection())
{
using (var res = await conn.QueryMultipleAsync(sql, spParams))
{
return true;
}
}
}
This is the extension class with generic methods that i created for mapping objects or a list of objects into Datatable
public static class DataTableExtensions
{
/// <summary>
/// Convert an IEnumerable into a Datatable
/// </summary>
/// <typeparam name="T"></typeparam>
/// <param name="listToDataTable"></param>
/// <returns></returns>
public static DataTable ListToDataTable<T>(this IEnumerable<T> listToDataTable)
{
DataTable dataTable = new DataTable();
AddToDataTableColumns<T>(dataTable);
foreach (var item in listToDataTable)
{
AddDataTableRows(dataTable, item);
}
return dataTable;
}
/// <summary>
/// Comvert a Type of Class to DataTable
/// </summary>
/// <typeparam name="T"></typeparam>
/// <param name="obj"></param>
/// <returns></returns>
public static DataTable ToDataTable<T>(this T obj) where T : class
{
DataTable dataTable = new DataTable();
AddToDataTableColumns<T>(dataTable);
AddDataTableRows(dataTable, obj);
return dataTable;
}
/// <summary>
/// Add values from of Type T to a given Datatable columns
/// </summary>
/// <typeparam name="T">Extract values from</typeparam>
/// <param name="dataTable">The datatable to add column values into</param>
/// <returns></returns>
private static DataTable AddToDataTableColumns<T>(DataTable dataTable)
{
try
{
PropertyInfo[] FilteredProps = GetFilteredProperties(typeof(T));
for (int i = 0; i < FilteredProps.Length; i++)
{
PropertyInfo prop = FilteredProps[i];
Type type = prop.PropertyType;
dataTable.Columns.Add(prop.Name, Nullable.GetUnderlyingType(type) ?? type);
}
}
catch (Exception ex)
{
new InfrastructuredException(ex.StackTrace);
}
return dataTable;
}
/// <summary>
/// Add values from of Type T to a given Datatable Rows
/// </summary>
/// <typeparam name="T">Extract values from</typeparam>
/// <param name="dataTable">The datatable to add Row values into</param>
/// <returns></returns>
private static DataTable AddDataTableRows<T>(DataTable dataTable, T obj)
{
try
{
PropertyInfo[] FilteredProps = GetFilteredProperties(typeof(T));
object[] values = new object[FilteredProps.Length];
for (int i = 0; i < values.Length; i++)
{
values[i] = FilteredProps[i].GetValue(obj);
}
dataTable.Rows.Add(values);
}
catch (Exception ex)
{
new InfrastructuredException(ex.StackTrace);
}
return dataTable;
}
/// <summary>
/// Return an array of Filterered Properties of a Type
/// </summary>
/// <param name="type"></param>
/// <returns>Properties that are filtered by Type</returns>
private static PropertyInfo[] GetFilteredProperties(Type type)
{
return type.GetProperties()
.Where(p => p.Name != "Id" && !p.PropertyType.IsSubclassOf(typeof(BaseEntity)) && !p.PropertyType.IsInterface)
.ToArray();
}
}
I would suggest that you use Bulk Insert function of SQL.
It can be combined with Dapper as well.
I was looking for the example like this for a long time, and finally found the pieces of solution, and combined them in one place.
You can see the example in my repo:
https://github.com/ayrat162/BulkInsert
It uses Dapper.Contrib, FastMember, and SqlBulkCopy for uploading large chunks of data to MS SQL Server.
Related
EF 6, .NET 4.51
I am trying to build a generic helper class that will help me "translate" each of the result sets into a type safe class as described here Handle multiple result from a stored procedure with SqlQuery
For my solution I want to pass the following to my helper class (MultiResultsetsHelper):
Generic Return Type
ObjectContext
DataReader
List of class types in order of the result sets coming back
and then have the helper class do the heavy lifting of populating 1. Below is the code so far:
Classes For Result
public class Set1ReturnDto
{
public int CruiseCount { get; set; }
public string DisplayText { get; set; }
public int DisplayValue { get; set; }
}
public class Set2ReturnDto
{
public string DepartingFrom { get; set; }
public string Port_Code { get; set; }
}
public class DummyReturnDto
{
public DummyReturnDto()
{
Set1 = new List<Set1ReturnDto>();
Set2 = new List<Set2ReturnDto>();
}
public List<Set1ReturnDto> Set1 { get; set; }
public List<Set2ReturnDto> Set2 { get; set; }
}
Low Level Database Call
public static DummyReturnDto DonoUspGetSideBarList(DbContext aDbContext, out int aProcResult)
{
SqlParameter procResultParam = new SqlParameter { ParameterName = "#procResult", SqlDbType = SqlDbType.Int, Direction = ParameterDirection.Output };
DbCommand dbCommand = aDbContext.Database.Connection.CreateCommand();
dbCommand.Parameters.Add(procResultParam);
dbCommand.CommandText = "EXEC #procResult = [dbo].[usp_GetSideBarList] ";
dbCommand.Transaction = aDbContext.Database.CurrentTransaction.UnderlyingTransaction;
DbDataReader reader = dbCommand.ExecuteReader();
aProcResult = -1;
// Drop down to the wrapped `ObjectContext` to get access to the `Translate` method
ObjectContext objectContext = ((IObjectContextAdapter)aDbContext).ObjectContext;
List<Type> containedDtos = new List<Type>
{
typeof (List<Set1ReturnDto>),
typeof (List<Set1ReturnDto>)
};
return MultiResultsetsHelper.Process<DummyReturnDto>(reader, objectContext, containedDtos);
}
The resulting datasets returned are:
Helper Class
public static class MultiResultsetsHelper
{
/// <summary>
/// Given a data reader that contains multiple result sets, use the supplied object context to serialise the
/// rows of data in the result set into our property.
/// </summary>
/// <typeparam name="T">Type of the containing object that contains all the various result sets.</typeparam>
/// <param name="aDbReader">Database reader that contains all the result sets returned from the database.</param>
/// <param name="aObjectContext">Data context associated with the data reader.</param>
/// <param name="aContainedDataSetReturnedTypes">
/// List of types in order of which the result sets are contained within the
/// data reader. We will serilize sequentially each result set the data reader contains
/// </param>
/// <returns>Retuns an object representing all the result sets returned by the data reader.</returns>
public static T Process<T>(DbDataReader aDbReader, ObjectContext aObjectContext, List<Type> aContainedDataSetReturnedTypes) where T : new()
{
//What we will be returning
T result = new T();
for (int datasetNdx = 0; datasetNdx < aContainedDataSetReturnedTypes.Count; datasetNdx++)
{
//Advance the reader if we are not looking at the first dataset
if (datasetNdx != 0)
aDbReader.NextResult();
//Get the property we are going to be updating based on the type of the class we will be filling
PropertyInfo propertyInfo = typeof (T).GetProperties().Single(p => p.PropertyType == aContainedDataSetReturnedTypes[datasetNdx]);
//Now get the object context to deserialize what is in the resultset into our type
var valueForProperty = aObjectContext.Translate <aContainedDataSetReturnedTypes[datasetNdx]> (aDbReader);
//Finally we update the property with the type safe information
propertyInfo.SetValue(result, valueForProperty, null);
}
return result;
}
}
However currently I cannot get this to compile.
Error 2 Operator '<' cannot be applied to operands of type 'method
group' and 'System.Type'
Can someone help out? Ultimately it has to do with how we use reflection and the passed in aContainedDataSetReturnedTypes. I am happy to change things around as long as it is still easy to call MultiResultsetsHelper.Process<>()
This code:
aObjectContext.Translate<aContainedDataSetReturnedTypes[datasetNdx]>
will not work, because generic type parameters are always resolved at compile-time. You can't pass a Type instance at runtime.
You can still call the generic method, but you'll have to use reflection.
With the help of all of the above I came up with the following (which can still be improved):
public static class MultiResultsetsHelper
{
/// <summary>
/// Given a data reader that contains multiple result sets, use the supplied object context to serialise the
/// rows of data in the result set into our property.
/// </summary>
/// <typeparam name="T">Type of the containing object that contains all the various result sets.</typeparam>
/// <param name="aDbReader">Database reader that contains all the result sets returned from the database.</param>
/// <param name="aDbContext">Data context associated with the data reader.</param>
/// <param name="aDataSetTypes">Type for each type to use when we call Translate() on the current result in the data reader.</param>
/// <param name="aContainedDataSetReturnedTypes">
/// List of types in order of which the result sets are contained within the
/// data reader. We will serilize sequentially each result set the data reader contains
/// </param>
/// <returns>Retuns an object representing all the result sets returned by the data reader.</returns>
public static T Process<T>(DbDataReader aDbReader, DbContext aDbContext, List<Type> aDataSetTypes, List<Type> aContainedDataSetReturnedTypes) where T : new()
{
//What we will be returning
T result = new T();
// Drop down to the wrapped `ObjectContext` to get access to the `Translate` method
ObjectContext objectContext = ((IObjectContextAdapter) aDbContext).ObjectContext;
//Iterate the passed in dataset types as they are in the same order as what the reader contains
for (int datasetNdx = 0; datasetNdx < aContainedDataSetReturnedTypes.Count; datasetNdx++)
{
//Advance the reader if we are not looking at the first dataset
if (datasetNdx != 0)
aDbReader.NextResult();
//Get the property we are going to be updating based on the type of the class we will be filling
PropertyInfo propertyInfo = typeof (T).GetProperties().Single(p => p.PropertyType == aContainedDataSetReturnedTypes[datasetNdx]);
//Now get the object context to deserialize what is in the resultset into our type
MethodInfo method = GetTranslateOverload(typeof (ObjectContext));
MethodInfo generic = method.MakeGenericMethod(aDataSetTypes[datasetNdx]);
//Invoke the generic method which we hvae constructed for Translate
object valueForProperty = generic.Invoke(objectContext, new object[] {aDbReader});
//Finally we update the property with the type safe information
propertyInfo.SetValue(result, valueForProperty);
}
return result;
}
/// <summary>
/// Internal helper method to get the necessary translate overload we need:
/// ObjectContext.Translate<T>(DbReader)
/// </summary>
/// <param name="aType">ObjectContext.GetType()</param>
/// <returns>Returns the method we require, null on error.</returns>
private static MethodInfo GetTranslateOverload(Type aType)
{
MethodInfo myMethod = aType
.GetMethods()
.Where(m => m.Name == "Translate")
.Select(m => new
{
Method = m,
Params = m.GetParameters(),
Args = m.GetGenericArguments()
})
.Where(x => x.Params.Length == 1
&& x.Args.Length == 1
&& x.Params[0].ParameterType == typeof (DbDataReader)
// && x.Params[0].ParameterType == x.Args[0]
)
.Select(x => x.Method)
.First();
return myMethod;
}
}
So assuming you have:
public class UspGetSideBarListReturnDto
{
public List<Set1ReturnDto> Dummy1 { get; set; }
public List<Set2ReturnDto> Dummy2 { get; set; }
}
public class Set1ReturnDto
{
public Int32 CruiseCount { get; set; }
public string DisplayText { get; set; }
public Int64 DisplayValue { get; set; }
}
public class Set2ReturnDto
{
public string DepartingFrom { get; set; }
public string Port_Code { get; set; }
}
You can call it as:
DbDataReader reader = dbCommand.ExecuteReader();
return MultiResultsHelper.Process<UspGetSideBarListReturnDto>(reader, myDbContext, new List<Type>{typeof(Set1ReturnDto), typeof(Set2ReturnDto)}, new List<Type>{typeof(List<Set1ReturnDto>), typeof(List<Set2ReturnDto>});
The order of the aDataSetTypes needs to correspond to the list of result sets in the aDbReader.
Improvements would be:
Only pass the list of dataset types. (And have the List properties automatically determined)
I have 2 excel files that I have converted into lists. The 1st file has a complete list of all items that I need. However, the 2nd list has a small list of items that need to be changed in the 1st list.
Here's how my 1st list is constructed:
IEnumerable<ExcelRow> queryListA = from d in datapullList
select new ExcelRow
{
Company = d.GetString(0),
Location = d.GetString(1),
ItemPrice = d.GetString(4),
SQL_Ticker = d.GetString(15)
};
The 2nd list is constructed in a very similar way:
IEnumerable<ExcelRow> queryListB = from dupes in dupespullList
select new ExcelRow
{
Company = d.GetString(0),
Location = d.GetString(1),
NewCompany = d.GetString(4)
};
So, if there is a company from a particular location in 1st list that matches 2nd list, then the company gets changed to the newcompany name.
Then, my final list should have everything in 1st list but with the changes specified from 2nd list.
I've been struggling with this for a few days now. Let me know if you need more details.
[Update:] I'm pretty new to LINQ and C#. I've found this code on the web regarding Excel reader for Office 2003. How can I create the 1 list (stated above) from all the following classes?
My ExcelRow class:
class ExcelRow
{
List<object> columns;
public ExcelRow()
{
columns = new List<object>();
}
internal void AddColumn(object value)
{
columns.Add(value);
}
public object this[int index]
{
get { return columns[index]; }
}
public string GetString(int index)
{
if (columns[index] is DBNull)
{
return null;
}
return columns[index].ToString();
}
public int Count
{
get { return this.columns.Count; }
}
}
My ExcelProvider class:
class ExcelProvider : IEnumerable<ExcelRow>
{
private string sheetName;
private string filePath;
private string columnName1;
private string columnName2;
private List<ExcelRow> rows;
public ExcelProvider()
{
rows = new List<ExcelRow>();
}
public static ExcelProvider Create(string filePath, string sheetName, string columnName1, string columnName2)
{
ExcelProvider provider = new ExcelProvider();
provider.sheetName = sheetName;
provider.filePath = filePath;
provider.columnName1 = columnName1;
provider.columnName2 = columnName2;
return provider;
}
private void Load()
{
string connectionString = #"Provider=Microsoft.Jet.OLEDB.4.0;Data Source={0};Extended Properties= ""Excel 8.0;HDR=YES;IMEX=1""";
connectionString = string.Format(connectionString, filePath);
rows.Clear();
using (OleDbConnection conn = new OleDbConnection(connectionString))
{
try
{
conn.Open();
using (OleDbCommand cmd = conn.CreateCommand())
{
cmd.CommandText = string.Format("SELECT * FROM [{0}$] WHERE {1} IS NOT NULL AND {2} <> \"{3}\"", sheetName, columnName1, columnName2, null);
using (OleDbDataReader reader = cmd.ExecuteReader())
{
while (reader.Read())
{
ExcelRow newRow = new ExcelRow();
for (int count = 0; count < reader.FieldCount; count++)
{
newRow.AddColumn(reader[count]);
}
rows.Add(newRow);
}
}
}
}
catch (Exception ex)
{ throw ex; }
finally
{
if (conn.State == System.Data.ConnectionState.Open)
conn.Close();
}
}
}
public IEnumerator<ExcelRow> GetEnumerator()
{
Load();
return rows.GetEnumerator();
}
System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
{
Load();
return rows.GetEnumerator();
}
}
So, using all this logic, how can I solve my problem?
//first create a dictionary of comapny whose name has been changed
var dict = queryListB.ToDictionary(x => x.Company, y => y.NewCompany);
//loop on the first list and do the changes in the first list
queryListA.ForEach( x =>
{
if(dict.Keys.Contains(x.Company))
x.Company = dict[x.Company];
});
Loop through queryListA and see if there is a matching company in queryListB. If so, then update the Company property.
Here's the code:
foreach (var companyA in queryListA)
{
var companyBMatch = queryListB.FirstOrDefault(x => x.Company == companyA.Company && x.Location == companyA.Location);
if (companyBMatch != null)
companyA.Company = companyBMatch.NewCompany;
}
I'm sure you can write simpler code to achieve the same goal but I've gone for a way that reduces the number of times you have to iterate through the first and second lists. If performance isn't an issue a simpler method that just searches the dupespullList for each element in datapullList might be appropriate.
var excelRowCreator = new ExcelRowCreator(dupespullList);
var finalRows = excelRowCreator.CreateExcelRows(datapullList);
// ...
public class ExcelRowCreator
{
/// <summary>
/// First key is company name, second is location
/// and final value is the replacement name.
/// </summary>
private readonly IDictionary<string, IDictionary<string, string>> nameReplacements;
/// <summary>
/// I don't know what type of objects your initial
/// lists contain so replace T with the correct type.
/// </summary>
public ExcelRowCreator(IEnumerable<T> replacementRows)
{
nameReplacements = CreateReplacementDictionary(replacementRows);
}
/// <summary>
/// Creates ExcelRows by replacing company name where appropriate.
/// </summary>
public IEnumerable<ExcelRow> CreateExcelRows(IEnumerable<T> inputRows)
{
// ToList is here so that if you iterate over the collection
// multiple times it doesn't create new excel rows each time
return inputRows.Select(CreateExcelRow).ToList();
}
/// <summary>
/// Creates an excel row from the input data replacing
/// the company name if required.
/// </summary>
private ExcelRow CreateExcelRow(T data)
{
var name = data.GetString(0);
var location = data.GetString(1);
IDictionary<string, string> replacementDictionary;
if (nameReplacements.TryGetValue(name, out replacementDictionary))
{
string replacementName;
if (replacementDictionary.TryGetValue(location, out replacementName))
{
name = replacementName;
}
}
return new ExcelRow
{
Company = name,
Location = location,
ItemPrice = data.GetString(4),
SQL_Ticker = data.GetString(15)
};
}
/// <summary>
/// A helper method to create the replacement dictionary.
/// </summary>
private static IDictionary<string, IDictionary<string, string>> CreateReplacementDictionary(IEnumerable<T> replacementRows)
{
var replacementDictionary = new Dictionary<string, IDictionary<string, string>>();
foreach (var dupe in replacementRows)
{
var name = dupe.GetString(0);
IDictionary<string, string> locationReplacements;
if (!replacementDictionary.TryGetValue(name, out locationReplacements))
{
locationReplacements = new Dictionary<string, string>();
replacementDictionary[name] = locationReplacements;
}
locationReplacements[dupe.GetString(1)] = dupe.GetString(4);
}
return replacementDictionary;
}
}
UPDATE : Packaged as a class and written in visual studio so there shouldn't be any grammatical errors.
I am trying to use a home grown web API to retrieve some data. The documentation is all written in PHP. The example I'm looking at is this:
$params = array(
'id' => 1
,'data' => array(
,'email' => 'example#hasoffers.com'
)
$url = "www.someapi.com/api?" . http_build_query( $params );
I'm using the C# WebClient class, but I can't figure out how to serialize the data parameter:
WebClient wc = new WebClient();
wc.QueryString["id"] = "1";
wc.QueryString["data"] = // I have no idea.
string json = wc.DownloadString(apiUrl);
I've tried a few variations:
wc.QueryString["data"] = "email=test#stackoverflow.com";
wc.QueryString["data"] = Uri.EscapeDataString("data[email]=test#stackoverflow.com");
wc.QueryString["data"] = Uri.EscapeDataString("email[0]=test#stackoverflow.com");
wc.QueryString["data"] = Uri.EscapeDataString("email=test#stackoverflow.com");
Of course, I don't have PHP setup anywhere to see what the http_build_query() is actually returning.
I finally figured it out. I guess posting this question rebooted my brain.
This is what worked:
wc.QueryString["data[email]"] = "test#stackoverflow.com";
Note, this question is very relevant to this answer, so I put this class as an answer there as well. That answer will receive any updates to this class.
As far as I know, there's nothing built in to do this. But, you can create your own class.
So I did:
/// <summary>
/// Helps up build a query string by converting an object into a set of named-values and making a
/// query string out of it.
/// </summary>
public class QueryStringBuilder
{
private readonly List<KeyValuePair<string, object>> _keyValuePairs
= new List<KeyValuePair<string, object>>();
/// <summary> Builds the query string from the given instance. </summary>
public static string BuildQueryString(object queryData, string argSeperator = "&")
{
var encoder = new QueryStringBuilder();
encoder.AddEntry(null, queryData, allowObjects: true);
return encoder.GetUriString(argSeperator);
}
/// <summary>
/// Convert the key-value pairs that we've collected into an actual query string.
/// </summary>
private string GetUriString(string argSeperator)
{
return String.Join(argSeperator,
_keyValuePairs.Select(kvp =>
{
var key = Uri.EscapeDataString(kvp.Key);
var value = Uri.EscapeDataString(kvp.Value.ToString());
return $"{key}={value}";
}));
}
/// <summary> Adds a single entry to the collection. </summary>
/// <param name="prefix"> The prefix to use when generating the key of the entry. Can be null. </param>
/// <param name="instance"> The instance to add.
///
/// - If the instance is a dictionary, the entries determine the key and values.
/// - If the instance is a collection, the keys will be the index of the entries, and the value
/// will be each item in the collection.
/// - If allowObjects is true, then the object's properties' names will be the keys, and the
/// values of the properties will be the values.
/// - Otherwise the instance is added with the given prefix to the collection of items. </param>
/// <param name="allowObjects"> true to add the properties of the given instance (if the object is
/// not a collection or dictionary), false to add the object as a key-value pair. </param>
private void AddEntry(string prefix, object instance, bool allowObjects)
{
var dictionary = instance as IDictionary;
var collection = instance as ICollection;
if (dictionary != null)
{
Add(prefix, GetDictionaryAdapter(dictionary));
}
else if (collection != null)
{
Add(prefix, GetArrayAdapter(collection));
}
else if (allowObjects)
{
Add(prefix, GetObjectAdapter(instance));
}
else
{
_keyValuePairs.Add(new KeyValuePair<string, object>(prefix, instance));
}
}
/// <summary> Adds the given collection of entries. </summary>
private void Add(string prefix, IEnumerable<Entry> datas)
{
foreach (var item in datas)
{
var newPrefix = String.IsNullOrEmpty(prefix)
? item.Key
: $"{prefix}[{item.Key}]";
AddEntry(newPrefix, item.Value, allowObjects: false);
}
}
private struct Entry
{
public string Key;
public object Value;
}
/// <summary>
/// Returns a collection of entries that represent the properties on the object.
/// </summary>
private IEnumerable<Entry> GetObjectAdapter(object data)
{
var properties = data.GetType().GetProperties();
foreach (var property in properties)
{
yield return new Entry()
{
Key = property.Name,
Value = property.GetValue(data)
};
}
}
/// <summary>
/// Returns a collection of entries that represent items in the collection.
/// </summary>
private IEnumerable<Entry> GetArrayAdapter(ICollection collection)
{
int i = 0;
foreach (var item in collection)
{
yield return new Entry()
{
Key = i.ToString(),
Value = item,
};
i++;
}
}
/// <summary>
/// Returns a collection of entries that represent items in the dictionary.
/// </summary>
private IEnumerable<Entry> GetDictionaryAdapter(IDictionary collection)
{
foreach (DictionaryEntry item in collection)
{
yield return new Entry()
{
Key = item.Key.ToString(),
Value = item.Value,
};
}
}
}
Like so:
// Age=19&Name=John%26Doe&Values%5B0%5D=1&Values%5B1%5D=2&Values%5B2%5D%5Bkey1%5D=value1&Values%5B2%5D%5Bkey2%5D=value2
QueryStringBuilder.BuildQueryString(new
{
Age = 19,
Name = "John&Doe",
Values = new object[]
{
1,
2,
new Dictionary<string, string>()
{
{ "key1", "value1" },
{ "key2", "value2" },
}
},
});
// 0=1&1=2&2%5B0%5D=one&2%5B1%5D=two&2%5B2%5D=three&3%5Bkey1%5D=value1&3%5Bkey2%5D=value2
QueryStringBuilder.BuildQueryString(new object[]
{
1,
2,
new object[] { "one", "two", "three" },
new Dictionary<string, string>()
{
{ "key1", "value1" },
{ "key2", "value2" },
}
}
);
I was wondering if there is a known technique for saving and using variables in an SqlLite database.
I am looking for something like the $something variables one can find under Oracle
Didn't find any builtin solution for this, so I solved it by having a global table with Key,Value pairs
Here is the C# class i made to wrap this nicely
public class SQLiteVariable
{
public SQLiteVariable() : this (null, string.Empty)
{}
public SQLiteVariable(SQLiteConnection connection) : this(connection, string.Empty)
{}
public SQLiteVariable(string name) : this(null, name)
{}
public SQLiteVariable(SQLiteConnection connection, string name)
{
Connection = connection;
Name = name;
}
/// <summary>
/// The table name used for storing the database variables
/// </summary>
private const string VariablesTable = "__GlobalDatabaseVariablesTable";
/// <summary>
/// Gets or sets the SQLite database connection.
/// </summary>
/// <value>The connection.</value>
public SQLiteConnection Connection { get; set; }
/// <summary>
/// Gets or sets the SQLite variable name.
/// </summary>
/// <value>The name.</value>
public string Name { get; set; }
/// <summary>
/// Gets or sets the SQLite variable value.
/// </summary>
/// <value>The value.</value>
public string Value
{
get
{
CheckEnviornemnt();
var cmd = new SQLiteCommand(Connection)
{
CommandText = "SELECT Value FROM " + VariablesTable + " WHERE Key=#VarName"
};
cmd.Parameters.Add(new SQLiteParameter("#VarName", Name));
var returnValue = cmd.ExecuteScalar();
return returnValue as string;
}
set
{
CheckEnviornemnt();
// Assume the variable exists and do an update
var cmd = new SQLiteCommand(Connection)
{
CommandText = "INSERT OR REPLACE INTO " + VariablesTable + " (Key, Value) VALUES(#VarName, #Value)"
};
cmd.Parameters.Add(new SQLiteParameter("#Value", value));
cmd.Parameters.Add(new SQLiteParameter("#VarName", Name));
var count = cmd.ExecuteNonQuery();
}
}
private void CheckEnviornemnt()
{
if (Connection == null) throw new ArgumentException("Connection was not initialized");
var cmd = new SQLiteCommand(Connection)
{
CommandText = "CREATE TABLE IF NOT EXISTS "+VariablesTable+" (Key VARCHAR(30) PRIMARY KEY, Value VARCHAR(256));"
};
cmd.ExecuteNonQuery();
}
}
I have a method which in essence converts a datatable to a list of objects which I call 'Bags'. This code is called many times per session, with many sessions concurrently running and with sometimes thousands of rows. Because of this I need it to be as quick as possible. I have an xml file which contains the DataColumn to Property mappings. The main method to optimize is ConvertRowToBag - the type parameter passed in is a type which derives from BagBase.
It's a long bit of code, but any tips would be much appreciated.
public class BagBase
{
/// <summary>
/// Dictionary of properties and names
/// </summary>
private static Dictionary<string, PropertyInfo> propertyDictionary = new Dictionary<string, PropertyInfo>();
/// <summary>
/// Table of column/property mappings
/// </summary>
private static DataTable mappings = new DataTable("Mappings");
/// <summary>
/// Returns true if the map exists
/// </summary>
/// <param name="columnName"></param>
/// <param name="type"></param>
/// <returns></returns>
private static bool MappingExists(string columnName, Type type)
{
DataRow [] rows = BagBase.mappings.Select(String.Format("Type = '{0}' and ColumnName = '{1}'", type.Name, columnName));
return (rows != null && rows.Length > 0);
}
/// <summary>
/// Converts the table to bags.
/// </summary>
/// <param name="table">The table.</param>
/// <param name="outputType">Type of the output.</param>
/// <returns></returns>
protected static List<BagBase> ConvertTableToBags(DataTable table, Type outputType)
{
Trace.TraceInformation(String.Format("ConvertTableToBags : table={0} Type={1}", table.TableName, outputType.Name));
// Create an empty list
List<BagBase> result = new List<BagBase>();
// Iterate through the rows
foreach (DataRow row in table.Rows)
{
// Add to the list
result.Add(ConvertRowToBag(outputType, row));
}
Trace.TraceInformation("ConvertTableToBags Finished.");
return result;
}
/// <summary>
/// Converts the row to bag.
/// </summary>
/// <param name="outputType">Type of the output.</param>
/// <param name="row">The row.</param>
/// <returns></returns>
protected static BagBase ConvertRowToBag(Type outputType, DataRow row)
{
// Create an instance of the child class and store in the base
BagBase bag = Activator.CreateInstance(outputType) as BagBase;
// Iterate through the columns
foreach (DataColumn column in row.Table.Columns)
{
// If this column has been mapped
if (BagBase.MappingExists(column.ColumnName, outputType))
{
PropertyInfo property;
string columnProperty = String.Format("{0}={1}", column.ColumnName, outputType.Name);
// Get the property as defined in the map
if (!propertyDictionary.ContainsKey(columnProperty))
{
// Get the property
property = outputType.GetProperty(BagBase.GetColumnMapping(column.ColumnName, outputType));
// Add the property to the dictionary
propertyDictionary.Add(columnProperty, property);
}
else
{
property = propertyDictionary[columnProperty];
}
if (property != null)
{
if (!row.IsNull(column))
{
// Set the value to the in the table
if (property.PropertyType.BaseType != null && property.PropertyType.BaseType == typeof(Enum))
{
if (column.DataType != typeof(String))
{
property.SetValue(bag, Enum.ToObject(property.PropertyType, row[column]), null);
}
else
{
property.SetValue(bag, Enum.ToObject(property.PropertyType, Convert.ToChar(row[column])), null);
}
}
else if (property.PropertyType == typeof(DateTime?))
{
property.SetValue(bag, (DateTime?)row[column], null);
}
else
{
property.SetValue(bag, Convert.ChangeType(row[column], property.PropertyType), null);
}
}
else // No nulls
{
if (column.DataType == typeof(String))
{
property.SetValue(bag, String.Empty, null);
}
}
// Generate the unique class.property name
string propertyKey = String.Format("{0}.{1}", outputType.Name, property.Name);
if (!columnCaptions.ContainsKey(propertyKey))
{
// Add to the caption map
columnCaptions.Add(propertyKey, column.Caption);
}
}
}
else
{
// If this column isn't mapped, add it to Other information
if (bag.OtherInformation == null)
{
bag.OtherInformation = new Dictionary<string, string>();
}
bag.OtherInformation.Add(column.ColumnName, !row.IsNull(column) ? row[column].ToString() : String.Empty);
}
}
return bag;
}
}
Use a profiler. There's no way for us to know what actually takes the most time in your code.
There's just really no use trying to optimize line-by-line and many people seem to not know this. Computers are always waiting for a resource, sometimes it's CPU or disk IO, and often it's the user. To make any piece of code faster, find the bottlenecks using a profiler and work on making that code faster.
Aside from the general advise of "use a profiler", there is probably not one bottleneck but either a series of slow calls or the very structure of the procedure is creating unnecessary iterations. At a glance:
The Select against a datatable is generally not very performant.
Reflection carries with it a lot of overhead, it looks like you are dependent on it but if you could limit its scope you will probably get better overall performance.