Develop Reference

Bulk insert of ICollection into an Oracle Table

I'm writing an API client from a spec that came from Nswag studio. I am able to retrieve that data using the supplied client.PlansAsync(apikey).GetAwaiter().GetResult(), but I'm struggling to turn the returned ICollection into something that I'm able to bulk insert into an Oracle database table.
I have attempted to create a DataTable but during the transformation into a dataTable, an exception is being thrown. I'm suspect it has something to do with nullable types in the collection.
My guess is that I should be attempting to do the inserts using Entity Framework, but it seems like adding all of the extra EF Core stuff is overkill for this particular client.
I feel like the Oracle Bulk copy methods are perfect for what I'm intending, but I've been running to the issue listed above.
Any help would be greatly appreciated.
TIA
EDIT: Here's the code in question.
//first in the calling class
ICollection<Plans> plansFromApi = client_.PlansAsync(apiKey).GetAwaiter().GetResult();
ListToDatTable listToDt = new();
List<Plans> ps = plansFromApi.ToList();
DataTable dt = listToDt.ToDataTable<Plans>(ps);
//second the List to Datatable class
public class ListToDataTable
{
public DataTable ToDataTable<T>(List<T> items)
{
DataTable dataTable = new(typeof(T).Name);
PropertyInfo[] Props = typeof(T).GetProperties(BindingFlags.Public | BindingFlags.Instance);
foreach(T item in items)
{
var values = new object[Props.Length];
for(int i = 0; i < Props.Length; i++)
{
values[i] = Props[i].GetValue(item);
}
//line where the exception is thrown
//System.ArgumentException: 'Input array is longer than the number of columns in this table.'
dataTable.Rows.Add(values);
}
}
}
EDIT 2:
Here is what came out of Nswag studio. This is just one of 15 datasets that I need to retrieve. This isn't the one I'm currently testing, as that one has 25 properties, so for brevity I'm including one of the smaller ones. In the end they all will be the same, since they are all going to be processed the exact same way, and yes, I have tested with this dataset as well, and received the same exception.
[System.CodeDom.Compiler.GeneratedCode("NJsonSchema", "10.5.2.0 (Newtonsoft.Json v12.0.0.0)")]
public partial class ContactGroupedManufacturer
{
[Newtonsoft.Json.JsonProperty("lastContacted", Required = Newtonsoft.Json.Required.Default, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
public System.DateTimeOffset? LastContacted { get; set; }
[Newtonsoft.Json.JsonProperty("vendorContactId", Required = Newtonsoft.Json.Required.DisallowNull, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
public int VendorContactId { get; set; }
[Newtonsoft.Json.JsonProperty("ManufacturerId", Required = Newtonsoft.Json.Required.DisallowNull, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
public int ManufacturerId { get; set; }
[Newtonsoft.Json.JsonProperty("website", Required = Newtonsoft.Json.Required.Default, NullValueHandling = Newtonsoft.Json.NullValueHandling.Ignore)]
public string Website { get; set; }
}
Here area few rows of data:
lastContacted
vendorContactId
manufacturerId
website
6575
1848
6599
2693
6604
8878
06/08/2018
6692
6879
6930
4040
some url
UPDATE 2021/11/10: I found a NuGet package called MoreLinq that contained an extension method that handled the transformation to a DataTable.
ICollection<ActionPlans> actionPlans = client.ActionPlansAsync(apiKey).GetAwaiter().GetResult();
_logger.LogInformation(${actionPlans.Count} APs returned");
DataTable actionPlansDt = actionPlans.ToDataTable();

You can wrap the IEnumerable in a IDataReader to pass to the BulkCopy method. See eg:
ObjectDataReader
Once you have an IDataReader, pass it to OracleBulkCopy.WriteToServer(IDataReader) See OracleBulkCopy.

You probably have problem with quantity of columns in the DataTable.
Try this code:
public class ListToDataTable
{
public DataTable ToDataTable<T>(List<T> items)
{
DataTable dataTable = new DataTable();
PropertyInfo[] Props = typeof(T).GetProperties(BindingFlags.Public | BindingFlags.Instance);
bool columnsAlreadyCreated=false;
foreach(T item in items)
{
if (columnsAlreadyCreated==false)
{
for(int i = 0; i < Props.Length; i++)
{
dataTable.Columns.Add(Props[i].Name,Nullable.GetUnderlyingType(
Props[i].PropertyType) ?? Props[i].PropertyType);
}
columnsAlreadyCreated=true;
}
var values = new object[Props.Length];
for(int i = 0; i < Props.Length; i++)
{
values[i] = Props[i].GetValue(item);
}
//line where the exception is thrown
//System.ArgumentException: 'Input array is longer than the number of columns in this table.'
dataTable.Rows.Add(values);
}
}
}

How to return two specific columns from database using LINQ?

I have table called Asset. It has lot of columns. I only want to select two of them and use them separately.
Both of these columns are strings.
Linq query :
public static List<string> GetAssetIdsWithNames()
{
using (var db = DbManager.Get())
{
var result = db.Assets.SelectMany(i=> new[] { i.AssetName, i.AssetId }).Distinct().ToList();
return result;
}
}
Where I want to use them :
var assetList = AssetManager.GetAssetIdsWithNames();
//CURRENCYBOX IS A DROPDOWN
CurrencyBox.DataSource = assetList;
CurrencyBox.DataBind();
foreach (var item in assetList)
{
CurrencyBox.DataValueField = //asset id goes here
CurrencyBox.DataTextField =//asset name goes here
break;
}

You cannot access the anonymous type outside of the local scope.
Anonymous types can only be returned as Object outside their local scope and their properties inspected via reflection.
So in this scenario, you are likely better off to use a typed data contract and map from your Asset entity instead and then access it from your calling method.
Your use of SelectMany seems odd too, you probably are after Select instead.
public class AssetDto
{
public string Name { get;set; }
public string Id { get; set; }
}
public static List<AssetDto> GetAssetIdsWithNames()
{
using (var db = DbManager.Get())
{
var result = db.Assets.Select(i=> new AssetDto { Name = i.AssetName, Id = i.AssetId }).ToList();
return result;
}
}

You could use named value tuples for that so you don't need to create an extra class
public static List<(string Name, int Id)> GetAssetWithIds()
{
using (var db = DbManager.Get())
{
var result = db.Assets
.Select(a => new { a.AssetName, a.AssetId })
.Distinct().AsEnumerable()
.Select(a => (a.AssetName, a.AssetId))
.ToList();
return result;
}
}
You will need to add System.ValueTuple

Importing an Excel Sheet and Validate the Imported Data with Loosely Coupled

I am trying to develop a module which will read excel sheets (possibly from other data sources too, so it should be loosely coupled) and convert them into Entities so to save.
The logic will be this:
The excel sheet can be in different format, for example column names in Excel sheet can be different so my system needs to be able to map different fields to my entities.
For now I will be assuming the format defined above will be same and hardcoded for now instead of coming from database dynamically after set on a configuration mapping UI kinda thing.
The data needs to be validated before even get mapped. So I should be able validate it beforehand against something. We're not using like XSD or something else so I should validate it against the object structure I am using as a template for importing.
The problem is, I put together some things together but I don't say I liked what I did. My Question is how I can improve the code below and make things more modular and fix the validation issues.
The code below is a mock-up and is not expected to work, just to see some structure of the design.
This is code I've come up with so far, and I've realized one thing that I need to improve my design patterns skills but for now I need your help, if you could help me:
//The Controller, a placeholder
class UploadController
{
//Somewhere here we call appropriate class and methods in order to convert
//excel sheet to dataset
}
After we uploaded file using an MVC Controller, there could be different controllers specialized to import certain behaviors, in this example I will uploading person related tables,
interface IDataImporter
{
void Import(DataSet dataset);
}
//We can use many other importers besides PersonImporter
class PersonImporter : IDataImporter
{
//We divide dataset to approprate data tables and call all the IImportActions
//related to Person data importing
//We call inserting to database functions here of the DataContext since this way
//we can do less db roundtrip.
public string PersonTableName {get;set;}
public string DemographicsTableName {get;set;}
public Import(Dataset dataset)
{
CreatePerson();
CreateDemograhics();
}
//We put different things in different methods to clear the field. High cohesion.
private void CreatePerson(DataSet dataset)
{
var personDataTable = GetDataTable(dataset,PersonTableName);
IImportAction addOrUpdatePerson = new AddOrUpdatePerson();
addOrUpdatePerson.MapEntity(personDataTable);
}
private void CreateDemograhics(DataSet dataset)
{
var demographicsDataTable = GetDataTable(dataset,DemographicsTableName);
IImportAction demoAction = new AddOrUpdateDemographic(demographicsDataTable);
demoAction.MapEntity();
}
private DataTable GetDataTable(DataSet dataset, string tableName)
{
return dataset.Tables[tableName];
}
}
I have IDataImporter and specialized concrete class PersonImporter. However, I am not sure it looks good so far since things should be SOLID so basically easy to extend later in the project cycle, this will be a foundation for future improvements, lets keep going:
IImportActions are where the magic mostly happens. Instead of designing things table based, I am developing it behavior based so one can call any of them to import things in more modular model. For example a table may have 2 different actions.
interface IImportAction
{
void MapEntity(DataTable table);
}
//A sample import action, AddOrUpdatePerson
class AddOrUpdatePerson : IImportAction
{
//Consider using default values as well?
public string FirstName {get;set;}
public string LastName {get;set;}
public string EmployeeId {get;set;}
public string Email {get;set;}
public void MapEntity(DataTable table)
{
//Each action is producing its own data context since they use
//different actions.
using(var dataContext = new DataContext())
{
foreach(DataRow row in table.Rows)
{
if(!emailValidate(row[Email]))
{
LoggingService.LogWarning(emailValidate.ValidationMessage);
}
var person = new Person(){
FirstName = row[FirstName],
LastName = row[LastName],
EmployeeId = row[EmployeeId],
Email = row[Email]
};
dataContext.SaveObject(person);
}
dataContext.SaveChangesToDatabase();
}
}
}
class AddOrUpdateDemographic: IImportAction
{
static string Name {get;set;}
static string EmployeeId {get;set;}
//So here for example, we will need to save dataContext first before passing it in
//to get the PersonId from Person (we're assuming that we need PersonId for Demograhics)
public void MapEntity(DataTable table)
{
using(var dataContext = new DataCOntext())
{
foreach(DataRow row in table.Rows)
{
var demograhic = new Demographic(){
Name = row[Name],
PersonId = dataContext.People.First(t => t.EmployeeId = int.Parse(row["EmpId"]))
};
dataContext.SaveObject(person);
}
dataContext.SaveChangesToDatabase();
}
}
}
And the validation, which mostly where I suck at unfortunately. The validation needs to be easy to extend and loosely coupled and also I need to be able to call this validation beforehand instead of adding everything.
public static class ValidationFactory
{
public static Lazy<IFieldValidation> PhoneValidation = new Lazy<IFieldValidation>(()=>new PhoneNumberValidation());
public static Lazy<IFieldValidation> EmailValidation = new Lazy<IFieldValidation>(()=>new EmailValidation());
//etc.
}
interface IFieldValidation
{
string ValidationMesage{get;set;}
bool Validate(object value);
}
class PhoneNumberValidation : IFieldValidation
{
public string ValidationMesage{get;set;}
public bool Validate(object value)
{
var validated = true; //lets say...
var innerValue = (string) value;
//validate innerValue using Regex or something
//if validation fails, then set ValidationMessage propert for logging.
return validated;
}
}
class EmailValidation : IFieldValidation
{
public string ValidationMesage{get;set;}
public bool Validate(object value)
{
var validated = true; //lets say...
var innerValue = (string) value;
//validate innerValue using Regex or something
//if validation fails, then set ValidationMessage propert for logging.
return validated;
}
}

I have done the same thing on a project. The difference is that I didn't have to import Excel sheets, but CSV files. I created a CSVValueProvider. And, therefore, the CSV data was bound to my IEnumerable model automatically.
As for validation, I figured that going through all rows, and cells, and validating them one by one is not very efficient, especially when the CSV file has thousands of records. So, what I did was that I created some validation methods that went through the CSV data column by column, instead of row by row, and did a linq query on each column and returned the row numbers of the cells with invalid data. Then, added the invalid row number/column names to ModelState.
UPDATE:
Here is what I have done...
CSVReader Class:
// A class that can read and parse the data in a CSV file.
public class CSVReader
{
// Regex expression that's used to parse the data in a line of a CSV file
private const string ESCAPE_SPLIT_REGEX = "({1}[^{1}]*{1})*(?<Separator>{0})({1}[^{1}]*{1})*";
// String array to hold the headers (column names)
private string[] _headers;
// List of string arrays to hold the data in the CSV file. Each string array in the list represents one line (row).
private List<string[]> _rows;
// The StreamReader class that's used to read the CSV file.
private StreamReader _reader;
public CSVReader(StreamReader reader)
{
_reader = reader;
Parse();
}
// Reads and parses the data from the CSV file
private void Parse()
{
_rows = new List<string[]>();
string[] row;
int rowNumber = 1;
var headerLine = "RowNumber," + _reader.ReadLine();
_headers = GetEscapedSVs(headerLine);
rowNumber++;
while (!_reader.EndOfStream)
{
var line = rowNumber + "," + _reader.ReadLine();
row = GetEscapedSVs(line);
_rows.Add(row);
rowNumber++;
}
_reader.Close();
}
private string[] GetEscapedSVs(string data)
{
if (!data.EndsWith(","))
data = data + ",";
return GetEscapedSVs(data, ",", "\"");
}
// Parses each row by using the given separator and escape characters
private string[] GetEscapedSVs(string data, string separator, string escape)
{
string[] result = null;
int priorMatchIndex = 0;
MatchCollection matches = Regex.Matches(data, string.Format(ESCAPE_SPLIT_REGEX, separator, escape));
// Skip empty rows...
if (matches.Count > 0)
{
result = new string[matches.Count];
for (int index = 0; index <= result.Length - 2; index++)
{
result[index] = data.Substring(priorMatchIndex, matches[index].Groups["Separator"].Index - priorMatchIndex);
priorMatchIndex = matches[index].Groups["Separator"].Index + separator.Length;
}
result[result.Length - 1] = data.Substring(priorMatchIndex, data.Length - priorMatchIndex - 1);
for (int index = 0; index <= result.Length - 1; index++)
{
if (Regex.IsMatch(result[index], string.Format("^{0}.*[^{0}]{0}$", escape)))
result[index] = result[index].Substring(1, result[index].Length - 2);
result[index] = result[index].Replace(escape + escape, escape);
if (result[index] == null || result[index] == escape)
result[index] = "";
}
}
return result;
}
// Returns the number of rows
public int RowCount
{
get
{
if (_rows == null)
return 0;
return _rows.Count;
}
}
// Returns the number of headers (columns)
public int HeaderCount
{
get
{
if (_headers == null)
return 0;
return _headers.Length;
}
}
// Returns the value in a given column name and row index
public object GetValue(string columnName, int rowIndex)
{
if (rowIndex >= _rows.Count)
{
return null;
}
var row = _rows[rowIndex];
int colIndex = GetColumnIndex(columnName);
if (colIndex == -1 || colIndex >= row.Length)
{
return null;
}
var value = row[colIndex];
return value;
}
// Returns the column index of the provided column name
public int GetColumnIndex(string columnName)
{
int index = -1;
for (int i = 0; i < _headers.Length; i++)
{
if (_headers[i].Replace(" ","").Equals(columnName, StringComparison.CurrentCultureIgnoreCase))
{
index = i;
return index;
}
}
return index;
}
}
CSVValueProviderFactory Class:
public class CSVValueProviderFactory : ValueProviderFactory
{
public override IValueProvider GetValueProvider(ControllerContext controllerContext)
{
var uploadedFiles = controllerContext.HttpContext.Request.Files;
if (uploadedFiles.Count > 0)
{
var file = uploadedFiles[0];
var extension = file.FileName.Split('.').Last();
if (extension.Equals("csv", StringComparison.CurrentCultureIgnoreCase))
{
if (file.ContentLength > 0)
{
var stream = file.InputStream;
var csvReader = new CSVReader(new StreamReader(stream, Encoding.Default, true));
return new CSVValueProvider(controllerContext, csvReader);
}
}
}
return null;
}
}
CSVValueProvider Class:
// Represents a value provider for the data in an uploaded CSV file.
public class CSVValueProvider : IValueProvider
{
private CSVReader _csvReader;
public CSVValueProvider(ControllerContext controllerContext, CSVReader csvReader)
{
if (controllerContext == null)
{
throw new ArgumentNullException("controllerContext");
}
if (csvReader == null)
{
throw new ArgumentNullException("csvReader");
}
_csvReader = csvReader;
}
public bool ContainsPrefix(string prefix)
{
if (prefix.Contains('[') && prefix.Contains(']'))
{
if (prefix.Contains('.'))
{
var header = prefix.Split('.').Last();
if (_csvReader.GetColumnIndex(header) == -1)
{
return false;
}
}
int index = int.Parse(prefix.Split('[').Last().Split(']').First());
if (index >= _csvReader.RowCount)
{
return false;
}
}
return true;
}
public ValueProviderResult GetValue(string key)
{
if (!key.Contains('[') || !key.Contains(']') || !key.Contains('.'))
{
return null;
}
object value = null;
var header = key.Split('.').Last();
int index = int.Parse(key.Split('[').Last().Split(']').First());
value = _csvReader.GetValue(header, index);
if (value == null)
{
return null;
}
return new ValueProviderResult(value, value.ToString(), CultureInfo.CurrentCulture);
}
}
For the validation, as I mentioned before, I figured that it would not be efficient to do it using DataAnnotation attributes. A row by row validation of the data would take a long time for CSV files with thousands of rows. So, I decided to validate the data in the Controller after the Model Binding is done. I should also mention that I needed to validate the data in the CSV file against some data in the database. If you just need to validate things like Email Address or Phone Number, you might as well just use DataAnnotation.
Here is a sample method for validating the Email Address column:
private void ValidateEmailAddress(IEnumerable<CSVViewModel> csvData)
{
var invalidRows = csvData.Where(d => ValidEmail(d.EmailAddress) == false).ToList();
foreach (var invalidRow in invalidRows)
{
var key = string.Format("csvData[{0}].{1}", invalidRow.RowNumber - 2, "EmailAddress");
ModelState.AddModelError(key, "Invalid Email Address");
}
}
private static bool ValidEmail(string email)
{
if(email == "")
return false;
else
return new System.Text.RegularExpressions.Regex(#"^[\w-\.]+#([\w-]+\.)+[\w-]{2,6}$").IsMatch(email);
}
UPDATE 2:
For validation using DataAnnotaion, you just use DataAnnotation attributes in your CSVViewModel like below (the CSVViewModel is the class that your CSV data will be bound to in your Controller Action):
public class CSVViewModel
{
// User proper names for your CSV columns, these are just examples...
[Required]
public int Column1 { get; set; }
[Required]
[StringLength(30)]
public string Column2 { get; set; }
}

How can I map the results of a sql query onto objects?

Currently, I am using something like this:
try
{
dr = SQL.Execute(sql);
if(dr != null) {
while(dr.Read()) {
CustomObject c = new CustomObject();
c.Key = dr[0].ToString();
c.Value = dr[1].ToString();
c.Meta = dr[2].ToString();
customerInfo.CustomerList.Add(c);
}
}
else
{
customerInfo.ErrorDetails="No records found";
}
Instead of me doing the assigments manually, is there a way to do this mapping directly (assume that the column names match with the field names).
One requirement, however is that I want to do this by my current approach of using sql queries and not by using pure LINQ based approaches. For one, the SQL queries are big enough, involve complex JOINs and have been tested thoroughly so I don't want to introduce more bugs at the moment. Any suggestions?

One simple solution would be to make a constructor for your CustomObject that takes a DataRow (from the example, so if it's another class, please correct me).
And in your new constructor, do as you do in your own example.
public CustomObject(DataRow row)
{
Key = row[0].ToString();
// And so on...
}
One other way would be to introduce generics, and make a new function in your SQL-class
Example (Took code from Passing arguments to C# generic new() of templated type):
// This function should reside in your SQL-class.
public IEnumerable<T> ExecuteObject<T>(string sql)
{
List<T> items = new List<T>();
var data = ExecuteDataTable(sql); // You probably need to build a ExecuteDataTable for your SQL-class.
foreach(var row in data.Rows)
{
T item = (T)Activator.CreateInstance(typeof(T), row);
items.Add(item);
}
return items;
}
Example usage:
public IEnumerable<CustomObject> GetCustomObjects()
{
return SQL.ExecuteObject<CustomObject>("SELECT * FROM CustomObject");
}
I have tested this code in LinqPad, it should work.

You can achieve by creating a generic method for your requirement. Also you can make your new method as the extension for the data table.
public static List<T> ToList<T>(this DataTable table) where T : class, new()
{
try
{
List<T> list = new List<T>();
foreach (var row in table.AsEnumerable())
{
T obj = new T();
foreach (var prop in obj.GetType().GetProperties())
{
try
{
PropertyInfo propertyInfo = obj.GetType().GetProperty(prop.Name);
propertyInfo.SetValue(obj, Convert.ChangeType(row[prop.Name], propertyInfo.PropertyType), null);
}
catch
{
continue;
}
}
list.Add(obj);
}
return list;
}
catch
{
return null;
}
}
}
Usage:
DataTable dtCustomer = GetCustomers();
List<CustomObject> CustomObjectList = dtCustomer.ToList<CustomObject>();

You should look into MicroORMs. Unlike regular ORMs, that provide an SDL you must use, MicroORMs allow you to use your own SQL queries and only provide the mapping from SQL result sets to C# objects and from C# objects to SQL parameters.
My favorite is PetaPoco, which also provides a query builder that uses your own SQL but does some neat manipulation of parameter numbers.

#user1553525's answer is great, however, if your column names do not match up exactly with your property names it does not work.
So first you would want to create a custom attribute. Then use the attribute in your class that you are trying to deserialize, finally, you want to deserialize the DataTable.
Custom Attribute
We create a custom attribute that will be applied to the properties inside of our class. We create the class to have the property Name that we will use later to get the correct column from our DataTable.
[AttributeUsage(AttributeTargets.Property, Inherited = false)]
public class MySqlColName : Attribute
{
private string _name = "";
public string Name { get => _name; set => _name = value; }
public MySqlColName(string name)
{
_name = name;
}
}
Class to deserialize
Next, in the class that we are going to populate, we are going to declare the column names that will link to the properties in the class using the attribute [MySqlColName] that we just created.
However, if the property name is the same as the database column we do not need to specify the column name in an attribute because the .ToList<>() function will assume the name of the column from the properties name.
public class EventInfo
{
[MySqlColName("ID")]
public int EventID { get; set; }
//Notice there is no attribute on this property?
public string Name { get; set; }
[MySqlColName("State")]
public string State { get; set; }
[MySqlColName("Start_Date")]
public DateTime StartDate { get; set; }
[MySqlColName("End_Date")]
public DateTime EndDate { get; set; }
}
DataTable ToList Extension Method
Finally, we modify #user1553525's answer by adding in a check to see if our custom attribute has been provided. If it is then we set the name of the column to the name provided, otherwise, we use the property name (see code inside of the try block).
public static List<T> ToList<T>(this DataTable table) where T : class, new()
{
try
{
List<T> list = new List<T>();
foreach (var row in table.AsEnumerable())
{
T obj = new T();
foreach (var prop in obj.GetType().GetProperties())
{
try
{
//Set the column name to be the name of the property
string ColumnName = prop.Name;
//Get a list of all of the attributes on the property
object[] attrs = prop.GetCustomAttributes(true);
foreach (object attr in attrs)
{
//Check if there is a custom property name
if (attr is MySqlColName colName)
{
//If the custom column name is specified overwrite property name
if (!colName.Name.IsNullOrWhiteSpace())
ColumnName = colName.Name;
}
}
PropertyInfo propertyInfo = obj.GetType().GetProperty(prop.Name);
//GET THE COLUMN NAME OFF THE ATTRIBUTE OR THE NAME OF THE PROPERTY
propertyInfo.SetValue(obj, Convert.ChangeType(row[ColumnName], propertyInfo.PropertyType), null);
}
catch
{
continue;
}
}
list.Add(obj);
}
return list;
}
catch
{
return null;
}
}//END METHOD
Usage
Finally, we can call the .ToList<>() method and get a list of serialized objects
List<EventInfo> CustomObjectList;
using (DataTable dtCustomer = GetDataTable("SELECT * FROM EventIndex"))
{
CustomObjectList = dtCustomer.ToList<EventInfo>();
}
Side Note: I have a few custom methods that I used
public static bool IsNullOrWhiteSpace(this string x)
{
return string.IsNullOrWhiteSpace(x);
}
public static DataTable GetDataTable(string Query)
{
MySqlConnection connection = new MySqlConnection("<Connection_String>");
try
{
DataTable data = new DataTable();
connection.Open();
using (MySqlCommand command = new MySqlCommand(Query, connection))
{
data.Load(command.ExecuteReader());
}
return data;
}
catch (Exception ex)
{
// handle exception here
Console.WriteLine(ex);
throw ex;
}
finally
{
connection.Close();
}
}

Assumption: if you need objects only for serialization or simple ad-hoc output.
You can use ExpandoObject and SqlDataReader.GetSchemaTable() like this:
private IEnumerable<dynamic> ReaderToAnonymmous(SqlCommand comm) {
using (var reader = comm.ExecuteReader()) {
var schemaTable = reader.GetSchemaTable();
List<string> colnames = new List<string>();
foreach (DataRow row in schemaTable.Rows) {
colnames.Add(row["ColumnName"].ToString());
}
while (reader.Read()) {
var data = new ExpandoObject() as IDictionary<string, Object>;
foreach (string colname in colnames) {
var val = reader[colname];
data.Add(colname, Convert.IsDBNull(val) ? null : val);
}
yield return (ExpandoObject)data;
}
}
}
Although there are posted faster solutions (i posted this as alternative lazy approach for ad-hoc SQL/Reader results/outputs).

The following function accepts a SQL string and an object, it requires the object to have a property for each column in the select statement. The object must be instantiated.
public object SqlToSingleObject(string sSql, object o)
{
MySql.Data.MySqlClient.MySqlDataReader oRead;
using (ConnectionHelper oDb = new ConnectionHelper())
{
oRead = oDb.Execute(sSql);
if (oRead.Read())
{
for (int i = 0; i < oRead.FieldCount; i++)
{
System.Reflection.PropertyInfo propertyInfo = o.GetType().GetProperty(oRead.GetName(i));
propertyInfo.SetValue(o, Convert.ChangeType(oRead[i], propertyInfo.PropertyType), null);
}
return o;
}
else
{
return null;
}
}
}

When searching for this answer I found that you can use Dapper library: https://dapper-tutorial.net/knowledge-base/44980945/querying-into-a-complex-object-with-dapper
You can use something like this:
using (var connection = new SqlConnection(ConnectionString))
{
connection.Open();
IList<CustomObject> result = connection.Query<CustomObject>(sql, commandType: CommandType.Text).ToList();
}

Although this question has been around I could not find a clean solution to this. For my purpose I came up with the following which works quite well in my case.
using System.Dynamic;
private IEnumerable<ExpandoObject> GetQueryToList()
{
try
{
using (var conn = new SqlConnection(ConnectionString))
using (var cmd = new SqlCommand(MyQuery, conn))
{
var list = new List<ExpandoObject>();
conn.Open();
var reader = cmd.ExecuteReader();
while (reader.Read())
{
var expandoObject = new ExpandoObject();
for (var i = 0; i < reader.FieldCount; i++)
{
((IDictionary<string, object>) expandoObject).Add(
reader.GetName(i), reader[i]);
}
list.Add(expandoObject);
}
reader.Close();
return list;
}
}
catch (Exception ex)
{
var m = MethodBase.GetCurrentMethod();
Console.WriteLine(ex.Message + " " + m.Name);
}
return null;
}

How to combine 2 LINQ dictionaries into 1?

I have 2 excel files that I have converted into lists. The 1st file has a complete list of all items that I need. However, the 2nd list has a small list of items that need to be changed in the 1st list.
Here's how my 1st list is constructed:
IEnumerable<ExcelRow> queryListA = from d in datapullList
select new ExcelRow
{
Company = d.GetString(0),
Location = d.GetString(1),
ItemPrice = d.GetString(4),
SQL_Ticker = d.GetString(15)
};
The 2nd list is constructed in a very similar way:
IEnumerable<ExcelRow> queryListB = from dupes in dupespullList
select new ExcelRow
{
Company = d.GetString(0),
Location = d.GetString(1),
NewCompany = d.GetString(4)
};
So, if there is a company from a particular location in 1st list that matches 2nd list, then the company gets changed to the newcompany name.
Then, my final list should have everything in 1st list but with the changes specified from 2nd list.
I've been struggling with this for a few days now. Let me know if you need more details.
[Update:] I'm pretty new to LINQ and C#. I've found this code on the web regarding Excel reader for Office 2003. How can I create the 1 list (stated above) from all the following classes?
My ExcelRow class:
class ExcelRow
{
List<object> columns;
public ExcelRow()
{
columns = new List<object>();
}
internal void AddColumn(object value)
{
columns.Add(value);
}
public object this[int index]
{
get { return columns[index]; }
}
public string GetString(int index)
{
if (columns[index] is DBNull)
{
return null;
}
return columns[index].ToString();
}
public int Count
{
get { return this.columns.Count; }
}
}
My ExcelProvider class:
class ExcelProvider : IEnumerable<ExcelRow>
{
private string sheetName;
private string filePath;
private string columnName1;
private string columnName2;
private List<ExcelRow> rows;
public ExcelProvider()
{
rows = new List<ExcelRow>();
}
public static ExcelProvider Create(string filePath, string sheetName, string columnName1, string columnName2)
{
ExcelProvider provider = new ExcelProvider();
provider.sheetName = sheetName;
provider.filePath = filePath;
provider.columnName1 = columnName1;
provider.columnName2 = columnName2;
return provider;
}
private void Load()
{
string connectionString = #"Provider=Microsoft.Jet.OLEDB.4.0;Data Source={0};Extended Properties= ""Excel 8.0;HDR=YES;IMEX=1""";
connectionString = string.Format(connectionString, filePath);
rows.Clear();
using (OleDbConnection conn = new OleDbConnection(connectionString))
{
try
{
conn.Open();
using (OleDbCommand cmd = conn.CreateCommand())
{
cmd.CommandText = string.Format("SELECT * FROM [{0}$] WHERE {1} IS NOT NULL AND {2} <> \"{3}\"", sheetName, columnName1, columnName2, null);
using (OleDbDataReader reader = cmd.ExecuteReader())
{
while (reader.Read())
{
ExcelRow newRow = new ExcelRow();
for (int count = 0; count < reader.FieldCount; count++)
{
newRow.AddColumn(reader[count]);
}
rows.Add(newRow);
}
}
}
}
catch (Exception ex)
{ throw ex; }
finally
{
if (conn.State == System.Data.ConnectionState.Open)
conn.Close();
}
}
}
public IEnumerator<ExcelRow> GetEnumerator()
{
Load();
return rows.GetEnumerator();
}
System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
{
Load();
return rows.GetEnumerator();
}
}
So, using all this logic, how can I solve my problem?

//first create a dictionary of comapny whose name has been changed
var dict = queryListB.ToDictionary(x => x.Company, y => y.NewCompany);
//loop on the first list and do the changes in the first list
queryListA.ForEach( x =>
{
if(dict.Keys.Contains(x.Company))
x.Company = dict[x.Company];
});

Loop through queryListA and see if there is a matching company in queryListB. If so, then update the Company property.
Here's the code:
foreach (var companyA in queryListA)
{
var companyBMatch = queryListB.FirstOrDefault(x => x.Company == companyA.Company && x.Location == companyA.Location);
if (companyBMatch != null)
companyA.Company = companyBMatch.NewCompany;
}

I'm sure you can write simpler code to achieve the same goal but I've gone for a way that reduces the number of times you have to iterate through the first and second lists. If performance isn't an issue a simpler method that just searches the dupespullList for each element in datapullList might be appropriate.
var excelRowCreator = new ExcelRowCreator(dupespullList);
var finalRows = excelRowCreator.CreateExcelRows(datapullList);
// ...
public class ExcelRowCreator
{
/// <summary>
/// First key is company name, second is location
/// and final value is the replacement name.
/// </summary>
private readonly IDictionary<string, IDictionary<string, string>> nameReplacements;
/// <summary>
/// I don't know what type of objects your initial
/// lists contain so replace T with the correct type.
/// </summary>
public ExcelRowCreator(IEnumerable<T> replacementRows)
{
nameReplacements = CreateReplacementDictionary(replacementRows);
}
/// <summary>
/// Creates ExcelRows by replacing company name where appropriate.
/// </summary>
public IEnumerable<ExcelRow> CreateExcelRows(IEnumerable<T> inputRows)
{
// ToList is here so that if you iterate over the collection
// multiple times it doesn't create new excel rows each time
return inputRows.Select(CreateExcelRow).ToList();
}
/// <summary>
/// Creates an excel row from the input data replacing
/// the company name if required.
/// </summary>
private ExcelRow CreateExcelRow(T data)
{
var name = data.GetString(0);
var location = data.GetString(1);
IDictionary<string, string> replacementDictionary;
if (nameReplacements.TryGetValue(name, out replacementDictionary))
{
string replacementName;
if (replacementDictionary.TryGetValue(location, out replacementName))
{
name = replacementName;
}
}
return new ExcelRow
{
Company = name,
Location = location,
ItemPrice = data.GetString(4),
SQL_Ticker = data.GetString(15)
};
}
/// <summary>
/// A helper method to create the replacement dictionary.
/// </summary>
private static IDictionary<string, IDictionary<string, string>> CreateReplacementDictionary(IEnumerable<T> replacementRows)
{
var replacementDictionary = new Dictionary<string, IDictionary<string, string>>();
foreach (var dupe in replacementRows)
{
var name = dupe.GetString(0);
IDictionary<string, string> locationReplacements;
if (!replacementDictionary.TryGetValue(name, out locationReplacements))
{
locationReplacements = new Dictionary<string, string>();
replacementDictionary[name] = locationReplacements;
}
locationReplacements[dupe.GetString(1)] = dupe.GetString(4);
}
return replacementDictionary;
}
}
UPDATE : Packaged as a class and written in visual studio so there shouldn't be any grammatical errors.