Develop Reference

Convert dynamically shaped query results into IEnumerabale or IQueryable

I am attempting to take the results of a Stored Procedure that that contains dynamic columns, different shapes of data, and convert that into an IEnumerable or IQueryable.
In the middleware, I am using EF and since I have never worked with dynamic queries, I am at a loss. I looked at the DataTable.ToSchemaTable() to see if that could convert to anything useful and hit a wall. Recently, I started looking into dynamics, however, I still can't seem to figure out how to convert to IEnumerable.
The clientside needs to have data in the form of {"FieldName":"FieldValue"}, and the result set needs to be pre-processed via an IEnumerable prior to being converted to JSON.

Here is the code I am working with that solved the issue.
public static class DataTableExtensions
{
public static IEnumerable<dynamic> AsDynamicEnumerable(this DataTable table)
{
if (table == null)
{
yield break;
}
foreach (DataRow row in table.Rows)
{
IDictionary<string, object> dRow = new ExpandoObject();
foreach (DataColumn column in table.Columns)
{
var value = row[column.ColumnName];
dRow[column.ColumnName] = Convert.IsDBNull(value) ? null : value;
}
yield return dRow;
}
}
}

replace List.foreach to LINQ

I'm new to LINQ and doing some experiments with it.
Sorry if it is a duplicate but I cant seem to find proper guide (for me) to it
I want to replace this code :
DataTable table
List<string> header = new List<string>();
table.Columns.Cast<DataColumn>().ToList().ForEach(col => header.Add(col.ColumnName));
with something LINQ like:
var LINQheader = from mycol in table.Columns select mycol.ColumnName;
LINQheader.tolist();
but it doesn't even compile.
what I want Is not a one line solution but would like some logic to understand how construct it with more complicated environments (Like choosing many node in XML with some logic)

here is the original code
table.Columns.Cast<DataColumn>().ToList().ForEach(col => header.Add(col.ColumnName));
Why Cast used?
because it allows you to treat DataColumnCollection items as a DataColumn not an object.
Why ToList used?
becuase it converts your IEnumerable to List and allows you to call ForEach because this function is special method that exists in List class.
Why ForEach used?
because it allows you to do what you want for each element on the list (in your case it adds column name of each column to another list(header)).
Simplified version:
now assume you want to add column names to header where they starts with "Student"
you can write something like this
DataTable table = new DataTable();
List<string> header = new List<string>();
foreach (DataColumn col in table.Columns)
{
if (col.ColumnName.StartsWith("Id")) // you can remove this line if you want to add all of them
header.Add(col.ColumnName);
}
you can also use this
table.Columns.Cast<DataColumn>()
.ToList()
.ForEach(col =>
{
if (col.ColumnName.StartsWith("Id"))
header.Add(col.ColumnName)
});
or
var headers = table.Columns.Cast<DataColumn>()
.Where(col => col.ColumnName.StartsWith("Id"))
.Select(col => col.ColumnName);
header.AddRange(headers);

You can use Enumerable.Aggregate() for this:
var header = table.Columns.Cast<DataColumn>().Aggregate(new List<string>(), (list, col) => { list.Add(col.ColumnName); return list; });
In general, Linq allows for retrieval and transformation of sequences of data from data sources. What you want to do in this question is to iterate over a sequence and return an immediate result. That isn't the primary focus of Linq, but there are methods that perform tasks like this, including Aggregate(), Average(), Count(), Max() and so on.

var LINQheader = from mycol in table.column select mycol.ColumnName;
LINQheader.tolist();
This will not compile as there is no such property in DataTable as column, there is only Columns, and you have to use .Cast() method as they are not implementing right interface (see #Uriil's answer).
Try this:
var LINQheader = from mycol in table.Columns.Cast<DataColumn>()
select mycol.ColumnName;
LINQheader.tolist();
If you want to use wrap it in an extension method, you can do it like this:
public static IEnumerable<string> GetHeaderColumns (this DataTable dataTable)
{
if (dataTable == null || !dataTable.Columns.Any())
{
yield break;
}
foreach (var col in dataTable.Columns.Cast<DataColumn>())
{
yield return col.ColumnName;
}
}

static void Main(string[] args)
{
DataTable tbl = new DataTable();
tbl.Columns.Add("A");
tbl.Columns.Add("B");
var p = from DataColumn col in tbl.Columns select col.ColumnName;
foreach(string a in p)
{
Console.WriteLine(a);
}
}
Here little code example. If you want to be List<string>, use ToList().
EDIT:
Like #Grundy says you missing to specify type of the col, which is DataColumn.
List<string> columnList = (from DataColumn mycol in table.Columns select mycol.ColumnName).ToList();
Here this will be your one line.

Why not simple select like
DataTable table;
IEnumerable<string> header = from mycol in table.Columns.Cast<DataColumn>()
select mycol.ColumnName;

You have some problems, which requires workaround:
ForEach is List specifict method, so can can not translate it into LINQ
LINQ is for data selection, aggregation, but not for data
modification
table.Columns, returns DataColumnCollection, which does not
implement IEnumerable<T>, so you will have to cast it anyway:
var LINQheader = from mycol in table.Columns.Cast<DataColumn>()
select name.ColumnName;

Problem in converting a generic list<string> or string[] array to datatable

I have the below function
public static DataTable ToTable<T>(this IEnumerable<T> listItem)
{
//Return null if the list is empty
if (listItem == null || listItem.Count() == 0) return null;
//Gets the type of the object
var listType = listItem.First().GetType();
//Initialize a new datatable
var dataTable = new DataTable(listType.Name);
//Create the datatable column names and types
listType.GetProperties().ToList().ForEach(col => dataTable.Columns.Add(col.Name, col.PropertyType));
//Get the datatable column names
var dataTableColumnNames = dataTable.GetDatatableColumnNames();
listItem.ToList().ForEach(item =>
{
//create a new datarow
var dataRow = dataTable.NewRow();
dataTableColumnNames
.Where(propName => listType.GetProperty(propName) != null)
.ToList()
.ForEach(columnName =>
//Exception happens here in the next line
dataRow[columnName] = listType.GetProperty(columnName).GetValue(item, null));
//Add the row to the data table
dataTable.Rows.Add(dataRow);
});
//Commit the changes to the datatable
dataTable.AcceptChanges();
return dataTable;
}
It works great for dictionary object and generic list as List<MyClass> .. but not for
List<string> or string[].
For those I am getting an exception as Parameter count mismatch.
The error is coming at
dataRow[columnName] = listType.GetProperty(columnName).GetValue(item, null));
What is the mistake that is happening?
Please help

Here's the deal. The index operator is actually considered a property when using reflection, hence parameter count mismatch.
If you break into your code and check the properties that are actually being enumerated by GetProperties(), you'll see the "Chars" property. That's the String's index operator. Since you didn't provide an index, you're getting a Parameter Count Mismatch error.
In essence, I assume string doesn't have any properties you want to put in your data table, but rather the string instance IS what you want to put in the data table.
You could create a model to store the string in, with the string as a property on the model, then the string would be stored with your current code. Otherwise, you will need to rethink your table generation algorithm for primitive types.
I hope this helps :)

Because one of the public properties of string is an indexer and you pass null as the index value. So you effectively end up doing this: string[null] which ends up in an exception.
I haven't verified this as I don't have VS available right now so I might be wrong but I'm pretty sure that's the problem.
Update: This question answers how you detect an indexed property: C# Reflection Indexed Properties

c# linq question

I have a data table and I want to select all distinct names from the result. I wrote following linq query for it.
var distinctRows = (from DataRow myDataRow in myDataTable.Rows
select new { col1 = myDataRow ["Name"]}).Distinct();
Now how can I iterate through distinctRows? Seems like I cannot do foreach(DataRow Row in distinctRows), It gives me "Cannot convert type 'AnonymousType#1' to 'System.Data.DataRow'" error

Since you're only selecting one field, you don't need an anonymous type here. Just select the names and then iterate over the distinct ones. To wit:
var distinctNames = (from DataRow myDataRow in myDataTable.Rows
select myDataRow.Field<string>("Name")
).Distinct();
foreach(var name in distinctNames) {
Console.WriteLine(name);
}
Note that the error makes it very clear what the problem is here. You are trying to convert an instance of an anonymous type to an instance of DataRow and that is impossible. Without changing your code, you could iterate this as
foreach(var item in distinctRows) {
Console.WriteLine((string)item.col1);
}
But I would change this as per the above as you don't need the anonymous type and your variable names and field names are poor.

Those aren't DataRows; they're anonymous objects.
To loop through them, you need to declare the variable using the var keyword.
However, there's no point in the anonymous type in the first place.
You can change your query to select myDataRow.Field<string>("Name") to get a set of strings.

You can use the keyword var to refer to anonymous types (which is what you're returning an IEnumerable<> of).
foreach(var row in distinctRows)
{
// do something with each anonymous type instance
}
Since you're only returning anonymous types with one string property however, you may as well project an IEnumerable<string>

That's because myDataRow["Name"] doesn't return a DataRow. Try
foreach(var item in distinctRows) {}

That's because the return value isn't a DataRow. It's an ad-hoc type that containes the property col1.

To build on SLaks answer . . .
var distinctRows = (from DataRow myDataRow in myDataTable.Rows
select new { col1 = myDataRow ["Name"]}).Distinct();
foreach(var row in distinctRows)
{
System.Console.Writeline(row.col1); //should work fine
}

The problem here is that you're selecting a new anonymous type by doing select new { col1 = myDataRow ["Name"]} and not the actual row itself. So when you try to iterate this as DataRow, it will error out because your anonymous type being selected is not of type DataRow.
If you want to be able to select a whole data row and not just the name field, you will need to implement a custom IEqualityComparer for data row to pass to the Distinct() extension method.
An example would be:
public class NameComparer : IEqualityComparer<DataRow>
{
public bool Equals(DataRow left, DataRow right)
{
return left.Field<string>("Name").Equals(right.Field<string>("Name"));
}
public int GetHashCode(DataRow obj)
{
return obj.ToString().GetHashCode();
}
}
Then using it:
var distinctRows = (from DataRow myDataRow in myDataTable.Rows
select myDataRow).Distinct(new NameComparer());

You can also foreach them but first you had to List them as the following :
List<string> rslt =(from DataRow myDataRow in myDataTable.Rows
select new { col1 = myDataRow ["Name"].ToString()}).Distinct().ToList();
foreach(string str in rlst)
{}
Hope this helped

How can I make this extension method more generic?

I'm having a brain fart trying to make the following method more generic such that any List<T> can be passed in for the columnValues parameter. Here's what I have:
public static DataRow NewRow(this DataTable dataTable, List<string> columnValues)
{
DataRow returnValue = dataTable.NewRow();
while (columnValues.Count > returnValue.Table.Columns.Count)
{
returnValue.Table.Columns.Add();
}
returnValue.ItemArray = columnValues.ToArray();
return returnValue;
}
I could change it to a List<object> and convert the original list prior to passing it to the method but I'm sure there is a better option :-)
Edit:
Frank's post made me rethink this. In most cases that source List<T> would be a List<object> since the column values will most likely be different types.
For my initial use a List<string> made sense because I was creating a dataset from a CSV parse which is all text at that point.

Why not just use params object[]:
public static DataRow NewRow(this DataTable dataTable, params object[] objects)
{
DataRow returnValue = dataTable.NewRow();
while (objects.Length > returnValue.Table.Columns.Count)
{
returnValue.Table.Columns.Add();
}
returnValue.ItemArray = objects;
return returnValue;
}
Then you can just call it like this:
myDataTable.NewRow(1,2,"hello");

You're basically out of luck, because the Item Array of the DataRow is an array of objects, that is, ultimately you can only pass in list of objects.
If you put in a generic parameter of the list all items of the list would have to be of that type, which is highly unlikely to be useful.
Having said that, in order to get numerous columns, all with different types, you could change your extension method to accept an object into which you instantiate an anonymous type:
table.NewRow(new { A = "Hello", B = 1, C = DateTime.Now })
With the aid to convert the anonymous type values to a string,object dictionary either by reflection or by a dynamic method it should be a fairly useful thing.

What about
IEnumerable<object>
in connection with
columnValues.Select(x => x.ToString()).ToArray();

What about using a closure to specify how to generate the ItemArray based upon your input type
public static DataRow NewRow<T>(this DataTable dataTable, List<T> columnValues, Func<T, string> itemArrayCriteria)
{
DataRow returnValue = dataTable.NewRow();
while (columnValues.Count > returnValue.Table.Columns.Count)
{
returnValue.Table.Columns.Add();
}
returnValue.ItemArray = columnValues.Select(x => itemArrayCriteria(x)).ToArray();
return returnValue;
}