Develop Reference

Update List of classes with data from a list of classes

I have a class:
public class DataMember {
public string ID{ get; set; }
public List<string> Versions { get; set; }
}
And another class:
public class MasterDataMember {
public string ID { get; set; }
public List<string> FoundVersions { get; set; }
}
I store both sets of data in a Cache as:
List<DataMember> datamembers
List<MasterDataMember> masterdatamembers
When originally built, the MasterDataMember is a list of partial "versions". These versions need to be confirmed and found in the list of DataMember's.
How can I update masterdatamembers with the confirmed versions found in datamembers?
(this code block is untested but it illustrates what I'm trying to do)
foreach (MasterDataMember item in masterdatamembers) {
List<string> confirmedvers = new List<string>();
foreach(string rawver in item.FoundVersions ){
foreach(DataMember checkitem in datamembers){
foreach (string confirmedver in checkitem.Versions) {
if (rawver.Contains(confirmedver)) {
confirmedvers.Add(confirmedver);
}
}
}
}
item.FoundVersions = vers;
}
Is there a LINQ that can accomplish this a lot easier, faster (I've already tried lots of ideas, iterations)?
Speed is the key here since both lists can be hundreds to thousands long.
Thank you in advance!

foreach (MasterDataMember item in masterdatamembers) {
IEnumerable<string> confirmedvers = item.FoundVersions.Where(rawver => rawver.Any(confirmedver => datamembers.Any(checkitem => checkitem.Versions.Contains(rawver)));
}
HOLY crap bro that was confusing as hell for me!
Awesome mind experiment though!

If speed really is your primary concern because of large lists, then you'll want to use hash table constructs. Using LINQ is slick, but won't necessarily make things faster (or clearer) for you. What you really need is to use the proper collection type.
Assumptions made for the code that follows:
datamembers cache cannot have duplicate DataMember entries (where more than one entry has the same ID).
masterdatamembers cache cannot have duplicate MasterDataMember entries (where more than one entry has the same ID).
In both DataMember and MasterDataMember, the Versions and FoundVersions lists cannot have duplicate version entries.
Algorithm Description
I still feel that your code block doesn't quite reflect your intent. And unfortunately, as a result, I think you got wrong answers.
This is the algorithm I followed, based on trying to interpret your intended result:
For each master data member, update its FoundVersions set (or list) by only keeping the versions in the list that can also be found in the matching data member's Versions set (or list). If no matching data member is found, then I assume you want the master data members FoundVersions set (or list) to be emptied, as none of the versions can be confirmed.
Implementation
Notice that I replaced a few uses of List<T> with Dictionary<K, V> or HashSet<T> where it would benefit performance. Of course, I am assuming that your lists can become large as you said. Otherwise, the performance will be similar as simple lists.
Your 2 classes, (notice the change in types):
public class DataMember
{
public string ID { get; set; }
public HashSet<string> Versions { get; set; } // using hashset is faster here.
}
public class MasterDataMember
{
public string ID { get; set; }
public HashSet<string> FoundVersions { get; set; } // used HashSet for consistency, but for the purposes of the algorithm, a List can still be used here if you want.
}
Your cached data, (notice the change to a Dictionary):
Dictionary<string, DataMember> datamembers; // using a Dictionary here, where your key is the DataMember's ID, is your fastest option.
List<MasterDataMember> masterdatamembers; // this can stay as a list if you want.
And finally, the work is done here:
foreach (var masterDataMember in masterdatamembers)
{
DataMember dataMember;
if (datamembers.TryGetValue(masterDataMember.ID, out dataMember))
{
HashSet<string> newSet = new HashSet<string>();
foreach (var version in masterDataMember.FoundVersions)
{
if (dataMember.Versions.Contains(version))
{
newSet.Add(version);
}
}
masterDataMember.FoundVersions = newSet;
}
else
{
masterDataMember.FoundVersions.Clear();
}
}

Your code will look like something like this in Linq
masterDataMembers.ForEach(q=>q.FoundVersions = (from rawver in q.FoundVersions from checkitem in dataMembers from confirmedver in checkitem.Versions where rawver.Contains(confirmedver) select confirmedver).ToList());

Transforming DataTable to List<T> in C#

I've already searched through StackOverflow (and other websites) about transforming a DataTable to List with reflection in C#.
My results until now are pretty good: I can reflect 200k lines in 3.5 seconds (0.5 seconds in hardcoded mode).
But my entities (the classes that represent my data, but I think you already know that) follow this pattern:
My database have columns like this (I don't actually do this, but you'll get the idea):
Table: Clients
Columns:
ClientID, ClientName, ClientPhone, CityID[FK]
I'm using SqlConnection (MySqlConnection), so I have to hardcode my entities and transform the database result in a list of this entity. Like:
Select *, cit.* from Clients cli
Inner join Cities cit on (cit.CityID == cli.CityID)
Inner join Countries cou on (cou.CountryID == cit.CountID)
I don't know if this SQL is correct, but I think you got the idea. This should return some fields like this:
ClientID, ClientName, ClientPhone, CityID, CityName, CountryID, CountryName
Shoud result a List<Client>.
Here's the problem: I have 2 inner joins and I represent this data in my entities like this (I like the expression "like this"):
public class Client
{
public int ClientID { get; set; }
public string ClientName { get; set; }
public string ClientPhone { get; set; }
public City ClientCity { get; set; }
}
public class City
{
public int CityID { get; set; }
public string CityName { get; set; }
public Country CityCountry { get; set; }
}
public class Country
{
public int ContryID { get; set; }
public string CountryName { get; set; }
}
So, if I have a Client object, I would get its country name by the expression client.ClientCity.CityCountry.CountryName. I call it a 3-level property acessor.
And I want to reflect it properly. Here is the main method to transform the DataTable into a List. My native language is Portuguese, but I tried to translate my comments to match my description above.
The idea of this code is: I try to find in the main class the column I have to set. If I don't find it, I search the property in the properties that are objects. Like CityName inside ClientCity inside Client. This code is a mess.
public List<T> ToList<T>(DataTable dt) where T : new()
{
Type type= typeof(T);
ReflectionHelper h = new ReflectionHelper(type);
insertPropInfo(tipo); //a pre-reflection work, I cache some delegates, etc..
List<T> list = new List<T>();
DataTableReader dtr = dt.CreateDataReader();
while (dtr.Read())
{
T obj = new T();
for (int i = 0; i < dtr.FieldCount; i++)
{
GetObject(ref obj, tipo, dtr.GetName(i), dtr.GetValue(i));
}
list.Add(obj);
}
return lista;
}
//ref T obj: the object I create before calling this method
//Type classType: the type of the object (say, Client)
//string colName: this is the Database Column i'm trying to fill. Like ClientID or CityName or CountryName.
//colLineData: the data I want to put in the colName.
public void GetObject<T>(ref T obj, Type classType, string colName, object colLineData) where T : new()
{
//I do some caching to reflect just once, and after the first iteration, I think all the reflection I need is already done.
foreach (PropertyInfo info in _classPropInfos[classType])
{
//If the current PropertyInfo is a valuetype (like int, int64) or string, and so on
if (info.PropertyType.IsValueType || info.PropertyType == typeof(string))
{
//I think string.Equals is a little faster, but i had not much difference using "string" == "string"
if (info.Name.Equals(colName)) //did I found the property?
if (info.PropertyType != typeof(char)) //I have to convert the type if this is a Char. MySql returns char as string.
{
_delegateSetters[info](obj, colLineData); //if it isn't a char, just set it.
}
else
{
_delegateSetters[info](obj, Convert.ChangeType(colLineData, typeof(char)));
}
break;
}
else //BUT, if the property is a class, like ClientCity:
{
//I reflect the City class, if it isn't reflected yet:
if (!_classPropInfos.ContainsKey(info.PropertyType))
{
insertPropInfo(info.PropertyType);
}
//now I search for the property:
Boolean foundProperty = false;
object instance = _delegateGetters[info](obj); //Get the existing instance of ClientCity, so I can fill the CityID and CityName in the same object.
foreach (PropertyInfo subInfo in _classPropInfos[info.PropertyType])
{
if (subInfo.Name.Equals(colName))//did I found the property?
{
if (instance == null)
{
//This will happen if i'm trying to set the first property of the class, like CityID. I have to instanciate it, so in the next iteration it won't be null, and will have it's CityID filled.
instance = _initializers[info.PropertyType]();//A very fast object initializer. I'm worried about the Dictionary lookups, but i have no other idea about how to cache it.
}
_delegateSetters[subInfo](instance, colLineData);//set the data. This method is very fast. Search about lambda getters & setters using System.Linq.Expression.
foundProperty = true;
break;//I break the loops when I find the property, so it wont iterate anymore.
}
}
if (foundProperty)//if I found the property in the code above, I set the instance of ClientCity to the Client object.
{
_delegateSetters[info](obj, instance);
break;
}
}
}
}
There is a problem with this code: I can reach the CityID and CityName, and fill it. But CountryID and CountryName wont. Because this code can do a 2-level reflection, I need some recursive-approach to fill many levels I need. I tried to do this BUT i got so many stack overflows and null reference exceptions I almost gave up.
This code would make it much easier to fetch database rows, Did you already find some library or anything that does what I want? If not, how could I achieve a n-level reflection to make a proper List from a DataTable?

Your problem is really common and practically every ORM in circulation addresses this question.
Of course changing an already written application to take advantage of an ORM is often unpractical, but there are some simple ORM that are really easy to add to an existing application and let you replace incrementally the already written code.
One of these ORMs is DAPPER. It consists of just one source file that you can include directly in the same project with your POCO classes and repository methods (Or just reference the compiled assembly). It is really easy to learn and it is incredibly fast considering the complexity of the work to be carried out. Not to mention that the authors of this little gem are regularly on this site answering questions on their work. Just do a search with the #dapper tag
The only nuisances that I have found to date are the mapping one-to-one from your POCO properties and the field names and also the sometime eluding rules between PK and FK when your keys are not named ID. But that's me that I still haven't fully understood these rules.

Consider to use EntityFramework. It will automate all this work.

This is based on you getting a dataset with the 3 tables and creating the proper DataRelation.
On your particular case(200k lines) i dont know how it will perform but shouldnt be that bad :).
Your calling code could be something like this:
List<Clients> clients = Test.CreateListFromTable<Clients>(ds.Tables["Clients"]);
Remember as i said its based in you fettching the dataset and creating the relations.
Next here is the class with the methods in question(ClientsToCity and CityToCountry are the names of the datarelations,you can place your own):
public class Test
{
// function that set the given object from the given data row
public static void SetItemFromRow<T>(T item, DataRow row) where T : new()
{
foreach (DataColumn c in row.Table.Columns)
{
PropertyInfo prop = item.GetType().GetProperty(c.ColumnName);
if (prop != null && row[c] != DBNull.Value)
{
prop.SetValue(item, row[c], null);
}
else
{
if (c.ColumnName == "CityID")
{
object obj = Activator.CreateInstance(typeof(City));
SetItemFromRow<City>(obj as City, row.GetChildRows("ClientsToCity")[0]);
PropertyInfo nestedprop = item.GetType().GetProperty("ClientCity");
nestedprop.SetValue(item, obj, null);
}
else if (c.ColumnName == "CountryID")
{
object obj = Activator.CreateInstance(typeof(Country));
SetItemFromRow<Country>(obj as Country, row.GetChildRows("CityToCountry")[0]);
PropertyInfo nestedprop = item.GetType().GetProperty("CityCountry");
nestedprop.SetValue(item, obj, null);
}
}
}
}
// function that creates an object from the given data row
public static T CreateItemFromRow<T>(DataRow row) where T : new()
{
T item = new T();
SetItemFromRow(item, row);
return item;
}
// function that creates a list of an object from the given data table
public static List<T> CreateListFromTable<T>(DataTable tbl) where T : new()
{
List<T> lst = new List<T>();
foreach (DataRow r in tbl.Rows)
{
lst.Add(CreateItemFromRow<T>(r));
}
return lst;
}
}

Datatable like custom collection in c#

In theory how would I do this.
short winded: store data like a datatable using custom collections, having variable amount of fields and columns...so long as the rows are consistent.
Long winded:
2 or 3 classes: field, row, optionally: table
Normally I would do something like List<Person> myList = new List<Person>;
Then that list could be bound to a datagridview and the columns would be based off the properties of the Person class.
Code to look at:
List<row> table = new List<row>;
List<field> row0 = new List<field>;
row0.Add(new field(col1,"value1"));
row0.Add(new field(col2,"value2"));
row0.add(new field(col3,"value3"));
table.Add(row0);
dataGridView1.DataSource = table;
theoretical Output:
| |col 1 | col 2| col 3|
___________________________
|row0|value1|value2|value3|
public class cField
{
public string Name { get; set; }
public string Content { get; set; }
public cField()
{
}
public cField(string name, string content)
{
Name = name;
Content = content;
}
}
public class cRow:BindingList<cField>
{
public cRow()
{
}
}
public class tables:BindingList<cRow>
{
public tables()
{
fillTestData();
}
private void fillTestData()
{
for (Int32 i = 0; i < 10; i++)
{
cRow tRow = new cRow();
for (Int32 x=0; x < 3; x++)
{
cField f1 = new cField("ColumnName" + x.ToString(), "content" + x.ToString());
tRow.Add(f1);
}
base.Items.Add(tRow);
}
}
}
//example class which shows the output of what I'd like.
public class eField
{
public string ColumnName0 { get; set; }
public string ColumnName1 { get; set; }
public string ColumnName2 { get; set; }
public eField(string colName0, string colName1, string colName2)
{
ColumnName0 = colName0;
ColumnName1 = colName1;
ColumnName2 = colName2;
}
}
public class eTable : BindingList<eField>
{
public eTable()
{
base.Add (new eField ("content","content", "content"));
base.Add(new eField("content", "content", "content"));
base.Add(new eField("content", "content", "content"));
}
}
Now Here is code for the form.
public partial class Form1 : Form
{
tables t;
public Form1()
{
InitializeComponent();
}
private void Form1_Load(object sender, EventArgs e)
{
t = new tables ();
dataGridView1.DataSource = t;
dataGridView2.DataSource = t[0];
eTable table3 = new eTable ();
dataGridView3.DataSource = table3;
}
}
If you make that code into a project...you will see the first binding....pulls some built in stuff from the bindinglist into grid1. Grid2 lists my fields vertically when I want them horizontal.
Grid 3 shows exactly how I want my output to be.....yet I can't achieve it with the collection structure I have going to mimic a dataTable....(provided in code)
Disclaimer:
I am short on keywords I would need to research this problem. I didn't find much. The closest thing I found was related to linq and pivots. But non of their outputs seemed to be as I described.
I use custom collections all over the place, so I would like to keep my code very similar instead of using a datatable. This is the first time I have needed my collections to behave in this manner.

It sounds like you are looking for a collection of objects to use in memory once you have loaded the data from a database. You can do calculations and the like on the built-in System.Data objects, but it is cumbersome, and it does not perform well with a large amount of data.
We use System.Data objects heavily to present data. We try to do calculations in the database later and present the results as a DataSet, so the client doesn't have to do any data manipulation.
A few of our modules need more sophisticated data processing. In one case, we used an array of objects that represented a large amount of data to be massaged on the fly. The columns were fixed, so they were easy to implement as properties on each object. When the app presented this data, it generated a small summary DataSet to be displayed in a grid.
We have another module in which there are fields that can have values, or they can also have calculations based on other fields. For this model, we opted to use objects that have dependencies on other objects that made a sort of web of calculations. Change one value, and the ValueChanged event notifies any dependent fields that they need to be calculated, which changes those values, etc. (This is a gross simplification.)
If I had to present a variable number of columns, I'd seriously consider sticking with a System.Data.DataSet. If that really doesn't work for you, you might consider a hashtable that maps a column name to a collection of row values for that column. I believe that is how the System.Data.DataTable is implemented; it stores values by column, not by row. Then a row object would know its row index and how to grab the values out of the column collections.

C# Database Mapper

I was looking to map my database query results to strongly type objects in my c# code. So i wrote a quick and dirty helper method on the SqlConnection class which runs the query on the database and uses reflection to map the record columns to the object properties. The code is below:
public static T Query<T>(this SqlConnection conn, string query) where T : new()
{
T obj = default(T);
using (SqlCommand command = new SqlCommand(query, conn))
{
using (SqlDataReader reader = command.ExecuteReader())
{
while (reader.Read())
{
obj = new T();
PropertyInfo[] propertyInfos;
propertyInfos = typeof(T).GetProperties();
for (int i = 0; i < reader.FieldCount; i++)
{
var name = reader.GetName(i);
foreach (var item in propertyInfos)
{
if (item.Name.Equals(name, StringComparison.InvariantCultureIgnoreCase) && item.CanWrite)
{
item.SetValue(obj, reader[i], null);
}
}
}
}
}
}
return obj;
}
public class User
{
public int id { get; set; }
public string firstname { get; set; }
public string lastname { get; set; }
public DateTime signupDate { get; set; }
public int age { get; set; }
public string gender { get; set; }
}
var user = conn.Query<User>("select id,firstname,lastname from users");
I just wanted a second opinion on my approach above of using reflection to tie the values together, if there's anything i can do better in the code above. Or if there's some other totally different approach i can take to get the same result?
I think i can probably improve the code in the helper method by removing the loop for propertyInfos and using a dictionary instead. Is there anything else that needs to be tweaked?
P.S: i'm aware of Dapper, i just wanted to implement something similar on my own to help me learn better.

What you've done is basically what linq-to-sql or other OR-mappers do under the hood. To learn the details of how it works it's always a good idea to write something from scratch.
If you want more inspiration or want to have something that's ready for production use out-of-the-box I'd recommend reading up on linq-to-sql. It is lightweight, yet competent.

There are a few of things I can think of:
I think that in order to skip the loop you can use:
reader[item.Name]
I've done something similar myself, but I never ran into dapper. I'm not sure if it uses reflection, but it's always a good idea to read someone else's code to sharpen your skill (Scott Hanselman frequently recommends doing so).
You can also look at:
http://www.codeproject.com/KB/database/metaquery_part1.aspx
You can implement an attribute that maps a field to a database column, but that's just for fun.
Edit:
5: You can also skip the while loop over the reader and just take the first row, and document the fact that your query only returns one object, so it doesn't pull a thousand rows if the query returns a thousand rows.

What is a better, cleaner way of using List<T>

I'm looking to implement a few nicer ways to use List in a couple of apps I'm working on. My current implementation looks like this.
MyPage.aspx.cs
protected void Page_Load(object sender, EventArgs e)
{
BLL.PostCollection oPost = new BLL.PostCollection();
oPost.OpenRecent();
rptPosts.DataSource = oArt;
rptPosts.DataBind();
}
BLL Class(s)
public class Post
{
public int PostId { get; set; }
public string PostTitle { get; set; }
public string PostContent { get; set; }
public string PostCreatedDate { get; set; }
public void OpenRecentInitFromRow(DataRow row)
{
this.PostId = (int) row["id"];
this.PostTitle = (string) row["title"];
this.PostContent = (string) row["content"];
this.PostCreatedDate = (DateTime) row["createddate"];
}
}
public class PostCollection : List<Post>
{
public void OpenRecent()
{
DataSet ds = DbProvider.Instance().Post_ListRecent();
foreach (DataRow row in ds.Tables[0].Rows)
{
Post oPost = new Post();
oPost.OpenRecentInitFromRow(row);
Add(oPost);
}
}
}
Now while this is working all well and good, I'm just wondering if there is any way to improve it, and just make it cleaner that having to use the two different classes do to something I think can happen in just one class or using an interface.

For one thing, I wouldn't derive from List<T> - you aren't really specializing the behaviour.
I'd also suggest that you could make Post immutable (at least externally), and write a static method (or constructor) to create one based on a DataRow:
public static Post FromDataRow(DataRow row)
Likewise you can have a list method:
public static List<Post> RecentPosts()
which returns them. Admittedly that might be better as an instance method in some sort of DAL class, which will allow mocking etc. Alternatively, in Post:
public static List<Post> ListFromDataSet(DataSet ds)
Now, as for the use of List<T> itself - are you using .NET 3.5? If so, you could make this considerably neater using LINQ:
public static List<Post> ListFromDataSet(DataSet ds)
{
return ds.Tables[0].AsEnumerable()
.Select(row => Post.FromDataRow(row))
.ToList();
}

Are you deriving from List<T> because you want to offer other consumers of PostCollection the ability to Add and Remove items? I'm guessing not, and that you actually just want a way to expose a collection you can bind to. If so, you could consider an iterator, perhaps:
class BLL {
...
public IEnumerable<Post> RecentPosts {
get {
DataSet ds = DbProvider.Instance().Post_ListRecent();
foreach (DataRow row in ds.Tables[0].Rows)
{
Post oPost = new Post();
oPost.OpenRecentInitFromRow(row);
yield return oPost;
}
}
}
...
}
Notwithstanding the fact that this might be considered poor form (in that we have a property getter that might be making a network call), this iterator approach will do away with the overhead of calling OpenRecentInitFromRow for Posts that are never enumerated.
You also become agnostic as to how potential consumers of your Posts might want to consume them. Code that absolutely, positively has to have every Post can do ToList(), but other code might want to use a LINQ query that short-circuits the enumeration after the right Post is found.

Edit: John Skeet's answer is probably a better option. But if you want to make just a few simple changes, read on:
Place the database access code, OpenRecentInitFromRow into the PostCollection and treat that as a Post manager class. That way the Post class is a plain old Data Transfer Object.
public class Post
{
public int PostId { get; set; }
public string PostTitle { get; set; }
public string PostContent { get; set; }
public string PostCreatedDate { get; set; }
}
public class PostCollection : List<Post>
{
public void OpenRecent()
{
DataSet ds = DbProvider.Instance().Post_ListRecent();
foreach (DataRow row in ds.Tables[0].Rows)
{
Add(LoadPostFromRow(row));
}
}
private Post LoadPostFromRow(DataRow row)
{
Post post = new Post();
post.PostId = (int) row["id"];
post.PostTitle = (string) row["title"];
post.PostContent = (string) row["content"];
post.PostCreatedDate = (DateTime) row["createddate"];
return post;
}
}

I'm looking to implement a few nicer ways to use List
That seems like an odd request. The "List" type is a means, rarely an end. With that in mind, one nicer way to accomplish your real end is to use IEnumerable rather than List, because that List forces you to keep your entire collection in memory while IEnumerable only requires one object at a time. The trick is just that you have to wire everything in your processing stream, from the data layer all the way up through presentation, to use it.
I have a good example in the link below about how to do this in a very clean way:
Fastest method for SQL Server inserts, updates, selects
Depending on your existing data layer code you may be able to skim much of the first half of the (long) post - the main point is that you use an iterator block to turn an SqlDataReader into an IEnumerable<IDataRecord>. Once you have that, it's pretty straightforward the rest of the way through.

You could do this:
protected void Page_Load(object sender, EventArgs e)
{
BLL.PostCollection oPost = new BLL.PostCollection();
rptPosts.DataSource = Post.OpenRecent();
rptPosts.DataBind();
}
public class Post
{
public int PostId { get; set; }
public string PostTitle { get; set; }
public string PostContent { get; set; }
public string PostCreatedDate { get; set; }
public void OpenRecentInitFromRow(DataRow row)
{
this.PostId = (int) row["id"];
this.PostTitle = (string) row["title"];
this.PostContent = (string) row["content"];
this.PostCreatedDate = (DateTime) row["createddate"];
}
public static List<Post> OpenRecent()
{
DataSet ds = DbProvider.Instance().Post_ListRecent();
foreach (DataRow row in ds.Tables[0].Rows)
{
Post oPost = new Post();
oPost.OpenRecentInitFromRow(row);
Add(oPost); //Not sure what this is doing
}
//need to return a List<Post>
}
}