Develop Reference

Creating strongly typed object from BigQuery Row in c#

After running my query in Google Cloud Platform, the results are of type BigQueryResults. To create an object from this results, I run a foreach loop across the rows of this results and keep adding them to the object as shown below:
var data = new List<DefectData>();
foreach (BigQueryRow row in results)
{
DefectData defectData = new DefectData();
defectData.DefectId = long.Parse(SqlUtility.GetValue(row, "DefectId"));
defectData.TrueSize = float.Parse(SqlUtility.GetValue(row, "TrueSize"));
defectData.WaferId = SqlUtility.GetValue(row, "WaferId");
defectData.DieX = int.Parse(SqlUtility.GetValue(row, "DieX"));
data.Add(defectData);
}
The results may have millions of records. The foreach loop takes too much time to create the object. I tried to use Parallel.foreach as well. That too doesn't help. How do I make this process of creating object to run faster?

Is there a way to make string a reference type in a collection?

I want to modify some strings that are contained in an object like say an array, or maybe the nodes in an XDocument (XText)XNode.Value.
I want to gather a subset of strings from these objects and modify them, but I don't know at runtime from what object type they come from.
Put another way, let's say I have objects like this:
List<string> fruits = new List<string>() {"apple", "banana", "cantelope"};
XDocument _xmlObject;
I want to be able to add a subset of values from the original collections to new lists like this:
List<ref string> myStrings1 = new List<ref string>();
myStrings1.Add(ref fruits[1]);
myStrings1.Add(ref fruits[2]);
List<ref string> myStrings2 = new List<ref string>();
IEnumerable<XNode> xTextNodes = getTargetTextNodes(targetPath); //some function returns a series of XNodes in the XDocument
foreach (XNode node in xTextNodes)
{
myStrings2.Add(((XText)node).Value);
}
Then change the values using a general purpose method like this:
public void Modify(List<ref string> mystrings){
foreach (ref string item in mystrings)
{
item = "new string";
}
}
Such that I can pass that method any string collection, and modify the strings in the original object without having to deal with the original object itself.
static void Main(string[] args)
{
Modify(myStrings1);
Modify(myStrings2);
}
The important part here is the mystrings collection. That can be special. But I need to be able to use a variety of different kinds of strings and string collections as the originals source data to go in that collection.
Of course, the above code doesn't work, and neither does any variation I've tried. Is this even possible in c#?

What you want is possible with C#... but only if you can fix every possible source for your strings. That would allow you to use pointers to the original strings... at a terrible cost, however, in terms of memory management and unsafe code throughout your application.
I encourage you to pursue a different direction for this.
Based on your edits, it looks like you're always working with an entire collection, and always modifying the entire collection at once. Also, this might not even be a string collection at the outset. I don't think you'll be able to get the exact result you want, because of the base XDocument type you're working with. But one possible direction to explore might look like this:
public IEnumerable<string> Modify(IEnumerable<string> items)
{
foreach(string item in items)
{
yield return "blah";
}
}
You can use a projection to get strings from any collection type, and get your modified text back:
fruits = Modify(fruits).ToList();
var nodes = Modify( xTextNodes.Select(n => (XText)n.Value));
And once you understand how to make a projection, you may find that the existing .Select() method already does everything you need.
What I really suggest, though, is that rather than working with an entire collection, think about working in terms of one record at a time. Create a common object type that all of your data sources understand. Create a projection from each data source into the common object type. Loop through each of the objects in your projection and make your adjustment. Then have another projection back to the original record type. This will not be the original collection. It will be a new collection. Write your new collection back to disk.
Used appropriately, this also has the potential for much greater performance than your original approach. This is because working with one record at a time, using these linq projections, opens the door to streaming the data, such that only one the one current record is ever held in memory at a time. You can open a stream from the original and a stream for the output, and write to the output just as fast as you can read from the original.

The easiest way to achieve this is by doing the looping outside of the method. This allows you to pass the strings by reference which will replace the existing reference with the new one (don't forget that strings are immutable).
And example of this:
void Main()
{
string[] arr = new[] {"lala", "lolo"};
arr.Dump();
for(var i = 0; i < arr.Length; i++)
{
ModifyStrings(ref arr[i]);
}
arr.Dump();
}
public void ModifyStrings(ref string item)
{
item = "blah";
}

How to read an array of objects

I have a C# list of objects and I'm trying to access its nested object. Here's the structure from Visual Studio's debugger (sorry I can't embed images since I'm a newbie to the site):
>product.Product {Product.AxdEntity_Product_EcoResProduct[1]}
>> - [0] {Product.EcoResProductMaster} // #1 - Please note curley braces
>>> - [Product.EcoResProductMaster] // #2 - Please note brackes
>>> + base {Product....
>>> + ModelingPolicy
>> + Identifier
To access the properties in #1, I would do the following:
var prod = product.Product[0];
Then I can access "Identifier" as such:
var identifier = prod.Identifier[0]...
To access the properties in #2 (such as ModelingPolicy), I'm not sure how to go about it:
var prod = product.Product[0][WhatShouldGoHere?].ModelingPolicy[0] ...?? I need help here
Eventually, I'd like to access the ModelingPolicy[0] like I did with prod.Identifier[0].
The Product class is being returned from a web service and I don't have access to its definition. At least I don't think I do.
Thank you for any guidance!

Looks to me like:
var ecoResProductMaster = product.Product[0].EcoResProductMaster; // Product type
var modelingPolicy = ecoResProductMaster.ModelingPolicy;
Have you tried this?

var productList = product.ToList()
foreach(var prod in productList)
{
var identifier = prod.identifier
foreach (var modelingPolicy in identifier.ModelingPolicy)
{
//do somthing with modeling policy
}
//do more stuff with product
}
As an aside please sort your naming conventions out. Having an array called product that contains a number of Product objects is just confusing

It's difficult to tell what you're going for here. Is this a List or array?
If it is a list then you can just foreach through the list to access each object.
Like if you have
List<SomeObject> someObjectList = new List<SomeObject>();
Then after you add a bunch of SomeObjects to the List you can just access the SomeObjects in a foreach loop.
foreach (SomeObject s in someObjectList)
{
// then to access nested objects
var n = s.SomeNestedObject;
// do something with n
}

Thanks all for your input. The answer from SmartDev helped me access the objects in the inner array. To do this, simply do the following:
var productMaster = (EcoResProductMaster)product.Product[0];
var modelingPolicy = productMaster.ModelingPolicy[0];
Of course, this is assuming that only one object is returned in the array, hence "[0]" but this should eventually go inside a foreach loop.
What I learned was if an object is returned without an index such as //#2 as noted above, a type cast should allow me to access it. However, if an index is returned such as //#1, then using [0] will allow me to access its properties.

Creation and recalling of objects in a foreach loop?

Beginner programmer here so please keep (explanation of) answers as simple as possible.
For an assignment we got a text file that contains a large amount of lines.
Each line is a different 'Order' with an ordernumber, location, frequency (amount of times per week), and a few other arguments.
Example:
18; New York; 3PWK; ***; ***; etc
I've made it so that each line is read and stored in a string with
string[] orders = System.IO.File.ReadAllLines(#<filepath here>);
And I've made a separate class called "Order" which has a get+set for all the properties like ordernumber, etc.
Now I'm stuck on how to actually get the values in there. I know how string splitting works but not how I can make unique objects in a loop.
I'd need something that created a new Order for every line and then assign Order.ordernumber, Order.location etc.
Any help would be MUCH appreciated!

An easy approach will be to make a class to define the orders like this:
public class Order{
public string OrderNumber{get;set;}
public string OrderId{get;set;}
public string OrderSomeThingElse{get;set;}
}
Then initialize a List:
var orderList = new List<Order>();
Then loop through and populate it:
foreach( var order in orders ){
var splitString = order.Split(';');
orderList.Add( new Order{
OrderNumber = splitString[0],
OrderId = splitString[1],
OrderSomeThingElse = splitString[2]
});
}
If you want an easy, but not that elegant approach, this is it.

In addition to all the good answers you've already received. I recommend you to use File.ReadLines() instead File.ReadAllLines() Because you are reading large file.
The ReadLines and ReadAllLines methods differ as follows: When you use ReadLines, you can start enumerating the collection of strings before the whole collection is returned; when you use ReadAllLines, you must wait for the whole array of strings be returned before you can access the array. Therefore, when you are working with very large files, ReadLines can be more efficient. MSDN

Unless I misunderstand... do you mean something like this?
var ordersCollection = new List<Order>();
foreach (var order in orders)
{
var o = new Order();
o.PropertyName = ""; // Assign your property values like this
ordersCollection.Add(o);
}
// ordersCollection is now full of your orders.

Why does the Foreach doesn't impact?

The code below doing a simple thing. manipulate the API's raw data and bind it to the view model's property.
I fetch data from a web service and save it to a object named calljsonObject. This is the raw data. Then I do a foreach loop to select the selected data from the raw data, and save it to a object named tmpList, and at the last line, bind tmpList to the view model property named docList
SingleBoardListRawData calljsonObject = JsonConvert.DeserializeObject<SingleBoardListRawData>(callbackjsonstring);
ObservableCollection<SinglePostViewModel> tmpList = new ObservableCollection<SinglePostViewModel>();
SinglePostViewModel tmpPost = new SinglePostViewModel();
foreach (List<object> item in calljsonObject.data)
{
tmpPost.doc_id = (long)item[0];
tmpPost.doc_title = (string)item[1];
tmpPost.doc_author = (string)item[2];
tmpPost.repliesCount = (long)item[4];
tmpPost.doc_post_date = DateTime.Parse((string)item[5]);
tmpPost.OnTop = (long)item[6];
tmpPost.CoolPost = (long)item[7];
tmpPost.doc_update_date = DateTime.Parse((string)item[9]);
tmpPost.PicInDoc = (long)item[10];
tmpPost.reply_user_name = (string)item[11];
tmpPost.doc_brief = (string)item[13];
tmpList.Add(tmpPost);
}
this.docList = tmpList;
But the result is the tmpList full of the same data, which is the result of the last foreach manipulation. But I thought that I've re-assigned the tmpPost value, it seems every Add method will take place the previous one.
I don't want to new a object everytime I do a foreach loop, I think it cost a lot,
My Question is :
1. Why?
2. How to solve it?

You're adding a reference to the same object on each iteration, and overwriting the data within that object on each iteration. You need to create a new object on each iteration. Move this line:
SinglePostViewModel tmpPost = new SinglePostViewModel();
into the loop.
foreach (List<object> item in calljsonObject.data)
{
SinglePostViewModel tmpPost = new SinglePostViewModel();
tmpPost.doc_id = (long)item[0];
...
}
I don't want to new a object everytime I do a foreach loop, I think it cost a lot,
How do you expect to maintain the different values if you don't create that many objects? Where do you expect the data to live? And what evidence do you have for your performance concern?
It's very important that you understand how reference types and value types work. Please read my article on this topic and think about how that applies in your situation.

If we assume that SinglePostViewModel, then you are adding the same reference multiple times in the list. There is only one instance of the SinglePostViewModel - you are simply over-writing it each time, hence the fail.
I don't want to new a object everytime I do a foreach loop, I think it cost a lot
Unless you are talking 10s of millions of records, your concerns are very unlikely to be justified. The data needs to go somewhere, after all.
My Question is : 1. Why? 2. How to solve it?
because you only have one object that you keep over-writing
don't do that