Develop Reference

c# class graph to Neo4j

I'm looking to transform my in memory Plain old C# classes into a neo4j database.
(Class types are node types and derive from, nodes have a List for "linkedTo")
Rather than write a long series of cypher queries to create nodes and properties then link them with relationships I am wondering if there is anything more clever I can do.
For example can I serialize them to json and then import that directly into neo4j?
I understand that the .unwind function in the C# neo4j driver may be of help here but do not see good examples of its use and then relationships need to be matched and created separately
Is there an optimal method for doing this? i expect to have around 50k nodes

OK, first off, I'm using Neo4jClient for this and I've added an INDEX to the DB using:
CREATE INDEX ON :MyClass(Id)
This is important for the way this works, as it makes inserting the data a lot quicker.
I have a class:
public class MyClass
{
public int Id {get;set;}
public string AValue {get;set;}
public ICollection<int> LinkToIds {get;set;} = new List<int>();
}
Which has an Id which I'll be keying off, and a string property - just because. The LinkToIds property is a collection of Ids that this instance is linked to.
To generate my MyClass instances I'm using this method to randomly generate them:
private static ICollection<MyClass> GenerateMyClass(int number = 50000){
var output = new List<MyClass>();
Random r = new Random((int) DateTime.Now.Ticks);
for (int i = 0; i < number; i++)
{
var mc = new MyClass { Id = i, AValue = $"Value_{i}" };
var numberOfLinks = r.Next(1, 10);
for(int j = 0; j < numberOfLinks; j++){
var link = r.Next(0, number-1);
if(!mc.LinkToIds.Contains(link) && link != mc.Id)
mc.LinkToIds.Add(link);
}
output.Add(mc);
}
return output;
}
Then I use another method to split this into smaller 'batches':
private static ICollection<ICollection<MyClass>> GetBatches(ICollection<MyClass> toBatch, int sizeOfBatch)
{
var output = new List<ICollection<MyClass>>();
if(sizeOfBatch > toBatch.Count) sizeOfBatch = toBatch.Count;
var numBatches = toBatch.Count / sizeOfBatch;
for(int i = 0; i < numBatches; i++){
output.Add(toBatch.Skip(i * sizeOfBatch).Take(sizeOfBatch).ToList());
}
return output;
}
Then to actually add into the DB:
void Main()
{
var gc = new GraphClient(new Uri("http://localhost:7474/db/data"), "neo4j", "neo");
gc.Connect();
var batches = GetBatches(GenerateMyClass(), 5000);
var now = DateTime.Now;
foreach (var batch in batches)
{
DateTime bstart = DateTime.Now;
var query = gc.Cypher
.Unwind(batch, "node")
.Merge($"(n:{nameof(MyClass)} {{Id: node.Id}})")
.Set("n = node")
.With("n, node")
.Unwind("node.LinkToIds", "linkTo")
.Merge($"(n1:{nameof(MyClass)} {{Id: linkTo}})")
.With("n, n1")
.Merge("(n)-[:LINKED_TO]->(n1)");
query.ExecuteWithoutResults();
Console.WriteLine($"Batch took: {(DateTime.Now - bstart).TotalMilliseconds} ms");
}
Console.WriteLine($"Total took: {(DateTime.Now - now).TotalMilliseconds} ms");
}
On my aging (5-6 years old now) machine it takes about 20s to put 50,000 nodes in and around about 500,000 relationships.
Let's break into that important call to Neo4j above. The key things are as you rightly suggesting UNWIND - here I UNWIND a batch and give each 'row' in that collection the identifier of node. I can then access the properties (node.Id) and use that to MERGE a node. In the first unwind - I always SET the newly created node (n) to be the node so all the properties (in this case just AValue) are set.
So up to the first With we have a new Node created with a MyClass label, and all it's properties set. Now. This does include having an array of LinkToIds which if you were a tidy person - you might want to remove. I'll leave that to yourself.
In the second UNWIND we take advantage of the fact that the LinkToIds property is an Array, and use that to create a 'placeholder' node that will be filled later, then we create a relationship between the n and the n1 placeholder. NB - if we've already created a node with the same id as n1 we'll use that node, and when we get to the same Id during the first UNWIND we'll set all the properties of the placeholder.
It's not the easiest to explain, but in the best things to look at are MERGE and UNWIND in the Neo4j Documentation.

C# Converting List to 2d list and adding additional values

Hello need some assistance with this issue. Hopefully i can describe it well.
I have a parser that goes though a document and find sessionID's, strips some tags from them and places them into a list.
while ((line = sr.ReadLine()) != null)
{
Match sID = sessionId.Match(line);
if (sID.Success)
{
String sIDString;
String sid = sID.ToString();
sIDString = Regex.Replace(sid, "<[^>]+>", string.Empty);
sessionIDList.Add(sIDString);
}
}
Then I go thought list and get the distinctSessionID's.
List<String> distinctSessionID = sessionIDList.Distinct().ToList();
Now I need to go thought he document again and add the lines that match the sessionID and add them to the list. This is the part that I am having issue with.
Do I need to create a 2d list so I can add the matching log lines to the corresponding sessionids.
I was looking at this but cannot seem to figure out a way that I could copy over my Distinct list then add the Lines I need into the new array.
From what I can test it looks like this would add the value into the masterlist
List<List<string>> masterLists = new List<List<string>>();
Foreach (string value in distinctSessionID)
{
masterLists[0].Add(value);
}
How do I add Lines I need to the corresponding Masterlist. Say masterList[0].Add value is 1, how do i add the lines to 1?
masterList[0][0].add(myLInes);
Basically i want
Sessionid1
-------> related log line
-------> Related log line
SessionID2
-------> related log line
-------> related log line.
So on and so forth. I have the parsing all working, it's just getting the values into a 2nd string list is the issue.
Thanks,

What you can do is, simple create a class with public properties, and make list of that custom class.
public class Session
{
public int SessionId{get;set;}
public List<string> SessionLog{get;set;}
}
List<Session> objList = new List<Session>();
var session1 = new Session();
session1.SessionId = 1;
session1.SessionLog.Add("description lline1");
objList.Add(session1);

Here is one way to do it:
public class MultiDimDictList: Dictionary<string, List<int>> { }
MultiDimDictList myDictList = new MultiDimDictList ();
Foreach (string value in distinctSessionID)
{
myDictList.Add(value, new List<int>());
for(int j=0; j < lengthofLines; j++)
{
myDictList[value].Add(myLine);
}
}
You would need to replace lengthofLines with a number to indicate how many iterations of lines you have.
See Charles Bretana's answer here

take one value from multiple loops

I have a list of arrays, of which i want to take one value from each array and build up a JSON structure. Currently for every managedstrategy the currency is always the last value in the loop. How can i take the 1st, then 2nd value etc while looping the names?
List<managedstrategy> Records = new List<managedstrategy>();
int idcnt = 0;
foreach (var name in results[0])
{
managedstrategy ms = new managedstrategy();
ms.Id = idcnt++;
ms.Name = name.ToString();
foreach (var currency in results[1]) {
ms.Currency = currency.ToString();
}
Records.Add(ms);
}
var Items = new
{
total = results.Count(),
Records
};
return Json(Items, JsonRequestBehavior.AllowGet);
JSON structure is {Records:[{name: blah, currency: gbp}]}

Assuming that I understand the problem correctly, you may want to look into the Zip method provided by Linq. It's used to "zip" together two different lists, similar to how a zipper works.
A related question can be found here.

Currently, you are nesting the second loop in the first, resulting in it always returning the last currency, you have to put it all in one big for-loop for it to do what you want:
for (int i = 0; i < someNumber; i++)
{
// some code
ms.Name = results[0][i].ToString();
ms.Currency = results[1][i].ToString();
}

Flat file normalization with a dynamic number of columns

I have a flat file with an unfortunately dynamic column structure. There is a value that is in a hierarchy of values, and each tier in the hierarchy gets its own column. For example, my flat file might resemble this:
StatisticID|FileId|Tier0ObjectId|Tier1ObjectId|Tier2ObjectId|Tier3ObjectId|Status
1234|7890|abcd|efgh|ijkl|mnop|Pending
...
The same feed the next day may resemble this:
StatisticID|FileId|Tier0ObjectId|Tier1ObjectId|Tier2ObjectId|Status
1234|7890|abcd|efgh|ijkl|Complete
...
The thing is, I don't care much about all the tiers; I only care about the id of the last (bottom) tier, and all the other row data that is not a part of the tier columns. I need normalize the feed to something resembling this to inject into a relational database:
StatisticID|FileId|ObjectId|Status
1234|7890|ijkl|Complete
...
What would be an efficient, easy-to-read mechanism for determining the last tier object id, and organizing the data as described? Every attempt I've made feels kludgy to me.
Some things I've done:
I have tried to examine the column names for regular expression patterns, identify the columns that are tiered, order them by name descending, and select the first record... but I lose the ordinal column number this way, so that didn't look good.
I have placed the columns I want into an IDictionary<string, int> object to reference, but again reliably collecting the ordinal of the dynamic columns is an issue, and it seems this would be rather non-performant.

I ran into a simular problem a few years ago. I used a Dictionary to map the columns, it was not pretty, but it worked.
First make a Dictionary:
private Dictionary<int, int> GetColumnDictionary(string headerLine)
{
Dictionary<int, int> columnDictionary = new Dictionary<int, int>();
List<string> columnNames = headerLine.Split('|').ToList();
string maxTierObjectColumnName = GetMaxTierObjectColumnName(columnNames);
for (int index = 0; index < columnNames.Count; index++)
{
if (columnNames[index] == "StatisticID")
{
columnDictionary.Add(0, index);
}
if (columnNames[index] == "FileId")
{
columnDictionary.Add(1, index);
}
if (columnNames[index] == maxTierObjectColumnName)
{
columnDictionary.Add(2, index);
}
if (columnNames[index] == "Status")
{
columnDictionary.Add(3, index);
}
}
return columnDictionary;
}
private string GetMaxTierObjectColumnName(List<string> columnNames)
{
// Edit this function if Tier ObjectId is greater then 9
var maxTierObjectColumnName = columnNames.Where(c => c.Contains("Tier") && c.Contains("Object")).OrderBy(c => c).Last();
return maxTierObjectColumnName;
}
And after that it's simply running thru the file:
private List<DataObject> ParseFile(string fileName)
{
StreamReader streamReader = new StreamReader(fileName);
string headerLine = streamReader.ReadLine();
Dictionary<int, int> columnDictionary = this.GetColumnDictionary(headerLine);
string line;
List<DataObject> dataObjects = new List<DataObject>();
while ((line = streamReader.ReadLine()) != null)
{
var lineValues = line.Split('|');
string statId = lineValues[columnDictionary[0]];
dataObjects.Add(
new DataObject()
{
StatisticId = lineValues[columnDictionary[0]],
FileId = lineValues[columnDictionary[1]],
ObjectId = lineValues[columnDictionary[2]],
Status = lineValues[columnDictionary[3]]
}
);
}
return dataObjects;
}
I hope this helps (even a little bit).

Personally I would not try to reformat your file. I think the easiest approach would be to parse each row from the front and the back. For example:
itemArray = getMyItems();
statisticId = itemArray[0];
fileId = itemArray[1];
//and so on for the rest of your pre-tier columns
//Then get the second to last column which will be the last tier
lastTierId = itemArray[itemArray.length -1];
Since you know the last tier will always be second from the end you can just start at the end and work your way forwards. This seems like it would be much easier than trying to reformat the datafile.
If you really want to create a new file, you could use this approach to get the data you want to write out.

I don't know C# syntax, but something along these lines:
split line in parts with | as separator
get parts [0], [1], [length - 2] and [length - 1]
pass the parts to the database handling code

I need to move from One Series to Two Series

We are using the dotNETCHARTING to portray our charts, they accepts Series for their SeriesCollection. This is a new chart I'm working on, all the previous ones have a 1:1 relation between value shown and value extracted. Now I have a list of values to show as a list of values 12:12.
I currently have my 2 lists of data showing (Actual vs Budgetted over the past 12 months) - but in a single Series, where they should be 2 Series. I have the data sorted and listed as needed, well almost listed right.
Restrictions: .NET 3.5 (VS2008), dotNETCHARTING.
It will be a very sad solution if I had to create 12 SQLs for each month and 12 for the budgetted. From what I see that is not necessary, as soon as I find a way to seperate each list into seperate Series.
Each Module has a List<ModuleValue>, I have tried with a Dictionary<int, List<ModuleValue>> so that each series of values (12months) could have a seperate List.
I have tried the For each list of Values, add each value in the list to a Series, repeat until out of List of values. (Foreach in a Foreach)
My question is: Can anyone give me some pointers to a possible solution. Graph below is per say correct, if there weren't lined up one after the other, but started and ended at the same timeframe (month). Eg budget for Jan compares to actual for Jan. I'm not asking about the dotNETCHARTING module, they have plenty of help. I'm asking for this mid-between and how it feeds the data to the module.
Main logic body:
protected override void CreateChildControls()
{
base.CreateChildControls();
//_chart.Type = ChartType.Combo;
_chart.DefaultSeries.Type = SeriesType.Line;
// Up for change - between here
IList listSeries = new List();
listSeries.Add(GetSeries(_module)); // This line should be listSeries = GetMultipleSeries(_module); or to that effect.
foreach (var series in listSeries)
{
_chart.SeriesCollection.Add(series);
}
// Up for change - and here
// This shows the title above the chart:
_chart.Title = _module.Title;
// This shows the title below the chart:
//_chart.XAxis.Label = new Label(_module.Title);
_chart.TitleBox.Line.Color = Charter.BackgroundColor;
base.SetAreaStyles();
base.SetLinkUrl(_module.LinkUrl);
}
This logic is the old logic, should remain as is - because all the other charts rely on it.
Can be used as a point of reference. Consider this logic locked.
protected Series GetSeries(FrontModule module)
{
Series series = new Series(module.Title);
foreach (var value in module.Values)
{
string sFieldTitle = value.Text;
Element element = new Element(sFieldTitle, value.Value);
element.Color = Charter.GetColor(value.ColorIndex);
series.Elements.Add(element);
string sToolTip = string.Format
("{0}: {1:N0}"
, value.Tooltip
, value.Value);
element.ToolTip = sToolTip;
if (!string.IsNullOrEmpty(value.LinkUrl))
{
element.URL = Page.ResolveUrl(value.LinkUrl);
}
ChartTooltip += string.Concat(sToolTip, ", ");
}
ChartTooltip += "\n";
return series;
}
This is the new Logic and should be changed to reflect the desired logic. Consider this as free as can be.
protected List GetMultipleSeries(FrontModule module)
{
List listSeries = new List();
Series series = new Series(module.Title);
foreach (var keyPair in module.DictionaryValues)
{
string sFieldTitle = keyPair.Value.Text;
Element element = new Element(sFieldTitle, keyPair.Value.Value);
element.Color = Charter.GetColor(keyPair.Value.ColorIndex);
series.Elements.Add(element);
string sToolTip = string.Format
("{0}: {1:N0}"
, keyPair.Value.Tooltip
, keyPair.Value.Value);
element.ToolTip = sToolTip;
if (!string.IsNullOrEmpty(keyPair.Value.LinkUrl))
{
element.URL = Page.ResolveUrl(keyPair.Value.LinkUrl);
}
ChartTooltip += string.Concat(sToolTip, ", ");
}
listSeries.Add(series);
ChartTooltip += "\n";
return listSeries;
}
This is how it shouldn't be, listing data in a sequal line. Though it shows it has all the required data.
I'd appreciate anything you could add. Thank you.

You need two Series objects:
protected List GetMultipleSeries(FrontModule module)
{
List listSeries = new List();
Series seriesActual = new Series(module.Title);
Series seriesBudgetted = new Series(module.Title);
foreach (var keyPair in module.DictionaryValues)
{
string sFieldTitle = keyPair.Value.Text;
Element element = new Element(sFieldTitle, keyPair.Value.Value);
element.Color = Charter.GetColor(keyPair.Value.ColorIndex);
// Is is actual or budgetted
if (keyPair.Value.IsActual)
seriesActual.Elements.Add(element);
else
seriesBudgetted.Elements.Add(element);
string sToolTip = string.Format
("{0}: {1:N0}"
, keyPair.Value.Tooltip
, keyPair.Value.Value);
element.ToolTip = sToolTip;
if (!string.IsNullOrEmpty(keyPair.Value.LinkUrl))
{
element.URL = Page.ResolveUrl(keyPair.Value.LinkUrl);
}
ChartTooltip += string.Concat(sToolTip, ", ");
}
listSeries.Add(seriesActual);
listSeries.Add(seriesBudgetted);
ChartTooltip += "\n";
return listSeries;
}
I'm assuming you have some way of testing whether the points are actual or budgetted for the if statement.