Develop Reference

System.ArgumentNullException when trying to access span with Xpath (C#)

So i've been trying to get a program working where I get info from google finance regarding different stock stats. So far I have not been able to get information out of spans. As of now I have hardcoded direct access to the apple stock.
Link to Apple stock: https://www.google.com/finance?q=NASDAQ%3AAAPL&ei=NgItWIG1GIftsAHCn4zIAg
What i can't understand is that I receive correct output when I trying it in the chrome console with the following command:
$x("//*[#id=\"appbar\"]//div//div//div//span");
This is my current code in Visual studio 2015 with Html Agility Pack installed(I suspect a fault in currDocNodeCompanyName):
class StockDataAccess
{
HtmlWeb web= new HtmlWeb();
private List<string> testList;
public void FindStock()
{
var histDoc = web.Load("https://www.google.com/finance/historical?q=NASDAQ%3AAAPL&ei=q9IsWNm4KZXjsAG-4I7oCA.html");
var histDocNode = histDoc.DocumentNode.SelectNodes("//*[#id=\"prices\"]//table//tr//td");
var currDoc = web.Load("https://www.google.com/finance?q=NASDAQ%3AAAPL&ei=CdcsWMjNCIe0swGd3oaYBA.html");
var currDocNodeCurrency = currDoc.DocumentNode.SelectNodes("//*[#id=\"ref_22144_elt\"]//div//div");
var currDocNodeCompanyName = currDoc.DocumentNode.SelectNodes("//*[#id=\"appbar\"]//div//div//div//span");
var histDocText = histDocNode.Select(node => node.InnerText);
var currDocCurrencyText = currDocNodeCurrency.Select(node => node.InnerText);
var currDocCompanyName = currDocNodeCompanyName.Select(node => node.InnerText);
List<String> result = new List<string>(histDocText.Take(6));
result.Add(currDocCurrencyText.First());
result.Add(currDocCompanyName.Take(2).ToString());
testList = result;
}
public List<String> ReturnStock()
{
return testList;
}
}
I have been trying the Xpath expression [text] and received an output that i can work with when using the chrome console but not in VS. I have also been experimenting with a foreach-loop, a few suggested it to others.
class StockDataAccess
{
HtmlWeb web= new HtmlWeb();
private List<string> testList;
public void FindStock()
{
///same as before
var currDoc = web.Load("https://www.google.com/finance?q=NASDAQ%3AAAPL&ei=CdcsWMjNCIe0swGd3oaYBA.html");
HtmlNodeCollection currDocNodeCompanyName = currDoc.DocumentNode.SelectNodes("//*[#id=\"appbar\"]//div//div//div//span");
///Same as before
List <string> blaList = new List<string>();
foreach (HtmlNode x in currDocNodeCompanyName)
{
blaList.Add(x.InnerText);
}
List<String> result = new List<string>(histDocText.Take(6));
result.Add(currDocCurrencyText.First());
result.Add(blaList[1]);
result.Add(blaList[2]);
testList = result;
}
public List<String> ReturnStock()
{
return testList;
}
}
I would really appreciate if anyone could point me in the right direction.

If you check the contents of currDoc.DocumentNode.InnerHtml you will notice that there is no element with the id "appbar", therefore the result is correct, since the xpath doesn't return anything.
I suspect that the html element you're trying to find is generated by a script (js for example), and that explains why you can see it on the browser and not on the HtmlDocument object, since HtmlAgilityPack does not render scripts, it only download and parse the raw source code.

Getting "mergable revisions" with SharpSvn

When setting up a merge, the TortoiseSvn client has a wonderful checkbox labeled "Hide non-mergable revisions". I'm looking to reproduce the list of revisions that shows up when it's enabled using SharpSvn.
The TortoiseSvn documentation explains this checkbox:
When merge tracking is used, the log dialog will show previously merged revisions, and revisions pre-dating the common ancestor point, i.e. before the branch was copied, as greyed out. The Hide non-mergeable revisions checkbox allows you to filter out these revisions completely so you see only the revisions which can be merged.
How can I reproduce this functionality in SharpSvn code? I need a list of SvnLogEventArgs (or similar) that are candidates for merging.
Current status: I've only gotten as far as pulling the logs for both branches. I can't figure out how to get the appropriate svn:mergeinfo attribute or what to do with it once I get it.

I kept plugging away, and following links, and here's what I ended up with:
using (var client = new SvnClient())
{
var release = SvnTarget.FromUri(new Uri(#"https://******/branches/Release"));
var trunk = SvnTarget.FromUri(new Uri(#"https://******/trunk"));
string trunkMergeinfo, releaseMergeinfo;
client.GetProperty(release, "svn:mergeinfo", out releaseMergeinfo);
client.GetProperty(trunk, "svn:mergeinfo", out trunkMergeinfo);
var relInfos = releaseMergeinfo.Split("\n");
var trunkInfos = trunkMergeinfo.Split("\n");
// This is here because I don't know what will happen once I merge something into trunk.
Debug.Assert(relInfos.Except(trunkInfos).Count() == 1,"Too many unknown merge paths");
var trunklist = relInfos.SingleOrDefault(i => i.StartsWith("/trunk:"));
var revisions = trunklist.Replace("/trunk:", "").Split(",").SelectMany(t =>
{
// If the log contains a range, break it out to it's specific revisions.
if (t.Contains("-"))
{
var match = Regex.Match(t, #"(\d+)-(\d+)");
var start = int.Parse(match.Groups[1].Value);
var end = int.Parse(match.Groups[2].Value);
return Enumerable.Range(start, end - start + 1).ToArray();
}
else
return new[] { int.Parse(t) };
}).Select(x => (long)x);
Collection<SvnLogEventArgs> baseRevs;
// Why can't this take "trunk" or a property thereof as an argument?
client.GetLog(new Uri(#"https://******/trunk"), new SvnLogArgs { Start = 1725, End = SvnRevisionType.Head }, out baseRevs);
baseRevs.Reverse().Where(r => !revisions.Contains(r.Revision) ).Select(x => x.LogMessage).Dump();
}
Hopefully, this helps someone else, although I'll note that it does not have a lot of the sanity checking that I'd put in production code - this is the quick-and-dirty version.

Try SvnClient.ListMergesEligible:
http://sharpsvn.qqn.nl/current/html/M_SharpSvn_SvnClient_ListMergesEligible_1.htm
Edit.
SharpSVN seems bugged for me, so I went for cmd.
Check this out:
private static void mergelhetőVerziókListája()
{
string revíziók = cmd("svn", "mergeinfo --show-revs eligible \".../branches/dev\" \".../trunk\"");
}
private static string cmd(string utasítás, string paraméter)
{
StringBuilder eredmény = new StringBuilder();
Process cmd = new Process()
{
StartInfo = new ProcessStartInfo
{
FileName = utasítás,
Arguments = paraméter,
UseShellExecute = false,
RedirectStandardOutput = true,
CreateNoWindow = true
}
};
cmd.Start();
while (!cmd.StandardOutput.EndOfStream)
{
string kimenet = cmd.StandardOutput.ReadLine();
eredmény.AppendLine(kimenet); //...
}
return eredmény.ToString();
}

Generating semantic code with roslyn

We try to figure out how to generate code with Roslyn. I'm not speaking about something like CSharpSyntaxTree.ParseText that will take some strings and convert them into an AST. Instead, I would like to build my model somehow like this (pseudo code):
Create file as compilation unit
Add class MyClass to file
Add method DoSomething to MyClass
Set body of DoSomething in a similar fashion like System.Linq.Expressions
We recently discovered Microsoft.CodeAnalysis.CSharp.SyntaxFactory, and it seemed to be promising. However, obviously we have to add trivia ourselves.
After building a tree with SyntaxFactory.CompilationUnit() and adding some members back and forth, the output of ToFullString() is just a bunch of text, that is neither readable, nor compilable (e.g., missing braces). Do we miss something when generating the text from the model?
EDIT:
When using workspaces, you can set options affecting the whitespace behavior:
public string Generate (CompilationNode rootNode)
{
var cw = new CustomWorkspace();
cw.Options.WithChangedOption (CSharpFormattingOptions.IndentBraces, true);
var formattedCode = Formatter.Format (CreateFile(rootNode), cw);
return formattedCode.ToFullString();
}
This already yields a better result. Can someone confirm this as a good solution or is it rather a hack?
One problem remains. We want to generate an auto-property, currently using SF.AccessorDeclaration but it misses the semicolon when converting to the full string.

You basically have to add block definitions, then Roslyn handles the trivia for you as long as you use the Formatter (as you have written)
Here is an example for a simple class that is generated correctly without myself having to specify any trivia
var consoleWriteLine = Syntax.MemberAccessExpression(
SyntaxKind.SimpleMemberAccessExpression,
Syntax.IdentifierName("Console"),
name: Syntax.IdentifierName("WriteLine"));
var arguments = Syntax.ArgumentList (
Syntax.SeparatedList (
new[]
{
Syntax.Argument (
Syntax.LiteralExpression (
SyntaxKind.StringLiteralExpression,
Syntax.Literal (#"""Goodbye everyone!""", "Goodbye everyone!")))
}));
var consoleWriteLineStatement = Syntax.ExpressionStatement (Syntax.InvocationExpression (consoleWriteLine, arguments));
var voidType = Syntax.ParseTypeName ("void");
var method = Syntax.MethodDeclaration (voidType, "Method").WithBody (Syntax.Block(consoleWriteLineStatement));
var intType = Syntax.ParseTypeName ("int");
var getterBody = Syntax.ReturnStatement (Syntax.DefaultExpression (intType));
var getter = Syntax.AccessorDeclaration (SyntaxKind.GetAccessorDeclaration, Syntax.Block (getterBody));
var property = Syntax.PropertyDeclaration (intType, "Property").WithAccessorList (Syntax.AccessorList (Syntax.SingletonList (getter)));
var #class = Syntax.ClassDeclaration ("MyClass").WithMembers (Syntax.List (new MemberDeclarationSyntax[] { method, property }));
var cw = new CustomWorkspace();
cw.Options.WithChangedOption (CSharpFormattingOptions.IndentBraces, true);
var formattedCode = Formatter.Format (#class, cw);
Console.WriteLine (formattedCode.ToFullString());
Note: Syntax = Microsoft.CodeAnalysis.CSharp.SyntaxFactory
This generates the following class definiton:
class MyClass
{
void Method()
{
Console.WriteLine("Goodbye everyone!");
}
int Property
{
get
{
return default(int);
}
}
}
Seems fine.

I had this same problem and found CustomWorkspace is now called AdhocWorkspace.
var cw = new AdhocWorkspace();
cw.Options.WithChangedOption(CSharpFormattingOptions.IndentBraces, true);
var formatter = Formatter.Format(cu, cw);
StringBuilder sb = new StringBuilder();
using (StringWriter writer = new StringWriter(sb))
{
formatter.WriteTo(writer);
}
var code = sb.ToString();

Rhino Mocks testing file system io

I have an application that takes a dictionary of files (file type, and list of file names) and copies the files from the original directory into another location.
I've already got the basic code for the copy process, but I need to do some unit tests so it is as robust as possible.
I have wrapper class that I am using so I can test that the System.IO methods are called as I expect, but I am having some difficulty figuring out how to form the tests as there are foreach and switch statements in the code.
Sample code below:
private IFileSystemIO _f;
public CopyFilesToDestination(IFileSystemIO f){
_f = f;
}
public void Cpy_Files(Dictionary<string, List<string>> files)
{
// get a list of the file types in the directory
var listOfFileTypes = new List<string>(files.Keys);
foreach (var fileType in listOfFileTypes){
var fileList = files[fileType].ToList();
foreach (var file in fileList){
switch(fileType){
case ".txt":
_f.Copy(file, #"c:\destination\text");
break;
case ".dat":
_.Copy(file, #"c:\destination\data");
break;
}
}
}
}
To test the above I had thought I would use a mock dictionary object, set up with a list of file types and paths:
public virtual Dictionary<string, List<string>> FakeFiles(){
return fakeDictionary = new Dictionary<string, List<string>>(){
{".txt", new List<string>(){
"c:\test\file1.txt",
"c:\test\file2.txt"
}
},
{".dat", new List<string>(){
"c:\test\file1.dat",
"c:\test\file2.dat"
}
};
}
}
The first test I came up with looks like this:
[Test]
public void Should_Copy_Text_Files(){
var dictionary = new FakeDictionary().FakeFiles();
var mockObject = MockRepository.GenerateMock<IFileSystemIO>();
var systemUnderTest = new CopyFileToDestination(mockObject);
systemUnderTest.Cpy_Files(dictionary);
// I think this means "test the operation, don't check the values in the arguments" but I also think I'm wrong
mockObject.AssertWasCalled(f => f.Copy("something", "something"), o => o.IgnoreArguments());
}
My first problem is: How do I test for a specific file type, such as ".txt"?
Then how do I test the loops? I know with the mocked dictionary that I only have two items, do I leverage this to form the test? How?
I think I may be close to a solution, but I am running out of time/patience hunting it down. Any help is greatly appreciated.
Thanks
Jim

I tried using Roberts solution, but as I stated, I have too many different file types to set up each test case individually. The next thing I tried was setting up a TestCaseSource, but every time I ran the test for that it marked the test as ignored:
[Test, TestCaseSource(typeof(FakeDictionary), "TestFiles")]
public void Cpy_Files_ShouldCopyAllFilesInDictionary(string extension, string fielName) {
// Arrange
var mockObject = MockRepository.GenerateMock<IFileSystemIO>();
var systemUnderTest = new CopyFileToDestination(mockObject);
// Act
systemUnderTest.Cpy_Files(dictionary);
// Assert
mockObject.AssertWasCalled(f => f.Copy(extension, fileName));
}
The data source is below:
public static Dictionary<string, string> TestFiles{
get
{
return new Dictionary<string, string>()
{
{".txt",
"C:\\test\\test1.txt"},
{".txt",
"c:\\test\\test2.txt"}
};
}
}
What I finally worked out uses the times to repeat option in Rhino and is really pretty simple:
[Test]
public void Cpy_Files_ShouldCopyAllFilesInDictionary(){
// Arrange
var mockObject = MockRepository.GenerateMock<IFileSystemIO>();
var systemUnderTest = new CopyFileToDestination(mockObject);
// Act
systemUnderTest.Cpy_Files(dictionary);
// Assert
// I know how many objects are in my fake dictionary so I set the times to repeat as a const
const int timesToRepeat = 2;
// now I just set the values below. I am not testing file names so the test will ignore arguments
mockObject.AssertWasCalled(f => f.Copy("",""), options => options.Repeat.Times(timesToRepeat).IgnoreArguments());
}
I hope this helps someone else with a similar problem.

I would try making use of the TestCase attribute:
[TestCase(".txt", "c:\test\file1.txt")]
[TestCase(".txt", "c:\test\file2.txt")]
[TestCase(".dat", "c:\test\file1.dat")]
[TestCase(".dat", "c:\test\file2.dat")]
public void Should_Copy_Text_Files(string extension, string fileName){
var dictionary = new FakeDictionary().FakeFiles();
var mockObject = MockRepository.GenerateMock<IFileSystemIO>();
var systemUnderTest = new CopyFileToDestination(mockObject);
systemUnderTest.Cpy_Files(dictionary);
mockObject.AssertWasCalled(f => f.Copy(extension, fileName));
}
This will run the test separately for each TestCase attribute, passing the parameters it contains into the test method. That way you can test that each item in your dictionary was "copied" without using multiple asserts in the same test.

Adding AsParallel() call cause my code to break on writing a file

I'm building a console application that have to process a bunch of document.
To stay simple, the process is :
for each year between X and Y, query the DB to get a list of document reference to process
for each of this reference, process a local file
The process method is, I think, independent and should be parallelized as soon as input args are different :
private static bool ProcessDocument(
DocumentsDataset.DocumentsRow d,
string langCode
)
{
try
{
var htmFileName = d.UniqueDocRef.Trim() + langCode + ".htm";
var htmFullPath = Path.Combine("x:\path", htmFileName;
missingHtmlFile = !File.Exists(htmFullPath);
if (!missingHtmlFile)
{
var html = File.ReadAllText(htmFullPath);
// ProcessHtml is quite long : it use a regex search for a list of reference
// which are other documents, then sends the result to a custom WS
ProcessHtml(ref html);
File.WriteAllText(htmFullPath, html);
}
return true;
}
catch (Exception exc)
{
Trace.TraceError("{0,8}Fail processing {1} : {2}","[FATAL]", d.UniqueDocRef, exc.ToString());
return false;
}
}
In order to enumerate my document, I have this method :
private static IEnumerable<DocumentsDataset.DocumentsRow> EnumerateDocuments()
{
return Enumerable.Range(1990, 2020 - 1990).AsParallel().SelectMany(year => {
return Document.FindAll((short)year).Documents;
});
}
Document is a business class that wrap the retrieval of documents. The output of this method is a typed dataset (I'm returning the Documents table). The method is waiting for a year and I'm sure a document can't be returned by more than one year (year is part of the key actually).
Note the use of AsParallel() here, but I never got issue with this one.
Now, my main method is :
var documents = EnumerateDocuments();
var result = documents.Select(d => {
bool success = true;
foreach (var langCode in new string[] { "-e","-f" })
{
success &= ProcessDocument(d, langCode);
}
return new {
d.UniqueDocRef,
success
};
});
using (var sw = File.CreateText("summary.csv"))
{
sw.WriteLine("Level;UniqueDocRef");
foreach (var item in result)
{
string level;
if (!item.success) level = "[ERROR]";
else level = "[OK]";
sw.WriteLine(
"{0};{1}",
level,
item.UniqueDocRef
);
//sw.WriteLine(item);
}
}
This method works as expected under this form. However, if I replace
var documents = EnumerateDocuments();
by
var documents = EnumerateDocuments().AsParrallel();
It stops to work, and I don't understand why.
The error appears exactly here (in my process method):
File.WriteAllText(htmFullPath, html);
It tells me that the file is already opened by another program.
I don't understand what can cause my program not to works as expected. As my documents variable is an IEnumerable returning unique values, why my process method is breaking ?
thx for advises
[Edit] Code for retrieving document :
/// <summary>
/// Get all documents in data store
/// </summary>
public static DocumentsDS FindAll(short? year)
{
Database db = DatabaseFactory.CreateDatabase(connStringName); // MS Entlib
DbCommand cm = db.GetStoredProcCommand("Document_Select");
if (year.HasValue) db.AddInParameter(cm, "Year", DbType.Int16, year.Value);
string[] tableNames = { "Documents", "Years" };
DocumentsDS ds = new DocumentsDS();
db.LoadDataSet(cm, ds, tableNames);
return ds;
}
[Edit2] Possible source of my issue, thanks to mquander. If I wrote :
var test = EnumerateDocuments().AsParallel().Select(d => d.UniqueDocRef);
var testGr = test.GroupBy(d => d).Select(d => new { d.Key, Count = d.Count() }).Where(c=>c.Count>1);
var testLst = testGr.ToList();
Console.WriteLine(testLst.Where(x => x.Count == 1).Count());
Console.WriteLine(testLst.Where(x => x.Count > 1).Count());
I get this result :
0
1758
Removing the AsParallel returns the same output.
Conclusion : my EnumerateDocuments have something wrong and returns twice each documents.
Have to dive here I think
This is probably my source enumeration in cause

I suggest you to have each task put the file data into a global queue and have a parallel thread take writing requests from the queue and do the actual writing.
Anyway, the performance of writing in parallel on a single disk is much worse than writing sequentially, because the disk needs to spin to seek the next writing location, so you are just bouncing the disk around between seeks. It's better to do the writes sequentially.

Is Document.FindAll((short)year).Documents threadsafe? Because the difference between the first and the second version is that in the second (broken) version, this call is running multiple times concurrently. That could plausibly be the cause of the issue.

Sounds like you're trying to write to the same file. Only one thread/program can write to a file at a given time, so you can't use Parallel.
If you're reading from the same file, then you need to open the file with only read permissions as not to put a write lock on it.
The simplest way to fix the issue is to place a lock around your File.WriteAllText, assuming the writing is fast and it's worth parallelizing the rest of the code.