Develop Reference

Finding all XPaths in a XQuery using Saxon-HE with C#

Situational Background: XSD with SCH
XML Schema (XSD)
I have an XML schema definition ("the schema") that includes several other XSDs, all in the same namespace. Some of those import other XSDs from foreign namespaces. All in all, the schema declares several global elements that can be instantiated as XML documents. Let's call them Global_1, Global_2 and Global_3.
Business Rules (SCH)
The schema is augmented by a Schematron file that defines the "business rules". It defines a number of abstract rules, and each abstract rule contains a number of assertions using the data model defined via XSD. For instance:
<sch:pattern>
<sch:rule id="rule_A" abstract="true">
<sch:assert test="if (abc:a/abc:b = '123') then abc:x/abc:y = ('aaa', 'bbb', 'ccc') else true()" id="A-01">Error message</sch:assert>
<sch:assert test="not(abc:c = 'abcd' and abc:d = 'zz')" id="A-02">Some other error message</sch:assert>
</sch:rule>
<!-- (...) -->
</sch:pattern>
Each abstract rule is extended by one or more non-abstract (concrete) rule that defines a specific context in which the abstract rule's assertions are to be validated. For example:
<sch:pattern>
<!-- (...) -->
<sch:rule context="abc:Global_1/abc:x/abc:y">
<sch:extends rule="rule_A"/>
</sch:rule>
<sch:rule context="abc:Global_2/abc:j//abc:k/abc:l">
<sch:extends rule="rule_A"/>
</sch:rule>
<!-- (...) -->
</sch:pattern>
In other words, all the assertions defined within the abstract rule_A are being applied to their specific contexts.
Both "the schema" and "the business rules" are subject to change - my program gets them at run-time and I don't know their content at design-time. The only thing I can safely assume is that there are no endless recursive structures in the schema: There is always one definite leaf node for every type and no type contains itself. Put differently, there are no "infinite loops" possible in the instances.
The Problem I want To Solve
Basically, I want to evaluate programmatically if each of the defined rules is correct. Since correctness can be quite a problematic topic, here by correctness I simply mean: Each XPath used in a rule (i.e. its context and within the XQueries of its inherited assertions) is "possible", meaning it can exist according to the data model defined in the schema. If, for instance, a namespace prefix is forgotten (abc:a/b instead of abc:a/abc:b), this XPath will never return anything other than an empty node set. The same is true if one step in the XPath is accidentally omitted, or spelled wrong, etc. This is obviously not a very strong claim for "correctness" of such a rule, but it'll do for a first step.
My Approach Towards A Solution For This
At least to me it doesn't seem like a trivial problem to evaluate an XPath (not to speak of the entire XQuery!) designed for the instance of a schema against the actual schema, given how it may contain axis steps like //, ancestor::, sibling::, etc. So I decided to construct something I would call a "maximum instance": By recursively iterating through all global elements and their children (and the structure of their respective complex types etc.), I build an XML instance at run-time that contains every possible element and attribute where it would be in the normal instance, but all at once. So every optional element/attribute, every element within a choice block and so on. So, said maximum instance would look something like this:
<maximumInstance>
<Global_1>
<abc:a>
<abc:b additionalAttribute="some_fixed_value">
<abc:j/>
<abc:k/>
<abc:l/>
</abc:b>
</abc:a>
</Global_1>
<Global_2>
<abc:x>
<abc:y>
<abc:a/>
<abc:z>
<abc:l/>
</abc:z>
</abc:y>
</abc:x>
</Global_2>
<Global_3>
<!-- ... -->
</Global_3>
<!-- ... -->
</maximumInstance>
All it takes now is to iterate over all abstract rules: And for every assertion in each abstract rule it must be checked that for every context the respective abstract rule is extended by, every XPath within an assertion results in a non-empty node set when evaluated against the maximum instance.
Where I'm stuck
I have written a C# (.NET Framework 4.8) program that parses "the schema" into said "maximum instance" (which is an XDocument at run-time). It also parses the business rules into a structure that makes it easy to get each abstract rule, its assertions, and the contexts these assertions are to be validated against.
But currently, I only have each complete XQuery (just like they are in the Schematron file) which effectively creates an assertion. But I actually need to break the XQuery down into its components (I guess I'd need the abstract syntax tree) so that I would have all individual XPaths. For instance, when given the XQuery if (abc:a/abc:b = '123') then abc:x/abc:y = ('aaa', 'bbb', 'ccc') else true(), I would need to retrieve abc:a/abc:b and abc:x/abc:y.
I assume that this could be done using Saxon-HE (or maybe another Parser/Compiler currently available for C# I don't know about). Unfortunately, I have yet to understand how to make use of Saxon well enough to even find at least a valid starting point for what I want to achieve. I've been trying to use the abstract syntax tree (so I can access the respective XPaths in the XQuery) seemingly accessible via XQueryExecutable:
Processor processor = new Processor();
XQueryCompiler xqueryCompiler = processor.NewXQueryCompiler();
XQueryExecutable exe = xqueryCompiler.Compile(xquery);
var AST = exe.getUnderlyingCompiledQuery();
var st = new XDocument();
st.Add(new XElement("root"));
XdmNode node = processor.NewDocumentBuilder().Build(st.CreateReader());
AST.explain((node); // <-- this is an error!
But that doesn't get me anywhere: I don't find any properties exposed I could work with? And while VS offers me to use AST.explain(...) (which seems promising), I'm unable to figure out what to parametrize here. I tried using a XdmNode which I thought would be a Destination? But also, I am using Saxon 10 (via NuGet), while Destination seems to be from Saxon 9: net.sf.saxon.s9api.Destination?!
Does anybody who was kind enough to read through all of this have any advice for me on how to tackle this? :-) Or, maybe there's a better way to solve my problem I haven't thought of - I'm also grateful for suggestions.
TL;DR
Sorry for the wall of text! In short: I have Schematron rules that augment an XML schema with business logic. To evaluate these rules (not: validate instances against the rules!) without actual XML instances, I need to break down the XQueries which make up the Schematron's assertions into their components so that I can handle all XPaths used in them. I think it can be done with Saxon-HE, but my knowledge is too limited to even understand what a good starting point what be for that. I'm also open for suggestions regarding a possibly better approach to solve my actual problem (as described in detail above).
Thank you for taking the time to read this.

If this were an XSD schema rather than a Schematron schema, then Saxon-EE would do the job for you automatically: this is very similar what a schema-aware XQuery processor attempts to do. But another difference is that in schema-aware XQuery, you can't assume that every element named foo is a valid instance of the element declaration named foo in the schema; it's quite legitimate, for example, for a query to transform valid instances into invalid instances, or vice versa. The input and output, after all, might conform to different schemas.
Saxon uses path analysis to do this: it looks at path expressions to see "where they might lead". Path analysis is also used to assess streamability, and to support document projection (building a trimmed-down tree representation of the source document that leaves out the parts that the query cannot reach). The path analysis in Saxon is by no means complete, for example it doesn't attempt to handle recursive functions. Although all these operations require Saxon-EE, the basic path analysis code is actually present in Saxon-HE, but I would offer no guarantee that it works for any purpose other than those described.
You're basically right that this is a tough problem you've set yourself, and I wish you luck with it.
Another approach you could adopt that wouldn't involve grovelling around the Saxon internals is to convert the XQuery to XQueryX, which is an XML representation of the parse tree, and then inspect the XQueryX (presumably using XQuery) to find the parts you need.

While XQueryX (as pointed out by Michael Kay) would theoretically have been exactly what I was looking for, unfortunately I could not find anything useful regarding an implementation for .NET during my research.
So I eventually solved the whole thing by creating my own parser using the XPath3.1 grammar for ANTLR4 as an ideal starting point. This way, I am now able to retrieve a syntax tree of any Schematron rule expression, allowing me to extract each contained XPath expression (and its sub expressions) separately.
Note that another stumbling block has been the fact that .NET still (!) only handles XPath 1.0 genuinely: While my parser does everything as supposed to, for some of the found expressions .NET gave me "illegal token" errors when trying to evaluate them. Installing the XPath2 NuGet package by Chertkov/Heyenrath was the solution.

Can't query/order on built-in rally fields "could not read all instances of class com.f4tech.slm.domain.Artifact"

I'm using v2.0 of the API via the C# dll. But this problem also happens when I pass a Query String to the v2.0 API via https://rally1.rallydev.com/slm/doc/webservice/
I'm querying at the Artifact level because I need both Defects and Stories. I tried to see what kind of query string the Rally front end is using, and it passes custom fields and built-in fields to the artifact query. I am doing the same thing, but am not finding any luck getting it to work.
I need to be able to filter out the released items from my query. Furthermore, I also need to sort by the custom c_ReleaseType field as well as the built-in DragAndDropRank field. I'm guessing this is a problem because those built-in fields are not actually on the Artifact object, but why would the custom fields work? They're not on the Artifact object either. It might just be a problem I'm not able to guess at hidden in the API. If I can query these objects based on custom fields, I would expect the ability would exist to query them by built-in fields as well, even if those fields don't exist on the Ancestor object.
For the sake of the example, I am leaving out a bunch of the setup code... and only leaving in the code that causes the issues.
var request = new Request("Artifact");
request.Order = "DragAndDropRank";
//"Could not read: could not read all instances of class com.f4tech.slm.domain.Artifact"
When I comment the Order by DragAndDropRank line, it works.
var request = new Request("Artifact");
request.Query = (new Query("c_SomeCustomField", Query.Operator.Equals, "somevalue").
And(new Query("Release", Query.Operator.Equals, "null")));
//"Could not read: could not read all instances of class com.f4tech.slm.domain.Artifact"
When I take the Release part out of the query, it works.
var request = new Request("Artifact");
request.Query = (((new Query("TypeDefOid", Query.Operator.Equals, "someID").
And(new Query("c_SomeCustomField", Query.Operator.Equals, "somevalue"))).
And(new Query("DirectChildrenCount", Query.Operator.Equals, "0"))));
//"Could not read: could not read all instances of class com.f4tech.slm.domain.Artifact"
When I take the DirectChildrenCount part out of the query, it works.
Here's an example of the problem demonstrated by an API call.
https://rally1.rallydev.com/slm/webservice/v2.0/artifact?query=(c_KanbanState%20%3D%20%22Backlog%22)&order=DragAndDropRank&start=1&pagesize=20
When I remove the Order by DragAndDropRank querystring, it works.

I think most of your trouble is due to the fact that in order to use the Artifact endpoint you need to specify a types parameter so it knows which artifact sub classes to include.
Simply adding that to your example WSAPI query above causes it to return successfully:
https://rally1.rallydev.com/slm/webservice/v2.0/artifact?query=(c_KanbanState = "Backlog")&order=DragAndDropRank&start=1&pagesize=20&types=hierarchicalrequirement,defect
However I'm not tally sure if the C# API allows you to encode additional custom parameters onto the request...

Your question already contains the answer.
UserStory (HierarchicalRequirement in WS API) and Defect inherit some of their fields from Artifact, e.g. FormattedID, Name, Description, LastUpdateDate, etc. You may use those fields in the context of Artifact type.
The fields that you are trying to access on Artifact object do not exist on it. They exist on a child level, e.g. DragAndDropRank, Release, Iteration. It is not possible to use those fields in the context of Artifact type.
Parent objects don't have access to attributes specific to child object.
Artifact is an abstract type.
If you need to filter by Release, you need to make two separate requests - one for stories, the other for defects.

Definition.Entity property is null

Whenever I create a ReloadableObjectRegistry with path to directory containing compiled modules, Definitions in the tree have Entity property set to null. The reason why I need to access this property is to be able to read the syntax of a definition.
I suspect, the Entity property only gets set after parsing a MIB... Is this a correct assumption? How else can I find out definition's syntax? The Type property is always 'Unknown'.
Sample code:
private ReloadableObjectRegistry Objects;
Objects = new ReloadableObjectRegistry(#"some_path");
P.S. By the looks of it, parsed module (.module) does not have any information about MIB Types in it.

The open source edition SharpSnmpLib.Mib was designed that way, where most of the classes are just placeholders.
If you do need to learn the syntax, you have to use SharpSnmpPro.Mib at http://sharpsnmp.com.
A sample project has been published on GitHub to demonstrate usage of the new APIs.

Updating Code, Find All Objects Of Type

Task:
Rip through all the code in the entire solution and wrap all webservice method-calls in another ws method-call that accepts a GUID (it's a login scenario)
Background :
Hundreds of web services, add token security. As explained to me when I was assigned to the task, we do it this way because if, in the future , some changes to security etc have to be made we can just do it in the WrappermethodClass in stead of having to change hundreds of web services
Tried and failed :
Find all references : too much data , returned more than 1000 hits , most of which are useless as they're only object references.
Rename WS so all references beak, build the project I'm working on and fix as I go : works well with the services not integral to the functionality but as soon as I do it with an important one it's like I shot the Solution through the brain, everything's f****d and and VS just gives up trying.
Current Solution :Open all relevant docs, Find ,select All Open Docs, skip through.
Question : How do I do this as efficiently as possible?
Code (before) :
wsGeneric wsGen = new wsGeneric();
wsGen.DoSomething();
Code (after) :
WrapperMethodClass.DoCheck takes params of (Action, GUID),
wsGeneric wsGen = new wGeneric();
wrapperMethodClass.DoCheck((g) =>
{ wsGen.UserInfo.token = g.ToString();
wsGen.DoSomething();
},Shell.token.Value);

Don´t you have some sort of interface or class where you changed the method signature already?
If you changed your webservice and your Code still compiles i´d say you did something wrong or i don´t understand the question.
Update:
I still don´t get it.
I think you have these options:
Change the method signature (all calls should be broken now, fix all the errors vs gives you and you should be done)
Find all references (of the method, not your webservice-class) and change the calls
If above isn´t possible use "Find in Files" and search for the method-name

If all your webservices inherit from an interface or base class you can refactor this method to add a parameter, all inheriting classes will also have the parameter.
If you pass a login object to each webservice, you can add a GUID element to this object and you're done.
It would be a lot easier if you showed us some code, some function interfaces that you have to change and how.

A better solution may be to just use PostSharp to add the checks to your services. This will solve your business problem (you only need to update your aspects) and is much less error prone then your current approach since you don't have to wory about some new developer forgetting to make the call to DoCheck.
Not having to find all references is a side benefit.

Integration Test for All References of a Method Invocation

So, I've been searching around on the internet for a bit, trying to see if someone has already invented the wheel here. What I want to do is write an integration test that will parse the current project, find all references to a certain method, find it's arguments, and then check the database for that argument. For example:
public interface IContentProvider
{
ContentItem GetContentFor(string descriptor);
}
public class ContentProvider : IContentProvider
{
public virtual ContentItem GetContentFor(string descriptor)
{
// Fetches Content from Database for descriptor and returns in
}
}
Any other class will get an IContentProvider injected into their constructor using IOC, such that they could write something like:
contentProvider.GetContentFor("SomeDescriptor");
contentProvider.GetContentFor("SomeOtherDescriptor");
Basically, the unit test finds all these references, find the set of text ["SomeDescriptor", "SomeOtherDescriptor"], and then I can check the database to make sure I have rows defined for those descriptors. Furthermore, the descriptors are hard coded.
I could make an enum value for all descriptors, but the enum would have thousands of possible options, and that seems like kinda a hack.
Now, this link on SO: How I can get all reference with Reflection + C# basically says it's impossible without some very advanced IL parsing. To clarify; I don't need Reflector or anything - it's just to be an automated test I can run so that if any other developers on my team check in code that calls for this content without creating the DB record, the test will fail.
Is this possible? If so, does anyone have a resource to look at or sample code to modify?
EDIT: Alternatively, perhaps a different method of doing this VS trying to find all references? The end result is I want a test to fail when the record doesnt exist.

This will be very difficult: your program may compute the value of the descriptor, which will mean your test is able to know which value are possible without executing said code.
I would suggest to change the way you program here, by using an enum type, or coding using the type safe enum pattern. This way, each and every use of a GetContentFor will be safe: the argument is part of the enum, and the languages type checker performs the check.
Your test can then easily iterate on the different enum fields, and check they are all declared in your database, very easily.
Adding a new content key requires editing the enum, but this is a small inconvenient you can live with, as it help a log ensuring all calls are safe.