I'm optimizing some code we use to query Active Directory. One of the methods fetches all of the AD users who have changed since a particular update, determined by the uSNCreated property of the Directory Entry. Essentially it's doing the C# equivalent of:
select * from PrincipalSearcher where uSNCreated > somevalue
The code is (more or less):
public IEnumerable<UserPrincipal> GetUpdatedUsers(string samAccountName, long lastUsnChanged)
{
using (var context = new PrincipalContext(ContextType.Domain))
using (var userSearcher = new PrincipalSearcher(new UserPrincipal(context)))
{
var items = userSearcher.FindAll().Cast<UserPrincipal>();
return items.Where(x => GetUsnChanged(x) > lastUsnChanged).ToArray();
}
}
private static long GetUsnChanged(Principal item)
{
var de = item.GetUnderlyingObject() as DirectoryEntry;
if (de == null)
return 0;
if (!de.Properties.Contains("uSNCreated"))
return 0;
var usn = de.Properties["uSNCreated"].Value;
var t = usn.GetType();
var highPart = (int)t.InvokeMember("HighPart", BindingFlags.GetProperty, null, usn, null);
var lowPart = (int)t.InvokeMember("LowPart", BindingFlags.GetProperty, null, usn, null);
return highPart * ((long)uint.MaxValue + 1) + lowPart;
}
Now this code DOES work, but the repeated calls to InvokeMember() are SLOW. What I'd like to do is get a reference to the HighPart and LowPart properties so that I can call them over and over without the overhead of needing to "rediscover" them every time when calling InvokeMember().
I'd though that I could do something along the lines of
static PropertyInfo highProp = highProp
?? t.GetProperty("HighPart", BindingFlags.GetProperty);
highPart = (int)highProp.GetValue(usn);
Unfortnately t.GetProperty() always returns null. Looking at the results returned by GetProperties(), GetMethods() and GetMembers(), there doesn't seem to be a visible "HighPart" or "LowPart" that I can get to, even when using BindingFlags.NonPublic - the __ComObject simply doesn't seem to expose them (even though I can call the using InvokeMember())
Is there a way to solve this, or is it time to admit defeat?
Classes from the System.DirectoryServices.AccountManagement namespace are designed for use in simple cases, e. g. you need to find a user or group. These classes have known perfomance issues. I'd recommend using DirectorySearcher or LdapConnection/SearchRequest. In this case you can filter objects on the server, not on the client which will significantly increase performance and reduce data sent over network. Here is an example of using DirectorySearcher to find all users: Get all users from AD domain
In your case the filter will look like (&(objectClass=user)(uSNCreated>=x+1)) where x is your last usn.
Be aware that you if you track objects with usnCreated attribute you will be getting only users that were created since last usn. To track changes use usnChanged attribute
Related
I've been grinding on this for quite a bit and keep hitting my head against the wall. I've read through tons of posts trying to figure it out but I'm obviously doing something wrong. When I use DirectorySearcher with Sizelimit under 1000, it works fine... (well, the code works fine, only returning the first 1000 presents a lot of issues :P) I'm trying to retrieve 4012 groups. So, I read that I need to use PageSize=1000 instead, which will return all objects on multiple pages. Great, that's what I needed.
My problem is, when I attempt to do that I get null values when trying to look at properties of each returned object. I'm not understanding why nor have I found other posts with the same issue... I imagine I'm doing something dumb as I'm new to C#, so I'm asking for someone to point it out to me :D
What I'm trying to accomplish:
I'm trying to get all distribution groups the currently logged in user either is the manager of or the co-manager of (in Exchange as an Owner, AD Value is "msExchCoManagedByLink") and display those groups in a ListBox.
Function I'm trying to use to do this:
private void FillGroups()
{
DL_listBox.Items.Clear();
DirectoryEntry entry = new DirectoryEntry("LDAP://<mydomain>");
string DN = GetDistinguishedName();
using (DirectorySearcher searcher = new DirectorySearcher(entry))
{
searcher.PageSize = 1000;
searcher.Filter = string.Format("(&(objectCategory=group)(!(groupType:1.2.840.113556.1.4.803:=-2147483648)))");
searcher.PropertiesToLoad.Add("displayname");
searcher.PropertiesToLoad.Add("managedby");
searcher.PropertiesToLoad.Add("msExchCoManagedByLink");
SearchResultCollection result = searcher.FindAll();
foreach (SearchResult grp in result)
{
List<string> DNs = new List<string>();
string mgr = grp.Properties["managedby"][0].ToString();
string g = grp.Properties["displayname"][0].ToString();
foreach (string t in grp.Properties["msExchCoManagedByLink"])
{
DNs.Add(t);
}
if (mgr == DN)
{
DL_listBox.Items.Add(g);
}
foreach (mgrc in DNs)
{
if (mgrc == DN)
{
DL_listBox.Items.Add(g);
}
}
}
}
}
Error I get when I run this:
On either of the lines using grp.Properties[" "][0]:
System.ArgumentOutOfRangeException: 'Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index'
Looking through the information in the error, the properties value being returned is null. I don't understand why that is if it works when it's limited to 1000.
I read up on PrincipalSearcher a bit, but was unable to validate I could poll the co-manager value using it. I'm also not sure I wouldn't hit the same issue. Any advice would be appreciated, I'm gonna keep trying other things but felt I'm at a point to ask someone more experienced to avoid wasting more days on it :)
Thank you for reading, this is my first post :D
Update:
I was able to bypass this issue after a few hours of playing with it by doing the below:
private void FillGroups()
{
DL_listBox.Items.Clear();
DirectoryEntry entry = new DirectoryEntry("LDAP://<mydomain>");
using (DirectorySearcher searcher = new DirectorySearcher(entry))
{
//searcher.PageSize = 1000;
searcher.SizeLimit = 200;
searcher.Filter = string.Format("(&(objectCategory=group)(!(groupType:1.2.840.113556.1.4.803:=-2147483648))(managedby={0}))", GetDistinguishedName());
searcher.PropertiesToLoad.Add("displayname");
SearchResultCollection result = searcher.FindAll();
foreach (SearchResult grp in result)
{
string g = grp.Properties["DisplayName"][0].ToString();
DL_listBox.Items.Add(g);
}
}
using (DirectorySearcher dsearcher = new DirectorySearcher(entry))
{
//searcher.PageSize = 1000;
dsearcher.SizeLimit = 200;
dsearcher.Filter = string.Format("(&(objectCategory=group)(!(groupType:1.2.840.113556.1.4.803:=-2147483648))(msExchCoManagedByLink:={0}))", GetDistinguishedName());
dsearcher.PropertiesToLoad.Add("displayname");
dsearcher.PropertiesToLoad.Add("managedby");
dsearcher.PropertiesToLoad.Add("msExchCoManagedByLink");
SearchResultCollection result = dsearcher.FindAll();
foreach (SearchResult grp in result)
{
string g = grp.Properties["DisplayName"][0].ToString();
DL_listBox.Items.Add(g);
}
}
}
I'd still like to understand why the initial code doesn't work though, if anyone could help me understand that I would be grateful! :)
I was able to find an answer somewhere else, but figured I'd post it here in case anyone else ran into this oddity ;) When I looked at the debugger of what was being returned in user.Properties when the exception hit, I saw it was pulling a contact object that had "\" in the displayname. Adding (objectClass=user) to the LDAP query removed these and everything started working perfectly.
Excluding contact objects is probably a good thing to do anyway, but you still could run into this issue or a similar one. The issue is that those attributes are optional - they don't have to be populated with anything.
The exception you saw ("Index was out of range") I would only expect to see for a multi-value attribute, which both displayName and managedBy are not. Maybe you were using grp.Properties["msExchCoManagedByLink"][0] at some point?
In the case of a multi-value attribute, you will see an empty collection when it is not set. So when you try to access the value at index 0, it throws that exception. You can avoid that by checking grp.Properties["msExchCoManagedByLink"].Count before trying to access the value.
However, even in your updated code, you could run into a null reference exception. The displayName attribute is not a required attribute either. When it is not set, the attribute will not exist in the Properties collection at all. (i.e. grp.Properties["displayName"] will be null) That means that grp.Properties["DisplayName"][0].ToString(); will throw a null reference exception if displayName is not set. You can avoid this by either checking grp.Properties.Contains("displayName") before setting it:
string g = grp.Properties.Contains("displayName") ? grp.Properties["DisplayName"][0].ToString() : null;
Or use the null conditional operator:
string g = grp.Properties["DisplayName"]?[0].ToString();
I've tried to reduce this example to remove the OrganizationServiceProxy/XrmServiceContext, but now I believe the issue is originating there. I'm pulling data from a Dynamics 365 instance using the code generated by CrmSvcUtil.exe from Microsoft. Here is the relevant snippet:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
CrmServiceClient client = new CrmServiceClient(userName, CrmServiceClient.MakeSecureString(password), string.Empty, orgName, isOffice365: true);
using (OrganizationServiceProxy service = client.OrganizationServiceProxy)
{
XrmServiceContext crm = new XrmServiceContext(service);
var q = crm.CreateQuery<Account>();
List<Account> accounts = q.Where(x => x.AccountNumber != x.dynamics_integrationkey).ToList();
}
I get the following exception:
variable 'x' of type '{company-specific}.Account' referenced from scope '', but it is not defined
This is happening on the last line when the Linq is actually executed. It's only when it's created through the Microsoft.Xrm.Sdk.Client.OrganizationServiceContext that I get the error.
Things I've checked:
I get the error when comparing any two fields on the same object (not just Account), even when both are the same exact data type.
If I replace crm.CreateQuery... with a hand-populated List...AsQueryable() the Linq works fine
Most frustratingly, if I work through an intermediary list with List<Account> a = q.ToList(); to rasterize the result set first then this all works fine. It takes much longer to run because I've forfeited any lazy loading, but it works without error.
I've looked at other answers related to the referenced from scope '' but not defined Linq error, but they are all about malformed Linq. I know this Linq is well formed because it works fine if I change the underlying Queryable.
Clearly something is different between a List...ToQueryable() and the Microsoft.Xrm.Sdk.Linq.Query<> created by the XrmServiceContext (which inherits CreateQuery from Microsoft.Xrm.Sdk.Client.OrganizationServiceContext), but I don't know what it is.
The query you are trying to build (compare a field value with another field value on the same entity) is not possible in Dynamics.
The LINQ Provider still uses QueryExpression under the hood, meaning that your LINQ condition will be converted in a ConditionExpression, and that comparison is not possible.
Similar questions:
https://social.microsoft.com/Forums/en-US/d1026ed7-56fd-4f54-b382-bac2fc5e46a7/linq-and-queryexpression-compare-entity-fields-in-query?forum=crmdevelopment
Dynamics QueryExpression - Find entity records where fieldA equals fieldB
This morning I realized that my answer contained the flaw of comparing two fields against each other in a LINQ query. My bad. Thank you Guido for calling that out as impossible.
I generate my proxy classes with a 3rd party tool, so I'm not familiar with the XrmServiceContext class. But, I have confirmed that XrmServiceContext inherits from Microsoft.Xrm.Client.CrmOrganizationServiceContext, which inherits from Microsoft.Xrm.Sdk.Client.OrganizationServiceContext
I use OrganizationServiceContext for LINQ queries.
If you retrieve all the accounts first by calling ToList() on the query, you can then do the comparison of the two fields. If you have too many Accounts to load all at once, you can page through them and store the mismatched ones as you go.
And, I would probably retrieve a subset of the fields rather than the whole record (as shown).
var client = new CrmServiceClient(userName, CrmServiceClient.MakeSecureString(password), string.Empty, orgName, isOffice365: true);
using (var ctx = new OrganizationServiceContext(client))
{
var q = from a in ctx.CreateQuery<Account>()
where a.AccountNumber != null
&& a.dynamics_integrationkey != null
select new Account
{
Id = a.AccountId,
AccountId = a.AccountId,
Name = a.Name,
AccountNumber = a.AccountNumber,
dynamics_integrationkey = a.dynamics_integrationkey
};
var accounts = q.ToList();
var mismatched = accounts.Where(a => a.AccountNumber != a.dynamics_integrationkey).ToList()
}
I'm using TweetInvi to grab a bunch of tweets that match a specified hashtag. I do this with the following:
var matchingTweets = Search.SearchTweets(hashtag);
This returns an IEnumerable (named ITweet, interface of Tweet), however I cannot create a List<> of Tweets, because Tweet is a static type.
I made, instead, a list of objects, using:
List<object> matches = matchingTweets.Cast<object>().ToList();
However, although each member of the matchingTweets IEnumerable has a number of properties, I cannot access them using:
long tweetID = matches[i].<property>;
Using matches[i].ToString() returns the tweet content, so how can I effectively cast the results in matchingTweets to a list, and subsequently access the properties of those list members? I would ideally like to avoid using dynamic.
In your example above you were trying to grab the ID from the tweet. ITweet implements ITweetIdentifier which contains the Id property. You can literally just access it by:
var matchingTweets = Search.SearchTweets(hashtag);
//Grab the first 5 tweets from the results.
var firstFiveTweets = matchingTweets.Take(5).ToList();
//if you only want the ids and not the entire object
var firstFiveTweetIds = matchingTweets.Take(5).Select(t => t.Id).ToList();
//Iterate through and do stuff
foreach (var tweet in matchingTweets)
{
//These are just examples of the properties accessible to you...
if(tweet.Favorited)
{
var text = tweet.FullText;
}
if(tweet.RetweetCount > 100)
{
//TODO: Handle popular tweets...
}
}
//Get item at specific index
matchingTweets.ElementAt(index);
I don't know exactly what you want to do with all the info, but since the SearchTweets returns a IEnumerable of ITweets you have access to anything an ITweet has defined.
I highly recommend looking through their wiki. It's pretty well organized and gives you clear examples of some basic tasks.
It makes sense you cannot access the properties. You cast it into object so you can only access the objects properties and methods (that like you said might have been overridden).
It should be fine to just access it like this:
List<ITweet> tweets = matchingTweets.Take(5).ToList();
What you can do is project it to a new object of yours:
var tweets = matchingTweets.Select(item => new {
property1 = item.property1,
property2 = item.property2
})
.Take(5).ToList();
Then you will be able to access what you need. Now, if you need to share this data outside the scope of that function create a DTO object and initialize it instead of the anonymous type.
Depending on the size of the project and amount of effort usually it is in any case a good practice to create a layer of DTO objects when you interact with an external service like this. Then if their models changed you can contain your changes only to the DTOs.
If all you want are the ids of the first 5 then:
var ids = matchingTweets.Take(5).Select(item => item.id).ToList();
I have a loop that retrieves some info from ActiveDirectory. It turned out to be a big performance bottleneck.
This snippet (inside a loop that executed it 31 times) took 00:01:14.6562500 (1 minute and 14 seconds):
SearchResult data = searcher.FindOne();
System.Diagnostics.Trace.WriteLine(PropsDump(data));
Replacing it with this snippet brought it down to 00:00:03.1093750 (3 secconds):
searcher.SizeLimit = 1;
SearchResultCollection coll = searcher.FindAll();
foreach (SearchResult data in coll)
{
System.Diagnostics.Trace.WriteLine(PropsDump(data));
}
The results are exactly identical, the same properties are returned in the same order. I found some info on memory leaks in another thread, but they did not mention performance (I'm on .Net 3.5).
The following is actually a different question but it gives some background on why I'm looping in the first place:
I wanted to get all the properties in one single query, but I cannot get the DirectorySearcher to return all the wanted properties in one go (it omits about 30% of the properties specified in PropertiesToLoad (also tried setting it in the constructor wich makes no difference), I found someone else had the same problem and this is his solution (to loop through them). When I loop through them like this, either using FindOne() or FindAll() I do get all the properties, But actually it all feels like a workaround.
Am I missing something?
Edit:
Seems like the problem was with the way I got the first DirectoryEntry on which I was using the DirectorySearcher.
This was the code that caused the DirectorySearcher only to return some of the properties:
private static DirectoryEntry GetEntry() {
DirectoryContext dc = new DirectoryContext(DirectoryContextType.DirectoryServer, "SERVERNAME", "USERNAME", "PASSWORD");
Forest forest = Forest.GetForest(dc);
DirectorySearcher searcher = forest.GlobalCatalogs[0].GetDirectorySearcher();
searcher.Filter = "OU=MyUnit";
searcher.CacheResults = true;
SearchResultCollection coll = searcher.FindAll();
foreach (SearchResult m in coll)
{
return m.GetDirectoryEntry();
}
throw new Exception("DirectoryEntry not found");
}
After replacing that big mouthfull with just this line, the DirectorySearcher returned all the properties and looping was no longer needed:
private static DirectoryEntry GetEntry2()
{
return new DirectoryEntry(#"LDAP://SERVERNAME/OU=MyUnit,DC=SERVERNAME,DC=local", "USERNAME", "PASSWORD");
}
Now it takes less than one 18th of a second to get all wanted properties of 31 entries.
So, it seems that two different instances of the same DirectoryEntry can give different results depending on the way it was constructed... feels a bit creepy!
Edit
Used JetBrains DotPeek to look at the implementation. The FindOne function starts like this:
public SearchResult FindOne()
{
SearchResult searchResult1 = (SearchResult) null;
SearchResultCollection all = this.FindAll(false);
...
My first reaction was Argh! no wonder... but then I noticed the argument. FindAll has a private version that accepts a boolean, this is the start of FindAll:
[TargetedPatchingOptOut("Performance critical to inline this type of method across NGen image boundaries")]
public SearchResultCollection FindAll()
{
return this.FindAll(true);
}
private SearchResultCollection FindAll(bool findMoreThanOne)
{
... // other code
this.SetSearchPreferences(adsSearch, findMoreThanOne);
So this gives slightly more insight, but does not really explain much.
New answer for the new stuff. Your first method was using the Global Catalog, so it was like using
private static DirectoryEntry GetEntry3()
{
return new DirectoryEntry(#"GC://SERVERNAME/OU=MyUnit,DC=SERVERNAME,DC=local", "USERNAME", "PASSWORD");
}
Also, Microsoft LDAP libraries usually have a way to tell it whether you're giving the server name, because it makes some optimizations that can be really slow if you don't say it was a server name. For DirectoryEntry, it's the constructor with the most arguments and AuthenticationTypes.ServerBind.
Looping is not a good idea. I'm going to analyze that guy's code:
objGroupEntry = sr.GetDirectoryEntry();
dso = new DirectorySearcher(objGroupEntry);
dso.ClientTimeout = TimeSpan.FromSeconds(30);
dso.PropertiesToLoad.Add("physicalDeliveryOfficeName");
dso.PropertiesToLoad.Add("otherFacsimileTelephoneNumber");
dso.PropertiesToLoad.Add("otherTelephone");
dso.PropertiesToLoad.Add("postalCode");
dso.PropertiesToLoad.Add("postOfficeBox");
dso.PropertiesToLoad.Add("streetAddress");
dso.PropertiesToLoad.Add("distinguishedName");
dso.SearchScope = SearchScope.OneLevel;
dso.Filter = "(&(objectClass=top)(objectClass=person)(objectClass=organizationalPerson)(objectClass=user))";
dso.PropertyNamesOnly = false;
SearchResult pResult = dso.FindOne();
if (pResult != null)
{
offEntry = pResult.GetDirectoryEntry();
foreach (PropertyValueCollection o in offEntry.Properties)
{
this.Controls.Add(new LiteralControl(o.PropertyName + " = " + o.Value.ToString() + "<br/>"));
}
}
I don't know why he's doing two searches, but let's assume there's a good reason. He should have gotten those properties from the SearchResult, not from the return value of pResult.GetDirectoryEntry, because its a completely new object.
string postalCode = pResult.Properties["postalCode"][0] as string;
List<string> otherTelephones = new List<string>();
foreach(string otherTelephone in pResult.Properties["otherTelephone"])
{
otherTelephones.Add(otherTelephone);
}
If you insist on getting the DirectoryEntry, then ask for all of the properties at once with RefreshCache:
offEntry = pResult.GetDirectoryEntry();
offEntry.RefreshCache(propertyNameArray);
If none of that helps, look at your filters and see if you can use the BaseLevel scope.
I have two goals with the below code
1) Get a list of Users who below to a specific AD group
2) Get the email/lastname/first name of all the users that belong to that group
If there is a better way to accomplish both please let me know.
I'm able to get the full DN but I'm not sure how to get the remaining data from the full DN, or if there is a better way to pull this info please let me know. below is the code I'm using but it gets error:
The value provided for adsObject does not implement IADs.
when I tried to do a DirectorySearcher using the full DN.
HashSet<string> User_Collection = new HashSet<string>();
SearchResultCollection sResults = null;
DirectoryEntry dEntryhighlevel = new DirectoryEntry("LDAP://CN=Global_Users,OU=Astrix,OU=Clients,OU=Channel,DC=astro,DC=net");
foreach (object dn in dEntryhighlevel.Properties["member"])
{
DirectoryEntry dEntry = new DirectoryEntry(dn);
Console.WriteLine(dn);
DirectorySearcher dSearcher = new DirectorySearcher(dEntry);
//filter just user objects
dSearcher.SearchScope = SearchScope.Base;
//dSearcher.Filter = "(&(objectClass=user)(dn="+dn+")";
dSearcher.PageSize = 1000;
sResults = dSearcher.FindAll();
foreach (SearchResult sResult in sResults)
{
string Last_Name = sResult.Properties["sn"][0].ToString();
string First_Name = sResult.Properties["givenname"][0].ToString();
string Email_Address = sResult.Properties["mail"][0].ToString();
User_Collection.Add(Last_Name + "|" + First_Name + "|" + Email_Address);
}
Speed is important and yes I understand I'm not using HashSet as it's designed.
I always use the System.DirectoryServices.AccountManagement.
One of the first things you will see is this: "Connections speeds are increased by using the Fast Concurrent Bind (FSB) feature when available. Connection caching decreases the number of ports used."
with that being said I did not test your code against this for speed you will have to do that your self but this is Microsoft's new library.
Here is my code example:
// Create the context for the principal object.
PrincipalContext ctx = new PrincipalContext(ContextType.Domain,
"fabrikam",
"DC=fabrikam,DC=com");
// Create an in-memory user object to use as the query example.
GroupPrincipal u = new GroupPrincipal(ctx) {DisplayName = "Your Group Name Here"};
// Set properties on the user principal object.
// Create a PrincipalSearcher object to perform the search.
PrincipalSearcher ps = new PrincipalSearcher {QueryFilter = u};
// Tell the PrincipalSearcher what to search for.
// Run the query. The query locates users
// that match the supplied user principal object.
PrincipalSearchResult<Principal> results = ps.FindAll();
foreach (UserPrincipal principal in ((GroupPrincipal)results.FirstOrDefault()).Members)
{
string email = principal.EmailAddress;
string name = principal.Name;
string surname = principal.Surname;
}
It looks like you're walking the group membership of some group in AD...(guessing this off of the member reference above)
Anyway, you need to decide what sort of API you're looking for. The one you're using now is a bit lower level (though you could go lower if you want :)). Going higher level is an option as the previous answer eludes to.
To shake out the code a bit more (and help with perf as you mentioned that it is impt to you):
Use the same connection you used for the group membership search itself (ie no additional connect/bind)
Do a base search where the base DN is the user DN, the search filter is (objectclass=*) and the attributes are ONLY the attributes you care about (no *)
You can remove page size. Paging is a way to ask for many objects in groups (aka pages) but a base search only returns 1 object so it doesn't actually do anything.
Base search result count should always be 1.
Keep in mind cross-domain issues too. Make sure you test your code with a 2 domain forest where a group of type domain local in domain1 has a member in it from domain 2. That will yield some additional work to get more properties as you need to connect to a DC in the other domain (or a GC if the few properties you care about are all in the GC partial attribute set...)
Also keep in mind security. If you don't have access to these properties for some user in your domain, what does the code do? The code above would fail in a nasty way. :) You might want to handle this more gracefully...
Hope this helps.
~Eric