I'm trying to call Lists.GetListItems with a CAML query. It works fine when I don't include a query parameter, but I get a time-out error when I include the query. Assuming that my web service is called TestWebService, code is:
public static void GetListItemsTest()
{
try
{
string listName = "TestList";
string[] fields = { "ID", "Title" };
string queryStr =
"<GroupBy collapse='true'>" +
"<FieldRef Name='" + fields[1] + "' />" +
"</GroupBy>" +
"<OrderBy>" +
"<FieldRef Name='ID' Ascending='False' />" +
"</OrderBy>";
XmlDocument xmlDoc = new System.Xml.XmlDocument();
XmlElement query = xmlDoc.CreateElement("Query");
XmlElement viewFields = xmlDoc.CreateElement("ViewFields");
query.InnerXml = queryStr;
viewFields.InnerXml = "";
TestWebService.Lists lists = new TestWebService.Lists();
lists.Url = "http://wss/sites/TestWebService/_vti_bin/lists.asmx";
lists.Credentials = System.Net.CredentialCache.DefaultCredentials;
XmlNode responseNode = lists.GetListItems(listName, null, query, viewFields, "1000000", null, null);
Console.WriteLine("ran successfully");
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
finally
{
Console.ReadLine();
}
}
The call to GetListItems hangs, then throws a WebException with the message "The operation has timed out." If I change the query.InnerXml assignment to:
query.InnerXml="";
then GetListItems successfully returns a value for responseNode. So it must have something to do with the CAML query fragment.
On further investigation, I found that the GroupBy element is the source of the problem; with the OrderBy element commented out, the query times out, and with the GroupBy element commented out, the query succeeds. I'm wondering if GroupBy can't handle large data sets, or it handles them too slowly; this set has about 20,000 elements, with generally not more than 5 or 6 duplicates of each Title value. Would LINQ be more applicable in this situation? Or is there a way to do this with straight CAML?
With 20,000 items on a SharePoint 2010 system you might risk getting an SPQueryThrottledException. A typical user is limited to 5000, and a few privileged users to 20,000 items returned by a query.
Have you set up an index on the Title column? This might improve performance on the GroupBy. Otherwise I suspect that the query forces the retrieval of all items, and then groups them. Try grouping them, and retrieving them in pages, rather than all at once.
I don't know it it's entirely relevant, but I did just blog about queries on large lists.
Related
I've got a LINQ query in a WCF service that runs and returns the correct number of results that I'm looking for, but repeats the first result 25 times instead of showing me all 25 different records.
The weird thing is that when I take the SQL query that it generates from the debugger and plug it into SQL Management studio, I get the correct results.
I have tried refreshing the view I'm querying from the edmx, and I've tried rewriting the query a few different ways, but I'm starting to run out of ideas.
I've included some of the code below. Any suggestions would be helpful. Thanks!
try
{
using (Entities db = new Entities())
{
var qInventory = db.vw_Web_Store_Inventory_Live
.Where(qi => qi.Sku_Number == inputSKU)
.ToList();
resultPInventory.SKU = inputSKU;
resultPInventory.StoreInventory = new List<StoreItem>();
foreach (var qi in qInventory)
{
resultPInventory.StoreInventory.Add(new StoreItem
{
StoreNum = qi.Store_Number,
Quantity = qi.Curr_Inv
});
}
}
}
catch (Exception e)
{
log.Error("[" + e.TargetSite + "] | " + e.Message);
}
log.Info("ProductInventory(" + inputSKU + ") returned " + resultPInventory.StoreInventory.Count + " results");
return resultPInventory;
As Gert's link in the comments points out, sometimes LINQ may do that if your primary keys are not set up well, or if there are multiple rows with no unique values in your database.
This link also shows a similar problem.
The solution, other than rewriting your database columns (although that would be better on the long run) with better primary / unique keys, is to select specific values anonymously (later you can assign them easily):
var qInventory = db.vw_Web_Store_Inventory_Live
.Where(qi => qi.Sku_Number == inputSKU)
.Select(qi => new { qi.Store_Number, qi.Curr_Inv })
.ToList();
resultPInventory.SKU = inputSKU;
resultPInventory.StoreInventory = new List<StoreItem>();
foreach (var qi in qInventory)
{
resultPInventory.StoreInventory.Add(new StoreItem
{
StoreNum = qi.Store_Number,
Quantity = qi.Curr_Inv
});
}
Of course this won't be the best way if you later need to use qInventory for other things. (In that case you can just select more fields)
PS, here is a way to shorten your code, but I am not sure if LINQ to Entities will allow it, so test it first:
resultPInventory.SKU = inputSKU;
resultPInventory.StoreInventory = db.vw_Web_Store_Inventory_Live
.Where(qi => qi.Sku_Number == inputSKU)
.Select(qi => new StoreItem { StoreNum = qi.Store_Number, Quantity = qi.Curr_Inv })
.ToList();
Assign new StoreItem() to a variable before adding it.
I'm using LINQ for querying the database. I'm using dynamic keyword in queries. I don't know this dynamic mechanism in depth, so I don't understand what's going on.
The situation follows. The following section of code:
var qGroup = qLocalOrd.GroupBy("new(...)", "it");
var qGroupCast = (qGroup as IQueryable<IGrouping<dynamic,dynamic>>).AsEnumerable();
var qAgg = from ordg in qGroupCast
select new {
ordg.Key,
Agg = (ordg.Key.OsID + " " + ordg.Key.NameOs + "\n" +
(ordg as IEnumerable<dynamic>).Aggregate("Составные части:\n", ...).Trim('\n') + ...)
};
Works just fine. But when I'm adding this to the end:
var qPlain = qAgg.Cast<dynamic>().AsQueryable().Select("new(Key.SubSchet, Key.NameSubSchet, ...)");
qPlain.Dump();
I'm receiving an error like "No field "Key" exists in type "Object"". It's the same if I use
(qAgg as IEnumerable<dynamic>)
So at this point dynamic treatment of qAgg elements is broken somewhy.
Why does this happen and how to make this thing work?
I am using xml linq on my project. I am dealing with very large xml's for easy understanding purpose I have mentioned small sample xml.
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Body>
<StackOverflowReply xmlns="http://xml.stack.com/RRAND01234">
<processStatus>
<statusCode1>P</statusCode1>
<statusCode2>P</statusCode2>
<statusCode3>P</statusCode3>
<statusCode4>P</statusCode4>
</processStatus>
</StackOverflowReply>
</soap:Body>
Following is C# xml linq
XNamespace x = "http://xml.stack.com/RRAND01234";
var result = from StackOverflowReply in XDocument.Parse(Myxml).Descendants(x + "Security_AuthenticateReply")
select new
{
status1 = StackOverflowReply.Element(x + "processStatus").Element(x + "statusCode1").Value,
status2 = StackOverflowReply.Element(x + "processStatus").Element(x + "statusCode2").Value,
status3 = StackOverflowReply.Element(x + "processStatus").Element(x + "statusCode3").Value,
status4 = StackOverflowReply.Element(x + "processStatus").Element(x + "statusCode4").Value,
status5 = StackOverflowReply.Element(x + "processStatus").Element(x + "statusCode5").Value,
};
Here I am getting exception like "Object reference not set to an instance of an object.". Because the tag
<statusCode5>
was not in my xml.In this case I want to get detail exception message like "Missing tag statusCode5". Please guide me how to get this message from my exception.
There's no easy way (that I'm aware of) to find out exactly what element(s) was/were missing in a LINQ to XML statement. What you can do however is use (string) on the element to handle missing elements - but that can get tricky if you have a chain of elements.
That wouldn't work in your current code:
status5 = (string)StackOverflowReply.Element(x + "processStatus").Element(x + "statusCode5")
Becuase (string) will only work on first element, and the second one is the one that is missing.
You could change your LINQ to focus only on the subnodes, like this:
XNamespace x = "http://xml.stack.com/RRAND01234";
var result = from StackOverflowReply in XDocument.Parse(Myxml).Descendants(x + "processStatus")
select new
{
status1 = (string)StackOverflowReply.Element(x + "statusCode1"),
status2 = (string)StackOverflowReply..Element(x + "statusCode2"),
status3 = (string)StackOverflowReply..Element(x + "statusCode3"),
status4 = (string)StackOverflowReply.Element(x + "statusCode4"),
status5 = (string)StackOverflowReply.Element(x + "statusCode5"),
};
However, if your XML is complex and you have different depths (nested elements), you'll need a more robust solution to avoid a bunch of conditional operator checks or multiple queries.
I have something that might help if that is the case - I'll have to dig it up.
EDIT For More Complex XML
I've had similar challenges with some XML I have to deal with at work. In lieu of an easy way to determine what node was the offending node, and not wanting to have hideously long ternary operators, I wrote an extension method that worked recursively from the specified starting node down to the one I was looking for.
Here's a somewhat simple and contrived example to demonstrate.
<SomeXML>
<Tag1>
<Tag1Child1>Value1</Tag1Child1>
<Tag1Child2>Value2</Tag1Child2>
<Tag1Child3>Value3</Tag1Child3>
<Tag1Child4>Value4</Tag1Child4>
</Tag1>
<Tag2>
<Tag2Child1>
<Tag2Child1Child1>SomeValue1</Tag2Child1Child1>
<Tag2Child1Child2>SomeValue2</Tag2Child1Child2>
<Tag2Child1Child3>SomeValue3</Tag2Child1Child3>
<Tag2Chidl1Child4>SomeValue4</Tag2Child1Child4>
<Tag2Child1>
<Tag2Child2>
<Tag2Child2Child1>
<Tag2Child2Child1Child1 />
<Tag2Child2Child1Child2 />
</Tag2Child2>
</Tag2>
</SomeXML>
In the above XML, I had no way of knowing (prior to parsing) if any of the children elements were empty, so I after some searching and fiddling I came up with the following extension method:
public static XElement GetChildFromPath(this XElement currentElement, List<string> elementNames, int position = 0)
{
if (currentElement == null || !currentElement.HasElements)
{
return currentElement;
}
if (position == elementNames.Count - 1)
{
return currentElement.Element(elementNames[position]);
}
else
{
XElement nextElement = currentElement.Element(elementNames[position]);
return GetChildFromPath(nextElement, elmenentNames, position + 1);
}
}
Basically, the method takes the XElement its called on, plus a List<string> of the elements in path order, with the one I want as the last one, and a position (index in the list), and then works it way down the path until it finds the element in question or runs out of elements in the path. It's not as elegant as I would like it to be, but I haven't had time to refactor it any.
I would use it like this (based on the sample XML above):
MyClass myObj = (from x in XDocument.Parse(myXML).Descendants("SomeXML")
select new MyClass() {
Tag1Child1 = (string)x.GetChildFromPath(new List<string>() {
"Tag1", "Tag1Child1" }),
Tag2Child1Child4 = (string)x.GetChildFromPath(new List<string>() {
"Tag2", "Tag2Child1", "Tag2Child1Child4" }),
Tag2Child2Child1Child2 = (string)x.GetChildFromPath(new List<string>() {
"Tag2", "Tag2Child2", "Tag2Child2Child1",
"Tag2Child2Child1Child2" })
}).SingleOrDefault();
Not as elegant as I'd like it to be, but at least it allows me to parse an XML document that may have missing nodes without blowing chunks. Another option was to do something like:
Tag2Child2Child1Child1 = x.Element("Tag2") == null ?
"" : x.Element("Tag2Child2") == null ?
"" : x.Element("Tag2Child2Child1") == null ?
"" : x.Element("Tag2Child2Child1Child2") == null ?
"" : x.Element("Tag2")
.Element("Tag2Child2")
.Element("Tag2Child2Child1")
.Element("Tag2Child2Child1Child2").Value
That would get really ugly for an object that had dozens of properties.
Anyway, if this is of use to you feel free to use/adapt/modify as you need.
I have a need to search a data source using LINQ.
In essence, the query that I want would be something like:
select * from table where id in ("1","2","4");
The added "complexity" is that the values inside the parenthesis will be coming out of string list.
This is my code:
public void getAdUserData(List<String> _inADUserDatas)
{
var records = from _adUserDatas in _adUserDataDBDataContex.ADUserDatas
where
new string[] { (char)34 + String.Join((char)34 + "," + (char)34, _inADUserDatas) + (char)34 }.Contains(_adUserDatas.fan)
orderby _adUserDatas.fan
select _adUserDatas;
var test = records.ToList();
Console.WriteLine((char)34 + String.Join((char)34 + "," + (char)34, _inADUserDatas) + (char)34);
as can be seen, I use String.join to convert my list into a comma-delimited string where every string is enclosed by a double-quote.
I've also tried using the code below without any luck.
new string[] {String.Join(",", _inADUserDatas) }.Contains(_adUserDatas.fan)
I also tried manually typing some of the ID in the list into the query and I am getting records.
I believe I am doing it right but it's not returning any records. The variable test has a count of 0.
Any ideas what could be wrong? I've gotten the ideas to my code by combining several discussions here in stackoverflow particularly this link here and this other link.
Thanks a lot
Call Contains on the list itself. Query provider should handle it out of the box:
var records = from _adUserDatas in _adUserDataDBDataContex.ADUserDatas
where _inADUserDatas.Contains(_adUserDatas.fan)
orderby _adUserDatas.fan
select _adUserDatas;
Just use contains on your collection:
var records = from _adUserDatas in _adUserDataDBDataContex.ADUserDatas
where _inADUserDatas.Contains(_adUserDatas.fan)
orderby _adUserDatas.fan
select _adUserDatas;
you can use below mentioned code
var list=_adUserDataDBDataContex.ADUserDatas.Where(p=>p.Contains(_adUserDatas.fan));
What's the data type for ids?
a simple collection.Where(s=> ids.Contains(s)) will translate to "in"
I'm not sure how linq to SQL translates these sorts of queries, but you don't need to sting join with commas in EF, try it without manually trying to create a comma separated list.
I need to query a list in SharePoint where the columns may be added in the future.
For instance at the moment I have the following columns
Name, Job, interests, address
I want to be able to query this string dynamically using a parameter from the browser so if columns are added in the future I don’t have to change the code but just the parameter.
The address may look like this www.contoso.com/sites/mypage.aspx?property=Interests
And the code something on the line of this:
var SiteParameter = Request.QueryString["property"];
var ItemsFromList = from item in ListItems where item[try to put the parameter in here] select item;
I use SPmetal to get the list details, so if I press item. Visual Studio2010 will return the columns within the list.
This may be easier without SPMetal.
var qy = new SPQuery();
qy.Query =
"<Where><Eq>" +
"<FieldRef Name=`" + siteParameter + "'/>" +
// You may have to worry about the type of the field here, too.
"<Value Type='Text'>" + desiredValue + "</Value>" +
"</Eq></Where>";
var itemCollection = myList.GetItems(qy);