Develop Reference

exception when trying to fetch more than 1 thousand customer(s) and invoice(s) from QBO API

I have more than 1000 customer(s) and invoice(s) and I am trying to fetch all those customers and invoice(s) into a drop-down list.
Documentation on the QBO site suggests that we should need to use pagination if I want to load all the customers in a grid, but what I want is to load all the customer(s) and invoice(s) in a drop-down list.
I am getting the following exception when I try to fetch more than 1000 customer(s) and invoice(s):
Validation Exception was thrown.
Details: QueryValidationError: value 100000 is too large. Max allowed value is 1000.
I am trying to fetch all the customers using the following code
public static List<Customer> GetAllQBOCustomers(ServiceContext context)
{
return Helper.FindAll<Customer>(context, new Customer(),1,100000);
}

I wrote the below code and solved my issue.
1. First I get the count of all the customers
2. Then I get all the customers in chunks and the chunk size is 1000
3. Create a List for customers.
4. Define 3 integer type variables for counting.
5. After that use do-while loop
6. Add all the customers are added to the main customer list
string strQuery = "Select Count(*) From Customer";
string custCount = qboAccess.GetCutomerCount(qboInz.QboServiceContext, strQuery);
List<qboData.Customer> customers = new List<Customer>();
int maxSize = 0;
int position = 1;
int count = Convert.ToInt32(custCount);
do
{
var custList = qboAccess.GetAllQBOEntityRecords(qboInz.QboServiceContext, new Customer(), position, 1000);
customers.AddRange(custList);
maxSize += custList.Count();
position += 1000;
} while (count > maxSize);

The straightforward answer is to loop enough times to get the records you need:
public static List<Customer> GetAllQBOCustomers(ServiceContext context)
{
var list = new List<Customer>();
for (int i=0; i<=10000; i+= 1000)
{
var results = Helper.FindAll<Customer>(context, new Customer(),i, 1000);
list.AddRange(results);
}
return list;
}
Or if you want to try to do it in parallel (and the API allows concurrent connections):
public static List<Customer> GetAllQBOCustomers(ServiceContext context)
{
var bag = new ConcurrentBag<Customer>();
Parallel.ForEach( Enumerable.Range(0, 10), i =>
{
var results = Helper.FindAll<Customer>(context, new Customer(),i * 1000, 1000);
bag.AddRange(results);
});
return bag.ToList();
}
Since the series of calls is likely to be expensive, I suggest you cache the results.

You can't download those records all at once. That is what the error is telling you - very clearly. There's no magic way to avoid the server's rules.
However, I really think you should not download them all at once anyway. A drop-down list is not a good way to display that amount of data to users. Consider the user experience - would you want to scroll through a list of thousands of customers to try and find the one you want? Or would it be easier to start typing part of the name and have it pop up a short list of possible matches to choose from?
A more user-friendly way to implement this would be use an auto-complete box instead of a drop-down list, and after the user has typed a few characters, it can use AJAX to search the API for customers whose names or IDs contain those characters. Then you'll only need to return a small number of records each time, and the user will not be stuck having to scroll for 10 minutes just to find a customer at the bottom of a list of 10,000 records.

Generating random collection from the database

I have the method below in a webapi that pulls data.
I am building an app which will have a listview with default data coming from this method.
I want this data to be changing each time any user starts the app.
How can I generate random data with this method. There are about 4 different categories.
public IEnumerable<ArticlesDto> Find(string category)
{
IEnumerable<ArticlesDto> objArticles = null;
var context = new ArticlesContext();
objArticles = (from j in context.Information
where j.Category == category
select new ArticlesDto()
{
Id = j.Id,
Headlines = j.Headlines,
Url = j.Url,
Category = j.Category,
Summary = j.Summary
});
return objArticles;
}
Example: first time I use the app, I see a list of data about 20 rows(default data).
Second time I use it, I see a different list of another 20 rows different from the last time I used the app.

Why don't you try using AutoFixture. This framework would help you generate random data every time your WebAPI call is made. Here the GITHub link. Please mark as answer if this helps.
https://github.com/AutoFixture

Just OrderBy a random number and then take as many as you like:
Random rnd = new Random();
objArticles = context.Information.Where(i=> i.Category == category)
.OrderBy(i=> rnd.Next())
.Select(i=> new ArticlesDto
{
Id = i.Id,
Headlines = i.Headlines,
Url = i.Url,
Category = i.Category,
Summary = i.Summary
}).Take(20);

how to get the index/ order of a defect in the Backlog, Rally Rest API C#

Task:
List all defects in the backlog of a project by their rank/index.
Here is my code:
var myRequest = new Request()
{
ArtifactName = "defect",
Limit = 2000,
Query = new Query("Project.OID", Query.Operator.Equals, MyDefectProjectOID),
Fetch = new List<string>() { "true" }
};
QueryResult queryMyResult = api.Query(myRequest);
Question:
1) How do I get the result set back in the order my users have organised them in Rally.
2) Is there a value on the defect item that tells me the rank/index (for example the Task item has a TaskIndex property)

1) Order by the DragAndDropRank field, ASC.
2) As long as you fetch DragAndDropRank as well that's your rank value. It's encoded as a string which is sortable in client code. The overall numeric index will be its index in your result set.
Another quick note- rather than specifying a query on Project.ObjectID to control scoping, you can just set the Project, ProjectScopeUp and ProjectScopeDown values:
Project = "/project/" + MyDefectProjectOID,
ProjectScopeUp = false,
ProjectScopeDown = false

Speeding up a linq query with 40,000 rows

In my service, first I generate 40,000 possible combinations of home and host countries, like so (clientLocations contains 200 records, so 200 x 200 is 40,000):
foreach (var homeLocation in clientLocations)
{
foreach (var hostLocation in clientLocations)
{
allLocationCombinations.Add(new AirShipmentRate
{
HomeCountryId = homeLocation.CountryId,
HomeCountry = homeLocation.CountryName,
HostCountryId = hostLocation.CountryId,
HostCountry = hostLocation.CountryName,
HomeLocationId = homeLocation.LocationId,
HomeLocation = homeLocation.LocationName,
HostLocationId = hostLocation.LocationId,
HostLocation = hostLocation.LocationName,
});
}
}
Then, I run the following query to find existing rates for the locations above, but also include empty the missing rates; resulting in a complete recordset of 40,000 rows.
var allLocationRates = (from l in allLocationCombinations
join r in Db.PaymentRates_AirShipment
on new { home = l.HomeLocationId, host = l.HostLocationId }
equals new { home = r.HomeLocationId, host = (Guid?)r.HostLocationId }
into matches
from rate in matches.DefaultIfEmpty(new PaymentRates_AirShipment
{
Id = Guid.NewGuid()
})
select new AirShipmentRate
{
Id = rate.Id,
HomeCountry = l.HomeCountry,
HomeCountryId = l.HomeCountryId,
HomeLocation = l.HomeLocation,
HomeLocationId = l.HomeLocationId,
HostCountry = l.HostCountry,
HostCountryId = l.HostCountryId,
HostLocation = l.HostLocation,
HostLocationId = l.HostLocationId,
AssigneeAirShipmentPlusInsurance = rate.AssigneeAirShipmentPlusInsurance,
DependentAirShipmentPlusInsurance = rate.DependentAirShipmentPlusInsurance,
SmallContainerPlusInsurance = rate.SmallContainerPlusInsurance,
LargeContainerPlusInsurance = rate.LargeContainerPlusInsurance,
CurrencyId = rate.RateCurrencyId
});
I have tried using .AsEnumerable() and .AsNoTracking() and that has sped things up quite a bit. The following code shaves several seconds off of my query:
var allLocationRates = (from l in allLocationCombinations.AsEnumerable()
join r in Db.PaymentRates_AirShipment.AsNoTracking()
But, I am wondering: How can I speed this up even more?
Edit: Can't replicate foreach functionality in linq.
allLocationCombinations = (from homeLocation in clientLocations
from hostLocation in clientLocations
select new AirShipmentRate
{
HomeCountryId = homeLocation.CountryId,
HomeCountry = homeLocation.CountryName,
HostCountryId = hostLocation.CountryId,
HostCountry = hostLocation.CountryName,
HomeLocationId = homeLocation.LocationId,
HomeLocation = homeLocation.LocationName,
HostLocationId = hostLocation.LocationId,
HostLocation = hostLocation.LocationName
});
I get an error on from hostLocation in clientLocations which says "cannot convert type IEnumerable to Generic.List."

The fastest way to query a database is to use the power of the database engine itself.
While Linq is a fantastic technology to use, it still generates a select statement out of the Linq query, and runs this query against the database.
Your best bet is to create a database View, or a stored procedure.
Views and stored procedures can easily be integrated into Linq.
Material Views ( in MS SQL ) can further speed up execution, and missing indexes are by far the most effective tool in speeding up database queries.

How can I speed this up even more?
Optimizing is a bitch.
Your code looks fine to me. Make sure to set the index on your DB schema where it's appropriate. And as already mentioned: Run your Linq against SQL to get a better idea of the performance.
Well, but how to improve performance anyway?
You may want to have a glance at the following link:
10 tips to improve LINQ to SQL Performance
To me, probably the most important points listed (in the link above):
Retrieve Only the Number of Records You Need
Turn off ObjectTrackingEnabled Property of Data Context If Not
Necessary
Filter Data Down to What You Need Using DataLoadOptions.AssociateWith
Use compiled queries when it's needed (please be careful with that one...)

Why am I getting index out of bounds error from database

I know what index out of bounds is all about. When I debug I see why as well. basically what is happening is I do a filter on my database to look for records that are potential/pending. I then gather a array of those numbers send them off to another server to check to see if those numbers have been upgraded to a sale. If it has been upgraded to a sale the server responds back with the new Sales Order ID and my old Pending Sales Order ID (SourceID). I then do a for loop on that list to filter it down that specific SourceID and update the SourceID to be the Sales Order ID and change a couple of other values. Problem is is that when I use that filter on the very first one it throws a index out of bounds error. I check the results returned by the filter and it says 0. Which i find kind of strange because I took the sales order number from the list so it should be there. So i dont know what the deal is. Here is the code in question that throws the error. And it doesn't do it all the time. Like I just ran the code this morning and it didn't throw the error. But last night it did before I went home.
filter.RowFilter = string.Format("Stage = '{0}'", Potential.PotentialSale);
if (filter.Count > 0)
{
var Soids = new int[filter.Count];
Console.Write("Searching for Soids - (");
for (int i = 0; i < filter.Count; i++)
{
Console.Write(filter[i][1].ToString() + ",");
Soids[i] = (int)filter[i][1];
}
Console.WriteLine(")");
var pendingRecords = Server.GetSoldRecords(Soids);
var updateRecords = new NameValueCollection();
for (int i = 0; i < pendingRecords.Length; i++)
{
filter.RowFilter = "Soid = " + pendingRecords[i][1];
filter[0].Row["Soid"] = pendingRecords[i][0];
filter[0].Row["SourceId"] = pendingRecords[i][1];
filter[0].Row["Stage"] = Potential.ClosedWon;
var potentialXML = Potential.GetUpdatePotentialXML(filter[0].Row["Soid"].ToString(), filter[0].Row["Stage"].ToString());
updateRecords.Add(filter[0].Row["ZohoID"].ToString(), potentialXML);
}
if i'm counting right line 17 is the error where the error is thrown. pendingRecords is a object[][] array. pendingRecords[i] is the individual records. pendingRecords[i][0] is the new Sales OrderID (SOID) and pendingRecords[i][1] is the old SOID (now the SourceID)
Any help on this one? is it because i'm changing the SOID to the new SOID, and the filter auto updates itself? I just don't know

Well I ended up changing how it worked all together and it actually sorts it a bit nicer now. The code i am about to post has a bunch of hard coded numbers due to the structure of my table that is returned. Sorry about that. I have learned since then to not do that, but i am working on a different project now and will change that when I have to change the program. But here is the solution.
var potentials = Server.GetNewPotentials(); //loads all records from server
for (int i = 0; i < potentials.Length; i++)
{
var filter = AllPotentials.DefaultView;
var result1 = CheckSoidOrSource(potentials[i].Soid, true);
var result2 = CheckSoidOrSource(potentials[i].SourceID,false) ;
//This potential can't be found at all so let's add it to our table
if (result1+result2==0)
{
Logger.WriteLine("Found new record. Adding it to DataTable and sending it to Zoho");
AllPotentials.Add(potentials[i]);
filter.RowFilter = string.Format("Soid = '{0}'", potentials[i].SourceID);
var index = AllPotentials.Rows.IndexOf(filter[0].Row);
ZohoPoster posterInsert = new ZohoPoster(Zoho.Fields.Potentials, Zoho.Calls.insertRecords);
AllPotentials.Rows[index]["ZohoID"] = posterInsert.PostNewPotentialRecord(3, filter[0].Row);
}
//This potential is not found, but has a SourceId that matches a Soid of another record.
if (result1==0 && result2 == 1)
{
Logger.WriteLine("Found a record that needs to be updated on Zoho");
ZohoPoster posterUpdate = new ZohoPoster(Zoho.Fields.Potentials, Zoho.Calls.updateRecords);
filter.RowFilter = string.Format("Soid = '{0}'", potentials[i].SourceID);
var index = AllPotentials.Rows.IndexOf(filter[0].Row);
AllPotentials.Rows[index]["Soid"] = potentials[i].Soid;
AllPotentials.Rows[index]["SourceId"] = potentials[i].SourceID;
AllPotentials.Rows[index]["PotentialStage"] = potentials[i].PotentialStage;
AllPotentials.Rows[index]["UpdateRecord"] = true;
AllPotentials.Rows[index]["Amount"] = potentials[i].Amount;
AllPotentials.Rows[index]["ZohoID"] = posterUpdate.UpdatePotentialRecord(3, filter[0].Row);
}
}
AllPotentials.AcceptChanges();
}
private int CheckSoidOrSource(string Soid, bool checkSource)
{
var filter = AllPotentials.DefaultView;
if (checkSource)
filter.RowFilter = string.Format("Soid = '{0}' OR SourceId = '{1}'",Soid, Soid);
else
filter.RowFilter = string.Format("Soid = '{0}'", Soid);
return filter.Count;
}
basically what is happening is that i noticed something about my data when I filter it this way. The two results would only return the following results (0,0) (0,1) and (1,0) (0,0) means that the record doesn't exist at all in this table so I need to add it. (1,0) means that the Sales Order ID (Soid) matches another Soid in the table so it already exists. Lastly (0,1) means that the Soid doesn't exist in this table but i found a record that has the Soid as it's source...which to me means that the one that had it as a source has been upgraded from a potential to a sale, which in turn means i have to update the record and Zoho. This worked out to much less work for me because now I don't have to search for won and lost records, i only have to search for lost records. less code same results is always a good thing :)