I have made a program that takes the longitude and latitude from several properties on a london property portal.
What I am to do next is send those long+lat to geocode and return the full formatted address. The below code works for a majority of URLs that I have in my geocodeURL list, but occasionally it returns a null reference exception.
When I check the URL that failed and returned the exception manually in a browser, it works fine.
If that is the case, what am I doing wrong?
for (int i = 0; i < longitude.Count; i++)
{
Console.WriteLine(longitude[i]);
Console.WriteLine(latitude[i]);
//Console.WriteLine("http://maps.google.com/maps/api/geocode/xml?latlng=" + latitude[i] + "," + longitude[i] + "&sensor=false");
geocodeURL.Add("https://maps.googleapis.com/maps/api/geocode/xml?latlng=" + latitude[i] + "," + longitude[i] + "&key=0000");
}
foreach (string i in geocodeURL)
{
try
{
var requestUri = string.Format(i);
var request = WebRequest.Create(requestUri);
var response = request.GetResponse();
var xdoc = XDocument.Load(response.GetResponseStream());
var result = xdoc.Element("GeocodeResponse").Element("result");
var fullAddy = result.Element("formatted_address").Value;
address.Add(fullAddy);
Console.WriteLine(fullAddy);
}
catch (Exception e)
{
Console.WriteLine(i);
Console.WriteLine(e);
}
}
The XMl response looks like this:
<GeocodeResponse>
<status>OK</status>
<result>
<type>street_address</type>
<formatted_address>4 Sydenham Ave, London SE26 6UH, UK</formatted_address>
One such example of the exception, and you can see the URL seems to be fine, but it throws an exception regardless... (I have blacked out the key fyi)
Google have a limit on how many results you can send in a certain amount of time. Since you are sending hundreds of requests at a time, you are probably running up against this limit and the error is google's way of telling you to either cough up some cash or gtfo.
If you don't want to pay up, you could put a Thread.Sleep(2000) (or some other wait period) every few requests to get around the limit.
Related
The application iterates through around 500 webrequests, at random the request returns a 500-error from the server. I belive their is either a problem with the amount of requests made or that at some point the information takes to long to read into the datatable that causes the connection to fail. But these are just guesses on my part. Is there a smarter way to iterate through all of the requests or to use another approach?
I have other iterations made in the same manner for other requests, none of them are as hefty as this one and they don't throw any errors from the server.
foreach (DataRow row in dt.Rows)
{
string url = row["href"].ToString();
HttpWebRequest productsDetails = (HttpWebRequest)WebRequest.Create(url);
productsDetails.Credentials = nc;
productsDetails.Method = "GET";
productsDetails.Timeout = 5000;
productsDetails.ContentType = "application/x-www-form-urlencoded;charset=utf-8";
using (HttpWebResponse productsDetailsResponse = (HttpWebResponse)productsDetails.GetResponse())
{
var detailedRespons = productsDetailsResponse.GetResponseStream();
XDocument detailedResponsDoc = XDocument.Load(detailedRespons);
//foreach (XElement xe4 in detailedResponsDoc.Descendants("product_option_value"))
//{
// DataRow row4 = dt4.NewRow();
// row4["href"] = xe4.Attribute(xlink + "href").Value;
// dt4.Rows.Add(row4);
//}
string p1 = detailedResponsDoc.Root.Element("combination").Element("id").Value;
string p2 = detailedResponsDoc.Root.Element("combination").Element("reference").Value;
string p3 = detailedResponsDoc.Root.Element("combination").Element("price").Value;
string p4;
foreach (XElement xe2 in detailedResponsDoc.Descendants("product_option_value"))
{
p4 = (xe2.Value);
DataRow row5 = test.NewRow();
row5["id"] = p1;
row5["referemce"] = p2;
row5["price"] = p3;
row5["POV"] = p4;
test.Rows.Add(row5);
DataRow row4 = dt4.NewRow();
row4["href"] = xe2.Attribute(xlink + "href").Value;
dt4.Rows.Add(row4);
}
productsDetailsResponse.Close();
}
}
}
}
}
catch (WebException webex)
{
WebResponse errResp = webex.Response;
using (Stream respStream = errResp.GetResponseStream())
{
StreamReader reader = new StreamReader(respStream);
string text = reader.ReadToEnd();
MessageBox.Show("yttre:" + text);
}
}
The error message is a a generic 500 exception from the server, referring me to contact the host. The host don't see anything and the little i have found in some kind of error log on the server don't contain any information.
Make sure the server is not blocking you, some servers have firewalls that block repetitive connections from a single IP address as they believe it will be an attack.
This is a normal and often cannot be disabled by hosts as it is a security feature.
Add a delay to the requests and see if the server responds correctly, if this works, then the server may be blocking you.
Try to make similar requests on a local server like XAMP, if the same errors occur this could be a code fault, like the information being passed to the server (Headers, Post, Get and etc).
Try reusing HttpWebRequest to avoid overhead if you repeatedly create an object, try using asynchronous methods.
There are many variables as to why there might be errors, but the chance of being server-related or HttpWebRequest is the most likely.
I have a very very strange issue going on with SSIS and wanted to know if anyone else has had something similar.
In a Data Flow I have a Source that gets data from a MSSQL table and then feeds it to a data transformation Script Task.
The script task is as follows:
Script Task
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
string SecurityHeader = (string)Variables.SecurityHeader;
string APIToken = (string)Variables.APIToken;
var AppFormID = (string)Row.AppFormID;
var AppClassName = (string)Row.AppClassName;
var Lat = Row.Latitude;
var Long = Row.Longitude;
var data = new
{
record = new
{
status = Row.Status.ToString(),
latitude = Row.Latitude,
longitude = Row.Longitude,
form_values = new Dictionary<string, string>()
}
};
if (Row.CreatedBy_IsNull == false) { data.record.form_values["1a09"] = Row.CreatedBy.ToString(); }
string jsonstring = JsonConvert.SerializeObject(data);
var client = new RestClient(AppURLRef);
var request = new RestRequest(Method.POST);
request.AddHeader(SecurityHeader, APIToken);
request.AddHeader("content-type", "application/json");
request.AddParameter("application/json", jsonstring, ParameterType.RequestBody);
try
{
IRestResponse dataresponse = client.Execute(request);
if (dataresponse.StatusCode.ToString() == "Created")
{
var listobject = JsonConvert.DeserializeObject<Lists>(dataresponse.Content);
}
else
{
Row.OErrorMsg = dataresponse.ErrorMessage.ToString();
};
}
catch (Exception ex)
{
MessageBox.Show(ex.Message + " Data: " + ex.Data + " Inner Exception: " + ex.InnerException);
}
}
The Issue
When I run this without any break points I get a Message Box with the error "Input string was not in correct format"
Where this gets really weird is that when I put a break point in the code works perfectly, thus making getting a line error impossible.
Has anyone ever come across an issue like this before and if so how did you fix it?
Update
After following some advice I moved the try catch block up to the start of the script, this resulted in no change the try catch block was never hit.
I attempted to but a MessageBox.Show("") at the very top of the input process row and again this was not hit. Something is happening at a higher level than both of these thought I am unable to find out where or how this is occurring.
Again to reiterate the error is in a MessageBox, not in the output and nothing gets logged in the process tab. So this means no line numbers, no useful error messages that discusses where this error is occurring.
I managed to solve this, after commenting EVERYTHING out and removing all of the input columns the error was still occurring.
I put it down to another way Microsoft wants to punish me, I copied the code deleted the script task, added a new script task and copied the old code back in. Worked perfectly...
Thanks Microsoft!!
Put delay in catch block in next line to MessageBox.Show
you able to view message box .
Some time message box goes in back ground ( USe ALT +TAB to find it).
Question Background:
I am making a call to Amazon's Product Advertising API. This gives back X number of paged results at a time with an node in the XML stating how many pages are available i.e 10. A page number is then specified and the API will responsed with the results for that given page.
Question:
I want to add all of the results from each page to a list that I then send to an MVC view. Currently I'm doing this through a For Loop specifying the upper limit as the maximum number of loops, for example if there are 10 pages available I'm looping round page 1, 2, 3, 4......and so on until 10.
The following code shows how the request is currently being made. Is this ok? It works but I have seen a StackOverflow exception a couple of times when debugging.
for (int i = 1; i <= 10; i++)
{
string requestUrl = "http://amazonApi.com/page"+i;
WebRequest amazonRequest = HttpWebRequest.Create(requestUrl);
try
{
WebResponse response = amazonRequest .GetResponse();
XmlDocument doc = new XmlDocument();
doc.Load(response.GetResponseStream());
XmlDocument xmlDoc = doc;
//Add Xml Document to List.....
AddToList(xmlDoc);
}
catch (Exception e)
{
System.Console.WriteLine("Caught Exception: " + e.Message);
System.Console.WriteLine("Stack Trace: " + e.StackTrace);
}
}
I am getting this error randomly when I attempt to upload an attachment.
"Unable to read data from the transport connection: The connection was
closed."
I have an import function using the C# RallyRestAPI which pulls data from Test Track and inserts it into Rally and copies in the attachments to Rally. My test data has 3 attachments which vary in size and are 350k, 63k and 43k. When I run my importer it will error on different uploads at different times. There is no consistency about it. Sometimes all three will fail, sometimes the 2nd one will and the third one will be successful. Creating and updating a story seem to be fine so it looks like a timeout but I am unsure how to change the timeout in the RallyRestAPI.
Has anyone else come across this issue using C# and the Rally RestAPI?
Here is my upload code. The call to Connect() returns a RallyRestAPI object and logs into that object. I re-login on every call to Rally (not sure if I need to do this or not).
private string AddAttachment(string reference, string name, string content, long contentSize, string type) {
var restAPI = Connect();
try {
var attachmentContent = new DynamicJsonObject();
attachmentContent["Content"] = content;
attachmentContent["Workspace"] = _workspace["_ref"];
attachmentContent["Project"] = _target["_ref"];
var result = restAPI.Create("AttachmentContent", attachmentContent);
if (result.Success) {
_logger.Info("Attached the relevant AttachmentContent.");
}
else {
throw new LoggedException("Could not attach attachment to '" + reference + "' due to the following errors\n" + GetErrorList(result.Errors));
}
var attachmentContentRef = result.Reference;
// DynamicJSONObject for Attachment Container
var myAttachment = new DynamicJsonObject();
myAttachment["Workspace"] = _workspace["_ref"];
myAttachment["Project"] = _target["_ref"];
myAttachment["Artifact"] = reference;
myAttachment["Content"] = attachmentContentRef;
myAttachment["Name"] = Path.GetFileName(name);
var contentType = "image/jpg";
if (!string.IsNullOrEmpty(type)) {
switch (type.Trim().ToLower()) {
case "doc":
contentType = "document/text";
break;
default:
contentType = type;
break;
}
}
myAttachment["ContentType"] = contentType;
myAttachment["Size"] = contentSize;
result = restAPI.Create("Attachment", myAttachment);
if (result.Success) {
_logger.Info("Attached the relevant attachment.");
}
else {
throw new LoggedException("Could not attach attachment to '" + reference + "' due to the following errors\n" + GetErrorList(result.Errors));
}
return attachmentContentRef;
}
catch (Exception ex) {
throw new LoggedException("Unhandled exception occurred: ",ex);
}
}
In my testing I've been able to upload attachments up to the Rally limit of 5 MB in size consistently without errors. Doesn't seem to relate to file type.
I'd recommend filing a Case with Rally Support (rallysupport#rallydev.com). Support has tools that can be used to identify bottlenecks - and try to see if they're server-side / data-related issues or client-connection issues.
I have a function which have a long execution time.
public void updateCampaign()
{
context.Session[processId] = "0|Fetching Lead360 Campaign";
Lead360 objLead360 = new Lead360();
string campaignXML = objLead360.getCampaigns();
string todayDate = DateTime.Now.ToString("dd-MMMM-yyyy");
context.Session[processId] = "1|Creating File for Lead360 Campaign on " + todayDate;
string fileName = HttpContext.Current.Server.MapPath("campaigns") + todayDate + ".xml";
objLead360.createFile(fileName, campaignXML);
context.Session[processId] = "2|Reading The latest Lead360 Campaign";
string file = File.ReadAllText(fileName);
context.Session[processId] = "3|Updating Lead360 Campaign";
string updateStatus = objLead360.updateCampaign(fileName);
string[] statusArr = updateStatus.Split('|');
context.Session[processId] = "99|" + statusArr[0] + " New Inserted , " + statusArr[1] + " Updated , With " + statusArr[2] + " Error , ";
}
So to track the Progress of the function I wrote a another function
public void getProgress()
{
if (context.Session[processId] == null)
{
string json = "{\"error\":true}";
Response.Write(json);
Response.End();
}else{
string[] status = context.Session[processId].ToString().Split('|');
if (status[0] == "99") context.Session.Remove(processId);
string json = "{\"error\":false,\"statuscode\":" + status[0] + ",\"statusmsz\":\"" + status[1] + "\" }";
Response.Write(json);
Response.End();
}
}
To call this by jQuery post request is used
reqUrl = "AjaxPages/lead360Campaign.aspx?processid=" + progressID + "&action=updatecampaign";
$.post(reqUrl);
setTimeout(getProgress, 500);
get getProgress is :
function getProgress() {
reqUrl = "AjaxPages/lead360Campaign.aspx?processid=" + progressID + "&action=getProgress";
$.post(reqUrl, function (response) {
var progress = jQuery.parseJSON(response);
console.log(progress)
if (progress.error) {
$("#fetchedCampaign .waitingMsz").html("Some error occured. Please try again later.");
$("#fetchedCampaign .waitingMsz").css({ "background": "url(common/images/ajax_error.jpg) no-repeat center 6px" });
return;
}
if (progress.statuscode == 99) {
$("#fetchedCampaign .waitingMsz").html("Update Status :"+ progress.statusmsz );
$("#fetchedCampaign .waitingMsz").css({ "background": "url(common/images/ajax_loded.jpg) no-repeat center 6px" });
return;
}
$("#fetchedCampaign .waitingMsz").html("Please Wait... " + progress.statusmsz);
setTimeout(getProgress, 500);
});
}
But the problem is that I can't see the intermediate message. Only the last message is been displayed after a long lime of ajax loading message
Also on the browser console I just see that after a long time first requested is completed and after that the second request is completed. but there should be for getProgress ?
I have checked jquery.doc and it says that $post is an asynchronous request.
Can anyone please explain what is wrong with the code or logic?
You are in a situation discussed here:
ASP.net session request queuing
While a request for a given user's session is processed, other requests for the same session are waiting. You need to run your long function in a background thread and let the request that initiates it finish. However, the background thread will not have access to session, and you will need a different mechanism to communicate its progress.
From the information you've provided, I would suspect that it's not your javascript code that's being synchronous, but rather the server-side code. You can test this by using Firebug or Chrome's dev tools to look at the start and end times of the two AJAX requests. If I'm right, you'll see that the second request begins after half a second, but doesn't complete until after the first one.
If that's the case, possible causes are:
Running in a dev environment in Visual Studio, especially in debug mode, seems to reduce the amount of asynchronicity. The dev environment seems to like to process one request at a time.
See Igor's answer about session request queueing.
You may have code that explicitly locks resources and causes the second request to block until the long-running request is done.
One other possible culprit is the fact that most browsers only allow a limited number of concurrent requests to a particular domain. If you have a few requests pending at any given moment, the browser may just be queuing up the remaining requests until they return.