The application iterates through around 500 webrequests, at random the request returns a 500-error from the server. I belive their is either a problem with the amount of requests made or that at some point the information takes to long to read into the datatable that causes the connection to fail. But these are just guesses on my part. Is there a smarter way to iterate through all of the requests or to use another approach?
I have other iterations made in the same manner for other requests, none of them are as hefty as this one and they don't throw any errors from the server.
foreach (DataRow row in dt.Rows)
{
string url = row["href"].ToString();
HttpWebRequest productsDetails = (HttpWebRequest)WebRequest.Create(url);
productsDetails.Credentials = nc;
productsDetails.Method = "GET";
productsDetails.Timeout = 5000;
productsDetails.ContentType = "application/x-www-form-urlencoded;charset=utf-8";
using (HttpWebResponse productsDetailsResponse = (HttpWebResponse)productsDetails.GetResponse())
{
var detailedRespons = productsDetailsResponse.GetResponseStream();
XDocument detailedResponsDoc = XDocument.Load(detailedRespons);
//foreach (XElement xe4 in detailedResponsDoc.Descendants("product_option_value"))
//{
// DataRow row4 = dt4.NewRow();
// row4["href"] = xe4.Attribute(xlink + "href").Value;
// dt4.Rows.Add(row4);
//}
string p1 = detailedResponsDoc.Root.Element("combination").Element("id").Value;
string p2 = detailedResponsDoc.Root.Element("combination").Element("reference").Value;
string p3 = detailedResponsDoc.Root.Element("combination").Element("price").Value;
string p4;
foreach (XElement xe2 in detailedResponsDoc.Descendants("product_option_value"))
{
p4 = (xe2.Value);
DataRow row5 = test.NewRow();
row5["id"] = p1;
row5["referemce"] = p2;
row5["price"] = p3;
row5["POV"] = p4;
test.Rows.Add(row5);
DataRow row4 = dt4.NewRow();
row4["href"] = xe2.Attribute(xlink + "href").Value;
dt4.Rows.Add(row4);
}
productsDetailsResponse.Close();
}
}
}
}
}
catch (WebException webex)
{
WebResponse errResp = webex.Response;
using (Stream respStream = errResp.GetResponseStream())
{
StreamReader reader = new StreamReader(respStream);
string text = reader.ReadToEnd();
MessageBox.Show("yttre:" + text);
}
}
The error message is a a generic 500 exception from the server, referring me to contact the host. The host don't see anything and the little i have found in some kind of error log on the server don't contain any information.
Make sure the server is not blocking you, some servers have firewalls that block repetitive connections from a single IP address as they believe it will be an attack.
This is a normal and often cannot be disabled by hosts as it is a security feature.
Add a delay to the requests and see if the server responds correctly, if this works, then the server may be blocking you.
Try to make similar requests on a local server like XAMP, if the same errors occur this could be a code fault, like the information being passed to the server (Headers, Post, Get and etc).
Try reusing HttpWebRequest to avoid overhead if you repeatedly create an object, try using asynchronous methods.
There are many variables as to why there might be errors, but the chance of being server-related or HttpWebRequest is the most likely.
Related
I have made a program that takes the longitude and latitude from several properties on a london property portal.
What I am to do next is send those long+lat to geocode and return the full formatted address. The below code works for a majority of URLs that I have in my geocodeURL list, but occasionally it returns a null reference exception.
When I check the URL that failed and returned the exception manually in a browser, it works fine.
If that is the case, what am I doing wrong?
for (int i = 0; i < longitude.Count; i++)
{
Console.WriteLine(longitude[i]);
Console.WriteLine(latitude[i]);
//Console.WriteLine("http://maps.google.com/maps/api/geocode/xml?latlng=" + latitude[i] + "," + longitude[i] + "&sensor=false");
geocodeURL.Add("https://maps.googleapis.com/maps/api/geocode/xml?latlng=" + latitude[i] + "," + longitude[i] + "&key=0000");
}
foreach (string i in geocodeURL)
{
try
{
var requestUri = string.Format(i);
var request = WebRequest.Create(requestUri);
var response = request.GetResponse();
var xdoc = XDocument.Load(response.GetResponseStream());
var result = xdoc.Element("GeocodeResponse").Element("result");
var fullAddy = result.Element("formatted_address").Value;
address.Add(fullAddy);
Console.WriteLine(fullAddy);
}
catch (Exception e)
{
Console.WriteLine(i);
Console.WriteLine(e);
}
}
The XMl response looks like this:
<GeocodeResponse>
<status>OK</status>
<result>
<type>street_address</type>
<formatted_address>4 Sydenham Ave, London SE26 6UH, UK</formatted_address>
One such example of the exception, and you can see the URL seems to be fine, but it throws an exception regardless... (I have blacked out the key fyi)
Google have a limit on how many results you can send in a certain amount of time. Since you are sending hundreds of requests at a time, you are probably running up against this limit and the error is google's way of telling you to either cough up some cash or gtfo.
If you don't want to pay up, you could put a Thread.Sleep(2000) (or some other wait period) every few requests to get around the limit.
I have four application server for my application.Application is working on all server using load balancing.If one of my server goes down I have to check it manually using my system hosts file.To avoid this manual process I have created one program using C#.I write server IP address one by one in host file and remove previous one.
private void RunWithUAC()
{
List<string> lstIPAddress = new List<string>();
lstIPAddress.Add("1.1.1.1 example.com");
lstIPAddress.Add("1.1.1.1 example.com");
lstIPAddress.Add("1.1.1.1 example.com");
lstIPAddress.Add("1.1.1.1 example.com");
var systemPath = Environment.GetFolderPath(Environment.SpecialFolder.System);
Console.WriteLine(systemPath);
var path = #"C:\Windows\System32\drivers\etc\hosts";
foreach (var item in lstIPAddress)
{
System.IO.File.WriteAllText(path, string.Empty);
try
{
File.WriteAllText(path, item);
WebRequest request = WebRequest.Create("https://example.com");
request.Timeout = 10000;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
}
catch (Exception)
{
MessageBox.Show(item);
}
Thread.Sleep(1000);
}
}
But When second server goes down.It will give me timeout error for third server.
Please check the code and let me know what is wrong with this code.
Probably some kind of connection pooling, HTTP pipelining or keep-alive. This is the wrong approach in the first place.
Connect directly to the right IP (WebRequest.Create("https://1.1.1.1")). If you need to send a Host header add that manually to the request.
I got out of memory exception problem for 4 months. My client use webservice, they wanna me test their webservice. In their webservice, there is a function call upload. I test that function on 1500 users who uploaded at the same time. I tried garbage collection function of visual studio (GC). With 2mb of file, there is not exception, but with 8mb of file there is still out of memory exception. I have tried many times and a lot of solutions but still happened. I gonna crazy now. When upload is on going, I watched memory of all test computers but memory is not out of. So I think that problem is from webservice and server. But my client said that i have to improve those reasons which is from webservice and server to them. I'm gonna crazy now. Do you guys have any solotions for this? In additional, Our client does not public code, I just use webservice's function to test. Additional, I have to use vps to connect their webservice and network rather slow when connect to vps.
I have to make sure that my test script doesn't have any problem. Here is my test script to test upload function.
public void UploadNewJob(string HalID, string fileUID, string jobUID, string fileName, out List errorMessages)
{
errorMessages = null;
try
{
int versionNumber;
int newVersionNumber;
string newRevisionTag;
datasyncservice.ErrorObject errorObj = new datasyncservice.ErrorObject();
PfgDbJob job = new PfgDbJob();
job.CompanyName = Constant.SEARCH_CN;
job.HalliburtonSalesOffice = Constant.SEARCH_SO;
job.HalliburtonOperationsLocation = Constant.SEARCH_OL;
job.UploadPersonHalId = HalID;
job.CheckOutState = Constant.CHECKOUT_STATE;
job.RevisionTag = Constant.NEW_REVISION_TAG;
var manifestItems = new List();
var newManifestItems = new List();
var manifestItem = new ManifestItem();
if (fileUID == "")
{
if (job.JobUid == Guid.Empty)
job.JobUid = Guid.NewGuid();
if (job.FileUid == Guid.Empty)
job.FileUid = Guid.NewGuid();
}
else
{
Guid JobUid = new Guid(jobUID);
job.JobUid = JobUid;
Guid fileUid = new Guid(fileUID);
job.FileUid = fileUid;
}
// Change the next line when we transfer .ssp files by parts
manifestItem.PartUid = job.FileUid;
job.JobFileName = fileName;
manifestItem.BinaryFileName = job.JobFileName;
manifestItem.FileUid = job.FileUid;
manifestItem.JobUid = job.JobUid;
manifestItem.PartName = string.Empty;
manifestItem.SequenceNumber = 0;
manifestItems.Add(manifestItem);
errorMessages = DataSyncService.Instance.ValidateForUploadPfgDbJobToDatabase(out newVersionNumber, out newRevisionTag, out errorObj, out newManifestItems, HalID, job, false);
if (manifestItems.Count == 0)
manifestItems = newManifestItems;
if (errorMessages.Count > 0)
{
if (errorMessages.Count > 1 || errorMessages[0].IndexOf("NOT AN ERROR") == -1)
{
return;
}
}
//upload new Job
Guid transferUid;
long a= GC.GetTotalMemory(false);
byte[] fileContents = File.ReadAllBytes(fileName);
fileContents = null;
GC.Collect();
long b = GC.GetTotalMemory(false);
//Assert.Fail((b - a).ToString());
//errorMessages = DataSyncService.Instance.UploadFileInAJob(out transferUid, out errorObj, job.UploadPersonHalId, job, manifestItem, fileContents);
DataSyncService.Instance.UploadPfgDbJobToDatabase(out errorObj, out versionNumber, job.UploadPersonHalId, job, false, manifestItems);
}
catch (Exception ex)
{
Assert.Fail("Error from Test Scripts: " + ex.Message);
}
}
Please review my test code. And if there is not any problem from my test code, I have to improve reason is not from my test code T_T
My guess would be that you hit the 2 GB object size limit of .NET (1500 * 8MB > 4GB).
You should consider to change to .NET 4.5 and use the large object mode - see here - the setting is called gcAllowVeryLargeObjects.
I'm developing a simple app in c#, that can check if a domain name is available to puchase for a specific tld.
The method: I downloaded a whois-server list, I send the domain name to its whois server with a TCP client on the protocol 43, and check the servers answer.
The problem: more countries has the same whois server: "whois.ripe.net" .
If I send the full domain name(with tld), the server's answer is always "No entries found in source RIPE.". If I send the domain name without tld, I dont get any tld specific data about the status of the domain name.
The method I use:
private string GetWhoisInformation(string whoisServer, string url)
{
try
{
StringBuilder stringBuilderResult = new StringBuilder();
TcpClient tcpClinetWhois = new TcpClient(whoisServer, 43);
NetworkStream networkStreamWhois = tcpClinetWhois.GetStream();
BufferedStream bufferedStreamWhois = new BufferedStream(networkStreamWhois);
StreamWriter streamWriter = new StreamWriter(bufferedStreamWhois);
streamWriter.WriteLine(url);
streamWriter.Flush();
StreamReader streamReaderReceive = new StreamReader(bufferedStreamWhois);
while (!streamReaderReceive.EndOfStream)
stringBuilderResult.AppendLine(streamReaderReceive.ReadLine());
return stringBuilderResult.ToString();
}
catch
{
return "lekérdezés sikertelen";
}
}
Example:
I do:
GetWhoisInformation("whois.ripe.net", "pokerstars.hu")
The server's answer:
%ERROR:101: no entries found
%
% No entries found in source RIPE.
for the next command:
GetWhoisInformation("whois.ripe.net", "pokerstars")
the result contains several blocks like this:
% Information related to '80.65.254.128 - 80.65.254.159'
inetnum: 80.65.254.128 - 80.65.254.159
netname: Pokerstars
descr: Hosting
country: GB
admin-c: DC77-RIPE
tech-c: JM2352-RIPE
status: assigned PA
mnt-by: manx-telecom-mnt
changed: bill.hogg#manx-telecom.com 20101123
source: RIPE
There's no information about the domain name "pokerstars.hu". Of course, I get exactly the same answers if I want to check pokerstars.va. Pokerstars.hu is a registred domain, pokerstars.va is not.
How can I find the correct status of a domain name?
RIPE does not serve as a ccTLD whois server for any domains; like ARIN, it contains only netblock information. Each ccTLD has its own root whois server (or, that is, some of them don't have a proper whois service -- for example, the Spanish .es registry requires that you use a web client, with an obnoxious CAPTCHA you have to fill in every time).
See also http://www.ripe.net/data-tools/db although it is not very explicit about what the database does not contain.
You can get the address of the authoritative whois server by requesting the ccTLD's information from whois.iana.org.
vnix$ whois -h whois.iana.org hu | fgrep whois:
whois: whois.nic.hu
See also http://www.iana.org/domains/root/db/
I tried your code against whois.melbourneit.net and it found one of my domains no trouble. I was able to reproduce your problem running against RIPE and so I tried the same query interactively on their website - and had the same result. There's nothing wrong with your code.
tripleee is right about whois.nic.hu, I successfully used it to resolve pokerstars.hu - which leaves me wondering what the blazes is purpose of the RIPE whois server.
Thanks to triplee for showing us how to obtain the whois server friendly-name for a ccTLD.
You may find this useful:
using System;
using System.IO;
using System.Net.Sockets;
using System.Text;
namespace Whois
{
class Program
{
static void Main(string[] args)
{
string tldWhoisServer = "whois.iana.org";
string ccTldServer, query = null;
Console.Write("Query> ");
while ((query = Console.ReadLine()) != string.Empty)
{
string tld = query.Substring(query.LastIndexOf('.') + 1);
string foo = GetWhoisInformation(tldWhoisServer, tld);
foo = foo.Remove(0, foo.IndexOf("whois:") + 6).TrimStart();
ccTldServer = foo.Substring(0, foo.IndexOf('\r'));
Console.WriteLine(GetWhoisInformation(ccTldServer, query));
Console.Write("Query> ");
}
}
static string GetWhoisInformation(string whoisServer, string url)
{
try
{
StringBuilder stringBuilderResult = new StringBuilder();
TcpClient tcpClinetWhois = new TcpClient(whoisServer, 43);
NetworkStream networkStreamWhois = tcpClinetWhois.GetStream();
BufferedStream bufferedStreamWhois = new BufferedStream(networkStreamWhois);
StreamWriter streamWriter = new StreamWriter(bufferedStreamWhois);
streamWriter.WriteLine(url);
streamWriter.Flush();
StreamReader streamReaderReceive = new StreamReader(bufferedStreamWhois);
while (!streamReaderReceive.EndOfStream)
stringBuilderResult.AppendLine(streamReaderReceive.ReadLine());
return stringBuilderResult.ToString();
}
catch
{
return "Query failed";
}
}
}
}
I'm running this application on a server that has assigned 5 IPs. I use HttpWebRequest to fetch some data from a website. But when I make the connection I have be able to specify which one of the 5 IPs to make the connection from. Does HttpWebRequest support this? If it doesn't can I inherit a class from it to change it's behavior? I need so ideas here.
My code right now is something like:
System.Net.WebRequest request = System.Net.WebRequest.Create(link);
((HttpWebRequest)request).Referer = "http://application.com";
using (System.Net.WebResponse response = request.GetResponse())
{
StreamReader sr = new StreamReader(response.GetResponseStream());
return sr.ReadToEnd();
}
According to this, no. You may have to drop down to using Sockets, where I know you can choose the local IP.
EDIT: actually, it seems that it may be possible. HttpWebRequest has a ServicePoint Property, which in turn has BindIPEndPointDelegate, which may be what you're looking for.
Give me a minute, I'm going to whip up an example...
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://stackoverflow.com");
req.ServicePoint.BindIPEndPointDelegate = delegate(
ServicePoint servicePoint,
IPEndPoint remoteEndPoint,
int retryCount) {
if (remoteEndPoint.AddressFamily == System.Net.Sockets.AddressFamily.InterNetworkV6) {
return new IPEndPoint(IPAddress.IPv6Any, 0);
} else {
return new IPEndPoint(IPAddress.Any, 0);
}
};
Console.WriteLine(req.GetResponse().ResponseUri);
Basically, the delegate has to return an IPEndPoint. You can pick whatever you want, but if it can't bind to it, it'll call the delegate again, up to int.MAX_VALUE times. That's why I included code to handle IPv6, since IPAddress.Any is IPv4.
If you don't care about IPv6, you can get rid of that. Also, I leave the actual choosing of the IPAddress as an exercise to the reader :)
Try this:
System.Net.WebRequest request = System.Net.WebRequest.Create(link);
request.ConnectionGroupName = "MyNameForThisGroup";
((HttpWebRequest)request).Referer = "http://application.com";
using (System.Net.WebResponse response = request.GetResponse())
{
StreamReader sr = new StreamReader(response.GetResponseStream());
return sr.ReadToEnd();
}
Then try setting the ConnectionGroupName to something distinct per source ip you wish to use.
edit: use this in conjunction with the IP binding delegate from the answer above.