Develop Reference

Https calls are not connecting to server

I am working on Windows Service in visual studio 2017. In the rest api's call, getting exceptions while debugging code. Sometimes first 2 3 calls working after that getting exceptions.
System.Net.WebException: 'The remote server returned an error: (503)
Server Unavailable.'
The remote server returned an error: (429)
Unable to connect to the remote server
When calling same api's from Postman, getting response successfully.
This is my code
private void timer1_Tick(object sender, ElapsedEventArgs e)
{
WriteToFile("timer1_Tick method called..");
try
{
string jsonString = "";
string jsonstring2 = "";
string prodfetchurl = HOST;
var req = WebRequest.Create(prodfetchurl) as HttpWebRequest;
req.Method = "GET";
InitializeRequest(req);
req.Accept = MIME_TYPE;
//System.Threading.Thread.Sleep(5000);
var response = (HttpWebResponse)req.GetResponse();
WriteToFile("First service called...");
if (response.StatusCode == HttpStatusCode.OK)
{
Stream responseStream = response.GetResponseStream();
StreamReader responseReader = new StreamReader(responseStream);
jsonString = responseReader.ReadToEnd();
}
var deserialsseobj = JsonConvert.DeserializeObject<ProductList>(jsonString).Products.Where(i => i.Failed > 0).ToList();
foreach (var a in deserialsseobj)
{
var pid = a.ID;
string url = FailedDevicesUrl + pid.Value + "/failed";
var req2 = WebRequest.Create(url) as HttpWebRequest;
req2.Method = "GET";
InitializeRequest(req2);
req2.Timeout = 300000;
req2.Accept = MIME_TYPE;
var response1 = (HttpWebResponse)req2.GetResponse();
Stream responsestream2 = response1.GetResponseStream();
WriteToFile("Second service called...");
if (response1.StatusCode == HttpStatusCode.OK)
{
StreamReader responsereader1 = new StreamReader(responsestream2);
jsonstring2 = responsereader1.ReadToEnd();
}
var output = JsonConvert.DeserializeObject<List<FailedDeviceList>>(jsonstring2); // Will get List of the Failed devices
List<int> deviceids = new List<int>();
Reprocessdata reproc = new Reprocessdata();
Reprocessdata.DeviceId rprod = new Reprocessdata.DeviceId();
reproc.ForceFlag = true;
reproc.ProductID = pid.Value;
foreach (var dd in output)
{
rprod.ID = dd.DeviceId;
reproc.DeviceIds.Add(rprod);
}
// Reprocess the Product in Devices
var req3 = WebRequest.Create(ReprocessUrl) as HttpWebRequest;
req3.Method = "POST";
InitializeRequest(req3);
req3.Accept = MIME_TYPE;
req3.Timeout = 300000;
req3.ContentType = "application/json";
using (StreamWriter writer = new StreamWriter(req3.GetRequestStream()))
{
string json = new JavaScriptSerializer().Serialize(reproc);
writer.Write(json);
writer.Close();
}
System.Threading.Thread.Sleep(5000);
var response5 = (HttpWebResponse)req3.GetResponse();
WriteToFile("Third service called...");
if (response5.StatusCode == HttpStatusCode.OK)
{
string result;
using (StreamReader rdr = new StreamReader(response5.GetResponseStream()))
{
result = rdr.ReadToEnd();
}
}
}
response.Close();
}
catch (Exception ex)
{
WriteToFile("Simple Service Error on: {0} " + ex.Message + ex.StackTrace);
}
}
Methods used in above code
protected override void OnStart(string[] args)
{
base.OnStart(args);
timer1 = new System.Timers.Timer();
timer1.Interval = 60000; //every 1 min
timer1.Elapsed += new System.Timers.ElapsedEventHandler(timer1_Tick);
timer1.Enabled = true;
WriteToFile("Service has started..");
}
public void InitializeRequest(HttpWebRequest request)
{
request.Headers.Add("aw-tenant-code", API_TENANT_CODE);
request.Credentials = new NetworkCredential(USER_NAME, PASSWORD);
request.KeepAlive = false;
request.AddRange(1024);
}
When I contacted service provide they said everything fine from there side. Is this my code is buggy or windows service not reliable? How can I fix this issue?
Note: All APIS are working fine from Angular application using Visual Studio Code. It means my code is not working.
Edit1: Three below services I am using from this document of VMware.
private const string HOST = "https:host/api/mdm/products/search?";
private const string FailedDevicesUrl = "https:host/api/mdm/products/";
private const string ReprocessUrl = "https:host/api/mdm/products/reprocessProduct";

Response http code 429 indicates that you sending too many requests on target web service.
This means service you trying to send requests has a policies that blocks some requests by request-per-time limit.
Also I admit that external service can be manually configured to throw 403 code in specific cases that you can't know about. If that, this information can be explained in external service documentation... or not :)
What you can do with this?
Fit in limitations
You can make detailed research what limits target webservice has and set up your code to fit in this limitations. For example if service has limitation for receiving only one request per 10 minutes - you must set up your timer to send one request each 10 or more minutes. If documentation not provide such information - you can test it manually by finding some patterns with external service responses.
Use proxy
Every limitation policy based on information about requests senders. Usually this information consists of IP address of sender only. This means if you send 2 requests from two different IP addresses - limitation policy will perceive that like 2 different computers sending these requests. So you can find/buy/rent some proxy IP addresses and send requests through there on target web server.
How to connect through proxy in C# using WebRequest you can see in this answer.
Negotiate with external service provider
If you have possibility to communicate with external service developers or help center, you can ask their to reduce limitations for your IP address (if it static) or provide some mechanisms to avoid limitation policy for you. If for some reason they cannot provide this opportunity, at least you can ask detailed information about limitations.
Repetition mechanism
Some times 503 error code that is outer exception you received may be caused by service unavailable. It means that server can be under maintenance or temporary overloaded. So you can write repetition mechanism to make continious sending requests to server until it'll be accessible.
Polly library may help you with repetition mechanism creation

The inner error of that 503 is:
The remote server returned an error: (429)
HTTP 429 indicates too many requests. Maybe your upstream server can’t process all requests sent.
This can happen when you reached rate limiting / throttling value if you’re calling a third party API.
UPDATE
As per page 28 in the API docs, you could configure throttling when creating a new API. Check if the throttling is too small or maybe turn off the throttling and see if that could fix the error?

Recover from a NameResolutionFailure error during a HTTPWebRequest

In our Outlook COM add-in, we're making an API call to our server using the .NET HTTPWebRequest method. One of our customers is running into a System.Net.WebException with the message The remote name could not be resolved: 'mydomain.com' and WebExceptionStatus.NameResolutionFailure as the status. All the users from this particular customer's company are using outlook/the addin from behind a VPN so we are piggy-backing proxy configuration from IE in order to make the request.
Our API calls work for a period of time but then it randomly disconnects and then does not allow future requests to go through either. Once the users close and restart Outlook though, it seems to work just fine again without changing any network configuration or reconnecting to wifi etc.
Other posts like this suggested retrying with a sleep in between. We have added a retry mechanism with 3 attempts, each with a sleep in between but that has not resolved the intermitent issue.
Our domain is hooked up to an AWS Classic Load Balancer so mydomain.com actually resolves a CNAME record to an AWS static domain, pointing to the ELB. I'm not sure if that would have any impact on the request or routing it.
The strange part is we also have a web browser component that loads a web page in a sidebar from the exact same domain as the API calls. It works perfectly and loads a URL from the same domain. The users can also load the URL in their browsers without any issues. It just appears that the HTTPWebRequest is running into the domain resolution issue. We've checked that it's not just a matter of a weak wifi signal. Since they are able to use IE which has the same proxy config to access the site just fine, I don't think it's that.
We're at a loss for how to gracefully recover and have the request try again. I've looked into suggestions from this answer and this other answer, we'll be trying those next. Unfortunately, we are not able to make the requests use direct IP addresses as some of the other answers suggest. That also eliminates the ability to edit the hosts file to point straight to it. The reason is we can't assign a static IP on a classic ELB.
We're considering trying to set the host to use the CNAME record from AWS directly but this is going to cause SSL errors as it doesn't have a valid cert for the CNAME entry. Is there a way to get around that by masking it via a header, similar to the IP approach?
Feel free to ask for more information, I will do my best to provide it.
Any suggestions on what to try / troubleshoot are welcome!
Update: We’re targeting .NET v4.5
Here's the code
var result = string.Empty;
bool retrying = false;
int retries = 0;
HttpWebRequest webRequest = null;
try
{
ServicePointManager.ServerCertificateValidationCallback =
CertificateCheck;
ServicePointManager.MaxServicePoints = 4;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
retry:
webRequest = (HttpWebRequest)WebRequest.Create(uriParam);
webRequest.Timeout = 120000;
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.Accept = acceptParam;
webRequest.Headers.Add("Cookie", cookieParam);
webRequest.UseDefaultCredentials = true;
webRequest.Proxy = null;
webRequest.KeepAlive = true; //default
webRequest.ServicePoint.ConnectionLeaseTimeout = webRequest.Timeout;
webRequest.ServicePoint.MaxIdleTime = webRequest.Timeout;
webRequest.ContentLength = dataParam.Length;
using (var reqStream = webRequest.GetRequestStream())
{
reqStream.Write(dataParam, 0, dataParam.Length);
reqStream.Flush();
reqStream.Close();
}
try
{
using (WebResponse webResponse = webRequest.GetResponse())
{
using (var responseStream = webResponse.GetResponseStream())
{
if (responseStream != null)
{
using (var reader = new StreamReader(responseStream))
{
result = reader.ReadToEnd();
}
}
}
webResponse.Close();
}
}
catch (WebException)
{
if (retrying && retries == 3)
{
//don't retry any more
return string.Empty;
}
retrying = true;
retries++;
webRequest.Abort();
System.Threading.Thread.Sleep(2000);
goto retry;
}
}
catch (Exception ex)
{
Log.Error(ex);
result = string.Empty;
}
finally
{
webRequest?.Abort();
}
return result;

HttpWebRequest random 'request timed out' exception

I am currently developing in Unity (in particular using C#) and I'm stuck with HttpWebRequest - HttpWebResponse random timeouts.
I have some methods that send a POST request to a server I host on my local machine (XAMPP) to use various php scripts which are going to fetch informations from MySQL Database (hosted with XAMPP) and give back those info in JSON format.
Then I handle these JSON informations with my C# scripts.
The problem is that when I run the first test all is good:I can get the JSON data from my Server and show it in the Debug Console.
When I run the second test,a WebException is raised with error:
WebException - The request timed out
After that second test,if I run again and again,the problem keeps presenting in a random way.
I followed all the guidelines I found on the internet on how to setup a webrequest - webresponse properly,in particular I tried to use ServicePoint.DefaultConnectionLimit and ServicePoint.MaxServicePointIdleTime,without any result.
The general structure of my methods (regarding the web request/response part) is something like that:
public void WebMethod(){
string post_url = "http://localhost/service.php?someparam=1&someparam=2";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(post_url);
request.Method = "POST";
request.KeepAlive = false;
request.Timeout = 5000;
request.Proxy = null;
string Response = "";
try
{
using (HttpWebResponse resp = request.GetResponse() as HttpWebResponse)
{
using (Stream objStream = resp.GetResponseStream())
{
using (StreamReader objReader = new StreamReader(objStream, Encoding.UTF8))
{
Response = objReader.ReadToEnd();
objReader.Close();
}
objStream.Flush();
objStream.Close();
}
resp.Close();
}
}catch(WebException e)
{
Debug.Log(e.Message);
}
finally
{
request.Abort();
}
//tried this one after reading some related answers here on StackOverflow,without results
//GC.Collect();
Debug.Log("SERVER RESPONSE:" + Response);
//Response Handling
}
I know that it may be something related to a wrong abort on the HttpWebRequest / Response or maybe related to the HTTP 1.1 connections limit,but I can't figure out any solution at the moment.
Any help is appreciated.

Ways of speeding up WebRequests? [duplicate]

This question already has answers here:
How to perform a fast web request in C#
(4 answers)
Closed 7 years ago.
I've made an app that can access and control Onvif cameras which it does well enough. However this is my first time making any app that uses web requests like this (or at all) so I assume I'm probably using quite basic techniques.
The part of code I'm curious about is this:
Uri uri = new Uri(
String.Format("http://" + ipAddr + "/onvif/" + "{0}", Service));
WebRequest request = WebRequest.Create((uri));
request.Method = "POST";
byte[] b = Encoding.ASCII.GetBytes(PostData);
request.ContentLength = b.Length;
//request.Timeout = 1000;
Stream stream = request.GetRequestStream();
//Send Message
XmlDocument recData = new XmlDocument();
try
{
using (stream = request.GetRequestStream())
{
stream.Write(b, 0, b.Length);
}
//Store response
var response = (HttpWebResponse) request.GetResponse();
if (response.GetResponseStream() != null)
{
string responsestring = new
StreamReader(response.GetResponseStream())
.ReadToEnd();
recData.LoadXml(responsestring);
}
}
catch (SystemException e)
{
MessageBox.Show(e.Message);
}
return recData;
}
The code works fine, however using the writeline statements I've found that the first request takes about 400ms to complete whereas the subsequent ones only take between 10 - 20ms. Is there anything I can do to speed up the first request?

You're doing it just fine. The reason for the difference in time to complete may be due to HTTP Keep-Alive. By default, the same connection is reused for subsequent requests. So the first request has to establish the connection, which is probably why it takes longer. The rest of the requests use the same already-open connection.

Aside from potential network and server issues, the request itself matters. You can opt to reduce the size of the request or break it down and asynchronously load your files.
Web Servers out of the box will not take 400ms to complete a simple request.

C# HTTP web request keeps timing out

I am making a Http Webrequest to an available site that I can visit fine, but the HTTP Web request keeps timing out. Is there any reason why this code might allow it to timeout when it shouldn't?
I've tried upping the timeout setting, but it still continues to timeout.
Uri CameraUrl = new Uri("http://" + cfg_cameraIps[i]);
HttpWebRequest myRequest = (HttpWebRequest)WebRequest.Create(CameraUrl);
myRequest.Timeout = 5000;
myRequest.Method = "HEAD";
try
{
HttpWebResponse webresponse;
webresponse = (HttpWebResponse)myRequest.GetResponse();
if (webresponse.StatusCode.ToString() == "OK")
{
continue;
}

You're not closing your web response - if you find that the first couple of requests work, but ones after that don't, then that's the problem. It's trying to reuse the existing connection to the server, but it can't because you haven't closed the response.
Change your code to:
using (HttpWebResponse webresponse = (HttpWebResponse) myRequest.GetResponse())
{
if (webresponse.StatusCode == HttpStatusCode.OK)
{
continue;
}
...
}
and see if that helps.
If it's failing on the very first request to the server, then that's something different. In that case, use Wireshark to see what's going on at the network level.
Note that in the code above I've also removed the string conversion in favour of comparing the status codes directly.