I have several Http Web Requests in various loops etc. The web requests get data from a variety of APIs.
These seem to work some of the time, but most of the time (recently) I am getting Timeout exception errors (Operation has timed out) and am not sure why.
I accept that every so often or once in a while you will get a time out error, but this is happening too often.
Here are two of my WebRequest codes:
public static EventList getEvents()
{
Uri myURI = new Uri("http://feeds.betway.com/events?key=XXX&keywords=horse-racing,uk-and-ireland&and=true");
WebRequest webRequest = WebRequest.Create(myURI);
webRequest.Timeout = 3000;
using (WebResponse webResponse = webRequest.GetResponse())
{
using (Stream stream = webResponse.GetResponseStream())
{
using (var reader = XmlReader.Create(stream))
{
XmlSerializer serializer = new XmlSerializer(typeof(EventList));
EventList data = (EventList)serializer.Deserialize(reader);
return data;
}
}
}
}
public static List<WilliamHillData.Event> GetAllCompetitionEvents(string compid)
{
string res = "";
Uri myURI = new Uri("https://gw.whapi.com/v2/sportsdata/competitions/" + compid + "/events/?&sort=startDateTime");
WebRequest webRequest = WebRequest.Create(myURI);
webRequest.Headers.Add("Content-Type", "application/json");
webRequest.Headers.Add("apiKey", "xxx");
webRequest.Timeout = 2000;
using (WebResponse webResponse = webRequest.GetResponse())
{
using (Stream stream = webResponse.GetResponseStream())
{
StreamReader reader = new StreamReader(stream, Encoding.UTF8);
res = reader.ReadToEnd();
}
}
JObject jobject = JObject.Parse(res);
List<WilliamHillData.Event> list = jobject["events"].ToObject<List<WilliamHillData.Event>>();
return list;
}
I cannot see anything wrong with my code as I am disposing objects correctly and have set the Timeout. Does the timeout need increasing or am I missing something?
Could this also possibly be a network issue on our end?
Related
I am using below code to make an API call from my C# code with WebRequest:
public object GetData()
{
object response = "";
string token = "EF232354";
string baseUrl = ConfigurationManager.AppSettings["BaseURL"].ToString();
string endPoint = ConfigurationManager.AppSettings["EndPoint"].ToString();
var httpWebRequest = (HttpWebRequest) WebRequest.Create(baseUrl + endPoint);
httpWebRequest.ContentType = "application/json";
httpWebRequest.Method = HttpVerb.GET.ToString();
httpWebRequest.Headers.Add("token", token);
var httpResponse = (HttpWebResponse) httpWebRequest.GetResponse();
Stream dataStream = httpResponse.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
using(JsonReader sdr = new JsonTextReader(reader))
{
JsonSerializer serializer = new JsonSerializer();
response = serializer.Deserialize(sdr);
}
return response;
}
Sometimes I am getting:
Message: The remote server returned an error: (504) Gateway Timeout.
Exception Type: System.Net.WebException
And how many requests can WebRequest make at a time?
I was adapting the question to demonstrate reading to a memory stream, when I noticed that the response was not being disposed. This is 95% likely to be your underlying problem. Streams and StreamReaders are also disposable and should be wrapped with using() closures.
public object GetData()
{
object response = "";
string token = "EF232354";
string baseUrl = ConfigurationManager.AppSettings["BaseURL"].ToString();
string endPoint = ConfigurationManager.AppSettings["EndPoint"].ToString();
var httpWebRequest = (HttpWebRequest) WebRequest.Create(baseUrl + endPoint);
httpWebRequest.ContentType = "application/json";
httpWebRequest.Method = HttpVerb.GET.ToString();
httpWebRequest.Headers.Add("token", token);
using (var httpResponse = (HttpWebResponse) httpWebRequest.GetResponse())
{
using (Stream dataStream = httpResponse.GetResponseStream())
{
using (StreamReader reader = new StreamReader(dataStream))
{
using(JsonReader sdr = new JsonTextReader(reader))
{
JsonSerializer serializer = new JsonSerializer();
response = serializer.Deserialize(sdr);
}
return response;
}
}
httpResponse.Close(); // For good measure. *should* be covered by Dispose.
}
}
I am attempting to create a console app that sends a WebRequest to a website so that I can get some information back from it in JSON format. Once I build up the request and try to get response I just want to simply print out the data, but when I call httpWebRequest.getResponse() it returns NULL.
I have tried multiple other methods of sending the data to the the url but those are all giving me like 404, or 400 errors, etc. This method at least isn't giving me any error, just a NULL.
Here is a snapshot of the documentation I am using for the API (albeit the docs aren't complete yet):
Here is the console app code that I have right now:
try
{
var httpWebRequest = (HttpWebRequest)WebRequest.Create("https://api.remot3.it/apv/v27/user/login");
httpWebRequest.ContentType = "application/json";
httpWebRequest.Method = "POST";
httpWebRequest.Headers.Add("developerkey", "***KEY***");
using (var streamWriter = new
StreamWriter(httpWebRequest.GetRequestStream()))
{
string json = new JavaScriptSerializer().Serialize(new
{
email = "***EMAIL***",
password = "***PASSWORD***"
});
Console.WriteLine(json);
streamWriter.Write(json);
}
var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var result = streamReader.ReadToEnd();
Console.WriteLine(result);
Console.ReadLine();
}
}catch(Exception e)
{
Console.WriteLine(e.Message);
Console.WriteLine(e.StackTrace);
Console.ReadLine();
}
Expected output is some JSON data, but I am getting a NULL back from getResponse().
Try to serialize the credential in your form and for header send as parameter for this class.
Check below for my code. It is not 100 % fit to your requirement, but atleast it will help to get through your logic.
Here is what I get Json Response from this code. Its work Perfect. Please remember to add timeout option on your webrequest and at the end close the streamreader and stream after completing your task. please check this code.
public static string httpPost(string url, string json)
{
string content = "";
byte[] bs;
if (json != null && json != string.Empty)
{
bs = Encoding.UTF8.GetBytes(json);
}
else
{
bs = Encoding.UTF8.GetBytes(url);
}
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(url);
req.Method = "POST";
if (json != string.Empty)
req.ContentType = "application/json";
else
req.ContentType = "application/x-www-form-urlencoded";
req.KeepAlive = false;
req.Timeout = 30000;
req.ReadWriteTimeout = 30000;
//req.UserAgent = "test.net";
req.Accept = "application/json";
req.ContentLength = bs.Length;
using (Stream reqStream = req.GetRequestStream())
{
reqStream.Write(bs, 0, bs.Length);
reqStream.Flush();
reqStream.Close();
}
using (WebResponse wr = req.GetResponse())
{
Stream s = wr.GetResponseStream();
StreamReader reader = new StreamReader(s, Encoding.UTF8);
content = reader.ReadToEnd();
wr.Close();
s.Close();
reader.Close();
}
return content;
}
I am doing an API Post request and cant seem to get it to work. I always get a sendFailure webexception and the response for the exception is always null so catching the exception is useless. It keeps happening when I try to get the httpWebResponse. I noticed too the request.contentlength gave errors at postream getrequeststream so i commented it out. Test.json is the file I use for the request body. I also tested this on different API testers by including the URL, body, and content-type in the header and they worked. I just cant seem to code it for myself. The credentials work I just dont know if im doing the request correctly?
JSON File:
{
"email": "abc#123.com",
"password": "12345",
"facilityNumber": "987654"
}
string filepath = "test.json";
string result = string.Empty;
using (StreamReader r = new StreamReader(filepath))
{
var json = r.ReadToEnd();
var jobj = JObject.Parse(json);
foreach (var item in jobj.Properties())
{
item.Value = item.Value.ToString().Replace("v1", "v2");
}
result = jobj.ToString();
Console.WriteLine(result);
}
try
{
string setupParameters;
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("https://www.test.com/abcde");
request.AllowAutoRedirect = true;
setupParameters = result;
ServicePointManager.ServerCertificateValidationCallback = (s, cert, chain, ssl) => true;
ASCIIEncoding encoding = new ASCIIEncoding();
var postData = setupParameters;
request.Method = "POST";
request.ContentType = "application/json";
byte[] data = encoding.GetBytes(postData);
//request.ContentLength = data.Length;
using (StreamWriter postStream = new StreamWriter(request.GetRequestStream()))//error if uncomment contentlength
{
postStream.Write(postData);
postStream.Flush();
postStream.Close();
}
HttpWebResponse wr = (HttpWebResponse)request.GetResponse();//error occurs
Stream receiveStream = wr.GetResponseStream();
StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8);
}
catch (WebException ex)
{
if (ex.Response != null)
{
using (var errorResponse = (HttpWebResponse)ex.Response)
{
using (var reader = new StreamReader(errorResponse.GetResponseStream()))
{
string error = reader.ReadToEnd();
result = error;
}
}
}
I suggest modifiying your request to follow this format. Especially pay attention to the request.Method and request.ContentType which have caught me out multiple times.
Also, handling the response is easier this way.
try
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(YOURURL);
request.ContentType = "application/json; charset=utf8";
request.Headers.Add(ADD HEADER HERE IF YOU NEED ONE);
request.Method = WebRequestMethods.Http.Post; // IMPORTANT
using (var streamWriter = new StreamWriter(request.GetRequestStream()))
{
streamWriter.Write(JsonConvert.SerializeObject(JSONBODYSTRING));
// I USUALLY YOU JSONCONVERT HERE TO SIMPLY SERIALIZE A STRING CONTAINING THE JSON INFO.
//BUT I GUESS YOUR METHOD WOULD ALSO WORK
streamWriter.Flush();
streamWriter.Close();
}
WebResponse response = request.GetResponse();
using (var streamReader = new StreamReader(response.GetResponseStream()))
{
string result = streamReader.ReadToEnd();
// DO WHATEVER YOU'D LIKE HERE
}
} catch (Exception ex)
{
// HANDLE YOUR EXCEPTIONS
}
http://ssw.com/profile/?apikey = skdwkdkfkkdj
I tried to use
public async Task<string> GetFromUriAsync(string requestUri, string token)
{
var client = new HttpClient();
client.BaseAddress = new Uri(BaseUri);
client.DefaultRequestHeaders.Authorization =
new AuthenticationHeaderValue("apikey", "=" + token);
HttpResponseMessage response = await client.GetAsync(requestUri);
response.EnsureSuccessStatusCode();
string responseBody = await response.Content.ReadAsStringAsync();
return responseBody;
}
Then it returns null
Am I missing something or is it just totally wrong?
Thanks
You're trying to pass the API key in the header information of your HTTP request. What you need to do is just pass that whole URL without any additional header information.
IE: use "http://ssw.com/profile?apikey=abcdef" as the requestUri and send token as null. Also, remove the setting of the client.DefaultRequestHeaders.Authorization property. Authorization was meant to be a user/pass system and not a token-based system.
To test this, download Fiddler 4 (https://www.telerik.com/download/fiddler). Once you have fiddler installed, on the "Composer" tab, you can test different queries you need by putting the URL directly into the URL box and clicking "Execute". You'll then be able to use the inspectors to see the responses and figure out where you need to go from there.
Here are the classes I use for HTTP GET and POST operations:
public static string HTTPGET(string url)
{
try
{
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
request.Timeout = 100000;
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
Stream responseStream = response.GetResponseStream();
if (responseStream != null)
using (StreamReader resStream = new StreamReader(responseStream))
return resStream.ReadToEnd();
return null;
}
catch (Exception e)
{
Console.WriteLine(url);
Console.WriteLine(e);
return null;
}
}
public static string HTTPPOST(string url, string postData)
{
try
{
HttpWebRequest webRequest = (HttpWebRequest) WebRequest.Create(url);
webRequest.Method = "POST";
webRequest.ContentType = "x-www-form-urlencoded";
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
using (Stream requestStream = webRequest.GetRequestStream())
requestStream.Write(byteArray, 0, byteArray.Length);
using (Stream responseStream = webRequest.GetResponse().GetResponseStream())
if (responseStream != null)
using (StreamReader responseReader = new StreamReader(responseStream))
return responseReader.ReadToEnd();
return null;
}
catch (Exception e)
{
Console.WriteLine(url);
Console.WriteLine(postData);
Console.WriteLine(e);
return null;
}
}
I am using HttpWebRequest to get webpage source code with POST method.
The page needs to be accessed for multiple times with different parameters.
When getting response in the second time in a short period, it always return the same response object to me. After debugging, if the second call is after around 30 seconds, the request can get the correct response object(source code).
public HtmlAgilityPack.HtmlDocument getHtmlData(string url, string cname)
{
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
try
{
HttpRequestCachePolicy policy = new HttpRequestCachePolicy(HttpRequestCacheLevel.Default);
HttpWebRequest.DefaultCachePolicy = policy;
HttpRequestCachePolicy noCachePolicy = new HttpRequestCachePolicy(HttpRequestCacheLevel.NoCacheNoStore);
HttpWebRequest req = WebRequest.Create(url) as HttpWebRequest;
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
req.KeepAlive = false;
req.Headers[HttpRequestHeader.CacheControl] = "no-cache";
req.Headers[HttpRequestHeader.Pragma] = "no-cache";
req.IfModifiedSince = DateTime.Now;
req.CachePolicy = noCachePolicy;
//add post data
string postData = "cname=" + cname;
byte[] byteArray = Encoding.GetEncoding("shift-jis").GetBytes(postData);
req.ContentLength = byteArray.Length;
using (Stream stream = req.GetRequestStream())
{
stream.Write(byteArray, 0, byteArray.Length);
}
// get response
string data = "";
HttpWebResponse response = req.GetResponse() as HttpWebResponse;
using (var stream = response.GetResponseStream())
{
StreamReader reader = new StreamReader(stream, Encoding.GetEncoding("shift-jis"));
data = reader.ReadToEnd();
reader.Close();
}
response.Close();
doc.LoadHtml(data);
//System.Threading.Thread.Sleep(30000);
}
catch(Exception ex)
{
Log.log.Debug(ex.ToString());
}
return doc;
}
Function call block:
getHtmlData(#"http://www.jra.go.jp/JRADB/accessS.html", "pw01sli00/AF");
getHtmlData(#"http://www.jra.go.jp/JRADB/accessS.html", "pw01skl00999999/B3");
getHtmlData(#"http://www.jra.go.jp/JRADB/accessS.html", "pw01skl00201604/E3");
I have stuck with the problem for a whole day. hope someone can give me a clue.
Many thanks!