Develop Reference

How to keep alive a WebClient object to avoid session expiration in C#

Could you help me with this question, I have a windows form in VS2019, which I want to get the html generated after send the captcha and other parameter (2 parameters), my windows form contains devexpress controls, a picture edit where I load the captcha, textedit1 where I put the captcha, textedit2 where i put the second parameter (e.g. 06892898), simplebutton1 where I clic to load the captcha to the picture edit, the simplebutton2 to request the website with the 2 parameters, the problem is that when i passed the two parameter in the url, so I got an html with the message "The session has finished", How could I get the session or keep alive the webclient object in C#. Thanks in advance, This is my code:
private WebClient myClient = new WebClient();
public FrmData()
{
InitializeComponent();
}
private void btnCaptcha_Click(object sender, EventArgs e)
{
myClient.DownloadFile(new Uri("http://ww4.essalud.gob.pe:7777/acredita/captcha.jpg"), #"c:\temp\captcha.jpg");
pictureCaptcha.Image = Image.FromFile(#"c:\temp\captcha.jpg");
}
private void btnLoad_Click(object sender, EventArgs e)
{
string line = string.Empty;
try
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://ww4.essalud.gob.pe:7777/acredita/servlet/Ctrlwacre?captchafield_doc=" + txtedit1.Text + "&td=1&nd=" + txtedit2.Text);
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
StreamReader sr = new StreamReader(resp.GetResponseStream(), System.Text.Encoding.UTF8);
line = sr.ReadToEnd();
sr.Close();
resp.Close();
XtraMessageBox.Show("" + line);
}
catch (Exception ex)
{
XtraMessageBox.Show("" + ex.Message);
}
}

I got the solution using HttpWebRequest, HttpWebResponse, CookieContainer class, I did this code:
private HttpWebRequest request;
private HttpWebResponse response;
private CookieContainer cookies = new CookieContainer();
private void btnCaptcha_Click(object sender, EventArgs e)
{
try
{
request = (HttpWebRequest)WebRequest.Create("http://ww4.essalud.gob.pe:7777/acredita/captcha.jpg");
request.CookieContainer = cookies;
request.Timeout = 10000;
request.ReadWriteTimeout = 10000;
request.KeepAlive = true;
response = (HttpWebResponse)request.GetResponse();
pictureCaptcha.Image = Image.FromStream(response.GetResponseStream());
}
catch (Exception ex)
{
XtraMessageBox.Show("" + ex.Message);
}
}
private void btnLoad_Click(object sender, EventArgs e)
{
string line = string.Empty;
try
{
request = (HttpWebRequest)WebRequest.Create("http://ww4.essalud.gob.pe:7777/acredita/servlet/Ctrlwacre?captchafield_doc=" + txtCaptcha.Text + "&td=1&nd=06915398");
request.CookieContainer = cookies;
response = (HttpWebResponse)request.GetResponse();
StreamReader sr = new StreamReader(response.GetResponseStream(), System.Text.Encoding.UTF8);
line = sr.ReadToEnd();
sr.Close();
response.Close();
}
catch (Exception ex)
{
XtraMessageBox.Show("" + ex.Message);
}
XtraMessageBox.Show("" + line);
}
Thanks for all the comments and the idea to use cookies and keep alive.

c# catch empty Json / 404 error

I have the following code for accessing a Api which returns a Json value. Now it's possible that i try to access the api but nothing is being returned, aka the given ID its trying to search doesnt exist. This ofcourse returns a 404 but i do not know how to handle this error so the code continious on going, now it breaks the program and crashes.
public RootObject GetApi(string url)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
try{
WebResponse response = request.GetResponse();
using (Stream responseStream = response.GetResponseStream()){
StreamReader reader = new StreamReader(responseStream, Encoding.UTF8);
var jsonReader = new JsonTextReader(reader);
var serializer = new JsonSerializer();
return serializer.Deserialize<RootObject>(jsonReader);
}
}
catch (WebException ex){
WebResponse errorResponse = ex.Response;
using (Stream responseStream = errorResponse.GetResponseStream()){
StreamReader reader = new StreamReader(responseStream, Encoding.GetEncoding("utf-8"));
String errorText = reader.ReadToEnd();
// log errorText
}
throw;
}
}
This is the button click event where the Url of the api is given.
private void button1_Click(object sender, EventArgs e)
{
result_rTBox.Text = "";
api_Handler api_Handler = new api_Handler();
string spidyApi_itemSearch = "http://www.gw2spidy.com/api/v0.9/json/item-search/";
string Gw2Api_allListings = "https://api.guildwars2.com/v2/commerce/listings/";
string userInput_itemName = userSearchInput_tBox.Text;
string spidyApi_searchIdByName = spidyApi_itemSearch + userInput_itemName;
if (!string.IsNullOrWhiteSpace(userSearchInput_tBox.Text)){
var spidyApi_searchIdByName_result = api_Handler.GetApi(spidyApi_searchIdByName);
var Gw2Api_isItemIdinListing_result = api_Handler.GetApi(Gw2Api_allListings + spidyApi_searchIdByName_result.results[0].data_id);
//result_rTBox.Text = Gw2Api_isItemIdinListing_result.results[0].data_id.ToString();
}
}
First i access the api with string "spidApi_itemSearch" and after that I have and ID that i need to check if exists in the api Gw2Api_allListings. If it doesnt exist, which will happen quite often, it returns nothing with a 404 error. How do i get around of making the code continue even if it returns nothing?
EDIT: code that i have now, still crashes on the break.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
try
{
var requesting = WebRequest.Create(url);
using (var response = request.GetResponse())
{
using (var responseStream = response.GetResponseStream())
{
StreamReader reader = new StreamReader(responseStream, Encoding.UTF8);
var jsonReader = new JsonTextReader(reader);
var serializer = new JsonSerializer();
return serializer.Deserialize<RootObject>(jsonReader);
}
}
}
catch (WebException ex)
{
if (ex.Status == WebExceptionStatus.ProtocolError &&
ex.Response != null)
{
var resp = (HttpWebResponse)ex.Response;
if (resp.StatusCode == HttpStatusCode.NotFound){
}
}
throw;
}
}

Use the HttpStatusCode Enumeration, specifically HttpStatusCode.NotFound
Instead of WebResponse, try using HttpWebResponse
HttpWebResponse errorResponse = we.Response as HttpWebResponse;
if (errorResponse.StatusCode == HttpStatusCode.NotFound) {
// handle the error here
}
Where we is a WebException

WebException when loading rss feed

I am attempting to load a page I've received from an RSS feed and I receive the following WebException:
Cannot handle redirect from HTTP/HTTPS protocols to other dissimilar ones.
with an inner exception:
Invalid URI: The hostname could not be parsed.
I wrote a code that would attempt loading the url via an HttpWebRequest. Due to some suggestions I received, when the HttpWebRequest fails I then set the AllowAutoRedirect to false and basically manually loop through the iterations of redirect until I find out what ultimately fails. Here's the code I'm using, please forgive the gratuitous Console.Write/Writeline calls:
Uri url = new Uri(val);
bool result = true;
System.Net.HttpWebRequest req = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create(url);
string source = String.Empty;
Uri responseURI;
try
{
using (System.Net.WebResponse webResponse = req.GetResponse())
{
using (HttpWebResponse httpWebResponse = webResponse as HttpWebResponse)
{
responseURI = httpWebResponse.ResponseUri;
StreamReader reader;
if (httpWebResponse.ContentEncoding.ToLower().Contains("gzip"))
{
reader = new StreamReader(new GZipStream(httpWebResponse.GetResponseStream(), CompressionMode.Decompress));
}
else if (httpWebResponse.ContentEncoding.ToLower().Contains("deflate"))
{
reader = new StreamReader(new DeflateStream(httpWebResponse.GetResponseStream(), CompressionMode.Decompress));
}
else
{
reader = new StreamReader(httpWebResponse.GetResponseStream());
}
source = reader.ReadToEnd();
reader.Close();
}
}
req.Abort();
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(source);
result = true;
}
catch (ArgumentException ae)
{
Console.WriteLine(url + "\n--\n" + ae.Message);
result = false;
}
catch (WebException we)
{
Console.WriteLine(url + "\n--\n" + we.Message);
result = false;
string urlValue = url.ToString();
try
{
bool cont = true;
int count = 0;
do
{
req = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create(urlValue);
req.Headers.Add("Accept-Language", "en-us,en;q=0.5");
req.AllowAutoRedirect = false;
using (System.Net.WebResponse webResponse = req.GetResponse())
{
using (HttpWebResponse httpWebResponse = webResponse as HttpWebResponse)
{
responseURI = httpWebResponse.ResponseUri;
StreamReader reader;
if (httpWebResponse.ContentEncoding.ToLower().Contains("gzip"))
{
reader = new StreamReader(new GZipStream(httpWebResponse.GetResponseStream(), CompressionMode.Decompress));
}
else if (httpWebResponse.ContentEncoding.ToLower().Contains("deflate"))
{
reader = new StreamReader(new DeflateStream(httpWebResponse.GetResponseStream(), CompressionMode.Decompress));
}
else
{
reader = new StreamReader(httpWebResponse.GetResponseStream());
}
source = reader.ReadToEnd();
if (string.IsNullOrEmpty(source))
{
urlValue = httpWebResponse.Headers["Location"].ToString();
count++;
reader.Close();
}
else
{
cont = false;
}
}
}
} while (cont);
}
catch (UriFormatException uriEx)
{
Console.WriteLine(urlValue + "\n--\n" + uriEx.Message + "\r\n");
result = false;
}
catch (WebException innerWE)
{
Console.WriteLine(urlValue + "\n--\n" + innerWE.Message+"\r\n");
result = false;
}
}
if (result)
Console.WriteLine("testing successful");
else
Console.WriteLine("testing unsuccessful");
Since this is currently just test code I hardcode val as http://rss.nytimes.com/c/34625/f/642557/s/3d072012/sc/38/l/0Lartsbeat0Bblogs0Bnytimes0N0C20A140C0A70C30A0Csarah0Ekane0Eplay0Eamong0Eofferings0Eat0Est0Eanns0Ewarehouse0C0Dpartner0Frss0Gemc0Frss/story01.htm
the ending url that gives the UriFormatException is: http:////www-nc.nytimes.com/2014/07/30/sarah-kane-play-among-offerings-at-st-anns-warehouse/?=_php=true&_type=blogs&_php=true&_type=blogs&_php=true&_type=blogs&_php=true&_type=blogs&_php=true&_type=blogs&_php=true&_type=blogs&_php=true&_type=blogs&partner=rss&emc=rss&_r=6&
Now I'm sure if I'm missing something or if I'm doing the looping wrong, but if I take val and just put that into a browser the page loads fine, and if I take the url that causes the exception and put it in a browser I get taken to an account login for nytimes.
I have a number of these rss feed urls that are resulting in this problem. I also have a large number of these rss feed urls that have no problem loading at all. Let me know if there is any more information needed to help resolve this. Any help with this would be greatly appreciated.
Could it be that I need to have some sort of cookie capability enabled?

You need to keep track of the cookies while doing all your requests. You can use an instance of the CookieContainer class to achieve that.
At the top of your method I made the following changes:
Uri url = new Uri(val);
bool result = true;
// keep all our cookies for the duration of our calls
var cookies = new CookieContainer();
System.Net.HttpWebRequest req = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create(url);
// assign our CookieContainer to the new request
req.CookieContainer = cookies;
string source = String.Empty;
Uri responseURI;
try
{
And in the exception handler where you create a new HttpWebRequest, you do the assignment from our CookieContainer again:
do
{
req = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create(urlValue);
// reuse our cookies!
req.CookieContainer = cookies;
req.Headers.Add("Accept-Language", "en-us,en;q=0.5");
req.AllowAutoRedirect = false;
using (System.Net.WebResponse webResponse = req.GetResponse())
{
This makes sure that on each successive call the already present cookies are resend again in the next request. If you leave this out, no cookies are sent and therefore the site you try to visit assumes you are a fresh/new/unseen user and gives you a kind of authentication path.
If you want to store/keep cookies beyond this method you could move the cookie instance variable to a static public property so you can use all those cookies program-wide like so:
public static class Cookies
{
static readonly CookieContainer _cookies = new CookieContainer();
public static CookieContainer All
{
get
{
return _cookies;
}
}
}
And to use it in a WebRequest:
var req = (System.Net.HttpWebRequest) WebRequest.Create(url);
req.CookieContainer = Cookies.All;

HttpWebRequest proper exception handling

So I'm using the HttpWebRequest API in the System.Net assembly but because C# has no checked exceptions, I'm not sure where to put my try-catch blocks to properly handle inevitable exceptions caused by common things like a network error. You know, in Java we would call these plain old checked IOExceptions.
This is what I have so far. Are my try-catch blocks properly set up to handle network errors? Am I wrapping the right method calls? Looking at the documentation, I think they are right, but I need a second pair of eyes.
HttpWebRequest request = WebRequest.CreateHttp(url);
request.Method = "POST";
request.BeginGetRequestStream(getRequestResult =>
{
HttpWebRequest getRequestRequest = (HttpWebRequest) getRequestResult.AsyncState;
try
{
Stream requestStream = getRequestRequest.EndGetRequestStream(getRequestResult);
requestStream.Write(parametersData, 0, parametersData.Length);
requestStream.Dispose();
getRequestRequest.BeginGetResponse(getResponseResult =>
{
HttpWebRequest getResponseRequest = (HttpWebRequest)getResponseResult.AsyncState;
try
{
WebResponse response = getResponseRequest.EndGetResponse(getRequestResult);
Stream responseStream = response.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);
string jsonString = reader.ReadToEnd();
reader.Dispose();
JObject jsonObject = JObject.Parse(jsonString);
onResult(StatusCode.Ok, jsonObject);
}
catch (WebException)
{
onResult(StatusCode.NetworkError);
}
}, getRequestRequest);
}
catch (IOException)
{
onResult(StatusCode.NetworkError);
}
}, request);

First off, unless there's some reason that you need to use HttpWebRequest, then you're better off using WebClient.UploadString instead, or any of WebClient's other UploadXXX overloads for uploading name/value pairs, files, binary data, etc. This will be much easier for you, and easier to troubleshoot and debug. Also, another problem is that you're treating exceptions during JSON parsing or during your onResult handler error as network errors.
Below are three examples of using WebClient that you might want to try: a synchronous version, an "old-style" async version, and a "new-style" async version that uses async/await. All three versions also try to fix the exception handling issue that I noted above. If you don't need async support, then the first version will be easiest.
static void PostSync (string url, string parametersData)
{
using (WebClient wc = new WebClient())
{
wc.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded"; // or "application/json" or ...
try
{
string htmlResult = wc.UploadString(url, parametersData); // or UploadValues, UploadFile, ...
JObject jsonObject = null;
try
{
jsonObject = JObject.Parse(htmlResult);
}
catch (JsonException ex)
{
onResult(StatusCode.JsonError);
}
onResult(StatusCode.Ok, jsonObject);
}
catch (System.Net.WebException ex)
{
onResult(StatusCode.NetworkError);
}
}
}
static void PostAsync(string url, string parametersData)
{
using (WebClient wc = new WebClient())
{
wc.UploadStringCompleted += (Object sender, UploadStringCompletedEventArgs e) =>
{
if (e.Error != null)
onResult(StatusCode.NetworkError);
JObject jsonObject = null;
try
{
jsonObject = JObject.Parse(e.Result);
}
catch (JsonException ex)
{
onResult(StatusCode.JsonError);
}
onResult(StatusCode.Ok, jsonObject);
};
try
{
wc.UploadStringAsync(new Uri(url, UriKind.Absolute), parametersData);
}
catch (System.Net.WebException ex)
{
onResult(StatusCode.NetworkError);
}
}
}
static async void PostTaskAsync(string url, string parametersData)
{
using (WebClient wc = new WebClient())
{
wc.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded"; // or "application/json" or ...
try
{
string htmlResult = await wc.UploadStringTaskAsync(url, parametersData); // or UploadValues, UploadFile, ...
JObject jsonObject = null;
try
{
jsonObject = JObject.Parse(htmlResult);
}
catch (JsonException ex)
{
onResult(StatusCode.JsonError);
}
onResult(StatusCode.Ok, jsonObject);
}
catch (System.Net.WebException ex)
{
onResult(StatusCode.NetworkError);
}
}
}

System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive

I'm trying to create a method in C# to return a string of a web pages html content from the url. I have tried several different ways, but I am getting the error System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive.
The following works fine locally, but gets the above error when running on a remote server:
public static string WebPageRead(string url)
{
string result = String.Empty;
WebResponse response = null;
StreamReader reader = null;
try
{
if (!String.IsNullOrEmpty(url))
{
HttpWebRequest request = HttpWebRequest.Create(url) as HttpWebRequest;
request.Method = "GET";
request.KeepAlive = false;
request.ProtocolVersion = HttpVersion.Version10;
response = request.GetResponse();
reader = new StreamReader(response.GetResponseStream(), Encoding.UTF8);
result = reader.ReadToEnd();
}
}
catch (Exception exc)
{
throw exc;
}
finally
{
if (reader != null)
{
reader.Close();
}
if (response != null)
{
response.Close();
}
}
return result;
}

This is probably not the problem, but try the following:
public static string WebPageRead(string url)
{
if (String.IsNullOrEmpty(url))
{
return null;
}
HttpWebRequest request = HttpWebRequest.Create(url) as HttpWebRequest;
if (request == null)
{
return null;
}
request.Method = "GET";
request.KeepAlive = false;
request.ProtocolVersion = HttpVersion.Version10;
using (WebResponse response = request.GetResponse())
{
using (Stream stream = response.GetResponseStream())
{
using (StreamReader reader =
new StreamReader(stream, Encoding.UTF8))
{
return reader.ReadToEnd();
}
}
}
}
I echo the earlier answer that suggests you try this with a known good URL. I'll add that you should try this with a known good HTTP 1.1 URL, commenting out the line that sets the version to 1.0. If that works, then it narrows things down considerably.

Thanks for the responses, the problem was due to a DNS issue on the remote server! Just to confirm, I went with the following code in the end:
public static string WebPageRead(string url)
{
string content = String.Empty;
if (!String.IsNullOrEmpty(url))
{
HttpWebRequest request = HttpWebRequest.Create(url) as HttpWebRequest;
if (request != null)
{
request.Method = "GET";
request.KeepAlive = false;
request.ProtocolVersion = HttpVersion.Version10;
try
{
using (WebResponse response = request.GetResponse())
{
using (Stream stream = response.GetResponseStream())
{
using (StreamReader reader = new StreamReader(stream, Encoding.UTF8))
{
content = reader.ReadToEnd();
}
}
}
}
catch (Exception exc)
{
throw exc;
}
}
}
return content;
}

Had a problem like this before that was solved by opening the url in IE on the machine with the problem. IE then asks you whether you want to add the url to the list of secure sites. Add it and it works for that url.
This is just one of the possible causes. Seriously a lot of other problems could cause this. Besides the problem described above, the best way I've found to solve this is the just catch the exception and retry the request.