Develop Reference

HttpWebRequest GetRequestStream SystemException after several calls randomly

I have a program in C# .NET 3.5 CompactFramework that requests data from a C# WebService:
public SynchronisationResult<J> Get<J>(IEnumerable<DomainProxy> existingObjects, string controller, string parameters) where J : IdJsonObject)
{
string existingObjectsJson = JsonConvert.SerializeObject(existingObjects);
string url = GenerateUrl(controller, parameters);
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.ContentType = "text/json";
request.Method = "POST";
request.KeepAlive = false;
request.ProtocolVersion = HttpVersion.Version11;
request.Timeout = 60000;
request.ContentLength = existingObjectsJson.Length;
using (Stream requestStream = request.GetRequestStream())
{
using (var streamWriter = new StreamWriter(requestStream))
{
streamWriter.Write(existingObjectsJson);
streamWriter.Flush();
streamWriter.Close();
}
requestStream.Close();
}
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
string responseData = "";
using (var streamReader = new StreamReader(response.GetResponseStream()))
{
responseData = streamReader.ReadToEnd();
}
if (response.StatusCode == HttpStatusCode.OK)
{
result = JsonConvert.DeserializeObject<SynchronisationResult<J>>(responseData);
}
else
{
throw new Exception(responseData);
}
response.Close();
}
}
I can call this method several times with different parameters (different controllers on the WebServer) and suddenly everything gets stuck. The application does not react anymore, when i press PAUSE in Visual Studio i see the program pointer at the location
using (Stream requestStream = request.GetRequestStream())
Sometimes a SystemException is thrown in Timer.ring of System.Net.Connection..., although in case, the application does not continue to run, even not by bubbling the exception to the next catch-Block. This means, that i have to reset the device it does never continue to run.
I have tried the following changes to solve the problem:
request.KeepAlive = true / false
request.Pipelines = true / false
ServicePointManager.DefaultConnectionLimit = 1000;
request.AutomaticDecompression = DecompressionMethods.GZip or nothing
Nothing, the request works fine in Postman for example.
Weird thing is, if i implement this in a for loop, asking for about 200 objects to be updated it crashes faster. In case i implement the request method on a button click and click it with a frequency of about 10 seconds it works way longer. I tried with a development IIS backend on port 888, with a production machine on port 80, firewall locally is turned off. There is no certain request that fails it could be a request for type A or B or C,... each run is different.
Would someone explain:
a) why code gets stuck and does not continue?
b) why code gets stuck even when an exception is thrown
c) how to configure ServicePointManager or Request to get things working properly
EDIT: This is the Exception that sometimes occurs when request.GetRequestStream() is executed:
at System.Threading.Timer.startTimer(UInt32 dueTime)
at System.Threading.Timer.Change(UInt32 dueTime, UInt32 period)
at System.Threading.Timer.Change(Int32 dueTime, Int32 period)
at System.Threading.ThreadPool.QueueUserWorkItem(WaitCallback callBack, Object state, Boolean IsHttpRequest)
at System.Net.Connection.actionSending()
at System.Net.Connection.changeState(ConnectionState state)
at System.Net.Connection.transitionRequestSent(Event e)
at System.Net.Connection.processEvent(Event e)
at System.Net.Connection.actionRequestSent()
at System.Net.Connection.changeState(ConnectionState state)
at System.Net.Connection.transitionIdle(Event e)
at System.Net.Connection.processEvent(Event e)
at System.Net.Connection.submitRequest(HttpWebRequest request)
at System.Net.ServicePoint.SubmitRequest(HttpWebRequest request, String connGroupName)
at System.Net.HttpWebRequest.SubmitRequest()
at System.Net.HttpWebRequest.finishGetRequestStream()
at System.Net.HttpWebRequest.GetRequestStream()

I was also stuck on this issue for couple of days. Finally what I found was if I run Fiddler in the background there was no exception thrown in the request.GetRequestStream(). Which means this is something related to the connection pool where fiddler is handling this. So I did some research and found the below link which solved my issue:
https://www.telerik.com/blogs/help!-running-fiddler-fixes-my-app-
Also after the request is completed make sure you abort that as well. What I did is:
if (webrequest != null) webrequest.Abort();
For me everything is working fine now.

Getting the Redirected URLs from the Original URL [duplicate]

Using the WebClient class I can get the title of a website easily enough:
WebClient x = new WebClient();
string source = x.DownloadString(s);
string title = Regex.Match(source,
#"\<title\b[^>]*\>\s*(?<Title>[\s\S]*?)\</title\>",
RegexOptions.IgnoreCase).Groups["Title"].Value;
I want to store the URL and the page title. However when following a link such as:
http://tinyurl.com/dbysxp
I'm clearly going to want to get the Url I'm redirected to.
QUESTIONS
Is there a way to do this using the WebClient class?
How would I do it using HttpResponse and HttpRequest?

If I understand the question, it's much easier than people are saying - if you want to let WebClient do all the nuts and bolts of the request (including the redirection), but then get the actual response URI at the end, you can subclass WebClient like this:
class MyWebClient : WebClient
{
Uri _responseUri;
public Uri ResponseUri
{
get { return _responseUri; }
}
protected override WebResponse GetWebResponse(WebRequest request)
{
WebResponse response = base.GetWebResponse(request);
_responseUri = response.ResponseUri;
return response;
}
}
Just use MyWebClient everywhere you would have used WebClient. After you've made whatever WebClient call you needed to do, then you can just use ResponseUri to get the actual redirected URI. You'd need to add a similar override for GetWebResponse(WebRequest request, IAsyncResult result) too, if you were using the async stuff.

I know this is already an answered question, but this works pretty to me:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://tinyurl.com/dbysxp");
request.AllowAutoRedirect = false;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
string redirUrl = response.Headers["Location"];
response.Close();
//Show the redirected url
MessageBox.Show("You're being redirected to: "+redirUrl);
Cheers.! ;)

With an HttpWebRequest, you would set the AllowAutoRedirect property to false. When this happens, any response with a status code between 300-399 will not be automatically redirected.
You can then get the new url from the response headers and then create a new HttpWebRequest instance to the new url.
With the WebClient class, I doubt you can change it out-of-the-box so that it does not allow redirects. What you could do is derive a class from the WebClient class and then override the GetWebRequest and the GetWebResponse methods to alter the WebRequest/WebResponse instances that the base implementation returns; if it is an HttpWebRequest, then set the AllowAutoRedirect property to false. On the response, if the status code is in the range of 300-399, then issue a new request.
However, I don't know that you can issue a new request from within the GetWebRequest/GetWebResponse methods, so it might be better to just have a loop that executes with HttpWebRequest/HttpWebResponse until all the redirects are followed.

I got the Uri for the redirected page and the page contents.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(strUrl);
request.AllowAutoRedirect = true;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream dataStream = response.GetResponseStream();
strLastRedirect = response.ResponseUri.ToString();
StreamReader reader = new StreamReader(dataStream);
string strResponse = reader.ReadToEnd();
response.Close();

In case you are only interested in the redirect URI you can use this code:
public static string GetRedirectUrl(string url)
{
HttpWebRequest request = (HttpWebRequest) HttpWebRequest.Create(url);
request.AllowAutoRedirect = false;
using (HttpWebResponse response = HttpWebResponse)request.GetResponse())
{
return response.Headers["Location"];
}
}
The method will return
null - in case of no redirect
a relative url - in case of a redirect
Please note: The using statement (or a final response.close()) is essential. See MSDN Library for details. Otherwise you may run out of connections or get a timeout when executing this code multiple times.

HttpWebRequest.AllowAutoRedirect can be set to false. Then you'd have to manually http status codes in the 300 range.
// Create a new HttpWebRequest Object to the mentioned URL.
HttpWebRequest myHttpWebRequest=(HttpWebRequest)WebRequest.Create("http://www.contoso.com");
myHttpWebRequest.MaximumAutomaticRedirections=1;
myHttpWebRequest.AllowAutoRedirect=true;
HttpWebResponse myHttpWebResponse=(HttpWebResponse)myHttpWebRequest.GetResponse();

The WebClient class has an option to follow redirects. Set that option and you should be fine.

Ok this is really hackish, but the key is to use the HttpWebRequest and then set the AllowAutoRedirect property to true.
Here's a VERY hacked together example
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://tinyurl.com/dbysxp");
req.Method = "GET";
req.AllowAutoRedirect = true;
WebResponse response = req.GetResponse();
response.GetResponseStream();
Stream responseStream = response.GetResponseStream();
// Content-Length header is not trustable, but makes a good hint.
// Responses longer than int size will throw an exception here!
int length = (int)response.ContentLength;
const int bufSizeMax = 65536; // max read buffer size conserves memory
const int bufSizeMin = 8192; // min size prevents numerous small reads
// Use Content-Length if between bufSizeMax and bufSizeMin
int bufSize = bufSizeMin;
if (length > bufSize)
bufSize = length > bufSizeMax ? bufSizeMax : length;
StringBuilder sb;
// Allocate buffer and StringBuilder for reading response
byte[] buf = new byte[bufSize];
sb = new StringBuilder(bufSize);
// Read response stream until end
while ((length = responseStream.Read(buf, 0, buf.Length)) != 0)
sb.Append(Encoding.UTF8.GetString(buf, 0, length));
string source = sb.ToString();string title = Regex.Match(source,
#"\<title\b[^>]*\>\s*(?<Title>[\s\S]*?)\</title\>",RegexOptions.IgnoreCase).Groups["Title"].Value;
enter code here

Connecting to .aspx site from c# application

I'm trying to build c# application, which notify me when there is an "update" in site.
The site login form contains 3 textboxes, and it's login.aspx.
My question is, how can I "send" the 3 details to the site and connect(authenticate) from the application I want to build in c#, and if it's possible, how can I do it?
I looked for any guide or something to read about this but haven't found.

you need to use the WebClient class. More info on this class can be found at http://msdn.microsoft.com/en-us/library/system.net.webclient(v=vs.80).aspx
And a nice example at http://msdn.microsoft.com/en-us/library/system.net.webclient(v=vs.80).aspx?cs-save-lang=1&cs-lang=csharp#code-snippet-4

First you need post a form using c#
HttpWebRequest request = (HttpWebRequest)WebRequest.Create (args[0]);
// Set some reasonable limits on resources used by this request
request.MaximumAutomaticRedirections = 4;
request.MaximumResponseHeadersLength = 4;
// Set credentials to use for this request.
request.Credentials = CredentialCache.DefaultCredentials;
HttpWebResponse response = (HttpWebResponse)request.GetResponse ();
Console.WriteLine ("Content length is {0}", response.ContentLength);
Console.WriteLine ("Content type is {0}", response.ContentType);
// Get the stream associated with the response.
Stream receiveStream = response.GetResponseStream ();
// Pipes the stream to a higher level stream reader with the required encoding format.
StreamReader readStream = new StreamReader (receiveStream, Encoding.UTF8);
Console.WriteLine ("Response stream received.");
Console.WriteLine (readStream.ReadToEnd ());
response.Close ();
readStream.Close ();
then try to save cookie, its required to store aspnet_session_id into client for future requests
private class CookieAwareWebClient : WebClient
{
public CookieAwareWebClient()
: this(new CookieContainer())
{ }
public CookieAwareWebClient(CookieContainer c)
{
this.CookieContainer = c;
}
public CookieContainer CookieContainer { get; set; }
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest request = base.GetWebRequest(address);
if (request is HttpWebRequest)
{
(request as HttpWebRequest).CookieContainer = this.CookieContainer;
}
return request;
}
}
ensure you send the and restore aspnet_session_id on each request.
And bingo!!
I recommend you to read this.

HTTPS web request failing

When I run the program contained below the first HTTPS request succeeds, but the second request fails. Both url's are valid and both can be accessed successfully in a browser. Any suggestions as to what needs to be done to access the second url successfully?
using System;
using System.IO;
using System.Net;
public class Program
{
private static void Main(string[] args)
{
var content = "";
bool status;
var url1 = "https://mail.google.com";
var url2 = "https://my.ooma.com";
status = DoHttpRequest(url1, out content);
OutputStatus(url1, status, content);
status = DoHttpRequest(url2, out content);
OutputStatus(url2, status, content);
Console.ReadLine();
}
private static void OutputStatus(string url, bool status, string content)
{
if (status) Console.WriteLine("Url={0}, Status=Success, content length = {1}", url, content.Length);
else Console.WriteLine("Url={0}, Status=Fail, ErrorMessage={1}", url, content);
}
private static bool DoHttpRequest(string url, out string content)
{
content = "";
var request = (HttpWebRequest) WebRequest.Create(url);
try
{
request.Method = "GET";
request.CookieContainer = null;
request.Timeout = 25000; // 25 seconds
var response = (HttpWebResponse) request.GetResponse();
var streamReader = new StreamReader(response.GetResponseStream());
content = streamReader.ReadToEnd();
return true;
}
catch (WebException ex)
{
content = ex.Message;
return false;
}
}
}

Historically, most problems of this description that I've seen occur when you forget to call .Close() on the object returned from GetResponseStream(). The problem exists because when you forget to close the first request, the second request deadlocks waiting for a free connection.
Typically this hang happens on the 3rd request, not the second.
Update: Looking at your repro, this has nothing to do with the order of the requests. You're hitting a problem because this site is sending a TLS Warning at the beginning of the HTTPS handshake, and .NET will timeout when that occurs. See http://blogs.msdn.com/b/fiddler/archive/2012/03/29/https-request-hangs-.net-application-connection-on-tls-server-name-indicator-warning.aspx. The problem only repros on Windows Vista and later, because the warning is related to a TLS extension that doesn't exist in the HTTPS stack on WinXP.

Increse your request TimeOut.
request.Timeout = 60000; //60 second.
May be your network connection is a bit slow. I run with 25 seconds, okay. (Yeah, the second url is a bit longer to get response, than the first one.)

C# Check url exist?

How can I check whether a page exists at a given URL?
I have this code:
private void check(string path)
{
try
{
Uri uri = new Uri(path);
WebRequest request = WebRequest.Create(uri);
request.Timeout = 3000;
WebResponse response;
response = request.GetResponse();
}
catch(Exception loi) { MessageBox.Show(loi.Message); }
}
But that gives an error message about the proxy. :(

First, you need to understand that your question is at least twofold,
you must first check if the server is responsive, using ping for example - that's the first check, while doing this, consider timeout, for which timeout you will consider a page as not existing?
second, try retrieving the page using many methods which are available on google, again, you need to consider the timeout, if the server taking long to replay, the page might still "be there" but the server is just under tons of pressure.

If the proxy needs to authenticate you with your Windows credentials (e.g. you are in a corporate network) use:
WebRequest request=WebRequest.Create(url);
request.UseDefaultCredentials=true;
request.Proxy.Credentials=request.Credentials;

try
{
Uri uri = new Uri(path);
HttpWebRequest request = HttpWebRequest.Create(uri);
request.Timeout = 3000;
HttpWebResponse response;
response = request.GetResponse();
if (response.StatusCode.Equals(200))
{
// great - something is there
}
}
catch (Exception loi)
{
MessageBox.Show(loi.Message);
}
You can check the content-type and length, see MSDN HTTPWebResponse.

At a guess, without knowing the specific error message or path, you could try casting the WebRequest to a HttpWebRequest and then setting the WebProxy.
See MSDN: HttpWebRequest - Proxy Property