Hello I am having problems with httpwebrequest, huge memory buildups when running a rather large request based program. The only real place the build up can be happening is reading the stream back after sending the request, how would I combat this or better yet set a limit to this sort of thing?
I also have various (10-15) if statements looking for various things in the body of the response but about to change them to a switch statement, although I doubt they're causing such high memory buildups I thought it'd be best to mention them. My request looks like so:
HttpWebResponse response = null;
try
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Headers.Set(HttpRequestHeader.CacheControl, "max-age=0");
request.Headers.Add("Upgrade-Insecure-Requests", #"1");
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36 OPR/63.0.3368.88";
response = (HttpWebResponse)request.GetResponse();
string _responseData = new StreamReader(response.GetResponseStream()).ReadToEnd();
catch (WebException e)
{
}
catch (UriFormatException p)
{
}
As you can see my stream is read back via _responseData which I then use .contains to see if what I'm looking for is there.
Set HttpWebRequest.AllowWriteStreamBuffering to false.
Dispose HttpWebResponse, Stream and StreamReader.
try
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.AllowWriteStreamBuffering = false;
request.Headers.Set(HttpRequestHeader.CacheControl, "max-age=0");
request.Headers.Add("Upgrade-Insecure-Requests", #"1");
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36 OPR/63.0.3368.88";
string _responseData;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream stream = response.GetResponseStream())
{
using (StreamReader sr = new StreamReader(stream, Encoding.UTF8))
{
_responseData = sr.ReadToEnd();
}
}
}
}
catch (WebException e)
{
}
catch (UriFormatException p)
{
}
Related
So I am trying to connect to http://cmyip.org because the content contains information about the proxy you are connecting with. Which is perfect because I am learning about making web-requests with proxies so it will show me if I actually connected with the proxy or not.
However instead of getting the content of the website I am making a request to I am getting the content of the proxy html
Before start thread
<html><head><title>Wowza Streaming Engine 4 Subscription Edition 4.7.2.01 build2
1094</title></head><body>Wowza Streaming Engine 4 Subscription Edition 4.7.2.01
build21094</body></html>
Working.
Why is it showing the contents of the proxy and not cmyip.org? I'm sure I am just too close to the code to see it but could really use some help.
If you try to connect to 192.99.46.182:1935 on your web browser you will see the same thing.
public static bool TestProxy(string proxyIP, int proxyPort)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.cmyip.org/");
WebProxy myproxy = new WebProxy(proxyIP, proxyPort);
myproxy.BypassProxyOnLocal = false;
request.Proxy = myproxy;
request.Method = "GET";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36";
request.Timeout = 2000;
try
{
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
using (StreamReader sr = new StreamReader(response.GetResponseStream()))
{
string Content = sr.ReadToEnd();
console.WriteLine(Content);
}
}
catch (Exception)
{
return false;
}
return true;
}
private static void SimpleTest()
{
var aBool = TestProxy("192.99.46.182", 1935);
if (aBool == true)
{
Console.WriteLine("Working.");
}
Console.ReadLine();
}
So without fiddler open, my code doesnt seem to do ANYTHING!
But with it open, it works fine?! What?
HttpWebRequest init = (HttpWebRequest)WebRequest.Create("https://accounts.spotify.com/en/login?continue=https:%2F%2Fwww.spotify.com%2Fus%2Faccount%2Foverview%2F");
init.Method = "GET";
init.UserAgent = "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36";
init.Timeout = 8000;
HttpWebResponse resp1 = (HttpWebResponse)init.GetResponse();
The thread (In my case ThreadPool thread), will just freeze, not end, will just freeze as its not able to connect, and the try catch doesnt even seem to work either.
If I change the Port on Fiddlers Settings to anything other than 8888 and have Fiddler open, it starts to freeze again, meaning somewhere its linked to 8888 somehow. Yet its not referenced in my code at all.
FIXED!
Found out it was caused by not closing! resp1.close().
Thanks everyone!
It's done right that, what you wrote in your code. Try debug this code and see what you get as a response, pay attention to responseBody and HtmlResult
HttpWebRequest init = (HttpWebRequest)WebRequest.Create("https://accounts.spotify.com/en/login?continue=https:%2F%2Fwww.spotify.com%2Fus%2Faccount%2Foverview%2F");
init.Method = "GET";
init.UserAgent = "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36";
init.Timeout = 8000;
HttpWebResponse resp1 = (HttpWebResponse)init.GetResponse();
HttpStatusCode sc = resp1.StatusCode;
String sd = resp1.StatusDescription;
string HtmlResult = "";
string responseBody = "";
using (System.IO.Stream rspStm = resp1.GetResponseStream())
{
using (System.IO.StreamReader reader = new System.IO.StreamReader(rspStm))
{
HtmlResult = HtmlResult +
"Response Description: " + resp1.StatusDescription;
HtmlResult = HtmlResult +
"Response Status Code: " + resp1.StatusCode;
HtmlResult = HtmlResult + "\r\n\r\n";
responseBody = reader.ReadToEnd();
}
}
I try to allow users to select images from the users google drive.
I follow this doc: https://developers.google.com/picker/docs/.
When i recive the image url from google ('/drive/v2/files/' + FileId) then i try to download tile image in my .NET code:
var WebRequest = (HttpWebRequest)System.Net.WebRequest.Create(url);
WebRequest.AllowWriteStreamBuffering = true;
WebRequest.Timeout = 30000;
WebRequest.KeepAlive = false;
WebRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36";
WebRequest.AllowAutoRedirect = true;
WebRequest.Headers.Add("Authorization", "Bearer " + token);
byte[] Arr;
using (var WebResponse = WebRequest.GetResponse())
{
using (var Stream = WebResponse.GetResponseStream())
{
Arr = Stream.StreamToByteArray();
}
WebResponse.Close();
}
return Arr;
But all i get is a empty (white) webpage. With the url i get from google, i can se the image in my browser, but i cant download it in my code.
How can i get the image downloaded?
Use instead WebClient:
client = new WebClient();
client.Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36");
client.Headers.Add("Authorization", "Bearer " + token);
client.AllowWriteStreamBuffering = true;
return client.DownloadData(url); /// DownloadData returns a byte array
My current job is to consume a RESTful API with OAuth2. Currently I worked out how to get the access token and it is working ok while I use the chrome extension Rest Console, but when I try to do it from my application I always get the error that I am sending an invalid OAuth request. Below you can see three of the ways I tried to consume the API, but to no success. The page always returns error 500. Any help will be appreciated, if I had missed something crucial.
var auth = "Bearer " + item.access_token;
/* First Attempt */
var client = new RestClient("http://<link>");
var request = new RestRequest("sample", Method.GET);
request.AddHeader("Authorization", auth);
request.AddHeader("Content-Type", "application/json;charset=UTF-8");
request.AddHeader("Pragma", "no-cache");
request.AddHeader("User-Agent", "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36");
request.AddHeader("Accept", "application/json");
request.RequestFormat = DataFormat.Json;
var response = client.Execute(request);
var content = response.Content;
/* Second Attempt */
string sURL = "http://<link>/sample";
string result = "";
using (WebClient client = new WebClient())
{
client.Headers["Authorization"] = auth;
client.Headers["Content-Type"] = "application/json;charset=UTF-8";
client.Headers["Pragma"] = "no-cache";
client.Headers["User-Agent"] = "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36";
client.Headers["Accept"] = "application/json";
byte[] byteArray = Encoding.UTF8.GetBytes(parameters);
var result1 = client.DownloadString(sURL);
}
/* Third Attempt */
var request = (HttpWebRequest)WebRequest.Create(sURL);
request.Method = "GET";
request.ContentType = "application/json;charset=UTF-8";
request.Accept = "application/json";
request.Headers["Authorization"] = auth;
request.UserAgent = "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36";
string content;
HttpStatusCode statusCode;
using (var response = request.GetResponse())
using (var stream = response.GetResponseStream())
{
var contentType = response.ContentType;
Encoding encoding = null;
if (contentType != null)
{
var match = Regex.Match(contentType, #"(?<=charset\=).*");
if (match.Success)
encoding = Encoding.GetEncoding(match.ToString());
}
encoding = encoding ?? Encoding.UTF8;
statusCode = ((HttpWebResponse)response).StatusCode;
using (var reader = new StreamReader(stream, encoding))
content = reader.ReadToEnd();
}
--------EDIT--------
For the first attempt I also tried to add the authentication to the client variable client.Authenticator = Authenticate; where OAuth2AuthorizationRequestHeaderAuthenticator Authenticate = new OAuth2AuthorizationRequestHeaderAuthenticator(item.access_token, item.token_type);
The code seems right. The fail attempts you did suggest that the issue is with the token and not the code. Bearer tokens have expiration time. So semms like your token expired between the first time you got it using chrome REST Console extension and when you wrote your code. But the strange situation here is the 500 error code you got. 401 is standard response when you token expired or not exist. 500 error code always mean server error.
I'm a little bit stuck here. I am busy making an Windows application that reads data from a website. However de website requires a login first and i don't seem to be able to get passes that. I'm fairly new to programming, so i hope someone know a solution.
This is the code i use to login:
private void btnLogin2_Click(object sender, EventArgs e)
{
HttpWebRequest request = WebRequest.Create(LoginPageURL) as HttpWebRequest;
request.KeepAlive = true;
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.2 Safari/537.36";
string postData = "j_username=" + number + "&j_password=" + password;
byte[] dataBytes = UTF8Encoding.UTF8.GetBytes(postData);
request.ContentLength = dataBytes.Length;
using (Stream postStream = request.GetRequestStream())
{
postStream.Write(dataBytes, 0, dataBytes.Length);
}
HttpWebResponse httpResponse = request.GetResponse() as HttpWebResponse;
request = WebRequest.Create(Page2URL) as HttpWebRequest;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.2 Safari/537.36";
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(httpResponse.Cookies);
request.Method = "GET";
HttpWebResponse httpResponse2 = request.GetResponse() as HttpWebResponse;
StreamReader stream = new StreamReader(httpResponse2.GetResponseStream(), System.Text.Encoding.UTF8);
string result = stream.ReadToEnd();
stream.Close();
tbOutput2.Text = result;
}
The point is that i get the HTML of the page after the login (Page2URL). But i keep getting the HTML from the Login Page.
You are adding cookies to the request from first response:
request.CookieContainer.Add(httpResponse.Cookies);
Probably the cookies in response are null! To cope up with this issue, read cookie values from response header and add them to the next request like this:
string response_header_cookies = response.Headers.Get("Set-Cookie")
req.Headers.Add("Cookie",response_header_cookies);
In most of the cases this is the more efficient way. Hope this helps!
Source: msdn