Develop Reference

C# HttpWebRequest Medium Blog returns 403 Forbidden but Site is Open

Medium blog pages are available on Chrome, IE etc... browsers but I can not send a web request with this code blog. It returns 403 Forbidden. By the way this method was working properly a couple of days ago. I changed my IP address numerious times, thought they might have banned my IP address but did not work.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://medium.com/#coinbaseblog");
request.CachePolicy = new HttpRequestCachePolicy(HttpRequestCacheLevel.NoCacheNoStore);
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
using (System.IO.Stream stream = response.GetResponseStream())
using (StreamReader reader = new StreamReader(stream))
{
return reader.ReadToEnd();
}

Try this:
var request = (HttpWebRequest)WebRequest.Create("https://medium.com/#coinbaseblog");
request.UserAgent = #"Mozilla/5.0 (compatible; Rigor/1.0.0; http://rigor.com)";
var result = (HttpWebResponse)request.GetResponse();

http status code 403 indicates that server has understood the request but refuses to authorize it. Even if you are reauthenticating it will make no difference. This is similar to http status code 401 but in this case reauthentication works just fine. You may be authenticated but the resource which you are trying to access is restricted for you.

.net MVC：How to hide the true URL?

I have an external URL, like http://a.com/?id=5 (not in my project)
and I want my website to show this URL's contents,
ex.
My website(http://MyWebsite.com/?id=123) shows 3rd party's url (http://a.com/?id=5) contents
but I don't want the client side to get a real URL(http://a.com/?id=5), I'll check the AUTH first and then shows the page.

I assume that you do not have control over the server of "http://a.com/?id=5". I think there's no way to completely hide the external link to users. They can always look at the HTML source code and http requests & trace back the original location.
One possible solution to partially hide that external site is using curl equivalent of MVC, on your controller: after auth-ed, you request the website from "http://a.com/?id=5" and then return that to your user:
ASP.NET MVC - Using cURL or similar to perform requests in application:
I assume the request to "http://a.com/?id=5" is in GET method:
public string GetResponseText(string userAgent) {
string url = "http://a.com/?id=5";
string responseText = String.Empty;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "GET";
request.UserAgent = userAgent;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
using (StreamReader sr = new StreamReader(response.GetResponseStream())) {
responseText = sr.ReadToEnd();
}
return responseText;
}
then, you just need to call this in your controller. Pass the same userAgent from client so that they can view the website exactly like they open it with their web browsers:
return GetResponseText( request.UserAgent);
//request is the request passed to the controller for http://MyWebsite.com/?id=123
PS: I may not using the correct MVC API, but the idea is there. Just need to look up MVC document on HttpWebRequest to make it work correctly.

JSON - No connection could be made because the target machine actively refused it

I’m trying to debug a problem.
I have been given a url to which I can send JSON commands via HTTP POST. If I stick the URL into Chrome I get a properly formatted response – all good.
If I put the same url into internet explorer (IE 11) I get a ‘The website is unable to display the webpage -This error (HTTP 501 Not Implemented or HTTP 505 Version Not Supported) means that the website you are visiting doesn’t currently have the ability to display the webpage, or support the HTTP version used to request the page.’ – Not sure whats going here
I ultimately need to send a command via a web service, to this end I have created a very simple app in visual studio.
I use this code, but then I get the error:
var httpWebRequest = (HttpWebRequest)WebRequest.Create("https://URL.....");
httpWebRequest.ContentType = "application/json";
httpWebRequest.Method = "POST";
using (var streamWriter = new StreamWriter(httpWebRequest.GetRequestStream()))
{
string json = new JavaScriptSerializer().Serialize(new
{
operation = "****",
username = "****",
password = "****"
});
streamWriter.Write(json);
}
var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var result = streamReader.ReadToEnd();
}
{"No connection could be made because the target machine actively refused it 1.1.1.1:443"}
There is a problem, but being new to this I have no idea where to start looking.
Any pointers appreciated.

Retrieve DOM data from site

Is there any chance to retrieve DOM results when I click older posts from the site:
http://www.facebook.com/FamilyGuy
using C# or Java? I heard that it is possible to execute a script with onclick and get results. How I can execute this script:
onclick="(JSCC.get('j4eb9ad57ab8a19f468880561') && JSCC.get('j4eb9ad57ab8a19f468880561').getHandler())(); return false;"

I think older posts link sends an Ajax request and appends the response to the page. (I'm not sure. You should check the page source).
You can emulate this behavior in C#, Java, and JavaScript (you already have the code for javascript).
Edit:
It seems that Facebook uses some sort of internal APIs (JSCC) to load the content and it's undocumented.
I don't know about Facebook Developers' APIs (you may want to check that first) but if you want to emulate exactly what happens in your browser then you can use TamperData to intercept GET requests when you click on more posts link and find the request URL and it's parameters.
After you get this information you have to Login to your account in your application and get the authentication cookie.
C# sample code as you requested:
private CookieContainer GetCookieContainer(string loginURL, string userName, string password)
{
var webRequest = WebRequest.Create(loginURL) as HttpWebRequest;
var responseReader = new StreamReader(webRequest.GetResponse().GetResponseStream());
string responseData = responseReader.ReadToEnd();
responseReader.Close();
// Now you may need to extract some values from the login form and build the POST data with your username and password.
// I don't know what exactly you need to POST but again a TamperData observation will help you to find out.
string postData =String.Format("UserName={0}&Password={1}", userName, password); // I emphasize that this is just an example.
// cookie container
var cookies = new CookieContainer();
// post the login form
webRequest = WebRequest.Create(loginURL) as HttpWebRequest;
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.CookieContainer = cookies;
// write the form values into the request message
var requestWriter = new StreamWriter(webRequest.GetRequestStream());
requestWriter.Write(postData);
requestWriter.Close();
webRequest.GetResponse().Close();
return cookies;
}
Then you can perform GET requests with the cookie you have, on the URL you've got from analyzing that JSCC.get().getHandler() requests using TamperData, and eventually you'll get what you want as a response stream:
var webRequest = WebRequest.Create(url) as HttpWebRequest;
webRequest.CookieContainer = GetCookieContainer(url, userName, password);
var responseStream = webRequest.GetResponse().GetResponseStream();
You can also use Selenium for browser automation. It also has C# and Java APIs (I have no experience using Selenium).

Facebook loads it's content dynamically with AJAX. You can use a tool like Firebug to examine what kind of request is made, and then replicate it.
Or you can use a browser render engine like webkit to process the JavaScript for you and expose the resulting HTML:
http://webscraping.com/blog/Scraping-JavaScript-webpages-with-webkit/

How to read HTTP header from response using .NET HttpWebRequest API?

My app currently uses OAuth to communicate with the Twitter API. Back in December, Twitter upped the rate limit for OAuth to 350 requests per hour. However, I am not seeing this. I am still getting 150 from the account/rate_limit_status method.
I was told that I needed to use the X-RateLimit-Limit HTTP header to get the new rate limit. However, in my code, I do not see that header.
Here is my code...
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(newURL);
request.Method = "GET";
request.ServicePoint.Expect100Continue = false;
request.ContentType = "application/x-www-form-urlencoded";
using (WebResponse response = request.GetResponse())
{
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
responseString = reader.ReadToEnd();
}
}
If I inspect the response, I can see that it has a property for Headers, and that there are 16 headers. However, I do not have X-RateLimit-Limit in the list.
(source: yfrog.com)
Any idea what I am doing wrong?

You should simple be able to use:
using (WebResponse response = request.GetResponse())
{
string limit = response.Headers["X-RateLimit-Limit"];
...
}
If that doesn't work as expected, you can do a watch on response.Headers and see what's in there.

Look at the raw response text (e.g., with Fiddler). If the header isn't there, no amount of C# code is going to make it appear. :) From what you've shown, it seems the header isn't in the response.
Update: When I go to: http://twitter.com/account/rate_limit_status.xml there is no X-RateLimit-Limit header. But when I go to http://twitter.com/statuses/public_timeline.xml, it's there. So I think you just need to use a different method.
It still says 150, though!