I'm building an AJAX proxy page to pull the information from a SSL Single Sign on Site. Unfortunately I feel like i'm getting stuck. I've tried using webclient and HttpWebRequest but i can't seem to get past the initial page.
I'm catch the a link click action and passing the URL of the link to my proxy page.
$("#Frame").load("/ProxyAjax?url=" + serializedData);
and executing the following code
string redirectUrl = Server.UrlDecode(Request.QueryString["url"]);
Uri myURI = new Uri(redirectUrl);
WebClient wc = new WebClient();
wc.Headers.Add("Cookie", Request.Headers["Cookie"]);
vd.String = wc.DownloadString(redirectUrl);
Have the page I'm calling is executing a redirect and I get stuck somewhere between. Any ideas of how to resolve this?
Related
When I upload an image, the sharepoint will give a link corresponding to that image.
I am working on a C # project that analyzes images, and want to use SHarepoint's image link.
But when executing the function to load the image from WebClient's url (), I was blocked by an error.
Error name: "the remote server returned an error (401) unauthorized"
this is image show link in my Sharepoint (the link that I am scanning): https://ibb.co/g9jYHKC
And this is code webclient() I used:
var webClient = new WebClient();
byte[] imageBytes = webClient.DownloadData("http://pmssd78/Animal/birddd.jpg"); //link copy from sharepoint like image show
Looking forward to hearing from everybody soon, thanks
HTTP 401 means unauthorized, which means you aren't authenticated with the resource that you are making the request to. You need to authenticate with the server for it to accept your request.
With the WebClient class, you can do this via the Credentials class member:
//Create a Credential Cache - you'll want this to be defined somewhere appropriate,
//maybe at a global level so other parts of your application can access it
CredentialCache credentialCache = new CredentialCache();
credentialCache.Add(new Uri("http://yourSharepointUrl.com"), "Basic", new NetworkCredential("yourUserName", "yourSecuredPassword"));
//Create WebClient, set credentials, perform download
WebClient webClient = new WebClient();
webClient.Credentials = credentialCache.GetCredential(new Uri("http://yourSharepoint.com"),
"Basic");
byte[] imageBytes = webClient.DownloadData("http://pmssd78/Animal/birddd.jpg");
Likely, the Uri used in the credential cache is going to be something along the lines of your URI being used in the .DownloadData signature.
I want to scrape the HTML of a website. When I access this website with my browser (no matter if it is Chrome or FireFox), I have no problem accessing the website + HTML.
When I try to parse the HTML with C# using methods like HttpWebRequest and HtmlAgilityPack, the website redirects me to another website and thus I parse the HTML of the redirected website.
Any idea how to solve this problem?
I thought the site recognises my program as a program and redirects immediately, so I tried using Selenium and a GoogleDriver and FireFoxDriver but also no luck, I get redirected immediately.
The Website: https://www.jodel.city/7700#!home
private void bt_load_Click(object sender, EventArgs e)
{
var url = #"https://www.jodel.city/7700#!home";
var req = (HttpWebRequest)WebRequest.Create(url);
req.AllowAutoRedirect = false;
// req.Referer = "http://www.muenchen.de/";
var resp = req.GetResponse();
StreamReader sr = new StreamReader(resp.GetResponseStream());
String returnedContent = sr.ReadToEnd();
Console.WriteLine(returnedContent);
return;
}
And of course, cookies are to blame again, because cookies are great and amazing.
So, let's look at what happens in Chrome the first time you visit the site:
(I went to https://www.jodel.city/7700#!home):
Yes, I got a 302 redirect, but I also got told by the server to set a __cfduid cookie (twice actually).
When you visit the site again, you are correctly let into the site:
Notice how this time a __cfduid cookie was sent along? That's the key here.
Your C# code needs to:
Go to the site once, get redirected, but obtain the cookie value from the response header.
Go BACK to the site with the correct cookie value in the request header.
You can go to the first link in this post to see an example of how to set cookie values for requests.
I did an application which parses an html document and then obtains some urls, the problem is the urls only can be downloaded directly from the navigator.
In VB.NET or C#, how I could redirect this url to obtain a direct link for later paste the link to download it in a Download Manager?
dim url as string = "http://m.mrtzcmp3.net/get.php?singer=Madonna&song=Like%20A%20Virgin%20&size=5242104&ids=687474703a2h2h63733434303876342g766s2g6f652h75323237363831362h617564696h732h3132323564303466333839622g6f7033"
I need to say that I'm not much experimented with http things, maybe I'm wrong and the url has anything to redirect or something similar fault, please just say me how can I redirect that kind of urls or If I'm wrong.
UPDATE:
Tried this, but I get the same url without any changes:
Dim url As String = _
"http://m.mrtzcmp3.net/get.php?singer=Madonna&song=Like%20A%20Virgin%20&size=5242104&ids=687474703a2h2h63733434303876342g766s2g6f652h75323237363831362h617564696h732h3132323564303466333839622g6f7033"
Dim request As HttpWebRequest = DirectCast(HttpWebRequest.Create(url), HttpWebRequest)
request.AllowAutoRedirect = True
Dim response As HttpWebResponse
Dim resUri As String
response = request.GetResponse
resUri = response.ResponseUri.AbsoluteUri
MsgBox(resUri)
UPDATE 2:
In the answer from here HttpWebRequest Login data Then Redirect
He says
If the redirect is handled transparently, the _response.ResponseURI
will contain the address it redirected to. If not, you have to read
the redirect header and decide yourself whether or not to request the
new page.
so...if I need to do thatm, how I can do that?
UPDATE 3:
DownloadThemAll plugin for Firefox can obtain the direct urls... as you can see all the urls finishes with an .mp3 file extension, that's what I need
To my knowledge, the url
http://m.mrtzcmp3.net/get.php?singer=Madonna&song=Like%20A%20Virgin%20&size=5242104&ids=687474703a2h2h63733434303876342g766s2g6f652h75323237363831362h617564696h732h3132323564303466333839622g6f7033
IS the direct url, a direct file url does not need to have the filetype in it.
you can download the file using
string url = "http://m.mrtzcmp3.net/get.php?singer=Madonna&song=Like%20A%20Virgin%20&size=5242104&ids=687474703a2h2h63733434303876342g766s2g6f652h75323237363831362h617564696h732h3132323564303466333839622g6f7033"
WebClient wc = new WebClient();
wc.DownloadFile(url, fileName);
you can get the fileName (Madonna-Like A Virgin -www.mrtzcmp3.net.mp3) by using
HttpWebRequest myHttpWebRequest = (HttpWebRequest)HttpWebRequest.Create(url);
string header = myHttpWebResponse.Headers.ToString();
fileName = header.Remove(0, header.IndexOf("filename=")+10);
fileName = fileName.Remove(fileName.IndexOf('"'));
that is untested, but it should work.
edit: I think this does what you want, but I may have misunderstood your question
you can perform a web request using web client to get the content (url) from that url, then you just need to perform the redirect.
Use an HttpWebRequest and use the AllowAutoRedirect=true to get the direct link and download the file.
Can you try to paste the URL to an URl shortener like tinyUrl or BitLy? Maybe there is a shortener Service that provides an API?
The file then will be downloaded at: http://tinyurl.com/phzhxsr
You will never get a direct URL from the site owner because the URL is dynamicaly parsed and the file is send with the retrun datastream, not by downloading a specific URL.
This is me publicly documenting my mistake so that if I or anyone does it again, they don't have to spend 3 hours tearing their hair out trying to fix such a simple thing.
Context
I was sending an HttpRequest from one C# MVC ASP.NET application to another.
The applications require an HTTPS connection, and we are using URLRewrite to redirect an HTTP request to an HTTPS url.
One application was sending a POST request with some JSON data in the body, pretty standard stuff. The other application was set up to receive this data with an MVC controller class (CollectionAction and Insert methods for GET and POST respectively).
Symptoms of the problem
The receiving application was running the GET method (CollectionAction) instead of the POST action (ItemAction). The reason for this was that the request coming in to the application was in fact a GET request, and to top it off the JSON data was missing too.
I sent the header "x-http-method" to override the request method from GET to POST (I was already setting the request httpmethod to POST but this was being ignored). This worked but still I had no data being sent.
So now I am stuck pulling my hair out, because I can see a POST request with content-length and data being sent out and I have a GET request with no data or content-length coming in (but the headers were preserved)
Turns out I was using UriBuilder to take a base URL and apply a resource path to it. For example I would have "google.com" in my web.config and then the UriBuilder would take a resource like Pages and construct the url "google.com/Pages". Unfortunately, I was not initializing the UriBuilder with the base URL, and instead was using a second UriBuilder to extract the host and add that to the path like so:
public Uri GetResourceUri(string resourceName)
{
var domain = new UriBuilder(GetBaseUrl());
var uribuilder = new UriBuilder()
{
Path = domain.Path.TrimEnd('/') + "/" + resourceName.TrimStart('/'),
Host = domain.Host
};
var resourceUri = uribuilder.Uri;
return resourceUri;
}
The problem with this code is that the scheme is ignored (HTTP:// vs HTTPS://) and it defaults to HTTP. So my client was sending out the request to an HTTP url instead of the required HTTPS url. This is the interesting part, URLRewrite was kicking in and saying that we needed to go to an HTTPS url instead so it redirected us there. But in doing so, it ignored the Http-Method and the POST data, which just got set to defaults GET and null. This is what the 2nd application could see at the receiving end.
So the function had to be rewritten to this which fixed the problem:
public Uri GetResourceUri(string resourceName)
{
var baseUrl = GetBaseUrl();
var domain = new UriBuilder(baseUrl);
var uribuilder = new UriBuilder(baseUrl)
{
Path = domain.Path.TrimEnd('/') + "/" + resourceName.TrimStart('/'),
};
var resourceUri = uribuilder.Uri;
return resourceUri;
}
I am trying to unshorten urls and have not been able to find code (vb.net/c#) to do this. These are the twitter shortened urls and I guess I could try and access one of the web services available and do a httpwebrequest but would prefer to find some programmatic way of doing this.
You can get it directly from response of the shortened url since it will return a status code MovedPermanently and the location for the real url.(This should work for most of the sites without the need for navigating to the real url)
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://t.co/xqbLEi6s");
req.AllowAutoRedirect = false;
var resp = req.GetResponse();
string realUrl = resp.Headers["Location"];
Other test data: http://goo.gl/zdf2n , http://tinyurl.com/8xc9vca , http://x.co/iEup, http://is.gd/vTOlz6 , http://bit.ly/FUA4YU
There is no magic way to unshorten a URL without asking the service which created the URL (and the way to ask will be different for each service), or more pragmatically, just opening the URL and watching where it redirects to.