I have an example URL (http://www.techspot.com/downloads.php?action=download_now&id=2991&evp=113a02f49ca8ac11b566336b984b1655&file=1). And when I click the link, the url will change to:
http://www.exisoftware.com/downloads/picture_finder/PictureFinderSetup.exe
Can anyone help me how to convert the php link to real url using vb.net or c#?
Because when I make a program to check file information in vb.net using HEAD method the file name not "PictureFinderSetup.exe" but "downloads.php?action=download_now&id=2991&evp=113a02f49ca8ac11b566336b984b1655&file=1".
It cannot be done without contacting the server. Only the server knows the exact mapping.
From Fiddler:
Request
GET http://www.techspot.com/downloads.php?action=download_now&id=2991&evp=113a02f49ca8ac11b566336b984b1655&file=1 HTTP/1.1
...
Response
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Fri, 27 Sep 2013 17:26:14 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Location: http://www.techspot.com/downloads/2991-extreme-picture-finder.html
To get the information you can use WebBrowser or HttpWebResponse to get data you need. How can I use VB.Net to read the content returned from a URL?
Related
We're building an ASP.NET web application which integrates with a DAM (digital asset management) system. The DAM stores files in Azure blob storage and makes them available publicly using Azure CDN.
We will be making these files (most are PDFs) available to download from our web app. When a user requests one of these files we will provide a custom URL which will run some code on the server (logging the download etc) before returning the relevant file for download.
The client requires that the file is always returned as a browser attachment (i.e. content disposition attachment header). I am curious about what options I have here.
My ideal would be that the CDN URL is abstracted and my custom URL is the public URL for the file. That would allow me to set relevant response headers etc. However, I assume the only solution here would be to download the file from CDN and cache it on my web server which would obfuscate the CDN's purpose. So presumably I have to redirect the client to the CDN public URL once I've done my server processing. But then is there a way I can ensure the file is returned by Azure with the correct response headers to ensure the browser's default download behaviour is delegated?
* Update *
In seeing the answers to this question I realised I was perhaps asking the wrong question. Thank you to those of you who answered here. Follow-up question is here.
TL;DR
You need to configure the default version on the blob storage in order for it to show the required header to non-authenticated clients. the question in this question has the code to make it work.
Once this is set, and working for anonymous clients the CDN will copy all of the headers across and it should work as expected.
Setting ContentDisposition
The functionality is present, you can set ContentDisposition on a blob property However, while this will set the property on the blob, it does not pass through to the header.
I tested this with Powershell using the following (just because its quicker than c#)
$context = New-AzureStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey
$container = Get-AzureStorageContainer -Name $ContainerName -Context $context
$blobref = ($script:container.CloudBlobContainer.GetBlobReferenceFromServer("images/pier.jpg"))
$blobref.Properties
$blobref.Properties.ContentDisposition = 'attachment; filename="fname.ext"'
$blobref.SetProperties()
$blobref = ($script:container.CloudBlobContainer.GetBlobReferenceFromServer("images/pier.jpg"))
$blobref.Properties
Which produces (amongst others)
ContentDisposition : attachment; filename="fname.ext"
However nothing is set when the headers are queried
([system.Net.HttpWebRequest]::Create($blobref.Uri.AbsoluteUri)).getresponse()
(to answer comment,. these are the headers returned - while experimenting I also tried with and without a contenttype - hence it being blank here)
IsMutuallyAuthenticated : False
Cookies : {}
Headers : {x-ms-request-id, x-ms-version, x-ms-lease-status, x-ms-blob-type...}
SupportsHeaders : True
ContentLength : 142224
ContentEncoding :
ContentType :
CharacterSet :
Server : Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
LastModified : 01/03/2016 11:29:04
StatusCode : OK
StatusDescription : OK
ProtocolVersion : 1.1
ResponseUri : https://xxxx.blob.core.windows.net/cdn/images/pier.jpg
Method : GET
IsFromCache : False
And since CDN will only copy the information from the HTTP headers themselves, this data isn't making it into CDN.
edited (after extended comment chatter!)
For reasons best known to itself Powershell wasn't sending the x-ms-version, so I fell back to telnet which did indeed produce the header -
HEAD /cdn/images/pier.jpg HTTP/1.1
HOST: xxxx.blob.core.windows.net
x-ms-version: 2015-04-05
HTTP/1.1 200 OK
Content-Length: 142224
Last-Modified: Tue, 01 Mar 2016 11:29:04 GMT
Accept-Ranges: bytes
ETag: "0x8D341C4B1C4F34F"
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: b4f41b01-0001-00d7-7cc9-7384c9000000
x-ms-version: 2015-04-05
x-ms-lease-status: unlocked
x-ms-lease-state: available
x-ms-blob-type: BlockBlob
Content-Disposition: attachment; filename="fname.ext"
Date: Tue, 01 Mar 2016 14:49:17 GMT
I am trying to handle a website programmatically. Lets say I visit the page www.example.com/something. On the website there is a button which I am pressing. The code of the button looks something like this:
<form action="/something" method="POST" enctype="text/plain">
<input type="submit" class="button" value="Click me" >
</form>
Pressing this button updates the information on the website.
Now I would like to do this procedure programatically to receive the content of the updated website after pressing the button.
Can someone lead me to the right direction on how to do this? preferably in C#.
Thank you in advance!
Edit:
I used Fiddler to capture the HTTP request and response, it looks like this:
POST /something HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Referer: http://example.com/something
Cookie: cookie1=cookiecontent; cookie2=cookiecontent
Connection: keep-alive
Content-Type: text/plain
Content-Length: 0
HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Encoding: gzip
Vary: Accept-Encoding
Server: Microsoft-IIS/8.0
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Thu, 05 Dec 2013 23:36:31 GMT
Content-Length: 2202
Although the requests includes cookies they don't appear to be relevant. I decompressed the received content with fiddler and found the wanted data to be included in the response.
I am not very experienced in HTTP requests and am therefore hoping that someone can help me convertion this into a C# http request to receive the content.
If the website in question is open and doesn't do any sort of cookie generation to validate requests (there are plenty of sites like this) then you can just use System.Net.WebRequest or similar to post the required form data, then examine the response. See this MSDN page for an example.
If the page does use cookies and so on you'll have to get a bit more creative. In some cases you can issue one web request to get the first page, examine the results for cookies and hidden form values and use those in your POST.
If all else fails then the Selenium WebDriver library will give you almost complete browser emulation with full access to the DOM. It's a bit more complex than using a WebRequest, but will work for pretty much everything you can use a web browser for.
Regardless of which method you use, Fiddler is a good debugging tool. Use it to compare what your C# code is doing to what the web browser is doing to see if there's anything your code isn't getting right.
Since it's a submit button then simulating the resulting HTTP Request would be easier than simulating a click. First, I would use a program like Fiddler to inspect what is being sent when you submit the form. Then I would replicate that request, just changing the values that I need changing, using HTTPWebRequest. You can find an example here.
The resultant HTTPWebResponse can then be parsed for data. Using something like HtmlAgilityPack makes that part easier.
You can do what you want with http://www.seleniumhq.org/projects/webdriver/. It is possible to do web automation with c# in a console program. I am using it for ui integration testing and it works fairly well
I would look into searching for a browser automation framework. I would usually do this in Python and have not used .Net for this, but a quick Google search yields quite a few results.
Included within these:
http://watin.org/
Web automation using .NET
Can we script and automate a browser, preferably with .Net?
I have a very simple app that sends an HttpWebRequest and gets a response. I need to know the exact request sent to the server. Is it possible?
Something like this:
POST /path/script.cgi HTTP/1.0
From: frog#jmarshall.com
User-Agent: HTTPTool/1.0
Content-Type: application/x-www-form-urlencoded
Content-Length: 32
Build a basic web server with the System.Net.Sockets.TcpListener. The example shows how to do this. Then, point your HttpWebRequest to that server and see the results.
I'm responsible for updating a client-side API using WCF. This is because of changes to the API on the server (an outside company). When I get the response, my client-side throws an exception. Using Fiddler, I came up with the following problem: a duplicate Content-Type.
HTTP/1.1 200 OK
Date: Thu, 05 Jan 2012 21:15:16 GMT
Connection: close
Content-Type: text/xml; charset=utf-8
Content-Type: text/xml; charset=UTF-8
Content-Length: 538
...
Using Fiddler, I removed the extra Content-Type, and the client continued happily. So, I wrote an IClientMessageInspector, with the intention of capturing the response and removing the duplicate. And therein lies my issue. My IClientMessageInspector gets a call to BeforeSendRequest, but not one to AfterReceiveRequest.
Is there some place other than AfterReceiveRequest that I should be handling my task of removing the extra Content-Type? Am I barking up the wrong tree altogether?
I'm trying to hunt down why a POST request from a C# script isn't working, when the same request works fine in Python. I want to be able to have all the data sent by the script, and the response from the sever, to be displayed on the screen so that I can work out what the difference is between what the C# and Python scripts are sending.
In Python I can do this with the standard httplib2 library by just using:
httplib2.debuglevel = 1
This produces the following output (as an example):
reply: 'HTTP/1.1 201 Created\r\n'
header: Date: Tue, 05 Oct 2010 09:25:42 GMT
header: Server: Apache/2.2.9 (Debian) PHP/5.2.6-1+lenny9 with Suhosin-Patch
header: X-Powered-By: PHP/5.2.6-1+lenny9
header: Location: http://example.org/api/2
header: Content-Length: 0
header: Content-Type: text/html
send: 'GET /api/2 HTTP/1.1\r\nHost: example.org\r\naccept-encoding: gzip, deflate\r\nuser-agent: Python-httplib2/$Rev$\r\n\r\n'
Is there a way to produce similar output in C# using the HttpWebRequest class?
I've seen mention of Fiddler in another question, but I'm running Linux and Fiddler appears to be for Windows only.
You can spin through resp.Headers.AllKeys and then dump the key and its value, though there is sometimes a degree of translation going on (most obviously when it is doing auto-redirect-following).
You can use System.Net tracing.
You can use ethereal which has a linux version. I don't use it for such things these days as Fiddler is indeed handier for such cases, but I used to use ethereal in the past, and sometimes seeing what is actually on the wire rather than what the code is saying is the best way to go (as you aren't depending on possibly buggy code to tell you if the code is buggy).
Yes you can and I have used it successfully before. If you are using C# I imagine there is a config file that you acn turn System.Net tracing on. I am not sure if Mono has this - if you are using Mono.
Have a look here:
http://msdn.microsoft.com/en-us/library/ty48b824.aspx