I am trying to handle a website programmatically. Lets say I visit the page www.example.com/something. On the website there is a button which I am pressing. The code of the button looks something like this:
<form action="/something" method="POST" enctype="text/plain">
<input type="submit" class="button" value="Click me" >
</form>
Pressing this button updates the information on the website.
Now I would like to do this procedure programatically to receive the content of the updated website after pressing the button.
Can someone lead me to the right direction on how to do this? preferably in C#.
Thank you in advance!
Edit:
I used Fiddler to capture the HTTP request and response, it looks like this:
POST /something HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Referer: http://example.com/something
Cookie: cookie1=cookiecontent; cookie2=cookiecontent
Connection: keep-alive
Content-Type: text/plain
Content-Length: 0
HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Encoding: gzip
Vary: Accept-Encoding
Server: Microsoft-IIS/8.0
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Thu, 05 Dec 2013 23:36:31 GMT
Content-Length: 2202
Although the requests includes cookies they don't appear to be relevant. I decompressed the received content with fiddler and found the wanted data to be included in the response.
I am not very experienced in HTTP requests and am therefore hoping that someone can help me convertion this into a C# http request to receive the content.
If the website in question is open and doesn't do any sort of cookie generation to validate requests (there are plenty of sites like this) then you can just use System.Net.WebRequest or similar to post the required form data, then examine the response. See this MSDN page for an example.
If the page does use cookies and so on you'll have to get a bit more creative. In some cases you can issue one web request to get the first page, examine the results for cookies and hidden form values and use those in your POST.
If all else fails then the Selenium WebDriver library will give you almost complete browser emulation with full access to the DOM. It's a bit more complex than using a WebRequest, but will work for pretty much everything you can use a web browser for.
Regardless of which method you use, Fiddler is a good debugging tool. Use it to compare what your C# code is doing to what the web browser is doing to see if there's anything your code isn't getting right.
Since it's a submit button then simulating the resulting HTTP Request would be easier than simulating a click. First, I would use a program like Fiddler to inspect what is being sent when you submit the form. Then I would replicate that request, just changing the values that I need changing, using HTTPWebRequest. You can find an example here.
The resultant HTTPWebResponse can then be parsed for data. Using something like HtmlAgilityPack makes that part easier.
You can do what you want with http://www.seleniumhq.org/projects/webdriver/. It is possible to do web automation with c# in a console program. I am using it for ui integration testing and it works fairly well
I would look into searching for a browser automation framework. I would usually do this in Python and have not used .Net for this, but a quick Google search yields quite a few results.
Included within these:
http://watin.org/
Web automation using .NET
Can we script and automate a browser, preferably with .Net?
Related
~(I found out when it happens, see bottom of question)
I am working with a traditional ASP.NET web application. There is an .aspx page that hosts an angular 11 application which loads fine 9/10 times but occasionally a bad response is returned with a 200 OK status. When this happens, in Firefox a page loads with a "content encoding error" and in Chrome and Edge, just a blank screen with the same verbiage in console.
Using Wireshark, I was able to determine that when the "content-encoding-error" occurs the response header has three comma separated "gzip" values appended to the Content-Encoding header, see below:
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store, must-revalidate
Content-Type: text/html; charset=utf-8
Content-Encoding: gzip, gzip, gzip
...
Whereas, a normal response from the .aspx page look like this.
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store, must-revalidate
Content-Type: text/html; charset=utf-8
Content-Encoding: gzip
..
I can duplicate the issue using one of aspx's [web method] calls:
var ctx = HttpContext.Current
var unused = ctx.Response.Filter // Because apparently you must access it before you can set it
ctx.Response.Filter = new GZipStream(ctx.Response.OutputStream, CompressionLevel.Optimal)
ctx.Response.AppendHeader("Content-Encoding", "gzip")
ctx.Response.AppendHeader("Content-Encoding", "gzip") // <--Gzip added twice here
The troubling part is that the multiple "gzip" values are on the response from the aspx page itself. I have search the entire code base and all web.config(s) in an attempt to find where this compression is being applied but to no avail. So, I am thinking it could be a third party doing this.
We use DevExtreme and I have been looking at these settings in our config:
<add key="DXEnableCallbackCompression" value="true" />
<add key="DXEnableResourceCompression" value="true" />
<add key="DXEnableResourceMerging" value="true" />
<add key="DXEnableHtmlCompression" value="true" />
I am still having trouble scanning the code for issues. Does anyone know of a trick using fiddler or Wireshark or any other tool that could reveal where these headers are sporadically showing triples at?
Edit: Here is the GET request header which returns a response which proper encoding ~90% of the time.
GET http://xxx/xxx.aspx?xxx=4 HTTP/1.1
Host: xxx.com
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36 Edg/92.0.902.73
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Referer: http://xxx/Home.aspx
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Cookie: ASP.NET_SessionId=x; .ASPXAUTH=x;
Found out when it happens:
I was able to duplicate this issue on a regular basis. If I close all browser sessions and recycle the app pool, the issue occurs on the first request. On subsequent requests, the issue does not happen.
Also, the culprit is a google script embedded on the HTML page. When this script is removed the page loads fine on first request after a recycle or not.
Culprit Code:
<script type="text/javascript" src="//maps.googleapis.com/maps/api/js?key=
<%=GoogleMapAPIKey%>&channel=<%=GoogleMappingChannel%>"></script>
I am sure it is not the js file itself. The keys are embedded into the tag via server side processors. Those two processor's call an API to get the keys and those calls are gzipped. I still don't know why the aspx's response header is getting three "gzips" when the js include statement is present in the page markup.
I may remove this wall of text and add a new question due to the new findings.
The problem seems to occur when you Gzip encoding was added to outgoing that were being triggered from markup on the aspx page. All web methods that are called after page load and in an async fasion from the angular client have no encoding issue.
There were two called via a page property that was triggered by page markup to access its value. These web methods had gzip applied and I guess since these were processed earlier in the page-lifecycle something was getting mixed up.
My problem was solved by removing the compression on those two calls.
There were two calls to a function that added Gzip encoding prior to page load and at that time the response was the aspx page itself.
I need to allow users to download files from our server, and I'd like to serve these files via an ASP.NET MVC 5 controller action. My action looks like this:
public FileContentResult Download(int fileId)
{
var myContent = GetContentForFile(fileId);
var myFileMeta = GetFileMeta(fileId);
if (myContent == null || myFileMeta == null)
throw new FriendlyException("The file or its associated data could not be found.");
return File(myContent.Content, myContent.MediaType, myFileMeta.FileName);
}
The above is as simple as I could get it, it works fine on PC and iPhone, but not on Android. Using Fiddler, I can see that the following response headers when I try to download one of my files - in this case a JPG file called "1447114384146-643143584.jpg":
HTTP/1.1 200 OK
Cache-Control: private, s-maxage=0
Content-Type: image/jpeg
Server: Microsoft-IIS/8.5
X-AspNetMvc-Version: 5.2
Content-Disposition: attachment; filename=1447114384146-643143584.jpg
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Thu, 12 Nov 2015 23:09:00 GMT
Content-Length: 1682868
Note that I don't have any reliable way to know the correct MIME-type - is this an issue and could it explain why the file isn't being downloaded in Android?
To clarify, when I attempt to download any file from the database using Android, I get a toast notification telling me "Download started", but then the download sits in the queue for a while on 0% before eventually just changing to "Failed".
What I've tried
I've tried all manner of things that people have suggested in similar questions, most of which are to do with the content-disposition header or the content-type header. I've tried forcing the content-type header to application/octet-stream for every file, I've tried sending the correct content-type header for the particular file. I've tried manually sending the content-disposition header. I've tried forcing the filename extension to uppercase.
None of the above has worked, in fact none of them have had any impact at all on the problem, for better OR worse. I'm amazed that this is so hard - I feel like I must be missing something obvious?
Additional information
Browser: latest Chrome on Android
OS: Android 5.1 (also occurs for a coworker on their Android phone which is at an earlier Android version (not sure which specifically), so I don't think this is tied to a specific Android version).
Update
After reading this blog entry: http://www.digiblog.de/2011/04/android-and-the-download-file-headers/ I tried following the advice and set my headers exactly as suggested:
HTTP/1.1 200 OK
Cache-Control: private, s-maxage=0
Content-Type: application/octet-stream
Server: Microsoft-IIS/8.5
X-AspNetMvc-Version: 5.2
Content-Disposition: attachment; filename="1447114384146-643143584.JPG"
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Thu, 12 Nov 2015 23:42:18 GMT
Content-Length: 1682868
Again, this had no impact on the problem at all.
Futher update
I have been able to test on a Marshmallow (Android v6.0) device and the download works. It seems to be a pre-Marshmallow issue.
Sadly this was caused by something very specific to my environment, but I'd like to put the answer here in case anyone else stumbles across this same problem.
It turns out the Android download manager doesn't like underscores in domain names, and our local domain address had an underscore in it. I used the server's IP address instead and everything worked as expected.
For example this: http://www.my_domain.com.au/file.png won't work. This: http://192.168.x.x/file.png does work.
Found as an answer on this question: Trouble downloading file from browser on Android
Disclaimer: I don't have enough rep to add to the comments so I am forced to comment here.
Have you tried different versions of Android using the emulator or
have you only tried using an actual device?
If only on a device, is the code in production or are using
connecting to your local development system through a local wireless
connection?
Have you tried to use Chrome Remote Debugging on the device?
https://developers.google.com/web/tools/chrome-devtools/debug/remote-debugging/remote-debugging?hl=en
As a way to rule out issues with the setup on your device would be to write a small Android app using Xamarin + RestSharp that does nothing but hits your download url to see if that works. If it does, then that helps to point the finger at Chrome itself. If it doesn't then at least you can run the app with the debugger attached to get better insight as to what is happening on the other end.
https://xamarin.com/
https://github.com/restsharp/RestSharp
UPDATE: Response headers as seen by Fiddler when calling a test served by my local machine
HTTP/1.1 200 OK
Cache-Control: private
Content-Type: application/octet-stream
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
Content-Disposition: attachment; filename=profile.jpg
Date: Fri, 13 Nov 2015 02:09:23 GMT
Content-Length: 218143
Update: Here are the incoming request server variable
ALL_HTTP=HTTP_CACHE_CONTROL:max-age=0
HTTP_CONNECTION:keep-alive
HTTP_ACCEPT:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
HTTP_ACCEPT_ENCODING:gzip, deflate, sdch
HTTP_ACCEPT_LANGUAGE:en-US,en;q=0.8
HTTP_COOKIE:_ga=GA1.1.420021277.1447377172
HTTP_HOST:192.168.1.2
HTTP_USER_AGENT:Mozilla/5.0 (Linux; Android 5.0.2; HTC One Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36
HTTP_UPGRADE_INSECURE_REQUESTS:1
HTTP_DNT:1
ALL_RAW=Cache-Control: max-age=0
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
Cookie: _ga=GA1.1.420021277.1447377172
Host: 192.168.1.2
User-Agent: Mozilla/5.0 (Linux; Android 5.0.2; HTC One Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36
Upgrade-Insecure-Requests: 1
DNT: 1
APPL_MD_PATH=/LM/W3SVC/2/ROOT
APPL_PHYSICAL_PATH=C:\development\rumble-strip\projects\net-framework\RumbleStrip.Website\
AUTH_TYPE=
AUTH_USER=
AUTH_PASSWORD=
LOGON_USER=
REMOTE_USER=
CERT_COOKIE=
CERT_FLAGS=
CERT_ISSUER=
CERT_KEYSIZE=
CERT_SECRETKEYSIZE=
CERT_SERIALNUMBER=
CERT_SERVER_ISSUER=
CERT_SERVER_SUBJECT=
CERT_SUBJECT=
CONTENT_LENGTH=0
CONTENT_TYPE=
GATEWAY_INTERFACE=CGI/1.1
HTTPS=off
HTTPS_KEYSIZE=
HTTPS_SECRETKEYSIZE=
HTTPS_SERVER_ISSUER=
HTTPS_SERVER_SUBJECT=
INSTANCE_ID=2
INSTANCE_META_PATH=/LM/W3SVC/2
LOCAL_ADDR=192.168.1.2
PATH_INFO=/
PATH_TRANSLATED=C:\development\rumble-strip\projects\net-framework\RumbleStrip.Website
QUERY_STRING=&REMOTE_ADDR=192.168.1.5&REMOTE_HOST=192.168.1.5
REMOTE_PORT=54748
REQUEST_METHOD=GET
SCRIPT_NAME=/
SERVER_NAME=192.168.1.2
SERVER_PORT=80
SERVER_PORT_SECURE=0
SERVER_PROTOCOL=HTTP/1.1
SERVER_SOFTWARE=Microsoft-IIS/10.0
URL=/
HTTP_CACHE_CONTROL=max-age=0
HTTP_CONNECTION=keep-alive
HTTP_ACCEPT=text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
HTTP_ACCEPT_ENCODING=gzip, deflate, sdch
HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.8
HTTP_COOKIE=_ga=GA1.1.420021277.1447377172
HTTP_HOST=192.168.1.2
HTTP_USER_AGENT=Mozilla/5.0 (Linux; Android 5.0.2; HTC One Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36
HTTP_UPGRADE_INSECURE_REQUESTS=1
HTTP_DNT=1
IS_LOGIN_PAGE=1
I have an example URL (http://www.techspot.com/downloads.php?action=download_now&id=2991&evp=113a02f49ca8ac11b566336b984b1655&file=1). And when I click the link, the url will change to:
http://www.exisoftware.com/downloads/picture_finder/PictureFinderSetup.exe
Can anyone help me how to convert the php link to real url using vb.net or c#?
Because when I make a program to check file information in vb.net using HEAD method the file name not "PictureFinderSetup.exe" but "downloads.php?action=download_now&id=2991&evp=113a02f49ca8ac11b566336b984b1655&file=1".
It cannot be done without contacting the server. Only the server knows the exact mapping.
From Fiddler:
Request
GET http://www.techspot.com/downloads.php?action=download_now&id=2991&evp=113a02f49ca8ac11b566336b984b1655&file=1 HTTP/1.1
...
Response
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Fri, 27 Sep 2013 17:26:14 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Location: http://www.techspot.com/downloads/2991-extreme-picture-finder.html
To get the information you can use WebBrowser or HttpWebResponse to get data you need. How can I use VB.Net to read the content returned from a URL?
I have a very simple app that sends an HttpWebRequest and gets a response. I need to know the exact request sent to the server. Is it possible?
Something like this:
POST /path/script.cgi HTTP/1.0
From: frog#jmarshall.com
User-Agent: HTTPTool/1.0
Content-Type: application/x-www-form-urlencoded
Content-Length: 32
Build a basic web server with the System.Net.Sockets.TcpListener. The example shows how to do this. Then, point your HttpWebRequest to that server and see the results.
I'm trying to hunt down why a POST request from a C# script isn't working, when the same request works fine in Python. I want to be able to have all the data sent by the script, and the response from the sever, to be displayed on the screen so that I can work out what the difference is between what the C# and Python scripts are sending.
In Python I can do this with the standard httplib2 library by just using:
httplib2.debuglevel = 1
This produces the following output (as an example):
reply: 'HTTP/1.1 201 Created\r\n'
header: Date: Tue, 05 Oct 2010 09:25:42 GMT
header: Server: Apache/2.2.9 (Debian) PHP/5.2.6-1+lenny9 with Suhosin-Patch
header: X-Powered-By: PHP/5.2.6-1+lenny9
header: Location: http://example.org/api/2
header: Content-Length: 0
header: Content-Type: text/html
send: 'GET /api/2 HTTP/1.1\r\nHost: example.org\r\naccept-encoding: gzip, deflate\r\nuser-agent: Python-httplib2/$Rev$\r\n\r\n'
Is there a way to produce similar output in C# using the HttpWebRequest class?
I've seen mention of Fiddler in another question, but I'm running Linux and Fiddler appears to be for Windows only.
You can spin through resp.Headers.AllKeys and then dump the key and its value, though there is sometimes a degree of translation going on (most obviously when it is doing auto-redirect-following).
You can use System.Net tracing.
You can use ethereal which has a linux version. I don't use it for such things these days as Fiddler is indeed handier for such cases, but I used to use ethereal in the past, and sometimes seeing what is actually on the wire rather than what the code is saying is the best way to go (as you aren't depending on possibly buggy code to tell you if the code is buggy).
Yes you can and I have used it successfully before. If you are using C# I imagine there is a config file that you acn turn System.Net tracing on. I am not sure if Mono has this - if you are using Mono.
Have a look here:
http://msdn.microsoft.com/en-us/library/ty48b824.aspx