I am planning create a movil application (for fun) that should use the result from this web page (http://consultawebvehiculos.carabineros.cl/index.php). is there any ways to create a instance of a browser in my Net code and read this result and publish it using a web service..
something like:
var IE= new broswer("http://consultawebvehiculos.carabineros.cl/index.php");
var result=IE.FindElementByID("txtIdentityCar").WriteText(YourIdentityCar);
publicToWebSerivce(result);
Update:
Using Fiddler i can see that http post is somthing like that:
POST http://consultawebvehiculos.carabineros.cl/index.php HTTP/1.1
Host: consultawebvehiculos.carabineros.cl
Connection: keep-alive
Content-Length: 61
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Origin: http://consultawebvehiculos.carabineros.cl
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.57 Safari/537.17
Content-Type: application/x-www-form-urlencoded
Referer: http://consultawebvehiculos.carabineros.cl/index.php
Accept-Encoding: gzip,deflate,sdch
Accept-Language: es-ES,es;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
accion=buscar&txtLetras=CL&txtNumeros1=sk&txtNumeros2=12&vin=
May be i need some .Net class like webclient in order connect with the php page...no sure.
UPDATE: I finally i found the solution using Fiddler to know the total parameters and I've used the code from http://www.hanselman.com/blog/HTTPPOSTsAndHTTPGETsWithWebClientAndCAndFakingAPostBack.aspx
If your are just interested in scraping the page, I suggest using Html Agility Pack.
If you also want to display the page, then you could use the WebBrowser control.
We've been using http://htmlunit.sourceforge.net/ for similair tasks. It allows you to send requests, receive response/status code/etc.
(it's a Java lib, so you could either google for a .Net port or use a converter to convert Java assembly into .Net assembly - see http://blog.stevensanderson.com/2010/03/30/using-htmlunit-on-net-for-headless-browser-automation/ for guidance. We've used the convertion approach).
Related
I have a .NET Core 1.1 MVC controller that somehow isn't getting called correctly when a request comes in.
The controller method looks like this:
Although I don't know if that really matters, because I have debugging lines in the controller's constructor (and have run in debug with breakpoints in the constructor, as well), and it looks like even the constructor is never getting called.
The application output contains a line like this, when the call comes into the server:
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker:Information: Executed action Namespace.NameController.GetData (AssemblyName) in 0.9161ms
Something that is suspicious there is that most of the lines I see logged like this for other controllers being called contain the argument information, as well, and this one doesn't.
I'm not getting an error from the client side, instead I'm getting a success response with an empty body. It's almost like an empty response is getting returned before any of my controller's code actually runs.
Here are the details of the request/response (the response body is empty):
Request URL: http://localhost:61410/path-to-controller/GetData?xtype=xtypeargument
Request Method: POST
Status Code: 200
Query Url
xtype: xtypeargument
Request Headers
accept: */*
Origin: http://localhost:61410
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36
Authorization: bearer <bearer-token>
Referer: http://localhost:61410/path-to/index.html
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8
Response Headers
Server: Kestrel
Access-Control-Allow-Origin: *
X-SourceFiles: =?UTF-8?B?...?=
X-Powered-By: ASP.NET
Date: Wed, 06 Sep 2017 14:08:10 GMT
Content-Length: 0
Any ideas of what might be going on here?
It turns out that the controller's constructor had arguments that were expected to be provided by Dependency Injection, but that weren't. Somehow this caused the behavior I was seeing, although I still don't really understand why I wasn't getting exceptions instead of these empty responses. Anyway, I fixed the code and it's working now.
I need to allow users to download files from our server, and I'd like to serve these files via an ASP.NET MVC 5 controller action. My action looks like this:
public FileContentResult Download(int fileId)
{
var myContent = GetContentForFile(fileId);
var myFileMeta = GetFileMeta(fileId);
if (myContent == null || myFileMeta == null)
throw new FriendlyException("The file or its associated data could not be found.");
return File(myContent.Content, myContent.MediaType, myFileMeta.FileName);
}
The above is as simple as I could get it, it works fine on PC and iPhone, but not on Android. Using Fiddler, I can see that the following response headers when I try to download one of my files - in this case a JPG file called "1447114384146-643143584.jpg":
HTTP/1.1 200 OK
Cache-Control: private, s-maxage=0
Content-Type: image/jpeg
Server: Microsoft-IIS/8.5
X-AspNetMvc-Version: 5.2
Content-Disposition: attachment; filename=1447114384146-643143584.jpg
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Thu, 12 Nov 2015 23:09:00 GMT
Content-Length: 1682868
Note that I don't have any reliable way to know the correct MIME-type - is this an issue and could it explain why the file isn't being downloaded in Android?
To clarify, when I attempt to download any file from the database using Android, I get a toast notification telling me "Download started", but then the download sits in the queue for a while on 0% before eventually just changing to "Failed".
What I've tried
I've tried all manner of things that people have suggested in similar questions, most of which are to do with the content-disposition header or the content-type header. I've tried forcing the content-type header to application/octet-stream for every file, I've tried sending the correct content-type header for the particular file. I've tried manually sending the content-disposition header. I've tried forcing the filename extension to uppercase.
None of the above has worked, in fact none of them have had any impact at all on the problem, for better OR worse. I'm amazed that this is so hard - I feel like I must be missing something obvious?
Additional information
Browser: latest Chrome on Android
OS: Android 5.1 (also occurs for a coworker on their Android phone which is at an earlier Android version (not sure which specifically), so I don't think this is tied to a specific Android version).
Update
After reading this blog entry: http://www.digiblog.de/2011/04/android-and-the-download-file-headers/ I tried following the advice and set my headers exactly as suggested:
HTTP/1.1 200 OK
Cache-Control: private, s-maxage=0
Content-Type: application/octet-stream
Server: Microsoft-IIS/8.5
X-AspNetMvc-Version: 5.2
Content-Disposition: attachment; filename="1447114384146-643143584.JPG"
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Thu, 12 Nov 2015 23:42:18 GMT
Content-Length: 1682868
Again, this had no impact on the problem at all.
Futher update
I have been able to test on a Marshmallow (Android v6.0) device and the download works. It seems to be a pre-Marshmallow issue.
Sadly this was caused by something very specific to my environment, but I'd like to put the answer here in case anyone else stumbles across this same problem.
It turns out the Android download manager doesn't like underscores in domain names, and our local domain address had an underscore in it. I used the server's IP address instead and everything worked as expected.
For example this: http://www.my_domain.com.au/file.png won't work. This: http://192.168.x.x/file.png does work.
Found as an answer on this question: Trouble downloading file from browser on Android
Disclaimer: I don't have enough rep to add to the comments so I am forced to comment here.
Have you tried different versions of Android using the emulator or
have you only tried using an actual device?
If only on a device, is the code in production or are using
connecting to your local development system through a local wireless
connection?
Have you tried to use Chrome Remote Debugging on the device?
https://developers.google.com/web/tools/chrome-devtools/debug/remote-debugging/remote-debugging?hl=en
As a way to rule out issues with the setup on your device would be to write a small Android app using Xamarin + RestSharp that does nothing but hits your download url to see if that works. If it does, then that helps to point the finger at Chrome itself. If it doesn't then at least you can run the app with the debugger attached to get better insight as to what is happening on the other end.
https://xamarin.com/
https://github.com/restsharp/RestSharp
UPDATE: Response headers as seen by Fiddler when calling a test served by my local machine
HTTP/1.1 200 OK
Cache-Control: private
Content-Type: application/octet-stream
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
Content-Disposition: attachment; filename=profile.jpg
Date: Fri, 13 Nov 2015 02:09:23 GMT
Content-Length: 218143
Update: Here are the incoming request server variable
ALL_HTTP=HTTP_CACHE_CONTROL:max-age=0
HTTP_CONNECTION:keep-alive
HTTP_ACCEPT:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
HTTP_ACCEPT_ENCODING:gzip, deflate, sdch
HTTP_ACCEPT_LANGUAGE:en-US,en;q=0.8
HTTP_COOKIE:_ga=GA1.1.420021277.1447377172
HTTP_HOST:192.168.1.2
HTTP_USER_AGENT:Mozilla/5.0 (Linux; Android 5.0.2; HTC One Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36
HTTP_UPGRADE_INSECURE_REQUESTS:1
HTTP_DNT:1
ALL_RAW=Cache-Control: max-age=0
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
Cookie: _ga=GA1.1.420021277.1447377172
Host: 192.168.1.2
User-Agent: Mozilla/5.0 (Linux; Android 5.0.2; HTC One Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36
Upgrade-Insecure-Requests: 1
DNT: 1
APPL_MD_PATH=/LM/W3SVC/2/ROOT
APPL_PHYSICAL_PATH=C:\development\rumble-strip\projects\net-framework\RumbleStrip.Website\
AUTH_TYPE=
AUTH_USER=
AUTH_PASSWORD=
LOGON_USER=
REMOTE_USER=
CERT_COOKIE=
CERT_FLAGS=
CERT_ISSUER=
CERT_KEYSIZE=
CERT_SECRETKEYSIZE=
CERT_SERIALNUMBER=
CERT_SERVER_ISSUER=
CERT_SERVER_SUBJECT=
CERT_SUBJECT=
CONTENT_LENGTH=0
CONTENT_TYPE=
GATEWAY_INTERFACE=CGI/1.1
HTTPS=off
HTTPS_KEYSIZE=
HTTPS_SECRETKEYSIZE=
HTTPS_SERVER_ISSUER=
HTTPS_SERVER_SUBJECT=
INSTANCE_ID=2
INSTANCE_META_PATH=/LM/W3SVC/2
LOCAL_ADDR=192.168.1.2
PATH_INFO=/
PATH_TRANSLATED=C:\development\rumble-strip\projects\net-framework\RumbleStrip.Website
QUERY_STRING=&REMOTE_ADDR=192.168.1.5&REMOTE_HOST=192.168.1.5
REMOTE_PORT=54748
REQUEST_METHOD=GET
SCRIPT_NAME=/
SERVER_NAME=192.168.1.2
SERVER_PORT=80
SERVER_PORT_SECURE=0
SERVER_PROTOCOL=HTTP/1.1
SERVER_SOFTWARE=Microsoft-IIS/10.0
URL=/
HTTP_CACHE_CONTROL=max-age=0
HTTP_CONNECTION=keep-alive
HTTP_ACCEPT=text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
HTTP_ACCEPT_ENCODING=gzip, deflate, sdch
HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.8
HTTP_COOKIE=_ga=GA1.1.420021277.1447377172
HTTP_HOST=192.168.1.2
HTTP_USER_AGENT=Mozilla/5.0 (Linux; Android 5.0.2; HTC One Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36
HTTP_UPGRADE_INSECURE_REQUESTS=1
HTTP_DNT=1
IS_LOGIN_PAGE=1
I am working on a desktop application which calls some web services via WebClient POST requests. These same web services are used in web application also.
I am facing a strange problem where in my desktop application request was successful and I got response but some of my requests parameters were not saved. But same request is updateing all the parameters we I call them from web application using jquery.
In web application I am calling web service like this
$.post("/MyService/Account/Register",accountModel, function (data) {
});
and I stingify my json object that is accountModel, my request looks like this when console.log
{"Name":"Lorem","Email":"abc#abc.com","interest":"[\"1\"]","sectors":"[\"1\",\"2\"]","subscribe":false}
Now when I used same request string to post data from my desktop application all the properties like name, email and subscribe were saved but interest and sectors were not saved.
I want to figure it out that why same request object is working via jquery and not in C# webclient.
Here is my code that I used to post data using WebClient
WebClient client = new WebClient();
string json = string.Format("{{\"Name\":\"{0}\",\"Email\":\"{1}\",\"interest\":\"[\"{2}\"]","sectors":"[\"{3}\",\"{4}\"]","subscribe":{5} }}","Lorem","abc#abc.com","1","1","2","false");
client.Headers[HttpRequestHeader.ContentType] = "application/json";
string result = client.UploadString("http://Server.com/MyService/Account/Register", json);
Please someone help me to resolve this issue that I am not getting any error but some of my parameters were not updates.
I want to clear that I do not have any code or documentation access to web service.
UPDATE
As per Jasen's comment here are requests captured with fiddler
JQuery request
POST http://Server.com/MyService/Account/Register HTTP/1.1
Host: server.com
Connection: keep-alive
Content-Length: 463
Accept: */*
Origin: http://server.com
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Referer: http://server.com/MyService/Account/Register/
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
Cookie: .ASPXAUTH=7A14FC68B72078BAE43A623B94A901180C72093CCE222BBD98EE2AE7E2612078D1E3B7D8860905A4F7B2D75FD67E9274A0A5C40760A5AF703F970504380EBAF8B3D09A15F0B70090ACF4882DC58885F7CF12473BF55647840F3080ADD2C19249
Name=Lorem&Email=abc#abc.com&interest=%5B%221%22%5D§ors=%5B%221%22%2C%222%22%5D&subscribe=false
WebClient Request
POST http://server.com/MyService/Account/Register HTTP/1.1
Content-Type: application/json
Host:server.com
Cookie: .ASPXAUTH=F586C63F64186E13EB6EC19AAB25A531A0EDA5B7B601013550ADD629C1481EC3F080DDB5F06D691CB8F81EE8631EF8859F82CF7DD3F2ED2A597AA971A53E80141EDD6EA549784AD7EAE8E144F0CD3196A44316F29C08E0C5383A7231A1B6C5EF
Content-Length: 536
Expect: 100-continue
{"Name":"Lorem","Email":"abc#abc.com","sectors":["1","2"],"interest":["1"],"subscribe":false}
Shall I send my webclient request as URL encoded string like we can see in jquery request?
Finally I got the solution with help of fiddler. Thanks to Jasen for suggesting fiddler to see requests
Here is my working code
WebClient client = new WebClient();
string result = client.UploadValues("http://Server.com/MyService/Account/Register", new NameValueCollection()
{
{"Name","Lorem"},
{"Email","abc#abc.com"},
{"interest","[\"1\"]"},
{"sectors","[\"1\",\"2\"]"},
{"interest","false"}
});
Yes I used UploadValues method instead of UploadData or UploadString. Also note that I have removed the content type json declaration from my code.
I've been searching for at least 2hrs but I can't find any pattern to extract following Urls using regex. I went with too many patterns which described in many articles. But I couldn't find something useful.
For Example : Urls like following patterns.
http://google.com
http://www.google.com
http://www.image.google.com
http://google.com:8080
http://google.com:8080/default.aspx?param=1
http://google.com/default.aspx?param=1¶m1=2
Update : Dear friends, It looks like I have to explain my issue in more details, I'm working on a simple proxy server using TCP components, My server listen to specific port when an incoming connection received. I'm extracting and reading all client request data.
data contains headers and content types and etc like following :
GET http://www.bing.com/ HTTP/1.1
Accept: text/html, application/xhtml+xml, */*
Accept-Language: en-US,en;q=0.7,fa;q=0.3
User-Agent: Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0)
Accept-Encoding: gzip, deflate
Host: www.bing.com
DNT: 1
Proxy-Connection: Keep-Alive
These are plain-text so I need to find and extract Urls for doing forwarding operations.
And any Url pattern you guess.
Please, Any advice will be helpful.
https?://[\w\.]+\.\w+(:\d{1,5})?(/[\w?&.=]+)?
Salam. Try this one:
https?://[^\s]+
EDIT:
I´ll be more specific. I want to do a script to download a group of files every day.
To do this programmatically, i need to click in a javascript button.
It´s simple when is just put the URL in WebRequest class, but in javascript button i don´t have the URL. How can i mount this URL?
Request (by Fiddler):
POST /SomeSite?Something.aspx HTTP/1.1
Accept: text/html, application/xhtml+xml, */*
Referer: http://www.Site.com/Stackoverflow/SomeSite?Something.aspx
Accept-Language: pt-BR
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip, deflate
Host: www.Site.com
Content-Length: 10616
Connection: Keep-Alive
Pragma: no-cache
Cookie: idioma=pt-br; WT_FPC=id=187.16.81.13-3324702672.30186643:lv=1320587789589:ss=1320587578749
__EVENTTARGET=ctl00%24contentPlaceHolderConteudo%24lnkDownloadArquivo&__EVENTARGUMENT=&__VIEWSTATE=%BlaBlaBla
Here you can see the _EVENTTARGET that is using postback with a link Button which name is "lnkDownloadArquivo". So far I understand you want to simulate same download request without button click. if so then you can check here a solution .
http://ciintelligence.blogspot.com/2011/01/fetching-aspnet-authenticated-page-with.html.
here you can get idea how asp.net button post back request works.
The built-in class you need is the HTTPWebRequest (or WebRequest) class. To create one, call System.Net.WebRequest.Create() and pass your URL, add the appropriate headers using the Headers collection, write to the Stream retrieved from WebRequest.GetRequestStream(), then retrieve the response using WebRequest.GetResponse(). From the retrieved response object, you can get the response Stream using WebResponse.GetResponseStream(). The Stream can then be read from like any other Stream object.