I'm trying to hunt down why a POST request from a C# script isn't working, when the same request works fine in Python. I want to be able to have all the data sent by the script, and the response from the sever, to be displayed on the screen so that I can work out what the difference is between what the C# and Python scripts are sending.
In Python I can do this with the standard httplib2 library by just using:
httplib2.debuglevel = 1
This produces the following output (as an example):
reply: 'HTTP/1.1 201 Created\r\n'
header: Date: Tue, 05 Oct 2010 09:25:42 GMT
header: Server: Apache/2.2.9 (Debian) PHP/5.2.6-1+lenny9 with Suhosin-Patch
header: X-Powered-By: PHP/5.2.6-1+lenny9
header: Location: http://example.org/api/2
header: Content-Length: 0
header: Content-Type: text/html
send: 'GET /api/2 HTTP/1.1\r\nHost: example.org\r\naccept-encoding: gzip, deflate\r\nuser-agent: Python-httplib2/$Rev$\r\n\r\n'
Is there a way to produce similar output in C# using the HttpWebRequest class?
I've seen mention of Fiddler in another question, but I'm running Linux and Fiddler appears to be for Windows only.
You can spin through resp.Headers.AllKeys and then dump the key and its value, though there is sometimes a degree of translation going on (most obviously when it is doing auto-redirect-following).
You can use System.Net tracing.
You can use ethereal which has a linux version. I don't use it for such things these days as Fiddler is indeed handier for such cases, but I used to use ethereal in the past, and sometimes seeing what is actually on the wire rather than what the code is saying is the best way to go (as you aren't depending on possibly buggy code to tell you if the code is buggy).
Yes you can and I have used it successfully before. If you are using C# I imagine there is a config file that you acn turn System.Net tracing on. I am not sure if Mono has this - if you are using Mono.
Have a look here:
http://msdn.microsoft.com/en-us/library/ty48b824.aspx
Related
I have to be able to send a HTTP request for a response like this:
HTTP/1.0 200 OK
/bin/sh: check_updatelock: not found
Cache-Control: no-cache, must-revalidate
Expires: ...
Content-Type: application/json
{"response": "..."}
Now here /bin/sh causes exception: Received an invalid header name: '/bin/sh' (at System.Net.Http.HttpConnection.ParseHeaderNameValue).
I've tried with HttpClient, WebClient, HttpWebRequest, and even with a 3rd party lib named Flurl, but that uses HttpClient under the hood.
The result is always the same expection.
Even in PowerShell, with Invoke-WebRequest, there is an execption: The server committed a protocol violation. Section=ResponseHeader Detail=Header name is invalid
How could I read such a response from a C# .NET Core console app?
(The server is out of my control)
Any idea highly appreciated!
Separator chars, like '/', are not part of the field name in the header. So HttpClient (and other libraries built on top of it) will throw an error.
You can try other third party libraries, which are not based on HttpClient like RestSharp (I'm not sure whether it supports non-standard header, but its not based on HttpClient)
I have an application that uploads videos to YouTube on behalf of our customers. We manage thousands of channels and upload thousands of videos a day. This has been working for years. The code is written in C#, .Net 4.7 (it was actually using a slightly older version, but I had to update it in order to recompile and test). It uses the REST API (not the SDK).
For some reason, the code broke the evening of Friday, Oct 12th, 2018. All of our requests to upload return with a 400 status and "Malformed multipart body." as the body.
We have not changed the code in at least 6 months (though likely over a year). I am able to reproduce the error on my dev machine. I looked at the raw http request in Fiddler and I don't see anything wrong with it, though I'm having difficulty finding the plain REST documentation to confirm the content-disposition headers. I know the JSON and video are fine (I validated the JSON and verified I'm able to upload the video directly to YouTube). Also, all of the other API calls we make work just fine. It's just uploading that is a problem.
Here's a copy of the HTTP request from Fiddler...
POST https://www.googleapis.com/upload/youtube/v3/videos?part=snippet,status HTTP/1.1
Authorization: Bearer <token>
Content-Type: multipart/form-data; boundary="590ce98e-6411-4e49-8dde-d7aa06cb067d"
Host: www.googleapis.com
Content-Length: 8305362
Expect: 100-continue
--590ce98e-6411-4e49-8dde-d7aa06cb067d
Content-Type: application/json; charset=utf-8
Content-Disposition: form-data; name=json; filename=file.json; filename*=utf-8''file.json
{"snippet":{"title":"77 Wakefield Street, Bald Hills, QLD, 4036","description":"77 Wakefield Street\r\n\r\nContact Brian Brewder for more information.\nTourFactory Corporate Headquarters\n123-456-7890\n\r\n","categoryId":19,"tags":["Tag1","Tag2"]},"status":{"privacyStatus":"public","embeddable":true}}
--590ce98e-6411-4e49-8dde-d7aa06cb067d
Content-Disposition: form-data; name="files"; filename="video"
Content-Type: video/x-msvideo
<video>
It seems like something must have changed on the YouTube side of the API, but I checked the blog and don't see anything listed. Since Google uses StackOverflow for support, I'm hoping somebody can help me identify the problem.
So, apparently what caused the issue (And still does) is that now the
Content-Type: multipart/mixed;
is no longer accepted. I managed to get an error message by changing some stuff around, and said to only send video/* or application/octet-stream. Unfortunately, while i still got OK from the server by doing so, resulted in a broken youtube video
At this point i tried playing with headers but with no success. The file would be accepted by YouTube but wouldn't be displayed correctly. I suspect they changed - or dropped - the non-resumable uploads API. If they did, i can't find an announcement on that.
So, my "solution" - well, a workaround - was to reimplement my upload function using the Resumable Upload protocol.
https://developers.google.com/youtube/v3/guides/using_resumable_upload_protocol
This works and didn't take too long. Just make sure to set both X-Upload-Content-Length and x-upload-content-type for BOTH requests (metadata first and payload after getting the upload put URL).
So you first POST the same metadata (snippet and status) and get the PUT url
QByteArray responseLocation = reply->rawHeader( "Location" );
and then create the PUT request using the same X-Upload-Content-Length and x-upload-content-type
newrequest.setRawHeader( "X-Upload-Content-Length", QByteArray::number( video->size() ) );
newrequest.setRawHeader( "x-upload-content-type", "video/*" );
That worked for me - I hope this helps!
We're building an ASP.NET web application which integrates with a DAM (digital asset management) system. The DAM stores files in Azure blob storage and makes them available publicly using Azure CDN.
We will be making these files (most are PDFs) available to download from our web app. When a user requests one of these files we will provide a custom URL which will run some code on the server (logging the download etc) before returning the relevant file for download.
The client requires that the file is always returned as a browser attachment (i.e. content disposition attachment header). I am curious about what options I have here.
My ideal would be that the CDN URL is abstracted and my custom URL is the public URL for the file. That would allow me to set relevant response headers etc. However, I assume the only solution here would be to download the file from CDN and cache it on my web server which would obfuscate the CDN's purpose. So presumably I have to redirect the client to the CDN public URL once I've done my server processing. But then is there a way I can ensure the file is returned by Azure with the correct response headers to ensure the browser's default download behaviour is delegated?
* Update *
In seeing the answers to this question I realised I was perhaps asking the wrong question. Thank you to those of you who answered here. Follow-up question is here.
TL;DR
You need to configure the default version on the blob storage in order for it to show the required header to non-authenticated clients. the question in this question has the code to make it work.
Once this is set, and working for anonymous clients the CDN will copy all of the headers across and it should work as expected.
Setting ContentDisposition
The functionality is present, you can set ContentDisposition on a blob property However, while this will set the property on the blob, it does not pass through to the header.
I tested this with Powershell using the following (just because its quicker than c#)
$context = New-AzureStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey
$container = Get-AzureStorageContainer -Name $ContainerName -Context $context
$blobref = ($script:container.CloudBlobContainer.GetBlobReferenceFromServer("images/pier.jpg"))
$blobref.Properties
$blobref.Properties.ContentDisposition = 'attachment; filename="fname.ext"'
$blobref.SetProperties()
$blobref = ($script:container.CloudBlobContainer.GetBlobReferenceFromServer("images/pier.jpg"))
$blobref.Properties
Which produces (amongst others)
ContentDisposition : attachment; filename="fname.ext"
However nothing is set when the headers are queried
([system.Net.HttpWebRequest]::Create($blobref.Uri.AbsoluteUri)).getresponse()
(to answer comment,. these are the headers returned - while experimenting I also tried with and without a contenttype - hence it being blank here)
IsMutuallyAuthenticated : False
Cookies : {}
Headers : {x-ms-request-id, x-ms-version, x-ms-lease-status, x-ms-blob-type...}
SupportsHeaders : True
ContentLength : 142224
ContentEncoding :
ContentType :
CharacterSet :
Server : Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
LastModified : 01/03/2016 11:29:04
StatusCode : OK
StatusDescription : OK
ProtocolVersion : 1.1
ResponseUri : https://xxxx.blob.core.windows.net/cdn/images/pier.jpg
Method : GET
IsFromCache : False
And since CDN will only copy the information from the HTTP headers themselves, this data isn't making it into CDN.
edited (after extended comment chatter!)
For reasons best known to itself Powershell wasn't sending the x-ms-version, so I fell back to telnet which did indeed produce the header -
HEAD /cdn/images/pier.jpg HTTP/1.1
HOST: xxxx.blob.core.windows.net
x-ms-version: 2015-04-05
HTTP/1.1 200 OK
Content-Length: 142224
Last-Modified: Tue, 01 Mar 2016 11:29:04 GMT
Accept-Ranges: bytes
ETag: "0x8D341C4B1C4F34F"
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: b4f41b01-0001-00d7-7cc9-7384c9000000
x-ms-version: 2015-04-05
x-ms-lease-status: unlocked
x-ms-lease-state: available
x-ms-blob-type: BlockBlob
Content-Disposition: attachment; filename="fname.ext"
Date: Tue, 01 Mar 2016 14:49:17 GMT
I am trying to handle a website programmatically. Lets say I visit the page www.example.com/something. On the website there is a button which I am pressing. The code of the button looks something like this:
<form action="/something" method="POST" enctype="text/plain">
<input type="submit" class="button" value="Click me" >
</form>
Pressing this button updates the information on the website.
Now I would like to do this procedure programatically to receive the content of the updated website after pressing the button.
Can someone lead me to the right direction on how to do this? preferably in C#.
Thank you in advance!
Edit:
I used Fiddler to capture the HTTP request and response, it looks like this:
POST /something HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Referer: http://example.com/something
Cookie: cookie1=cookiecontent; cookie2=cookiecontent
Connection: keep-alive
Content-Type: text/plain
Content-Length: 0
HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Encoding: gzip
Vary: Accept-Encoding
Server: Microsoft-IIS/8.0
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Thu, 05 Dec 2013 23:36:31 GMT
Content-Length: 2202
Although the requests includes cookies they don't appear to be relevant. I decompressed the received content with fiddler and found the wanted data to be included in the response.
I am not very experienced in HTTP requests and am therefore hoping that someone can help me convertion this into a C# http request to receive the content.
If the website in question is open and doesn't do any sort of cookie generation to validate requests (there are plenty of sites like this) then you can just use System.Net.WebRequest or similar to post the required form data, then examine the response. See this MSDN page for an example.
If the page does use cookies and so on you'll have to get a bit more creative. In some cases you can issue one web request to get the first page, examine the results for cookies and hidden form values and use those in your POST.
If all else fails then the Selenium WebDriver library will give you almost complete browser emulation with full access to the DOM. It's a bit more complex than using a WebRequest, but will work for pretty much everything you can use a web browser for.
Regardless of which method you use, Fiddler is a good debugging tool. Use it to compare what your C# code is doing to what the web browser is doing to see if there's anything your code isn't getting right.
Since it's a submit button then simulating the resulting HTTP Request would be easier than simulating a click. First, I would use a program like Fiddler to inspect what is being sent when you submit the form. Then I would replicate that request, just changing the values that I need changing, using HTTPWebRequest. You can find an example here.
The resultant HTTPWebResponse can then be parsed for data. Using something like HtmlAgilityPack makes that part easier.
You can do what you want with http://www.seleniumhq.org/projects/webdriver/. It is possible to do web automation with c# in a console program. I am using it for ui integration testing and it works fairly well
I would look into searching for a browser automation framework. I would usually do this in Python and have not used .Net for this, but a quick Google search yields quite a few results.
Included within these:
http://watin.org/
Web automation using .NET
Can we script and automate a browser, preferably with .Net?
I have an example URL (http://www.techspot.com/downloads.php?action=download_now&id=2991&evp=113a02f49ca8ac11b566336b984b1655&file=1). And when I click the link, the url will change to:
http://www.exisoftware.com/downloads/picture_finder/PictureFinderSetup.exe
Can anyone help me how to convert the php link to real url using vb.net or c#?
Because when I make a program to check file information in vb.net using HEAD method the file name not "PictureFinderSetup.exe" but "downloads.php?action=download_now&id=2991&evp=113a02f49ca8ac11b566336b984b1655&file=1".
It cannot be done without contacting the server. Only the server knows the exact mapping.
From Fiddler:
Request
GET http://www.techspot.com/downloads.php?action=download_now&id=2991&evp=113a02f49ca8ac11b566336b984b1655&file=1 HTTP/1.1
...
Response
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Fri, 27 Sep 2013 17:26:14 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Location: http://www.techspot.com/downloads/2991-extreme-picture-finder.html
To get the information you can use WebBrowser or HttpWebResponse to get data you need. How can I use VB.Net to read the content returned from a URL?