I have a zip file that I can read with DotNetZipLib from the file system. However, when I POST it via a form to my MVC application it can't be read as a stream. My best guess at the moment is that the HTTP upload is somehow corrupting the zip file. There's no shortage of questions with the same problem, and I thought I'd accounted for the stream properly but perhaps I'm not using the .NET object(s) here as intended.
Here's my WebAPI POST handler:
public void Post(HttpRequestMessage request)
{
using(var fileData = request.Content.ReadAsStreamAsync().Result)
if (fileData.Length > 0)
{
var zip = ZipFile.Read(fileData); // exception
}
}
The exception, of course, is from the DotNetZipLib ZipFile just saying that the stream can't be read as a zip. If I replace fileData with just a path to the file (this is all being tested on the same machine) then it reads it, so it has to be the HTTP upload.
In FireBug, the headers for the POST are:
Response Headers:
Cache-Control no-cache
Content-Length 1100
Content-Type application/xml; charset=utf-8
Date Sat, 01 Feb 2014 23:18:32 GMT
Expires -1
Pragma no-cache
Server Microsoft-IIS/8.0
X-AspNet-Version 4.0.30319
X-Powered-By ASP.NET
X-SourceFiles =?UTF-8?B?QzpcRGF0YVxDb2RlXE9yZ1BvcnRhbFxPcmdQb3J0YWxTZXJ2ZXJcYXBpXGFwcHg=?=
Request Headers
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding gzip, deflate
Accept-Language en-US,en;q=0.5
Connection keep-alive
Cookie uvts=ENGUn8FXEnEQFeS
Host localhost:48257
Referer http://localhost:48257/Home/Test
User-Agent Mozilla/5.0 (Windows NT 6.3; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0
Request Headers From Upload Stream
Content-Length 31817
Content-Type multipart/form-data; boundary=---------------------------265001916915724
And the form is simple enough:
<form action="/api/appx" method="post" enctype="multipart/form-data">
<input name="postedFile" type="file" />
<input type="submit" />
</form>
Am I doing something wrong with the steam? Pulling data from the HttpRequestMessage incorrectly? Or perhaps I should be receiving the upload in an entirely different way?
When you post a file using a HTML form the media type is multipart/form-data which has some special formatting syntax, as you can see from your Firebug details. You can't just read it as a stream and expect it to match the file that was sent. There are a set of ReadAsMultipartAsync extension methods for handling this media type.
The below code worked fine for both Zip and Text file. you may try this out
public HttpStatusCode Post(string fileName)
{
var task = this.Request.Content.ReadAsStreamAsync();
task.Wait();
Stream requestStream = task.Result;
try
{
Stream fileStream = File.Create(HttpContext.Current.Server.MapPath("~/" + fileName));
requestStream.CopyTo(fileStream);
fileStream.Close();
requestStream.Close();
}
catch (IOException)
{
throw new HttpResponseException(HttpStatusCode.InternalServerError);
}
HttpResponseMessage response = new HttpResponseMessage();
response.StatusCode = HttpStatusCode.Created;
return response.StatusCode;
}
Related
I am trying to create an attachment using the Support Bee API as documented here:
https://supportbee.com/api#create_attachment
I have written a service that uses an HttpClient to create and send the request using a filename.
If I test in in Postman, it succeeds. I am using form-data for the body and just selecting the file to upload from the UI:
It doesn't work when I try to upload it via my HttpClient Service:
public async Task<string> CreateAttachmentAsync(string fileName)
{
// "client" is HttpClient provided via D.I.
MultipartFormDataContent content = new MultipartFormDataContent();
content.Add(new StreamContent(new FileStream(fileName, FileMode.Open)), "files[]");
using (HttpResponseMessage response = await client.PostAsync(
"https://xxx.supportbee.com/attachments?auth_token=xxx",
content))
{
string responseString = await response.Content.ReadAsStringAsync();
return responseString;
}
}
This results in a 500 Internal Server Error. Inspecting the MultipartFormDataContent object I can see that it's header values are automatically being set:
{
Content-Type: multipart/form-data; boundary="c9be3778-4de5-4460-9929-adcaa6bdda79"
Content-Length: 164
}
I have also tried reading the file to a byte array first and using ByteArrayContent instead of StreamContent to no avail. The response doesn't provide anything helpful, but since the request works in Postman I must have something wrong with my code, but I don't know what else to try.
Edit: I tested with Fiddler to compare the successful Postman request to my code. Here is the request with Postman:
POST
https://xxx.supportbee.com/attachments?auth_token=xxx
HTTP/1.1 User-Agent: PostmanRuntime/7.22.0 Accept: / Cache-Control:
no-cache Postman-Token: f84d22fa-b4b1-4bf5-b183-916a786c6385 Host:
xx.supportbee.com Content-Type: multipart/form-data;
boundary=--------------------------714700821471353664787346
Accept-Encoding: gzip, deflate, br Content-Length: 241 Connection:
close
----------------------------714700821471353664787346 Content-Disposition: form-data; name="files[]"; filename="sample.txt"
Content-Type: text/plain
This contains example text.
----------------------------714700821471353664787346--
And the failing request from my code:
POST
https://xxx.supportbee.com/attachments?auth_token=xxx
HTTP/1.1 Host: xxx.supportbee.com Accept: / Accept-Encoding:
gzip, deflate, br Connection: close Content-Type: multipart/form-data;
boundary="ea97cbc1-70ea-4cc4-9801-09f5feffc763" Content-Length: 206
--ea97cbc1-70ea-4cc4-9801-09f5feffc763 Content-Disposition: form-data; name="files[]"; filename=sample; filename*=utf-8''sample
This contains example text.
--ea97cbc1-70ea-4cc4-9801-09f5feffc763--
The difference I can see is that the individual part in Postman has its own Content-Type: text/plain header for the file, and mine doesn't. I'm unable to add this because if I try content.Headers.Add("Content-Type", "text/plain"); It fails with 'Cannot add value because header 'Content-Type' does not support multiple values.'
First, it's important to note that a 500 response is akin to an unhandled exception, i.e. it's a bug on their end and more or less impossible to know for sure what you did wrong. I would suggest reporting it to them and, although I'm not familiar with Support Bee, I would hope they have good support people who can help you troubleshoot. :)
But if you want to play the guessing game, I agree that subtle differences between your successful Postman call and your code are a good place to start. For that header, note that content is the MultipartFormDataContent. You actually want to set it on the StreamContent object.
Also, look at the request headers Postman is sending and see if Content-Disposition includes a filename. You might need to add that to your code too, if the API is expecting it.
Here's how to do both:
var fileContent = new StreamContent(File.OpenRead(path));
fileContent.Headers.ContentType = new MediaTypeHeaderValue("text/plain");
content.Add(fileContent, "files[]", Path.GetFileName(path));
If that's not the problem, look at the "raw" version of the request body in Postman, as well as those 11 request headers, and see if you can spot anything else you might be missing.
I have a small problem and I can not figure out how to solve my problem.
I need to download a file from a site in my WPF application, but my problem is i can't access direct link of file because the file is updated daily and the URL respond different file each day. link is correct and i can download in IDM or any web browser, but when I use C# WPF classes like WebClient, it download something else and when I open the file with office it says that file is corrupted. can anyone offer me how to download my excel file from this link using C# in WPF?
Here is the link
Also another problem is I don't know respond file's name, is it possible to figure out the file name too ?
I will appreciate any response, thanks a lot.
From the looks of it, the server is not respecting the Accept-Encoding header sent by the request. It just always sends the response with gzip encoding. I was able to download the file sucessfully with HttpClient, once I set AutomaticDecompression to GZip.
static void Main()
{
var task = DownloadFileAsync("http://members.tsetmc.com/tsev2/excel/MarketWatchPlus.aspx?d=0");
task.Wait();
}
static async Task DownloadFileAsync(string url)
{
HttpClient client = new HttpClient(new HttpClientHandler { AutomaticDecompression = DecompressionMethods.GZip });
HttpResponseMessage response = await client.GetAsync(url);
// Get the file name from the content-disposition header.
// This is nasty because of bug in .net: http://stackoverflow.com/questions/21008499/httpresponsemessage-content-headers-contentdisposition-is-null
string fileName = response.Content.Headers.GetValues("Content-Disposition")
.Select(h => Regex.Match(h, #"(?<=filename=).+$").Value)
.FirstOrDefault()
.Replace('/', '_');
using (FileStream file = File.Create(fileName))
{
await response.Content.CopyToAsync(file);
}
}
I had a look at the http headers that the server sends back:
Content-Encoding: gzip
Vary: *
Content-Disposition: attachment; filename=MarketWatchPlus-1393/4/25.xlsx
Content-Length: 77100
Cache-Control: public, max-age=60
Content-Type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Date: Tue, 15 Jul 2014 20:59:46 GMT
Expires: Tue, 15 Jul 2014 21:00:44 GMT
Last-Modified: Tue, 15 Jul 2014 20:59:44 GMT
Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
Notice how it says gzip there. I renamed the file to .gz and unzipped it, looks fine then. Looks that while web browsers sort that out by themselves, the WebRequest classes do not.
To answer your second question, the file name is in the headers as well.
I'm developing some API for testing, and I have a problem when I make a webrequest and especially when i retrieve the webresponse.
I use this code:
string request = HttpPost("http://iunlocker.net/check_imei.php", "ime_i=013270000134001");
public static string HttpPost(string URI, string Parameters)
{
try
{
System.Net.WebRequest req = System.Net.WebRequest.Create(URI);
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = System.Text.Encoding.ASCII.GetBytes(Parameters);
req.ContentLength = bytes.Length;
System.IO.Stream os = req.GetRequestStream();
os.Write(bytes, 0, bytes.Length);
os.Close();
System.Net.WebResponse resp= req.GetResponse();
if (resp == null) return null;
System.IO.StreamReader sr = new System.IO.StreamReader(resp.GetResponseStream());
return sr.ReadToEnd().Trim();
}
catch (Exception ex) { }
return null;
}
The website in the call is an example, because with this and with other websites I can't retrieve the result correctly. I receive an exception "Error 403"
Can anyone maybe help me by telling what I may be doing wrong?
I thought the problem was on encoding/decoding -- in fact using Fiddler it asks me if I want to decode before see the text -- but with another website, used for examples, I receive the same message from Fiddler but I can retrieve the response without a problem.
Thanks in advance.
HTTP 403 error means "access forbidden". The destination website is refusing to fulfill your request, for reasons of its own.
Given this particular website http://iunlocker.net/, I'm going to hazard a guess that it may be checking the HTTP_REFERER. In other words it's refusing to fulfill your request because it knows it didn't come from a browser that was viewing the form.
[EDIT] After viewing the response from
curl --form ime_i=013270000134001 -i http://iunlocker.net/check_imei.php
I can see that the immediate response is setting a cookie and a redirect.
HTTP/1.1 307 Temporary Redirect
Server: nginx
Date: Wed, 03 Jul 2013 04:00:27 GMT
Content-Type: text/html
Content-Length: 180
Connection: keep-alive
Set-Cookie: PMBC=35e9e4cd3a7f9d50e7f3bb39d43750d1; path=/
Location: http://iunlocker.net/check_imei.php?pmtry=1
<html>
<head><title>307 Temporary Redirect</title></head>
<body bgcolor="white">
<center><h1>307 Temporary Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>
This site does not want you scraping it; if you wish to defeat this you will have to make use of its cookies.
http://en.wikipedia.org/wiki/HTTP_403 - The web server is denying you access to that URL.
Perhaps the IP address you are using, is not allowed to access that resource. Check web server.
I have developed web api which accept files using POST method, makes manipulation and return them back using HTTP Response. The web api return additional data in http header like output file name. The problem is that then I am posting and receiving response with HttpWebResponse I get scrambled file name in response header value and unicode characters are lost.
For example if I submit наталья.docx file I get наÑалÑÑ.pdf.
The full response header
Pragma: no-cache
Transfer-Encoding: chunked
Access-Control-Allow-Origin: *
Result: True
StoreFile: false
Timeout: 300
OutputFileName: наÑалÑÑ.pdf
Content-Disposition: attachment; filename=наÑалÑÑ.pdf
Cache-Control: no-cache, no-store
Content-Type: application/pdf
Date: Wed, 12 Sep 2012 07:21:37 GMT
Expires: -1
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4
I am reading header values like this
HttpWebResponse webResponse = FormUpload.MultipartFormDataPost(postdatatoserver);
using (Stream clientResponse = webResponse.GetResponseStream())
if (webResponse.StatusCode == HttpStatusCode.OK)
{
Helpers.CopyStream(clientResponse, outStream);
webHeaderCollection = webResponse.Headers;
}
I am not sure should I just decode scrambled characters to unicode when I read them from response header or maybe I need to include encoding into response header when I send data from web api server?
See http://msdn.microsoft.com/en-us/library/system.net.webresponse.getresponsestream.aspx:
Stream ReceiveStream = myWebResponse.GetResponseStream();
Encoding enc = System.Text.Encoding.UTF8;
// Pipe the stream to a higher level stream reader with the required encoding format.
StreamReader readStream = new StreamReader(ReceiveStream, enc);
You might also try
System.Text.Encoding.Default
or
System.Text.Encoding.UTF7
or
System.Text.Encoding.Unicode
or
System.Text.Encoding.GetEncoding(1251)
or
System.Text.Encoding.GetEncoding(1252)
or
System.Text.Encoding.GetEncoding(20866)
See here for a longer list:
http://www.pcreview.co.uk/forums/system-text-encoding-getencoding-whatvalidstrings-t1406242.html
Edit:
Current [RFC 2045] grammar restricts parameter values (and hence
Content-Disposition filenames) to US-ASCII.
So the HTTP-Headers are always transmitted in ASCII format, irrespective of the StreamReader encoding.
IE doesn't conform to the standard, so there is a workaround: UrlEncode the filename
So you need to do this when you write the file back:
// IE needs url encoding, FF doesn't support it, Google Chrome doesn't care
if (Request.Browser.IsBrowser ("IE"))
{
fileName = Server.UrlEncode(fileName);
}
Response.Clear ();
Response.AddHeader ("content-disposition", String.Format ("attachment;filename=\"{0}\"", fileName));
Response.AddHeader ("Content-Length", data.Length.ToString (CultureInfo.InvariantCulture));
Response.ContentType = mimeType;
Response.BinaryWrite(data);
As per
Unicode in Content-Disposition header
you can add an asterisk, and append the proper encoding.
I'm working on a small C#/WPF application that interfaces with a web service implemented in Ruby on Rails, using handcrafted HttpWebRequest calls and JSON serialization. Without caching, everything works as it's supposed to, and I've got HTTP authentication and compression working as well.
Once I enable caching, by setting request.CachePolicy = new HttpRequestCachePolicy(HttpRequestCacheLevel.CacheIfAvailable);, things go awry - in the production environment. When connecting to a simple WEBrick instance, things work fine, I get HTTP/1.1 304 Not Modified as expected and HttpWebRequest delivers the cached content.
When I try the same against the production server, running nginx/0.8.53 + Phusion Passenger 3.0.0, the application breaks. First request (uncached) is served properly, but on the second request which results in the 304 response, I get a WebException stating that "The request was aborted: The request was canceled." as soon as I invoke request.GetResponse().
I've run the connections through fiddler, which hasn't helped a whole lot; both WEBrick and nginx return an empty entity body, albeit different response headers. Intercepting the request and changing the response headers for nginx to match those of WEBrick didn't change anything, leading me to think that it could be a keep-alive issue; setting request.KeepAlive = false; changes nothing, though - it doesn't break stuff when connecting to WEBrick, and it doesn't fix stuff when connecting to nginx.
For what it's worth, the WebException.InnerException is a NullReferenceException with the following StackTrace:
at System.Net.HttpWebRequest.CheckCacheUpdateOnResponse()
at System.Net.HttpWebRequest.CheckResubmitForCache(Exception& e)
at System.Net.HttpWebRequest.DoSubmitRequestProcessing(Exception& exception)
at System.Net.HttpWebRequest.ProcessResponse()
at System.Net.HttpWebRequest.SetResponse(CoreResponseData coreResponseData)
Headers for the (working) WEBrick connection:
########## request
GET /users/current.json HTTP/1.1
Authorization: Basic *REDACTED*
Content-Type: application/json
Accept: application/json
Accept-Charset: utf-8
Host: testbox.local:3030
If-None-Match: "84a49062768e4ca619b1c081736da20f"
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
########## response
HTTP/1.1 304 Not Modified
X-Ua-Compatible: IE=Edge
Etag: "84a49062768e4ca619b1c081736da20f"
Date: Wed, 01 Dec 2010 18:18:59 GMT
Server: WEBrick/1.3.1 (Ruby/1.8.7/2010-08-16)
X-Runtime: 0.177545
Cache-Control: max-age=0, private, must-revalidate
Set-Cookie: *REDACTED*
Headers for the (exception-throwing) nginx connection:
########## request
GET /users/current.json HTTP/1.1
Authorization: Basic *REDACTED*
Content-Type: application/json
Accept: application/json
Accept-Charset: utf-8
Host: testsystem.local:8080
If-None-Match: "a64560553465e0270cc0a23cc4c33f9f"
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
########## response
HTTP/1.1 304 Not Modified
Connection: keep-alive
Status: 304
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 3.0.0
ETag: "a64560553465e0270cc0a23cc4c33f9f"
X-UA-Compatible: IE=Edge,chrome=1
X-Runtime: 0.240160
Set-Cookie: *REDACTED*
Cache-Control: max-age=0, private, must-revalidate
Server: nginx/0.8.53 + Phusion Passenger 3.0.0 (mod_rails/mod_rack)
UPDATE:
I tried doing a quick-and-dirty manual ETag cache, but turns out that's a no-go: I get a WebException when invoking request.GetResponce(), telling me that "The remote server returned an error: (304) Not Modified." - yeah, .NET, I kinda knew that, and I'd like to (attempt to) handle it myself, grr.
UPDATE 2:
Getting closer to the root of the problem. The showstopper seems to be a difference in the response headers for the initial request. WEBrick includes a Date: Wed, 01 Dec 2010 21:30:01 GMT header, which isn't present in the nginx reply. There's other differences as well, but intercepting the initial nginx reply with fiddler and adding a Date header, the subsequent HttpWebRequests are able to process the (unmodified) nginx 304 replies.
Going to try to look for a workaround, as well as getting nginx to add the Date header.
UPDATE 3:
It seems that the serverside issue is with Phusion Passenger, they have an open issue about lack of the Date header. I'd still say that HttpWebRequest's behavior is... suboptimal.
UPDATE 4:
Added a Microsoft Connect ticket for the bug.
I think the designers find it reasonable to throw an exception when the "expected behavior"---i.e., getting a response body---cannot be completed. You can handle this somewhat intelligently as follows:
catch (WebException ex)
{
if (ex.Status == WebExceptionStatus.ProtocolError)
{
var statusCode = ((HttpWebResponse)ex.Response).StatusCode;
// Test against HttpStatusCode enumeration.
}
else
{
// Do something else, e.g. throw;
}
}
So, it turns out to be Phusion Passenger (or nginx, depending on how you look at it - and Thin as well) that doesn't add a Date HTTP response header, combined with what I see as a bug in .NET HttpWebRequest (in my situation there's no If-Modified-Since, thus Date shouldn't be necessary) leading to the problem.
The workaround for this particular case was to edit our Rails ApplicationController:
class ApplicationController < ActionController::Base
# ...other stuff here
before_filter :add_date_header
# bugfix for .NET HttpWebRequst 304-handling bug and various
# webservers' lazyness in not adding the Date: response header.
def add_date_header
response.headers['Date'] = Time.now.to_s
end
end
UPDATE:
Turns out it's a bit more complex than "just" setting HttpRequestCachePolicy - to repro, I also need to have manually constructed HTTP Basic Auth. So the involved components are the following:
HTTP server that doesn't include a HTTP "Date:" response header.
manual construction of HTTP Authorization request header.
use of HttpRequestCachePolicy.
Smallest repro I've been able to come up with:
namespace Repro
{
using System;
using System.IO;
using System.Net;
using System.Net.Cache;
using System.Text;
class ReproProg
{
const string requestUrl = "http://drivelog.miracle.local:3030/users/current.json";
// Manual construction of HTTP basic auth so we don't get an unnecessary server
// roundtrip telling us to auth, which is what we get if we simply use
// HttpWebRequest.Credentials.
private static void SetAuthorization(HttpWebRequest request, string _username, string _password)
{
string userAndPass = string.Format("{0}:{1}", _username, _password);
byte[] authBytes = Encoding.UTF8.GetBytes(userAndPass.ToCharArray());
request.Headers["Authorization"] = "Basic " + Convert.ToBase64String(authBytes);
}
static public void DoRequest()
{
var request = (HttpWebRequest) WebRequest.Create(requestUrl);
request.Method = "GET";
request.CachePolicy = new HttpRequestCachePolicy(HttpRequestCacheLevel.CacheIfAvailable);
SetAuthorization(request, "user#domain.com", "12345678");
using(var response = request.GetResponse())
using(var stream = response.GetResponseStream())
using(var reader = new StreamReader(stream))
{
string reply = reader.ReadToEnd();
Console.WriteLine("########## Server reply: {0}", reply);
}
}
static public void Main(string[] args)
{
DoRequest(); // works
DoRequest(); // explodes
}
}
}