I'd like to cache the response I get from a HttpWebRequest. I need both the ResponseStream and the headers. Using HttpRequestCachePolicy(HttpCacheAgeControl.MaxAge, TimeSpan.FromDays(1)) when creating the request doesn't seem to work (IsFromCache is never true for some reason), and I'm a bit scared of manually caching the entire HttpWebResponse since it's containing a stream of gzipped data, and I have a bad feeling about storing streams in the ASP.NET cache.
The response is mapped to an object like this (simplified):
public readonly Stream Response;
public readonly string Etag;
private MyObject(Stream response, string etag)
{
this.Response = response;
this.Etag = etag;
}
Since the object also contains the response stream I face the same issue here.
How do I cache this?
A Stream is a pipe, not a bucket. If you need to store it, you must first get the actual contents. For example, you could read it all into a byte[]. For example:
using(var ms = new MemoryStream()) {
response.CopyTo(ms);
byte[] payload = ms.ToArray();
}
You can store the byte[] trivially. Then if you want a Stream later, you can use new MemoryStream(payload).
Can you create a custom class that contains a byte array for storing the data and another field with HttpWebRequest.Headers for the headers? Then cache that.
Related
I am using ActionFilterAttribute to get the request before hitting the controller as below :
public override void OnActionExecuting(HttpActionContext actionContext)
{
using (var stream = new MemoryStream())
{
HttpContextBase context = (HttpContextBase)actionContext.Request.Properties["MS_HttpContext"];
context.Request.InputStream.Seek(0, SeekOrigin.Begin);
context.Request.InputStream.CopyTo(stream);
requestBody = Encoding.UTF8.GetString(stream.ToArray());
}
}
The above method is working for small request but for a large json it is giving me this error :
Either BinaryRead, Form, Files, or InputStream was accessed before the internal storage was filled by the caller of HttpRequest.GetBufferedInputStream.
And the input stream gives this error
context.Request.InputStream threw an exception of type System.InvalidOperationException System.IO.Stream {System.InvalidOperationException}
As I found in my research that it is an issue with the timeout but I am unable to change the timeout in the code. I tried changing the values in the web.config file maxRequestLength="102400000" and maxAllowedContentLength="209715100" but still I am facing the same error.
If I read the GetBufferedInputStream but still same issue it is reading just a part of the buffer, not the entire stream.
I also tried the below :
Stream InStream;
int Len;
InStream = HttpContext.Current.Request.InputStream;
Len = System.Convert.ToInt32(InStream.Length);
byte[] ByteArray = new byte[Len + 1];
InStream.Seek(0, SeekOrigin.Begin);
InStream.Read(ByteArray, 0, Len);
var jsonParam = System.Text.Encoding.UTF8.GetString(ByteArray);
Note that if I set the content type application/xml or application/x-www-form-urlencoded it works, but if I set it to application/json it gives me this error!!
Please advise!
There are couple of points:
First, if you try and read 0 bytes from a stream, then it will throw a System.InvalidOperationException exception. So, I will change your code like below and add a check for ContentLength > 0.
using (var stream = new MemoryStream())
{
HttpContextBase context = (HttpContextBase)actionContext.Request.Properties["MS_HttpContext"];
if(context.Request.Contentlength > 0)
{
context.Request.InputStream.Seek(0, SeekOrigin.Begin);
context.Request.InputStream.CopyTo(stream);
requestBody = Encoding.UTF8.GetString(stream.ToArray());
}
}
Also, I once experienced the same issue and increasing the maxRequestLength in web.config seems to have resolved the issue.
This link further provides more info here
This is how I do it inside my Model binder but I'm not sure how it will work with your Action filter. I checked online and there's conflicting information; Some say you cannot read the input stream since it's not seekable and ASP.NET would need to read it to bind the model. Some say it's indeed seekable, and use the method you shared above. So the only way to figure out what would really work is to test.
I hope my code sample helps you figure it out.
object request = null;
if (actionContext.Request.Method == HttpMethod.Post && "application/json".Equals(actionContext.Request.Content.Headers.ContentType.MediaType))
{
var jsonContentTask = actionContext.Request.Content.ReadAsStringAsync();
Task.WaitAll(jsonContentTask);
string jsonContent = jsonContentTask.Result;
//... other stuff
}
I may be wrong but here is what I found. The action filters are executed after the model binding, which means the request stream has already been read. In your case i am not sure what that means. https://exceptionnotfound.net/the-asp-net-web-api-2-http-message-lifecycle-in-43-easy-steps-2/ explains the lifecycle in detail. Changing the content type wouldn't change the lifecycle events but would rather change the request content which in turn might affect the model binding.
If you have a model set for the action, then How to get current model in action filter should help. So the solution would be to get model object from the actionContext and then modify it accordingly.
I'm accessing the tomtom json api, and the api either returns me an array of objects, or a single object, when an error has happen.
Example:
[{"driverno": "...
Error Example:
{"errorCode": "8011","errorMsg": "request quota reached, error code: 8011"}
The data is accessed WebRequest, WebResponse and they return a stream, which can then be passed to a DataContractJsonSerializer. However, I can't create a serialization class, which accepts both forms of JSON, and the stream can't be passed twice, because the seek function is not supported.
Is there a way, to create a serialization class which supports both types of JSON input?
I found a workaround, where I copy the Stream to a MemoryStream, which enables seeking. I'm not completly settisfied with th solution, becuase it does a Stream copying and the DataContractJsonSerializer twice.
Sample:
string text = File.ReadAllText(PAHT);
text = Regex.Replace(text, "\\{[\\n\\r ]*\"__type", "{\"__type");
// copy to MemoryStream
using (MemoryStream dataStream = new MemoryStream(Encoding.UTF8.GetBytes(text)))
{
DataContractJsonSerializer errorDeserializer = new DataContractJsonSerializer(typeof(RequestError));
RequestError errorSerilaized = (RequestError)errorDeserializer.ReadObject(dataStream);
// check if an error happened
if (errorSerilaized.errorCode == null)
{
// seek the stream to position 0
dataStream.Position = 0;
DataContractJsonSerializer _deserializer = new DataContractJsonSerializer(typeof(NoneErrorSerializationClass));
NoneErrorSerializationClass tripReportsSerialized = (NoneErrorSerializationClass)_deserializer.ReadObject(dataStream);
// ...
}
else
{
MessageBox.Show(errorSerilaized.errorMsg);
}
}
In ASP.NET WebForms 4.5, I have WebAPI Controller with a GET method for getting a PDF.
Then down in the business layer of the application, I have an API class with a method that contains the logic for actually finding and returning the PDF to the controller.
So MyController class basically has:
public HttpResponseMessage GetStatement(string acctNumber, string stmtDate) {
MyApi myApi = new MyApi();
HttpResponseMessage response = new HttpResponseMessage(HttpStatusCode.OK);
FileStream stream = myApi.GetStatement(acctNumber, stmtDate);
...set the response.Content = stream...
... set the mime type..
... close the stream...
return response;
}
And MyApi class has:
public FileStream GetStatement(string acctNumber, string stmtDate) {
... makes an HttpWebRequest to get a PDF from another system ...
HttpWebRequest req = WebRequest.Create(......)....
FileStream stream = new FileStream(accountNumber +"_" + stmtDate + ".pdf", FileMode.Create);
response.GetResponseStream().CopyTo(stream);
return stream;
}
The API class is not in the web layer of the application because it's used by other (non-web) parts of the software.
I guess my concern is there's no explicit closing of the FileStream in the API method. I could do it in the Controller method, but I'd be relying on others to do the same when they're calling it from other areas.
Is there a better way to return the PDF file from the API method? Possibly just as a byte array or something like that? Preferably as little overhead as possible.
Thanks-
You should not be returning a file stream but instead an array of bytes. This way you can correctly dispose of the object correctly and not worry about other calling methods in the stack.
byte[] currentFile = ....
You can then deliver your file as follows this, a byte array is easy to convert to anything. Below example is for MVC4.
return new FileContentResult(currentFile, "application/pdf");
It's not uncommon for methods to return FileStreams, hoping that the caller will remember to put the method call in a using statement. But it's understandable to not want to make that assumption. One alternative is to use an interesting form of Inversion of Control, where you require the caller to give you the callback function that knows what to do with the FileStream, and then you wrap a call to that handler inside a using statement.
public T GetStatement<T>(string acctNumber, string stmtDate,
Func<FileStream, T> callback) {
... makes an HttpWebRequest to get a PDF from another system ...
HttpWebRequest req = WebRequest.Create(......)....
using(FileStream stream = new FileStream(accountNumber +"_" + stmtDate + ".pdf", FileMode.Create))
{
response.GetResponseStream().CopyTo(stream);
return callback(stream);
}
}
However, this is going to require a little extra hackery in your use case because you're returning a response whose content is feeding through the stream, so you'd need to find a way to cause your message to push out the entire response before the callback returns.
It may be best in this case to just throw comments on your method to document the fact that you're expecting the caller to ensure the stream gets closed. That's what we do in our application, and it's worked well so far.
Generally, you should put your FileStream inside a using block as already described by other answers or have some faith that it will be disposed of by other parts of your code. However, this is tricky when you are returning a FileStream from your controller. For example:
public ActionResult GetImage()
{
Stream stream = //get the stream
return base.File( stream, "image/png" );
}
Happily, the stream is disposed of by the framework once written, so you don't need to worry about the disposing of it. See here for details.
This is not an answer to your exact requirement, but thought of sharing it so that it can give you an idea of other ways of doing this.
public async Task<HttpResponseMessage> Get(string acctNumber, string stmtDate)
{
HttpResponseMessage response2 = new HttpResponseMessage();
HttpClient client= new HttpClient();
string url = "http://localhost:9090/BusinessLayer?acctNumber=" + acctNumber + "&stmtDate=" + stmtDate;
// NOTE 1: here we are only reading the response1's headers and not the body. The body of response1 would be later be
// read by response2.
HttpResponseMessage response1 = await client.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
// NOTE 2: here we need not close response1's stream as Web API would close it when disposing
// response2's stream content.
response2.Content = new StreamContent(await response1.Content.ReadAsStreamAsync());
response2.Content.Headers.ContentLength = response1.Content.Headers.ContentLength;
response2.Content.Headers.ContentType = response1.Content.Headers.ContentType;
return response2;
}
I have a PDF which is hosted in say http://test.com/mypdf.pdf.
How can I convert the PDF to Stream and then using this Stream convert it back to PDF.
I tried the following but got an exception(see image):
private static Stream ConvertToStream(string fileUrl)
{
HttpWebResponse aResponse = null;
try
{
HttpWebRequest aRequest = (HttpWebRequest)WebRequest.Create(fileUrl);
aResponse = (HttpWebResponse)aRequest.GetResponse();
}
catch (Exception ex)
{
}
return aResponse.GetResponseStream();
}
This will work:
private static Stream ConvertToStream(string fileUrl)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(fileUrl);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
try {
MemoryStream mem = new MemoryStream();
Stream stream = response.GetResponseStream();
stream.CopyTo(mem,4096);
return mem;
} finally {
response.Close();
}
}
However you are entirely responsible for the lifetime of the returned memory stream.
A better approach is:
private static void ConvertToStream(string fileUrl, Stream stream)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(fileUrl);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
try {
Stream response_stream = response.GetResponseStream();
response_stream.CopyTo(stream,4096);
} finally {
response.Close();
}
}
You can then do something like:
using (MemoryStream mem = new MemoryStream()) {
ConvertToStream('http://www.example.com/',mem);
mem.Seek(0,SeekOrigin.Begin);
... Do something else ...
}
You may also be able to return the response stream directly but you'd have to check on the lifetime of that, releasing the response may release the stream, hence the mem copy.
You may want to take a look at WebClient.DownloadFile.
You give it a URL and local file name and it saves the file straight to disk. Might save you a step or two.
You could also try WebClient.DownloadData which saves the file to an in-memory byte[].
EDIT
You did not specify the protocol of the web-service you are posting the file to. The simplest form (RESTful) would be just to POST the file to data to another URL. Here is how you would do that.
using (WebClient wc = new WebClient())
{
// copy data to byte[]
byte[] data = wc.DownloadData("http://somesite.com/your.pdf");
// POST data to another URL
wc.Headers.Add("Content-Type","application/pdf");
wc.UploadData("http://anothersite.com/your.pdf", data);
}
If you are using SOAP, you would have to convert the file to a Base64 string, but hopefully you are using a generated client which takes care of that for you. If you could elaborate on the type of web-service you are sending the file to, I could probably provide some more information..
I need to download a large file (2 GB) over HTTP in a C# console application. Problem is, after about 1.2 GB, the application runs out of memory.
Here's the code I'm using:
WebClient request = new WebClient();
request.Credentials = new NetworkCredential(username, password);
byte[] fileData = request.DownloadData(baseURL + fName);
As you can see... I'm reading the file directly into memory. I'm pretty sure I could solve this if I were to read the data back from HTTP in chunks and write it to a file on disk.
How could I do this?
If you use WebClient.DownloadFile you could save it directly into a file.
The WebClient class is the one for simplified scenarios. Once you get past simple scenarios (and you have), you'll have to fall back a bit and use WebRequest.
With WebRequest, you'll have access to the response stream, and you'll be able to loop over it, reading a bit and writing a bit, until you're done.
From the Microsoft documentation:
We don't recommend that you use WebRequest or its derived classes for
new development. Instead, use the System.Net.Http.HttpClient class.
Source: learn.microsoft.com/WebRequest
Example:
public void MyDownloadFile(Uri url, string outputFilePath)
{
const int BUFFER_SIZE = 16 * 1024;
using (var outputFileStream = File.Create(outputFilePath, BUFFER_SIZE))
{
var req = WebRequest.Create(url);
using (var response = req.GetResponse())
{
using (var responseStream = response.GetResponseStream())
{
var buffer = new byte[BUFFER_SIZE];
int bytesRead;
do
{
bytesRead = responseStream.Read(buffer, 0, BUFFER_SIZE);
outputFileStream.Write(buffer, 0, bytesRead);
} while (bytesRead > 0);
}
}
}
}
Note that if WebClient.DownloadFile works, then I'd call it the best solution. I wrote the above before the "DownloadFile" answer was posted. I also wrote it way too early in the morning, so a grain of salt (and testing) may be required.
You need to get the response stream and then read in blocks, writing each block to a file to allow memory to be reused.
As you have written it, the whole response, all 2GB, needs to be in memory. Even on a 64bit system that will hit the 2GB limit for a single .NET object.
Update: easier option. Get WebClient to do the work for you: with its DownloadFile method which will put the data directly into a file.
WebClient.OpenRead returns a Stream, just use Read to loop over the contents, so the data is not buffered in memory but can be written in blocks to a file.
i would use something like this
The connection can be interrupted, so it is better to download the file in small chunks.
Akka streams can help download file in small chunks from a System.IO.Stream using multithreading. https://getakka.net/articles/intro/what-is-akka.html
The Download method will append the bytes to the file starting with long fileStart. If the file does not exist, fileStart value must be 0.
using Akka.Actor;
using Akka.IO;
using Akka.Streams;
using Akka.Streams.Dsl;
using Akka.Streams.IO;
private static Sink<ByteString, Task<IOResult>> FileSink(string filename)
{
return Flow.Create<ByteString>()
.ToMaterialized(FileIO.ToFile(new FileInfo(filename), FileMode.Append), Keep.Right);
}
private async Task Download(string path, Uri uri, long fileStart)
{
using (var system = ActorSystem.Create("system"))
using (var materializer = system.Materializer())
{
HttpWebRequest request = WebRequest.Create(uri) as HttpWebRequest;
request.AddRange(fileStart);
using (WebResponse response = request.GetResponse())
{
Stream stream = response.GetResponseStream();
await StreamConverters.FromInputStream(() => stream, chunkSize: 1024)
.RunWith(FileSink(path), materializer);
}
}
}