Upload file using a virtual path provider and Amazon S3 SDK - c#

The background to this question is based on a virtual file system I'm developing. The concept I'm using is virutal path providers for different types of storage type i.e local file system, dropbox and amazon s3. My base class for a virtual file looks like this:
public abstract class CommonVirtualFile : VirtualFile {
public virtual string Url {
get { throw new NotImplementedException(); }
}
public virtual string LocalPath {
get { throw new NotImplementedException(); }
}
public override Stream Open() {
throw new NotImplementedException();
}
public virtual Stream Open(FileMode fileMode) {
throw new NotImplementedException();
}
protected CommonVirtualFile(string virtualPath) : base(virtualPath) { }
}
The implementation of the second Open method is what my question is all about. If we look at my implementation for the local file system i.e saving a file on disk it looks like this:
public override Stream Open(FileMode fileMode) {
return new FileStream("The_Path_To_The_File_On_Disk"), fileMode);
}
If I would like to save a file on the local file system this would look something like this:
const string virtualPath = "/assets/newFile.txt";
var file = HostingEnvironment.VirtualPathProvider.GetFile(virtualPath) as CommonVirtualFile;
if (file == null) {
var virtualDir = VirtualPathUtility.GetDirectory(virtualPath);
var directory = HostingEnvironment.VirtualPathProvider.GetDirectory(virtualDir) as CommonVirtualDirectory;
file = directory.CreateFile(VirtualPathUtility.GetFileName(virtualPath));
}
byte[] fileContent;
using (var fileStream = new FileStream(#"c:\temp\fileToCopy.txt", FileMode.Open, FileAccess.Read)) {
fileContent = new byte[fileStream.Length];
fileStream.Read(fileContent, 0, fileContent.Length);
}
// write the content to the local file system
using (Stream stream = file.Open(FileMode.Create)) {
stream.Write(fileContent, 0, fileContent.Length);
}
What I want is that if I switch to my amazon s3 virtual path provider I want this code to work directly without any changes so to sum things up, how can I solve this using the amazon s3 sdk and how should i implement my Open(FileMode fileMode) method in my amazon s3 virtual path provider?

Hey i stood for this problem, too, and i solved it implementing a stream.
Here is my way i did it maybe it helps:
public static Stream OpenStream(S3TransferUtility transferUtility, string key)
{
byte[] buffer = new byte[Buffersize + Buffersize/2];
S3CopyMemoryStream s3CopyStream =
new S3CopyMemoryStream(key, buffer, transferUtility)
.WithS3CopyFileStreamEvent(CreateMultiPartS3Blob);
return s3CopyStream;
}
My Stream with constructor overrides the close and write(array, offset, count) methods and upload the stream to amazon s3 partly.
public class S3CopyMemoryStream : MemoryStream
{
public S3CopyMemoryStream WithS3CopyFileStreamEvent(StartUploadS3CopyFileStreamEvent doing)
{
S3CopyMemoryStream s3CopyStream = new S3CopyMemoryStream(this._key, this._buffer, this._transferUtility);
s3CopyStream.StartUploadS3FileStreamEvent = new S3CopyMemoryStream.StartUploadS3CopyFileStreamEvent(CreateMultiPartS3Blob);
return s3CopyStream;
}
public S3CopyMemoryStream(string key, byte[] buffer, S3TransferUtility transferUtility)
: base(buffer)
{
if (buffer.LongLength > int.MaxValue)
throw new ArgumentException("The length of the buffer may not be longer than int.MaxValue", "buffer");
InitiatingPart = true;
EndOfPart = false;
WriteCount = 1;
PartETagCollection = new List<PartETag>();
_buffer = buffer;
_key = key;
_transferUtility = transferUtility;
}
The event StartUploadS3FileStreamEvent invokes a call that initiate, uploadpart and complete the upload.
Alternatively you could implement a FileStream which is much easier because you can use
TransferUtilityUploadRequest request =
new TransferUtilityUploadRequest()
.WithAutoCloseStream(false).WithBucketName(
transferUtility.BucketName)
.WithKey(key)
.WithPartSize(stream.PartSize)
.WithInputStream(stream) as TransferUtilityUploadRequest;
transferUtility.Upload(request);
at the close method of the overriden FileStream. The disadvantage is that you have to write the whole data to the disk first and then you can upload it.

Related

How to allow multiple threads to read and write to a file without locking issues

I have a web application that checks for the modification of a config.json file on each page load. It checks the modified date of the file and compares it against the last processing time that was recorded. If it differs, it proceeds to allow the reading of the file contents, processing those contents and updating config.json as well as writing to a separate file. I want to ensure that multiple simultaneous connections both reading and writing to these two files won't cause an issue.
var lastWriteTime = File.GetLastWriteTimeUtc(ConfigJsonPath);
if (CacheTime != lastWriteTime)
{
var config = new ConfigWriter().ReadData(ConfigJsonPath);
var model = JsonConvert.DeserializeObject<StyleModel>(config);
// This method writes to a another file
ProcessConfig(model, page);
var serialized = JsonConvert.SerializeObject(model, Formatting.Indented);
new ConfigWriter().WriteData(serialized, ConfigJsonPath);
CacheTime = File.GetLastWriteTimeUtc(ConfigJsonPath);
}
ConfigWriter Class
public class ConfigWriter
{
private static readonly ReaderWriterLockSlim Lock = new ReaderWriterLockSlim();
public void WriteData(string data, string path)
{
Lock.EnterWriteLock();
try
{
using (var fs = new FileStream(path, FileMode.OpenOrCreate, FileAccess.Write))
{
fs.SetLength(0);
var dataAsByteArray = new UTF8Encoding(true).GetBytes(data);
fs.Write(dataAsByteArray , 0, dataAsByteArray .Length);
}
}
finally
{
Lock.ExitWriteLock();
}
}
public string ReadData(string filePath)
{
Lock.EnterReadLock();
try
{
string config;
using (var fs = new FileStream(filePath, FileMode.Open, FileAccess.Read))
{
using (var r = new StreamReader(fs, Encoding.UTF8))
{
config = r.ReadToEnd();
}
}
return config;
}
finally
{
Lock.ExitReadLock();
}
}
}
So in the end there is one file being read, and two files being altered (write), and I'm using the same lock on the write.
Did I overdo it here? I started with a simple filestream with FileShare set to read/write, and then I got paranoid and second guessing myself. Is there a better way to implement this?

Request big file stream failed with wcf rest

I have a REST GET API that is written using WCF library to return Stream of a specific requested file that is located on API server that hosts that web service application. The service works well if the size of the requested file is small; that is less than 100 MB. But if file size is greater than > 100 MB, then the service returns 0 bytes without any logged information I can get the library method (saying, the "catch" block).
The library method (the class library project) returns Stream of needed file is
public Stream GetFile(string fileId, string seekStartPosition=null)
{
_lastActionResult = string.Empty;
Stream fileStream = null;
try
{
Guid fileGuid;
if (Guid.TryParse(fileId, out fileGuid) == false)
{
_lastActionResult = string.Format(ErrorMessage.FileIdInvalidT, fileId);
}
else
{
ContentPackageItemService contentItemService = new ContentPackageItemService();
string filePath = DALCacheHelper.GetFilePath(fileId);
if (File.Exists(filePath))
{
fileStream = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.Read);
long seekStart = 0;
// if seek position is specified, move the stream pointer to that location
if (string.IsNullOrEmpty(seekStartPosition) == false && long.TryParse(seekStartPosition, out seekStart))
{
// make sure seek position is smaller than file size
FileInfo fi = new FileInfo(filePath);
if (seekStart >= 0 && seekStart < fi.Length)
{
fileStream.Seek(seekStart, SeekOrigin.Begin);
}
else
{
_lastActionResult = string.Format(ErrorMessage.FileSeekInvalidT, seekStart, fi.Length);
}
}
}
else
{
_lastActionResult = string.Format(ErrorMessage.FileNotFoundT, fileId);
Logger.Write(_lastActionResult,
"General", 1, Constants.LogId.RESTSync, System.Diagnostics.TraceEventType.Error, System.Reflection.MethodBase.GetCurrentMethod().Name);
}
}
}
catch(Exception ex)
{
Logger.Write(ex,"General", 1, Constants.LogId.RESTSync, System.Diagnostics.TraceEventType.Error, System.Reflection.MethodBase.GetCurrentMethod().Name);
}
return fileStream;
}
API method on the client side project (where .svc file is):
[WebGet(UriTemplate = "files/{fileid}")]
public Stream GetFile(string fileid)
{
ContentHandler handler = new ContentHandler();
Stream fileStream = null;
try
{
fileStream = handler.GetFile(fileid);
}
catch (Exception ex)
{
Logger.Write(string.Format("{0} {1}", ex.Message, ex.StackTrace), "General", 1, Constants.LogId.RESTSync, System.Diagnostics.TraceEventType.Error, System.Reflection.MethodBase.GetCurrentMethod().Name);
throw new WebFaultException<ErrorResponse>(new ErrorResponse(HttpStatusCode.InternalServerError, ex.Message), HttpStatusCode.InternalServerError);
}
if (fileStream == null)
{
throw new WebFaultException<ErrorResponse>(new ErrorResponse(handler.LastActionResult), HttpStatusCode.InternalServerError);
}
return fileStream;
}
As you are using REST, I presume you are using the WebHttpBinding. You need to set the MaxReceivedMessageSize on the client binding to be sufficient for the maximum expected response size. The default is 64K. Here's the msdn documentation for the property if you are creating your binding in code. If you are creating your binding in your app.config, then this is the documentation you need.

Create a filestream without a file c#

Is it possible to create a filestream without an actual file?
I'll try to explain:
I know how to create a stream from a real file:
FileStream s = new FileStream("FilePath", FileMode.Open, FileAccess.Read);
But can I create a fileStream with a fake file?
meaning:
define properties such as name, type, size, whatever else is necessary, to some file object (is there such thing?), without a content, just all the properties,
and after that to create a fileStream from this "file"? to have the result similar to the above code?
edit.
I am using an API sample that has that code:
FileStream s = new FileStream("FilePath", FileMode.Open, FileAccess.Read);
try
{
SolFS.SolFSStream stream = new SolFS.SolFSStream(Storage, FullName, true, false, true, true, true, "pswd", SolFS.SolFSEncryption.ecAES256_SHA256, 0);
try
{
byte[] buffer = new byte[1024*1024];
long ToRead = 0;
while (s.Position < s.Length)
{
if (s.Length - s.Position < 1024*1024)
ToRead = s.Length - s.Position;
else
ToRead = 1024 * 1024;
s.Read(buffer, 0, (int) ToRead);
stream.Write(buffer, 0, (int) ToRead);
}
So it is basically writes fileStream "s" somewhere.
I don't want to take an existing file and write it, but I want to "create" a different file without the content (I don't need the content) but to have the properties of the real file such as size, name, type
Apparently, you want to have a FileStream (explicitly with its FileStream-specific properties such as Name) that does not point to a file.
This is, to my knowledge, not possible based on the implementation of FileStream.
However, creating a wrapper class with the required properties would be a straightforward solution:
You could store all the properties you need in the wrapper.
The wrapper could wrap an arbitrary Stream, so you would be free to choose between FileStream, MemoryStream, or any other stream type.
Here is an example:
public class StreamContainer
{
public StreamContainer(string name, Stream contents)
{
if (name == null) {
throw new ArgumentNullException("name");
}
if (contents == null) {
throw new ArgumentNullException("contents");
}
this.name = name;
this.contents = contents;
}
private readonly string name;
public string Name {
get {
return name;
}
}
private readonly Stream contents;
public Stream Contents {
get {
return contents;
}
}
}
Of course, you could then add some courtesy creation methods for various stream types (as static methods in the above class):
public static StreamContainer CreateForFile(string path)
{
return new StreamContainer(path, new FileStream(path, FileMode.Open, FileAccess.Read));
}
public static StreamContainer CreateWithoutFile(string name)
{
return new StreamContainer(name, new MemoryStream());
}
In your application, whereever you want to use such a named stream, pass around the StreamContainer rather than expecting a Stream directly.

Writing to ZipArchive using the HttpContext OutputStream

I've been trying to get the "new" ZipArchive included in .NET 4.5 (System.IO.Compression.ZipArchive) to work in a ASP.NET site. But it seems like it doesn't like writing to the stream of HttpContext.Response.OutputStream.
My following code example will throw
System.NotSupportedException: Specified method is not supported
as soon as a write is attempted on the stream.
The CanWrite property on the stream returns true.
If I exchange the OutputStream with a filestream, pointing to a local directory, it works. What gives?
ZipArchive archive = new ZipArchive(HttpContext.Response.OutputStream, ZipArchiveMode.Create, false);
ZipArchiveEntry entry = archive.CreateEntry("filename");
using (StreamWriter writer = new StreamWriter(entry.Open()))
{
writer.WriteLine("Information about this package.");
writer.WriteLine("========================");
}
Stacktrace:
[NotSupportedException: Specified method is not supported.]
System.Web.HttpResponseStream.get_Position() +29
System.IO.Compression.ZipArchiveEntry.WriteLocalFileHeader(Boolean isEmptyFile) +389
System.IO.Compression.DirectToArchiveWriterStream.Write(Byte[] buffer, Int32 offset, Int32 count) +94
System.IO.Compression.WrappedStream.Write(Byte[] buffer, Int32 offset, Int32 count) +41
Note: This has been fixed in .Net Core 2.0. I'm not sure what is the status of the fix for .Net Framework.
Calbertoferreira's answer has some useful information, but the conclusion is mostly wrong. To create an archive, you don't need seek, but you do need to be able to read the Position.
According to the documentation, reading Position should be supported only for seekable streams, but ZipArchive seems to require this even from non-seekable streams, which is a bug.
So, all you need to do to support writing ZIP files directly to OutputStream is to wrap it in a custom Stream that supports getting Position. Something like:
class PositionWrapperStream : Stream
{
private readonly Stream wrapped;
private long pos = 0;
public PositionWrapperStream(Stream wrapped)
{
this.wrapped = wrapped;
}
public override bool CanSeek { get { return false; } }
public override bool CanWrite { get { return true; } }
public override long Position
{
get { return pos; }
set { throw new NotSupportedException(); }
}
public override void Write(byte[] buffer, int offset, int count)
{
pos += count;
wrapped.Write(buffer, offset, count);
}
public override void Flush()
{
wrapped.Flush();
}
protected override void Dispose(bool disposing)
{
wrapped.Dispose();
base.Dispose(disposing);
}
// all the other required methods can throw NotSupportedException
}
Using this, the following code will write a ZIP archive into OutputStream:
using (var outputStream = new PositionWrapperStream(Response.OutputStream))
using (var archive = new ZipArchive(outputStream, ZipArchiveMode.Create, false))
{
var entry = archive.CreateEntry("filename");
using (var writer = new StreamWriter(entry.Open()))
{
writer.WriteLine("Information about this package.");
writer.WriteLine("========================");
}
}
If you compare your code adaptation with the version presented in MSDN page you'll see that the ZipArchiveMode.Create is never used, what is used is ZipArchiveMode.Update.
Despite that, the main problem is the OutputStream that doesn't support Read and Seek which is need by the ZipArchive in Update Mode:
When you set the mode to Update, the underlying file or stream must
support reading, writing, and seeking. The content of the entire
archive is held in memory, and no data is written to the underlying
file or stream until the archive is disposed.
Source: MSDN
You weren't getting any exceptions with the create mode because it only needs to write:
When you set the mode to Create, the underlying file or stream must support writing, but does not have to support seeking. Each entry in the archive can be opened only once for writing. If you create a single entry, the data is written to the underlying stream or file as soon as it is available. If you create multiple entries, such as by calling the CreateFromDirectory method, the data is written to the underlying stream or file after all the entries are created.
Source: MSDN
I believe you can't create a zip file directly in the OutputStream since it's a network stream and seek is not supported:
Streams can support seeking. Seeking refers to querying and modifying the current position within a stream. Seek capability depends on the kind of backing store a stream has. For example, network streams have no unified concept of a current position, and therefore typically do not support seeking.
An alternative could be writing to a memory stream, then use the OutputStream.Write method to send the zip file.
MemoryStream ZipInMemory = new MemoryStream();
using (ZipArchive UpdateArchive = new ZipArchive(ZipInMemory, ZipArchiveMode.Update))
{
ZipArchiveEntry Zipentry = UpdateArchive.CreateEntry("filename.txt");
foreach (ZipArchiveEntry entry in UpdateArchive.Entries)
{
using (StreamWriter writer = new StreamWriter(entry.Open()))
{
writer.WriteLine("Information about this package.");
writer.WriteLine("========================");
}
}
}
byte[] buffer = ZipInMemory.GetBuffer();
Response.AppendHeader("content-disposition", "attachment; filename=Zip_" + DateTime.Now.ToString() + ".zip");
Response.AppendHeader("content-length", buffer.Length.ToString());
Response.ContentType = "application/x-compressed";
Response.OutputStream.Write(buffer, 0, buffer.Length);
EDIT: With feedback from comments and further reading, you could be creating large Zip files, so the memory stream could cause you problems.
In this case i suggest you create the zip file on the web server then output the file using Response.WriteFile .
A refinement to svick's answer of 2nd February 2014. I found that it was necessary to implement some more methods and properties of the Stream abstract class and to declare the pos member as long. After that it worked like a charm. I haven't extensively tested this class, but it works for the purposes of returning a ZipArchive in the HttpResponse. I assume I've implemented Seek and Read correctly, but they may need some tweaking.
class PositionWrapperStream : Stream
{
private readonly Stream wrapped;
private long pos = 0;
public PositionWrapperStream(Stream wrapped)
{
this.wrapped = wrapped;
}
public override bool CanSeek
{
get { return false; }
}
public override bool CanWrite
{
get { return true; }
}
public override long Position
{
get { return pos; }
set { throw new NotSupportedException(); }
}
public override bool CanRead
{
get { return wrapped.CanRead; }
}
public override long Length
{
get { return wrapped.Length; }
}
public override void Write(byte[] buffer, int offset, int count)
{
pos += count;
wrapped.Write(buffer, offset, count);
}
public override void Flush()
{
wrapped.Flush();
}
protected override void Dispose(bool disposing)
{
wrapped.Dispose();
base.Dispose(disposing);
}
public override long Seek(long offset, SeekOrigin origin)
{
switch (origin)
{
case SeekOrigin.Begin:
pos = 0;
break;
case SeekOrigin.End:
pos = Length - 1;
break;
}
pos += offset;
return wrapped.Seek(offset, origin);
}
public override void SetLength(long value)
{
wrapped.SetLength(value);
}
public override int Read(byte[] buffer, int offset, int count)
{
pos += offset;
int result = wrapped.Read(buffer, offset, count);
pos += count;
return result;
}
}
An simplified version of svick's answer for zipping a server-side file and sending it via the OutputStream:
using (var outputStream = new PositionWrapperStream(Response.OutputStream))
using (var archive = new ZipArchive(outputStream, ZipArchiveMode.Create, false))
{
var entry = archive.CreateEntryFromFile(fullPathOfFileOnDisk, fileNameAppearingInZipArchive);
}
(In case this seems obvious, it wasn't to me!)
Presumably this is not an MVC app, where you could easily just use the FileStreamResult class.
I'm using this currently with ZipArchive created using a MemoryStream, so I know it works.
With that in mind, have a look at the FileStreamResult.WriteFile() method:
protected override void WriteFile(HttpResponseBase response)
{
// grab chunks of data and write to the output stream
Stream outputStream = response.OutputStream;
using (FileStream)
{
byte[] buffer = newbyte[_bufferSize];
while (true)
{
int bytesRead = FileStream.Read(buffer, 0, _bufferSize);
if (bytesRead == 0)
{
// no more data
break;
}
outputStream.Write(buffer, 0, bytesRead);
}
}
}
(Entire FileStreamResult on CodePlex)
Here is how I'm generating and returning the ZipArchive.
You should have no issues replacing the FSR with the guts of the WriteFile method from above, where FileStream becomes resultStream from the code below:
var resultStream = new MemoryStream();
using (var zipArchive = new ZipArchive(resultStream, ZipArchiveMode.Create, true))
{
foreach (var doc in req)
{
var fileName = string.Format("Install.Rollback.{0}.v{1}.docx", doc.AppName, doc.Version);
var xmlData = doc.GetXDocument();
var fileStream = WriteWord.BuildFile(templatePath, xmlData);
var docZipEntry = zipArchive.CreateEntry(fileName, CompressionLevel.Optimal);
using (var entryStream = docZipEntry.Open())
{
fileStream.CopyTo(entryStream);
}
}
}
resultStream.Position = 0;
// add the Response Header for downloading the file
var cd = new ContentDisposition
{
FileName = string.Format(
"{0}.{1}.{2}.{3}.Install.Rollback.Documents.zip",
DateTime.Now.Year, DateTime.Now.Month, DateTime.Now.Day, (long)DateTime.Now.TimeOfDay.TotalSeconds),
// always prompt the user for downloading, set to true if you want
// the browser to try to show the file inline
Inline = false,
};
Response.AppendHeader("Content-Disposition", cd.ToString());
// stuff the zip package into a FileStreamResult
var fsr = new FileStreamResult(resultStream, MediaTypeNames.Application.Zip);
return fsr;
Finally, if you will be writing large streams (or a larger number of them at any given time), then you may want to consider using anonymous pipes to write the data to the output stream immediately after you write it to the underlying stream in the zip file. Because you will be holding all the file contents in memory on the server. The end of this answer to a similar question has a nice explanation of how to do that.

IFilter ErrorCode FILTER_E_ACCESS when reading .rtf (Rick Text format) from byte array

I'm using this http://www.codeproject.com/Articles/31944/Implementing-a-TextReader-to-extract-various-files and it's mostly working.
I wrote this test to check if the Filter would read as expected from a byte array.
private const string ExpectedText = "This is a test!";
[Test]
public void FilterReader_RtfBytes_TextMatch()
{
var bytes = File.ReadAllBytes(#"Test Documents\DocTest.rtf");
var reader = new FilterReader(bytes, ".rtf");
reader.Init();
var actualText = reader.ReadToEnd();
StringAssert.Contains(ExpectedText, actualText);
}
The test fails with ErrorCode : FILTER_E_ACCESS, it works fine when I give it the filename.
new FilterReader(#"Test Documents\DocTest.rtf", ".rtf"); <-- works
I'm puzzled as to why that is. I looked through the code and it seems it's the rtf filter dll that returns the error. Which is even more puzzling.
It works fine for other file types, like; .doc, .docx, .pdf
under the hood the using of concrete way of work with iFilter is defined by constructor: when you use constructor FilterReader(byte[] bytes, string extension) is used IPersistStream for content loading from memory, when FilterReader(string path, string extension) - IPersistFile for loading from file.
why rtf-ifilter returns error when used with IPersistStream i'm afraid we will not able to know because source code isn't opened
in my case i encapsulate filter's specific in constructor and refactor code next way:
remove all constructors
remove public void Init()-method
implement one custom constructor public FilterReader(string fileName, string extension, uint blockSize = 0x2000):
#region Contracts
Contract.Requires(!string.IsNullOrEmpty(fileName));
Contract.Requires(!string.IsNullOrEmpty(extension));
Contract.Requires(blockSize > 1);
#endregion
const string rtfExtension = ".rtf";
FileName = fileName;
Extension = extension;
BufferSize = blockSize;
_buffer = new char[ActBufferSize];
// ! Take into account that Rtf-file can be loaded only using IPersistFile.
var doUseIPersistFile = string.Compare(rtfExtension, extension, StringComparison.InvariantCultureIgnoreCase) == 0;
// Initialize _filter instance.
try
{
if (doUseIPersistFile)
{
// Load content using IPersistFile.
_filter = FilterLoader.LoadIFilterFromIPersistFile(FileName, Extension);
}
else
{
// Load content using IPersistStream.
using (var stream = new FileStream(path: fileName, mode: FileMode.Open, access: FileAccess.Read, share: FileShare.Read))
{
var buffer = new byte[stream.Length];
stream.Read(buffer, 0, buffer.Length);
_filter = FilterLoader.LoadIFilterFromStream(buffer, Extension);
}
}
}
catch (FileNotFoundException)
{
throw;
}
catch (Exception e)
{
throw new AggregateException(message: string.Format("Filter Not Found or Loaded for extension '{0}'.", Extension), innerException: e);
}

Categories

Resources