Amazon S3 Save Response Stream - c#

I am trying to load a .gz file out of a bucket.
Connection and authentication work finde, I even do get a file, but the problem is, the file is a lot bigger then the file should be. it is, original size, 155MB within the bucket but when it comes onto my hard disk it gets up to about 288MB
here is the function code:
public bool SaveBucketToFile(string Filename)
{
//Response check into file
using (StreamReader StRead = new StreamReader(_ObjResponse.ResponseStream))
{
string TempFile = Path.GetTempFileName();
StreamWriter StWrite = new StreamWriter(TempFile, false);
StWrite.Write(StRead.ReadToEnd());
StWrite.Close();
StRead.Close();
// Move to real destination
if (File.Exists(Filename))
{
File.Delete(Filename);
}
File.Move(TempFile, Filename);
}
return true;
}
the download and filling of _ObjResponse is made over usage of the AmazonS3 Client from their SDK. I am using a proxy but the same code on a different machine without proxy brings back the same result.
Any hints what to do here? the object request is simple:
_ObjRequest = new GetObjectRequest
{
BucketName = BucketName,
Key = Key
};
glad for any help...

for everyone to stumble upon this.
I needed to first save the stream via bufferedStream into a memorystream.
the code looks like this:
MemoryStream MemStream = new MemoryStream();
BufferedStream Stream2 = new BufferedStream(_ObjResponse.ResponseStream);
byte[] Buffer = new byte[0x2000];
int Count;
while ((Count = Stream2.Read(Buffer, 0, Buffer.Length)) > 0)
{
MemStream.Write(Buffer, 0, Count);
}
// Pfad auslesen
string TempFile = Path.GetTempFileName();
//Stream zum Tempfile öffnen
FileStream Newfile = new FileStream(TempFile,FileMode.Create);
//Stream wieder auf Position 0 ziehen
MemStream.Position = 0;
// in Tempdatei speichern
MemStream.CopyTo(Newfile);
Newfile.Close();
// Endgültigen Speicherpunkt prüfen und Tempdatei dorthin schieben
if (File.Exists(Filename))
{
File.Delete(Filename);
}
File.Move(TempFile, Filename);
I found this somewhere here:
http://www.codeproject.com/Articles/186132/Beginning-with-Amazon-S under the Caption "Get a file from Amazon S3"

Related

The server does not allow messages larger than 5242880 bytes

Elmah has recently reported this bug;
Microsoft.SharePoint.Client.ServerException: The request message is too big. The server does not allow messages larger than 5242880 bytes.
The code where it fell over was;
public SharepointFileInfo Save(byte[] file, string fileName)
{
using (var context = new ClientContext(this.SharepointServer))
{
context.Credentials = new NetworkCredential(this.UserName, this.Password, this.Domain);
var list = context.Web.Lists.GetByTitle(this.DocumentLibrary);
var fileCreationInformation = new FileCreationInformation
{
Content = file,
Overwrite = true,
Url = fileName
};
var uploadFile = list.RootFolder.Files.Add(fileCreationInformation);
var listItem = uploadFile.ListItemAllFields;
listItem.Update();
context.ExecuteQuery();
if (this.Metadata.Count > 0)
{
this.SaveMetadata(uploadFile, context);
}
return GetSharepointFileInfo(context, list, uploadFile);
}
}
I am using Sharepoint 2013.
How do I fix this?
It's a normal problme. You use the classic API (new FileCreationInformation [...] context.ExecuteQuery()) which sent a HTTP requet to the server. You file is up to 5 Mb. So, IIS receive a huge request, and reject it.
To Upload a file to SharePoint you need to use :
File.SaveBinaryDirect
(with this you don't need to change settings ;) )
using (FileStream fs = new FileStream(filePath, FileMode.Open))
{
Microsoft.SharePoint.Client.File.SaveBinaryDirect(ctx, string.Format("/{0}/{1}", libraryName, System.IO.Path.GetFileName(filePath)), fs, true);
}
Check this links to see how to upload a file to SharePoint using CSOM :
Upload large files sample app for SharePoint
good luck
There is several approche to do that (upload file with metaData). I propose 2 methods to you (one simple, second more complex)
In 2 Times (simple)
Upload the the file with the File.SaveBinaryDirect
Get the SPFile with CSOM by the file URL with SP.Web.getFileByServerRelativeUrl and File.listItemAllFields methodes.
Here an exemple : get listitem by file URL
With FileCreationInformation but more complex.
You need to use : File.StartUpload, File.ContinueUpload and File.FinishUpload
The code is from Microsoft the last part of the tuto, not mine
public Microsoft.SharePoint.Client.File UploadFileSlicePerSlice(ClientContext ctx, string libraryName, string fileName,int fileChunkSizeInMB = 3){
// Each sliced upload requires a unique ID.
Guid uploadId = Guid.NewGuid();
// Get the name of the file.
string uniqueFileName = Path.GetFileName(fileName);
// Ensure that target library exists, and create it if it is missing.
if (!LibraryExists(ctx, ctx.Web, libraryName))
{
CreateLibrary(ctx, ctx.Web, libraryName);
}
// Get the folder to upload into.
List docs = ctx.Web.Lists.GetByTitle(libraryName);
ctx.Load(docs, l => l.RootFolder);
// Get the information about the folder that will hold the file.
ctx.Load(docs.RootFolder, f => f.ServerRelativeUrl);
ctx.ExecuteQuery();
// File object.
Microsoft.SharePoint.Client.File uploadFile;
// Calculate block size in bytes.
int blockSize = fileChunkSizeInMB * 1024 * 1024;
// Get the information about the folder that will hold the file.
ctx.Load(docs.RootFolder, f => f.ServerRelativeUrl);
ctx.ExecuteQuery();
// Get the size of the file.
long fileSize = new FileInfo(fileName).Length;
if (fileSize <= blockSize)
{
// Use regular approach.
using (FileStream fs = new FileStream(fileName, FileMode.Open))
{
FileCreationInformation fileInfo = new FileCreationInformation();
fileInfo.ContentStream = fs;
fileInfo.Url = uniqueFileName;
fileInfo.Overwrite = true;
uploadFile = docs.RootFolder.Files.Add(fileInfo);
ctx.Load(uploadFile);
ctx.ExecuteQuery();
// Return the file object for the uploaded file.
return uploadFile;
}
}
else
{
// Use large file upload approach.
ClientResult<long> bytesUploaded = null;
FileStream fs = null;
try
{
fs = System.IO.File.Open(fileName, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
using (BinaryReader br = new BinaryReader(fs))
{
byte[] buffer = new byte[blockSize];
Byte[] lastBuffer = null;
long fileoffset = 0;
long totalBytesRead = 0;
int bytesRead;
bool first = true;
bool last = false;
// Read data from file system in blocks.
while ((bytesRead = br.Read(buffer, 0, buffer.Length)) > 0)
{
totalBytesRead = totalBytesRead + bytesRead;
// You've reached the end of the file.
if (totalBytesRead == fileSize)
{
last = true;
// Copy to a new buffer that has the correct size.
lastBuffer = new byte[bytesRead];
Array.Copy(buffer, 0, lastBuffer, 0, bytesRead);
}
if (first)
{
using (MemoryStream contentStream = new MemoryStream())
{
// Add an empty file.
FileCreationInformation fileInfo = new FileCreationInformation();
fileInfo.ContentStream = contentStream;
fileInfo.Url = uniqueFileName;
fileInfo.Overwrite = true;
uploadFile = docs.RootFolder.Files.Add(fileInfo);
// Start upload by uploading the first slice.
using (MemoryStream s = new MemoryStream(buffer))
{
// Call the start upload method on the first slice.
bytesUploaded = uploadFile.StartUpload(uploadId, s);
ctx.ExecuteQuery();
// fileoffset is the pointer where the next slice will be added.
fileoffset = bytesUploaded.Value;
}
// You can only start the upload once.
first = false;
}
}
else
{
// Get a reference to your file.
uploadFile = ctx.Web.GetFileByServerRelativeUrl(docs.RootFolder.ServerRelativeUrl + System.IO.Path.AltDirectorySeparatorChar + uniqueFileName);
if (last)
{
// Is this the last slice of data?
using (MemoryStream s = new MemoryStream(lastBuffer))
{
// End sliced upload by calling FinishUpload.
uploadFile = uploadFile.FinishUpload(uploadId, fileoffset, s);
ctx.ExecuteQuery();
// Return the file object for the uploaded file.
return uploadFile;
}
}
else
{
using (MemoryStream s = new MemoryStream(buffer))
{
// Continue sliced upload.
bytesUploaded = uploadFile.ContinueUpload(uploadId, fileoffset, s);
ctx.ExecuteQuery();
// Update fileoffset for the next slice.
fileoffset = bytesUploaded.Value;
}
}
}
} // while ((bytesRead = br.Read(buffer, 0, buffer.Length)) > 0)
}
}
finally
{
if (fs != null)
{
fs.Dispose();
}
}
}
return null;}
hope this help you

RAM doesn't increase to normal value after downloading a file

I made a server and client (As training). They both communicate perfect. The only problem that appears comes when Iam uploading a file to my server from my client. When the server is downloading my servers-RAM goes up around 860MB (--> Downloading a 299MByte file). I noticed that everytime a client has finished uploading my RAM doesnt go back to normal value. Instead it is adding to the current ram value of my server (2 clients finished uploading --> server ram goes over 1300MB)
Here is my Code from my server where it is reciving the file
private void DownloadFileFromClient(string path, string nameAndExtension, TcpClient client)
{
string currFileName = "Download";
string savingPath = path;
if (File.Exists(savingPath)) //Wenn der Pfad einen Namen für die Datei enthält
{
FileInfo fi = new FileInfo(savingPath);
currFileName = fi.Name;
}
else if (Directory.Exists(savingPath)) //Wenn der Pfad zu einem Ordner führt
{
int counter = 0; //zählt die doppelten Datein nachoben
foreach (string file in Directory.GetFiles(savingPath))
{
FileInfo fi = new FileInfo(file);
string[] name = fi.Name.ToString().Split('.');
if (name[0].ToUpperInvariant() == currFileName.ToUpperInvariant()) //Wenn schon eine Datei mit dem selben Namen vorhanden ist
{
counter++;
currFileName = "download" + counter.ToString();
}
}
savingPath = savingPath + "\\" + currFileName + "." + nameAndExtension;
}
using (NetworkStream stream = client.GetStream())
{
BinaryFormatter bf = new BinaryFormatter();
object op;
op = bf.Deserialize(stream); // Deserialize the Object from Stream
BinaryReader br = new BinaryReader(stream);
byte[] buffer = br.ReadBytes(MaxDownloadBytes); //Maximale Dateiengröße in Bytes 10MB = 10485760Bytes | 50MB = 52428800Bytes | 100MB = 104857600Bytes | 500MB = 524288000Bytes
br.Dispose();
br.Close();
using (FileStream filestream = new FileStream(savingPath, FileMode.CreateNew, FileAccess.Write))
{
filestream.Write(buffer, 0, buffer.Length);
}
}
LogMessage("Succefully downloaded file from Client", client.GetHashCode().ToString(), "Manuel", "test", client);
}
I tryied to dispose all, but it didnt help alot. Thanks for any answer!
I didn't necessarily see anything wrong, but there could be some weirdness with all of the streams and reading you have going on that aren't necessarily useful. Below is a tweak to what you have to remove what appears to be unnecessary code / stream reading.
using (NetworkStream stream = client.GetStream())
{
using(BinaryReader br = new BinaryReader(stream))
{
byte[] buffer = br.ReadBytes(MaxDownloadBytes);
using (FileStream filestream = new FileStream(savingPath, FileMode.CreateNew, FileAccess.Write))
{
filestream.Write(buffer, 0, buffer.Length);
}
}
}
I would highly recommend doing this in chunks. Doing that will allow you to remove the necessity for BinaryReader. You can see an example of what just using the NetworkStream looks like here. It would also make your download consume much less memory because you aren't downloading the entire file into memory before you write it.
I solved it...almost. I just added GC.Collect(); after filestream gets closed.
(codesource from TyCobb, thanks)
using (NetworkStream stream = client.GetStream())
{
using(BinaryReader br = new BinaryReader(stream))
{
byte[] buffer = br.ReadBytes(MaxDownloadBytes);
using (FileStream filestream = new FileStream(savingPath, FileMode.CreateNew, FileAccess.Write))
{
filestream.Write(buffer, 0, buffer.Length);
}
GC.Collect();
}
}
But somehow after the 299MB Download the RAM is up to 300MB, but doesnt change if iam downloading another file.

Delete a Temp File after FileStream Pop-Up response

I use Interop to SaveAs(D:/Temp) a template excel sheet after the changes are made.
Then I use FileStream to send the user a Pop-up to save this file. But that file in D:\Temp still exists.
Is there a way to delete this file on Pop-up response?
//Save the Excel File
SaveExcelFile(exportPath, sourceFile, excelWorkBook,
excelApllication, excelWorkSheet);
#region Pop Up and File Open
if (System.IO.File.Exists(sourceFile))
{
FileStream fsSource =
new FileStream(sourceFile, FileMode.Open, FileAccess.Read);
return File(fsSource, "application/vnd.ms-excel", "FileName" + .xls");
}
else
{
return View();
}
#endregion
To delete one file
string filePath)= #"C:\MyDir\filename.txt";
public bool RemoveFile(string filePath)
{
try
{
if (File.Exists(filePath))
{
File.Delete(filePath);
return true;
}
else
return true;
}
catch (Exception ex)
{
return false;
}
}
Delete all files
string[] filePaths = Directory.GetFiles(#"c:\MyDir\");
foreach (string filePath in filePaths)
File.Delete(filePath);
To delete all files using one code line
Array.ForEach(Directory.GetFiles(#"c:\MyDir\"),
delegate(string path) { File.Delete(path); });
You can use File.Delete method.
if (File.Exists("File_Path"))
{
File.Delete("File_Path");
}
Updated
For downloading binary files,
using (FileStream fs = File.OpenRead(path))
{
int length = (int)fs.Length;
byte[] buffer;
using (BinaryReader br = new BinaryReader(fs))
{
buffer = br.ReadBytes(length);
}
Response.Clear();
Response.Buffer = true;
Response.AddHeader("content-disposition", String.Format("attachment;filename={0}", Path.GetFileName(path)));
Response.ContentType = "application/" + Path.GetExtension(path).Substring(1);
Response.BinaryWrite(buffer);
Response.Flush();
Response.End();
}
Found this code from here
Instead of creating a temp file, loading it to stream, and then trying to delete it, I suggest that you create the file directly in memory stream (i.e. System.IO.MemoryStream) in the first place, so you don't have to load it and delete it.
If you cannot create it directly in memory stream, the main issue is that you cannot delete the temp file while you're using it in the FileStream. In this case, you copy the FileStream to a MemoryStream, close and dispose the FileStream, delete the temp file, and then return the MemoryStream to the user.
You can use the function bellow to copy streams correctly.
// Author: Racil Hilan.
/// <summary>Copies data from a source stream to a target stream.</summary>
private static void CopyStream(Stream SourceStream, Stream TargetStream) {
const int BUFFER_SIZE = 4096;
byte[] buffer = new byte[BUFFER_SIZE];
//Reset the source stream in order to process all data.
if (SourceStream.CanSeek)
SourceStream.Position = 0;
//Copy data from the source stream to the target stream.
int BytesRead = 0;
while ((BytesRead = SourceStream.Read(buffer, 0, BUFFER_SIZE)) > 0)
TargetStream.Write(buffer, 0, BytesRead);
//Reset the source stream and the target stream to make them ready for any other operation.
if (SourceStream.CanSeek)
SourceStream.Position = 0;
if (TargetStream.CanSeek)
TargetStream.Position = 0;
}
You can use File.Delete() for this. Just make sure you've closed the stream before you try to delete the file, and preferably, that you have been able to send whatever you need to.
I'm guessing you don't want to delete the file if the main operation fails.

download zip files by use of reader in c#

I have been working on this application that enables user to log in into another website, and then download specified file from that server. So far I have succeeded in logging on the website and download the file. But everything ruins when it comes to zip files.
Is there any chunk of code that could be helpful in reading the .zip files byte by byte or by using stream reader?
I m using downloadfile() but its not returning the correct zip file.
I need a method by which I can read zip files. Can I do it by using ByteReader()
The code used to download zip file is
string filename = "13572_BranchInformationReport_2012-05-22.zip";
string filepath = "C:\\Documents and Settings\\user\\Desktop\\" + filename.ToString();
WebClient client = new WebClient();
string user = "abcd", pass = "password";
client.Credentials = new NetworkCredential(user, pass);
client.Encoding = System.Text.Encoding.UTF8;
try
{
client.DownloadFile("https://web.site/archive/13572_BranchInformationReport_2012-05-22.zip", filepath);
Response.Write("Success");
}
catch (Exception ue)
{
Response.Write(ue.Message);
}
Thanks in advance.
is there any chunk of code that could be helpful in reading the zip files bytes by bytes aur by using stream reader.
Absolutely not. StreamReader - and indeed any TextReader is for reading text content, not binary content. A zip file is not text - it's composed of bytes, not characters.
If you're reading binary content such as zip files, you should be using a Stream rather than a TextReader of any kind.
Note that WebClient.DownloadFile and WebClient.DownloadData can generally make things easier for downloading binary content.
Another simple way to downlaod zip file
<asp:HyperLink ID="HyperLink1" runat="server" NavigateUrl="~/DOWNLOAD/Filename.zip">Click To Download</asp:HyperLink>
Another solution
private void DownloadFile()
{
string getPath = "DOWNLOAD/FileName.zip";
System.IO.Stream iStream = null;
byte[] buffer = new Byte[1024];
// Length of the file:
int length;
// Total bytes to read:
long dataToRead;
// Identify the file to download including its path.
string filepath = Server.MapPath(getPath);
// Identify the file name.
string filename = System.IO.Path.GetFileName(filepath);
try
{
// Open the file.
iStream = new System.IO.FileStream(filepath, System.IO.FileMode.Open,
System.IO.FileAccess.Read, System.IO.FileShare.Read);
// Total bytes to read:
dataToRead = iStream.Length;
// Page.Response.ContentType = "application/vnd.android.package-archive";
// Page.Response.ContentType = "application/octet-stream";
Page.Response.AddHeader("Content-Disposition", "attachment; filename=" + filename);
// Read the bytes.
while (dataToRead > 0)
{
// Verify that the client is connected.
if (Response.IsClientConnected)
{
// Read the data in buffer.
length = iStream.Read(buffer, 0, 1024);
// Write the data to the current output stream.
Page.Response.OutputStream.Write(buffer, 0, length);
// Flush the data to the HTML output.
Page.Response.Flush();
// buffer = new Byte[1024];
dataToRead = dataToRead - length;
}
else
{
//prevent infinite loop if user disconnects
dataToRead = -1;
}
}
}
catch (Exception ex)
{
// Trap the error, if any.
Page.Response.Write(ex.Message);
}
finally
{
if (iStream != null)
{
//Close the file.
iStream.Close();
Page.Response.Close();
}
}
}
Your answer
WebRequest objRequest = System.Net.HttpWebRequest.Create(url);
objResponse = objRequest.GetResponse();
byte[] buffer = new byte[32768];
using (Stream input = objResponse.GetResponseStream())
{
using (FileStream output = new FileStream ("test.doc",
FileMode.CreateNew))
{
int bytesRead;
while ( (bytesRead=input.Read (buffer, 0, buffer.Length)) > 0)
{
output.Write(buffer, 0, bytesRead);
}
}
}
This is how i achieved it. Thanks everyone for ur help

Create new FileStream out of a byte array

I am attempting to create a new FileStream object from a byte array. I'm sure that made no sense at all so I will try to explain in further detail below.
Tasks I am completing:
1) Reading the source file which was previously compressed
2) Decompressing the data using GZipStream
3) copying the decompressed data into a byte array.
What I would like to change:
1) I would like to be able to use File.ReadAllBytes to read the decompressed data.
2) I would then like to create a new filestream object usingg this byte array.
In short, I want to do this entire operating using byte arrays. One of the parameters for GZipStream is a stream of some sort, so I figured I was stuck using a filestream. But, if some method exists where I can create a new instance of a FileStream from a byte array - then I should be fine.
Here is what I have so far:
FolderBrowserDialog fbd = new FolderBrowserDialog(); // Shows a browser dialog
fbd.ShowDialog();
// Path to directory of files to compress and decompress.
string dirpath = fbd.SelectedPath;
DirectoryInfo di = new DirectoryInfo(dirpath);
foreach (FileInfo fi in di.GetFiles())
{
zip.Program.Decompress(fi);
}
// Get the stream of the source file.
using (FileStream inFile = fi.OpenRead())
{
//Create the decompressed file.
string outfile = #"C:\Decompressed.exe";
{
using (GZipStream Decompress = new GZipStream(inFile,
CompressionMode.Decompress))
{
byte[] b = new byte[blen.Length];
Decompress.Read(b,0,b.Length);
File.WriteAllBytes(outfile, b);
}
}
}
Thanks for any help!
Regards,
Evan
It sounds like you need to use a MemoryStream.
Since you don't know how many bytes you'll be reading from the GZipStream, you can't really allocate an array for it. You need to read it all into a byte array and then use a MemoryStream to decompress.
const int BufferSize = 65536;
byte[] compressedBytes = File.ReadAllBytes("compressedFilename");
// create memory stream
using (var mstrm = new MemoryStream(compressedBytes))
{
using(var inStream = new GzipStream(mstrm, CompressionMode.Decompress))
{
using (var outStream = File.Create("outputfilename"))
{
var buffer = new byte[BufferSize];
int bytesRead;
while ((bytesRead = inStream.Read(buffer, 0, BufferSize)) != 0)
{
outStream.Write(buffer, 0, bytesRead);
}
}
}
}
Here is what I ended up doing. I realize that I did not give sufficient information in my question - and I apologize for that - but I do know the size of the file I need to decompress as I am using it earlier in my program. This buffer is referred to as "blen".
string fi = #"C:\Path To Compressed File";
// Get the stream of the source file.
// using (FileStream inFile = fi.OpenRead())
using (MemoryStream infile1 = new MemoryStream(File.ReadAllBytes(fi)))
{
//Create the decompressed file.
string outfile = #"C:\Decompressed.exe";
{
using (GZipStream Decompress = new GZipStream(infile1,
CompressionMode.Decompress))
{
byte[] b = new byte[blen.Length];
Decompress.Read(b,0,b.Length);
File.WriteAllBytes(outfile, b);
}
}
}

Categories

Resources