Help appreciated in resolving the 'Out of memory exception' from the below source at line return Convert.ToBase64String(stream.ToArray());
private static string GetSPFileBinary(ClientContext ctx, string fileUrlPath)
{
ctx.RequestTimeout = Int32.MaxValue;
var spFile = ctx.Web.GetFileByServerRelativeUrl(fileUrlPath);
var spFileContent = spFile.OpenBinaryStream();
ctx.Load(spFile);
ctx.ExecuteQuery();
MemoryStream stream = new MemoryStream();
if (spFileContent != null && spFileContent.Value != null)
{
spFileContent.Value.CopyTo(stream);
}
return Convert.ToBase64String(stream.ToArray());
}
Assuming that stream.ToArray() was executed successfully and Convert.ToBase64String crashes, you could always convert a batch to 64 string and store it somewhere from where you can reload it. And then load the converted chunks one-by-one and store them into the result file. If your application crashes on stream.ToArray(), then you will need to allow your application to use more memory if available or apply the chunk loading idea when you load from stream as well.
even the below throws the same old 'Out of memory exception', help appreciated.
private static string GetSPFileBinary(ClientContext ctx, string fileUrlPath)
{
ctx.RequestTimeout = Int32.MaxValue;
var spFile = ctx.Web.GetFileByServerRelativeUrl(fileUrlPath);
var spFileContent = spFile.OpenBinaryStream();
ctx.Load(spFile);
ctx.ExecuteQuery();
MemoryStream stream = new MemoryStream();
if (spFileContent != null && spFileContent.Value != null)
{
spFileContent.Value.CopyTo(stream);
}
byte[] docBytes = stream.ToArray();
return Convert.ToBase64String(docBytes);
}
Related
I have asp.net mvc application which has file upload functionality. While uploading the file, I am performing few validations on the uploaded content before moving it to database and file system location.
Here goes my code:
[HttpPost]
[ValidateAntiForgeryToken]
public ActionResult AddImage([Bind(Include = "image,ImageName,ImageType,CountryId,Keyword,Source,Copyright,Description")] CreateImageViewModel model)
{
if (!this.ModelState.IsValid)
{
return View("Images");
}
if (model != null && model.image.ContentType.Contains(Constants.Image) && !ValidateUploadedImageContent(model.image, model.image.FileName))
{
var dto = new ImageDTO();
model.FilePath = model.image.FileName;
dto.ImageFile = model.image;
dto.Name = model.ImageName;
dto.FilePath = model.image.FileName;
dto.FileType = Path.GetExtension(model.FilePath);
dto.ImageType = model.ImageType;
dto.CountryId = model.CountryId;
dto.Keyword = model.Keyword;
dto.Source = model.Source;
dto.Copyright = model.Copyright;
dto.Description = model.Description;
dto.CreatedBy = UserDto.emailId;
try
{
_imageService.SaveImage(dto);
}
catch (Exception ex)
{
if (ex.Message.Equals(Constants.InvalidImageType))
return GetSafeRedirect(Url.Action("AddImage", model) + "#onload-errors");
throw ex;
}
return RedirectToAction(Constants.Actions.Images.ToString());
}
else
{
return GetSafeRedirect(Url.Action("AddImage", model) + "#onload-errors");
}
}
private bool ValidateUploadedImageContent(HttpPostedFileBase uploadedFile, string imageFileName)
{
if (Path.GetExtension(imageFileName).Equals(".svg", StringComparison.OrdinalIgnoreCase))
{
if (uploadedFile.ContentLength > 0)
{
byte[] data;
//using (Stream inputStream = uploadedFile.InputStream)
//{
Stream inputStream = uploadedFile.InputStream;
var memoryStream = inputStream as MemoryStream;
if (memoryStream == null)
{
memoryStream = new MemoryStream();
inputStream.CopyTo(memoryStream);
}
data = memoryStream.ToArray();
//}
var parsedData = Encoding.UTF8.GetString(data, 0, data.Length).TrimEnd('\0');
var result = parsedData.ContainsAny(Constants.InsecureStrings, StringComparison.CurrentCultureIgnoreCase);
return result;
}
}
return false;
}
Here in the above method: ValidateUploadedImageContent(), I tried to dispose the stream object with the help of using statement but I found that if I keep the below code in the method: ValidateUploadedImageContent(), then in that case post validation process, I found on debugging that the ContentLength property is set with 0 value and finally corrupted image gets saved in the file system location.
Updated :
private bool ValidateUploadedImageContent(HttpPostedFileBase uploadedFile, string imageFileName)
{
if (Path.GetExtension(imageFileName).Equals(".svg", StringComparison.OrdinalIgnoreCase))
{
if (uploadedFile.ContentLength > 0)
{
byte[] data;
using (Stream inputStream = uploadedFile.InputStream)
{
Stream inputStream = uploadedFile.InputStream;
var memoryStream = inputStream as MemoryStream;
if (memoryStream == null)
{
memoryStream = new MemoryStream();
inputStream.CopyTo(memoryStream);
}
data = memoryStream.ToArray();
}
var parsedData = Encoding.UTF8.GetString(data, 0, data.Length).TrimEnd('\0');
var result = parsedData.ContainsAny(Constants.InsecureStrings, StringComparison.CurrentCultureIgnoreCase);
return result;
}
}
return false;
}
Can anyone help me to know how to fix this issue?
First to address your issue, which I now understand is that after the call to ValidateUploadedImageContent the image stream is invalid.
That is because the stream gained from the HttpPostedFileBase is "read-only sequential (non-seekable)" as detailed in this SO answer. This explains why the stream's ContentLength is 0 - the stream has been consumed by the validation call.
If you have flexibility with the ImageDTO class, modifying the validation method such that it returns the image bytes would be a workaround.
For example,
// on success, buffer contains the image data. Otherwise it is null.
private bool ValidateUploadedImageContent(
out byte[] buffer,
HttpPostedFileBase uploadedFile,
string imageFileName)
{
buffer = null;
if (Path.GetExtension(imageFileName).Equals(".svg", StringComparison.OrdinalIgnoreCase))
{
if (uploadedFile.ContentLength > 0)
{
var reader = new BinaryReader(inputStream);
buffer = reader.ReadBytes((int)uploadedFile.ContentLength);
var parsedData = Encoding.UTF8.GetString(buffer, 0, buffer.Length).TrimEnd('\0');
return parsedData.ContainsAny(Constants.InsecureStrings, StringComparison.CurrentCultureIgnoreCase);
}
}
return false;
}
I've used BinaryReader to simplify the code.
Then back to the calling method,
byte[] imageBuffer = null;
if (model != null && model.image.ContentType.Contains(Constants.Image)
&& !ValidateUploadedImageContent(out imageBuffer, model.image, model.image.FileName)) {
var dto = new ImageDTO();
using(var imageStream = new MemoryStream(imageBuffer)) {
// pass along imageStream to your ImageDTO and save.
}
}
Again, hopefully you have some flexibility with the ImageDTO class.
I am trying to download files from a SharePoint library using the client object model. I seem to be able to access the files using OpenBinaryStream() and then executing the query, but when I try to access the stream, it is a stream of Length = 0. I've seen many examples and I've tried several, but I can't get the files to download. I've uploaded successfully, and credentials and permissions aren't the problem. Anyone have any thoughts?
public SharepointFileContainer DownloadFolder(bool includeSubfolders, params object[] path)
{
try
{
List<string> pathStrings = new List<string>();
foreach (object o in path)
pathStrings.Add(o.ToString());
var docs = _context.Web.Lists.GetByTitle(Library);
_context.Load(docs);
_context.ExecuteQuery();
var rootFolder = docs.RootFolder;
_context.Load(rootFolder);
_context.ExecuteQuery();
var folder = GetFolder(rootFolder, pathStrings);
var files = folder.Files;
_context.Load(files);
_context.ExecuteQuery();
SharepointFileContainer remoteFiles = new SharepointFileContainer();
foreach (Sharepoint.File f in files)
{
_context.Load(f);
var file = f.OpenBinaryStream();
_context.ExecuteQuery();
var memoryStream = new MemoryStream();
file.Value.CopyTo(memoryStream);
remoteFiles.Files.Add(f.Name, memoryStream);
}
...
}
SharepointFileContainer is just a custom class for my calling application to dispose of the streams when it has finished processing them. GetFolder is a recursive method to drill down the given folder path. I've had problems with providing the direct url and have had the most success with this.
My big question is why "file.Value" is a Stream with a Length == 0?
Thanks in advance!
EDIT:
Thanks for your input so far...unfortunately I'm experiencing the same problem. Both solutions pitched make use of OpenBinaryDirect. The resulting FileInformation class has this for the stream...
I'm still getting a file with 0 bytes downloaded.
You need to get the list item of the file (as a ListItem object) and then use it's property File. Something like:
//...
// Previous code
//...
var docs = _context.Web.Lists.GetByTitle(Library);
var listItem = docs.GetItemById(listItemId);
_context.Load(docs);
clientContext.Load(listItem, i => i.File);
clientContext.ExecuteQuery();
var fileRef = listItem.File.ServerRelativeUrl;
var fileInfo = Microsoft.SharePoint.Client.File.OpenBinaryDirect(clientContext, fileRef);
var fileName = Path.Combine(filePath,(string)listItem.File.Name);
using (var fileStream = System.IO.File.Create(fileName))
{
fileInfo.Stream.CopyTo(fileStream);
}
After that you do whatever you need to do with the stream. The current one just saves it to the specified path, but you can also download it in the browser, etc..
We can use the following code to get the memory stream.
var fileInformation = Microsoft.SharePoint.Client.File.OpenBinaryDirect(clientContext, file.ServerRelativeUrl);
if (fileInformation != null && fileInformation.Stream != null)
{
using (MemoryStream memoryStream = new MemoryStream())
{
byte[] buffer = new byte[32768];
int bytesRead;
do
{
bytesRead = fileInformation.Stream.Read(buffer, 0, buffer.Length);
memoryStream.Write(buffer, 0, bytesRead);
} while (bytesRead != 0);
}
}
Reference: https://praveenkasireddy.wordpress.com/2012/11/11/download-document-from-document-set-using-client-object-model-om/
I know that this is may be a question without one 'right' answer
I have a C# windows application that has an embedded resource included in the assembly. I've been trying to come up with a way to compare the contents of my resource stream to determine if the contents of that stream matches a particular file on the file system.
e.g.
using(var resourceStream = Assembly.GetExecutingAssembly().GetManifestResourceStream(#"Manifest/Resource/Path/thing.exe"))
using(var fileStream = new FileStream(#"File/System/Path/thing.exe", FileMode.Read))
// Compare Contents (thing.exe may be an older version)
if(CompareStreamContents(resourceStream, fileStream))
{
/* Do a thing */
}
else
{
/* Do another thing*/
}
Is there a better way than simply doing a byte-by-byte comparison? Thoughts? (and thanks in advance!)
Per my comment:
private bool CompareStreamContents(Stream resourceStream, Stream fileStream)
{
var sha = new SHA256CryptoServiceProvider();
var hash1 = Convert.ToBase64String(sha.ComputeHash(ReadToEnd(resourceStream)));
var hash2 = Convert.ToBase64String(sha.ComputeHash(ReadToEnd(fileStream)));
return hash1 == hash2;
}
private byte[] ReadToEnd(Stream stream)
{
var continueRead = true;
var buffer = new byte[0x10000];
var ms = new MemoryStream();
while (continueRead)
{
var size = stream.Read((byte[])buffer, 0, buffer.Length);
if (size > 0)
{
ms.Write(buffer, 0, size);
}
else
{
continueRead = false;
}
}
return ms.ToArray();
}
If you plan on doing something else with the streams after the compare, you may want to set the stream position back to origin before returning from the compare method.
I'm using ServiceStack to create a Service. In one of the methods I write some data in response's output stream like this: await response.OutputStream.WriteAsync(Consts.DATA, 0, Consts.DATA.Length); where data is a byte[]. Ok, now the problem is when using this service from a client and GET the HttpResponseMessage for the specified method, I don't know how to get the data out. I wan't to unit test that the actual response contains the data I passed when writing to output content.
(I've tried to ReadAsByteArray but it throws an exception. When reading as Stream or String I don't know how to make it a byte[] to compare it in the test with the excpected Consts.DATA byte array.)
I'm pretty new to the field, excuse my ignorance if i miss something. Any help is appreciated.
Thanks!
If you want to use the typed client, you can do (from the documentation):
As raw bytes:
byte[] responseBytes = client.Get<byte[]>("/poco/World");
var dto = responseBytes.FromUtf8Bytes().FromJson<PocoResponse>();
dto.Result //Hello, World
Or as a Stream:
using (Stream responseStream = client.Get<Stream>("/poco/World")) {
var dto = responseStream.ReadFully()
.FromUtf8Bytes()
.FromJson<PocoResponse>();
dto.Result //Hello, World
}
Or even access the populated HttpWebResponse object:
HttpWebResponse webResponse = client.Get<HttpWebResponse>("/poco/World");
webResponse.Headers["X-Response"] //World
using (var stream = webResponse.GetResponseStream())
using (var sr = new StreamReader(stream)) {
var dto = sr.ReadToEnd().FromJson<PocoResponse>();
dto.Result //Hello, World
}
You can also use untyped ServiceStack client to access raw response, it is described in the documentation with samples.
If all you need is load the data from an URL as a Stream and youre using a HttpClient you can just do this to get a stream back:
var result = await client.GetStreamAsync(URL);
I think you can also use GetResponseStream() on your HttpWebResponse.
var result = message.GetResponseStream();
Run your output stream through this static method:
public static byte[] ReadFullStream(Stream st)
{
var lockTaken = false;
try
{
Monitor.Enter(_lock, ref lockTaken);
var size = 0;
var continueRead = true;
var buffer = (byte[])Array.CreateInstance(typeof(byte), 0x10000);
using (MemoryStream ms = new MemoryStream())
{
while (continueRead)
{
size = st.Read(buffer, 0, buffer.Length);
if (size > 0)
{
ms.Write(buffer, 0, size);
}
else
{
continueRead = false;
}
}
return ms.ToArray();
}
}
finally
{
if (lockTaken) { Monitor.Exit(_lock); }
}
}
EDIT: forgot, you'll need this defined at class scope:
private static readonly object _lock = new object();
Read response stream as follow-
using (var reader = new System.IO.StreamReader(response.GetResponseStream()))
{
responseContent = reader.ReadToEnd();
if (response.StatusCode == HttpStatusCode.OK && response.StatusDescription.ToUpper() == "OK" && !string.IsNullOrEmpty(responseContent))
{
objFSLstGetAgent = JsonConvert.DeserializeObject<YourObjectClass>(responseContent);
}
}
I'm having troubles to make my program work correctly - here I explain :
I have, on one hand a C# WinForms app which launches an instance of IE by using the "Navigate" method : myWebBrowser.Navigate(myUrl, "_blank", intarray, "");, with intarray defined like this : byte[] intarray = BitConverter.GetBytes(id);. On this side, it works.
On the other side, I have an ASP .NET WebForms application which has to retrieve this intarray. I've tried this.
if (HttpContext.Current != null)
{
if (Session["Authenticated"] == null)
{
var current = HttpContext.Current;
byte[] postdata = getpostdata(current);
}
}
private byte[] getpostdata(HttpContext CurrentContext)
{
MemoryStream ms = new MemoryStream();
CurrentContext.Request.InputStream.CopyTo(ms);
byte[] postdata = ms.ToArray();
return postdata;
}
// Convert a byte array to an Object
public int ByteArrayToInt(byte[] arrBytes)
{
if (BitConverter.IsLittleEndian) Array.Reverse(arrBytes);
int i = BitConverter.ToInt32(arrBytes, 0);
return i;
}
The problem seems to be in retrieving the Data in the getpostdata(HttpContext) function... I get a byte array with length = 0 instead of the one which is sent with length = 4...
Does anyone know how to make it work ?
Yann
var current = HttpContext.Current;
var sr = new StreamReader(Request.InputStream, Encoding.Default);
var postdata = sr.ReadToEnd();
above