Develop Reference

Extract the file header signature as it is being streamed directly to disk in ASP.NET Core

I have an API method that streams uploaded files directly to disk to be scanned with a virus checker. Some of these files can be quite large, so IFormFile is a no go:
Any single buffered file exceeding 64 KB is moved from memory to a
temp file on disk.
Source: https://learn.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-3.1
I have a working example that uses multipart/form-data and a really nice NuGet package that takes out the headache when working with multipart/form-data, and it works well, however I want to add a file header signature check, to make sure that the file type defined by the client is actually what they say it is. I can't rely on the file extension to do this securely, but I can use the file header signature to make it at least a bit more secure. Since I'm am streaming directly to disk, how can I extract the first bytes as it's going through the file stream?
[DisableFormValueModelBinding] // required for form binding
[ValidateMimeMultipartContent] // simple check to make sure this is a multipart form
[FileUploadOperation(typeof(SwaggerFileItem))] // used to define the Swagger schema
[RequestSizeLimit(31457280)] // 30MB
[RequestFormLimits(MultipartBodyLengthLimit = 31457280)]
public async Task<IActionResult> PostAsync([FromRoute] int customerId)
{
// place holders
var uploadLocation = string.Empty;
var trustedFileNameForDisplay = string.Empty;
// this is using a nuget package that does the hard work on reading the multipart form-data.... using UploadStream;
var model = await this.StreamFiles<FileItem>(async x =>
{
// never trust the client
trustedFileNameForDisplay = WebUtility.HtmlEncode(Path.GetFileName(x.FileName));
// determien the quarantine location
uploadLocation = GetUploadLocation(trustedFileNameForDisplay);
// stream the input stream to the file stream
// importantly this should never load the file into memory
// it should be a straight pass through to disk
await using var fs = System.IO.File.Create(uploadLocation, BufSize);
// --> How do I extract the file signature? I.e. a copy of the header bytes as it is being streamed??? <--
await x.OpenReadStream().CopyToAsync(fs);
});
// The model state can now be checked
if (!ModelState.IsValid)
{
// delete the file
DeleteFileIfExists(uploadLocation);
// return a bad request
ThrowProblemDetails(ModelState, StatusCodes.Status400BadRequest);
}
// map as much as we can
var request = _mapper.Map<CreateAttachmentRequest>(model);
// map the remaining properties
request.CustomerId = customerId;
request.UploadServer = Environment.MachineName;
request.uploadLocation = uploadLocation;
request.FileName = trustedFileNameForDisplay;
// call mediator with this request to send it over WCF to Pulse Core.
var result = await _mediator.Send(request);
// build response
var response = new FileResponse { Id = result.FileId, CustomerId = customerId, ExternalId = request.ExternalId };
// return the 201 with the appropriate response
return CreatedAtAction(nameof(GetFile), new { fileId = response.Id, customerId = response.customerId }, response);
}
The section I'm stuck on is around the line await x.OpenReadStream().CopyToAsync(fs);. I would like to pull out the file header here as the stream is being copied to the FileStream. Is there a way to add some kind of inspector? I don't want to read the entire stream again, just the header.
Update
Based on the answer given by #Ackdari I have successfully switched the code to extract the header from the uploaded file stream. I don't know if this could be made any more efficient, but it does work:
//...... removed for clarity
var model = await this.StreamFiles<FileItem>(async x =>
{
trustedFileNameForDisplay = WebUtility.HtmlEncode(Path.GetFileName(x.FileName));
quarantineLocation = QuarantineLocation(trustedFileNameForDisplay);
await using (var fs = System.IO.File.Create(quarantineLocation, BufSize))
{
await x.OpenReadStream().CopyToAsync(fs);
fileFormat = await FileHelpers.GetFileFormatFromFileHeader(fs);
}
});
//...... removed for clarity
and
// using https://github.com/AJMitev/FileTypeChecker
public static async Task<IFileType> GetFileFormatFromFileHeader(FileStream fs)
{
IFileType fileFormat = null;
fs.Position = 0;
var headerData = new byte[40];
var bytesRead = await fs.ReadAsync(headerData, 0, 40);
if (bytesRead > 0)
{
await using (var ms = new MemoryStream(headerData))
{
if (!FileTypeValidator.IsTypeRecognizable(ms))
{
return null;
}
fileFormat = FileTypeValidator.GetFileType(ms);
}
}
return fileFormat;
}

You may want to consider reading the header yourself dependent on which file type is expected
int n = 4; // length of header
var headerData = new byte[n];
var bytesRead = 0;
while (bytesRead < n)
bytesRead += await x.ReadAsync(headerData.AsMemory(bytesRead));
CheckHeader(headerData);
await fs.WriteAsync(headerData.AsMemory());
await x.CopyToAsync(fs);

Azure blob file got corrupted post uploading file using UploadFromStreamAsync

I tried below code to upload file to azure blob container but uploaded file got corrupted.
public async void UploadFile(Stream memoryStream, string fileName, string containerName)
{
try
{
memoryStream.Position = 0;
CloudBlockBlob file = GetBlockBlobContainer(containerName).GetBlockBlobReference(fileName);
file.Metadata["FileType"] = Path.GetExtension(fileName);
file.Metadata["Name"] = fileName;
await file.UploadFromStreamAsync(memoryStream).ConfigureAwait(false);
}
catch (Exception ex)
{
throw ex;
}
}
How can I resolve it.
Unable to open excel file which was uploaded to blob using above code.
Error:
Stream streamData= ConvertDataSetToByteArray(sourceTable); // sourceTable is the DataTable
streamData.Position = 0;
UploadFile(streamData,'ABCD.xlsx','sampleBlobContainer'); //calling logic to upload stream to blob
private Stream ConvertDataSetToByteArray(DataTable dataTable)
{
StringBuilder sb = new StringBuilder();
IEnumerable<string> columnNames = dataTable.Columns.Cast<DataColumn>().
Select(column => column.ColumnName);
sb.AppendLine(string.Join(",", columnNames));
foreach (DataRow row in dataTable.Rows)
{
IEnumerable<string> fields = row.ItemArray.Select(field => (field.ToString()));
sb.AppendLine(string.Join(",", fields));
}
var myByteArray = System.Text.Encoding.UTF8.GetBytes(sb.ToString());
var streamData = new MemoryStream(myByteArray);
return streamData;
}

Your code above creates a .csv file, not an .xlsx file. You can easily test this out by creating something similar to what your code builds, e.g.:
Then if you rename it to .xlsx, to replicate what you do, you get:
You have two solutions:
You either need to build an actual .xlsx file, you can do this with the https://github.com/JanKallman/EPPlus package for example
or
You need to save your file as a .csv, because that's what it really is.
The fact the you upload it to azure blob storage is completely irrelevant here - there's no issue with the upload.

Since the stream is instantiated outside this method I assume the file is handled there and added to the stream, however, here you are returning the position of the stream to 0, thus invalidating the file.

First of all, are you sure the file got corrupted? Save both the MemoryStream contents and the blog to local files and compare them. You could also save the MemoryStream contents to a file and use UploadFromFileAsync.
To check for actual corruption you should calculate the content's MD5 hash in advance and compare it with the blob's hash after upload.
To calculate the stream's MD5 hash use ComputeHash.
var hasher=MD5.Create();
memoryStream.Position = 0;
var originalHash=Convert.ToBase64String(hasher.ComputeHash(memoryStream));
To get the client to calculate an blob has you need to set the BlobRequestOptions.StoreBlobContentMD5 option while uploading :
memoryStream.Position = 0;
var options = new BlobRequestOptions()
{
StoreBlobContentMD5 = testMd5
};
await file.UploadFromStreamAsync(memoryStream,null,options,null).ConfigureAwait(false);
To retrieve and check the uploaded hash use FetchAttributes or FetchAttributesAsync and compare the BlobProperties.ContentMD5 value with the original :
file.FetchAttributes();
var blobHash=file.Properties.ContentMD5;
if (blobHash != originalHash)
{
//Ouch! Retry perhaps?
}

It seems that your method don't have fatal problems. I guess the part of your Stream conversion has gone wrong.
This is my code:
using System;
using System.IO;
using Microsoft.WindowsAzure.Storage;
namespace ConsoleApp7
{
class Program
{
public static class Util
{
public async static void UploadFile(Stream memoryStream, string fileName, string containerName)
{
memoryStream.Position = 0;
var storageAccount = CloudStorageAccount.Parse("xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx");
var blockBlob = storageAccount.CreateCloudBlobClient()
.GetContainerReference(containerName)
.GetBlockBlobReference(fileName);
blockBlob.UploadFromStreamAsync(memoryStream);
}
}
static void Main(string[] args)
{
//Open the file
FileStream fileStream = new FileStream("C:\\Users\\bowmanzh\\Desktop\\Book1.xlsx", FileMode.Open);
//Read the byte[] of File
byte[] bytes = new byte[fileStream.Length];
fileStream.Read(bytes,0,bytes.Length);
fileStream.Close();
//turn from byte[] to Stream
Stream stream = new MemoryStream(bytes);
Util.UploadFile(stream,"Book2.xlsx","test");
Console.WriteLine("Hello World!");
Console.ReadLine();
}
}
}

Checking if the file is rar through its bytes

I am trying to verify that the file is a .rar file through its bytes for security purposes. Th following code is my code the only problem is that the sub-header is not matching with the one generated from the file. I noticed that is different for different file. Could you please explain to me why?
static bool IsRARFile(string filePath)
{
bool isDocFile = false;
//
// File sigs from: http://www.garykessler.net/library/file_sigs.html
//
string msOfficeHeader = "52-61-72-21-1A-07-00-CF";
string docSubHeader = "64-2E-63-73";
using (Stream stream = File.OpenRead(filePath))
{
//get file header
byte[] headerBuffer = new byte[8];
stream.Read(headerBuffer, 0, headerBuffer.Length);
string headerString = BitConverter.ToString(headerBuffer);
if (headerString.Equals(msOfficeHeader, StringComparison.InvariantCultureIgnoreCase))
{
//get subheader
byte[] subHeaderBuffer = new byte[4];
stream.Seek(512, SeekOrigin.Begin);
stream.Read(subHeaderBuffer, 0, subHeaderBuffer.Length);
string subHeaderString = BitConverter.ToString(subHeaderBuffer);
if (subHeaderString.Equals(docSubHeader, StringComparison.InvariantCultureIgnoreCase))
{
isDocFile = true;
}
}
}
return isDocFile;
}

This is because you have just copied a function from somewhere for a different filetype and not every filetype has any notion of a "subheader". You only need to check the main header in the case of RAR.
I also suggest modifying the naming of the variables, it is quite a mismash if a function says it's checking for RAR type and internally all variables refer to DOCs.

Convert .db to binary

I'm trying to convert a .db file to binary so I can stream it across a web server. I'm pretty new to C#. I've gotten as far as looking at code snippets online but I'm not really sure if the code below puts me on the right track. How I can write the data once I read it? Does BinaryReader automatically open up and read the entire file so I can then just write it out in binary format?
class Program
{
static void Main(string[] args)
{
using (FileStream fs = new FileStream("output.bin", FileMode.Create))
{
using (BinaryWriter bw = new BinaryWriter(fs))
{
long totalBytes = new System.IO.FileInfo("input.db").Length;
byte[] buffer = null;
BinaryReader binReader = new BinaryReader(File.Open("input.db", FileMode.Open));
}
}
}
}
Edit: Code to stream the database:
[WebGet(UriTemplate = "GetDatabase/{databaseName}")]
public Stream GetDatabase(string databaseName)
{
string fileName = "\\\\computer\\" + databaseName + ".db";
if (File.Exists(fileName))
{
FileStream stream = File.OpenRead(fileName);
if (WebOperationContext.Current != null)
{
WebOperationContext.Current.OutgoingResponse.ContentType = "binary/.bin";
}
return stream;
}
return null;
}
When I call my server, I get nothing back. When I use this same type of method for a content-type of image/.png, it works fine.

All the code you posted will actually do is copy the file input.db to the file output.bin. You could accomplish the same using File.Copy.
BinaryReader will just read in all of the bytes of the file. It is a suitable start to streaming the bytes to an output stream that expects binary data.
Once you have the bytes corresponding to your file, you can write them to the web server's response like this:
using (BinaryReader binReader = new BinaryReader(File.Open("input.db",
FileMode.Open)))
{
byte[] bytes = binReader.ReadBytes(int.MaxValue); // See note below
Response.BinaryWrite(bytes);
Response.Flush();
Response.Close();
Response.End();
}
Note: The code binReader.ReadBytes(int.MaxValue) is for demonstrating the concept only. Don't use it in production code as loading a large file can quickly lead to an OutOfMemoryException. Instead, you should read in the file in chunks, writing to the response stream in chunks.
See this answer for guidance on how to do that
https://stackoverflow.com/a/8613300/141172

create a file in Virtual Memory

Hi i am trying to upload a local into Sharepoint documentLibrary.
The following code works well to upload a file into document Libray.
public void UploadFile(string srcUrl, string destUrl)
{
if (!File.Exists(srcUrl))
{
throw new ArgumentException(String.Format("{0} does not exist",
srcUrl), "srcUrl");
}
SPWeb site = new SPSite(destUrl).OpenWeb();
FileStream fStream = File.OpenRead(srcUrl);
byte[] contents = new byte[fStream.Length];
fStream.Read(contents, 0, (int)fStream.Length);
fStream.Close();
site.Files.Add(destUrl, contents);
}
But i need to create a text file in document Library which contains a content like "This is a new file" without saving it in local disk.

You can use a MemoryStream instead of FileStream.

You can encode the string into a byte array and create the file from that array.
As an aside, note that your code leaks an SPSite and an SPWeb, which is quite dangerous since those objects can take a lot of memory. You need to properly dispose of them, e.g. with nested using statements:
using System.Text;
public void AddNewFile(string destUrl)
{
using (SPSite site = new SPSite(destUrl)) {
using (SPWeb web = site.OpenWeb()) {
byte[] bytes = Encoding.GetEncoding("UTF-8").GetBytes(
"This is a new file.");
web.Files.Add(destUrl, bytes);
}
}
}

Something like that:
public void UploadText(string text, Encoding encoding, string destUrl)
{
SPWeb site = new SPSite(destUrl).OpenWeb();
site.Files.Add(destUrl, encoding.GetBytes(text));
}
PS: you will need an encoding to convert from a string to an array of bytes. You can hardcode one or pass it as a parameter just like I did.