Develop Reference

How can I save a stream of Json data to a text file in C# Windows Forms app?

I've got a stream of data incoming as a Json file and I'm trying to save it to a text file, I've got it working here below however, when i check the file, it only has the last Json message received saved, I am trying to get it so that once it saves a line it goes onto a new line and prints the latest Json message below. at the moment it will print let's say 1000 lines but they are all the same and they match the latest Json received.
Any help would be much appreciated.
void ReceiveData() //This function is used to listen for messages from the flight simulator
{
while (true)
{
NetworkStream stream = client.GetStream(); //sets the network stream to the client's stream
byte[] buffer = new byte[256]; //Defines the max amount of bytes that can be sent
int bytesRead = stream.Read(buffer, 0, buffer.Length);
if (bytesRead > 0)
{
string jsonreceived = Encoding.ASCII.GetString(buffer, 0, bytesRead); //Converts the received data into ASCII for the json variable
JavaScriptSerializer serializer = new JavaScriptSerializer();
TelemetryUpdate telemetry = serializer.Deserialize<TelemetryUpdate>(jsonreceived);
this.Invoke(new Action(() => { TelemetryReceivedLabel.Text = jsonreceived;
})) ;
Updatelabels(telemetry); //runs the update labels function with the telemetry data as an argument
File.Delete(#"c:\temp\BLACKBOX.txt"); // this deletes the original file
string path = #"c:\temp\BLACKBOX.txt"; //this stores the path of the file in a string
using (StreamWriter sw = File.CreateText(path)) // Create a file to write to.
{
for (int i = 0; i<10000; i++)
{
sw.Write(jsonreceived.ToString()); //writes the json data to the file
}
}
}
}
}

As per the .NET documentation for File.CreateText:
Creates or opens a file for writing UTF-8 encoded text. If the file already exists, its contents are overwritten.
So, every time you call File.CreateText you're creating a new StreamWriter that's going to overwrite the contents of your file. Try using File.AppendText instead to pick up where you left off.

Extract the file header signature as it is being streamed directly to disk in ASP.NET Core

I have an API method that streams uploaded files directly to disk to be scanned with a virus checker. Some of these files can be quite large, so IFormFile is a no go:
Any single buffered file exceeding 64 KB is moved from memory to a
temp file on disk.
Source: https://learn.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-3.1
I have a working example that uses multipart/form-data and a really nice NuGet package that takes out the headache when working with multipart/form-data, and it works well, however I want to add a file header signature check, to make sure that the file type defined by the client is actually what they say it is. I can't rely on the file extension to do this securely, but I can use the file header signature to make it at least a bit more secure. Since I'm am streaming directly to disk, how can I extract the first bytes as it's going through the file stream?
[DisableFormValueModelBinding] // required for form binding
[ValidateMimeMultipartContent] // simple check to make sure this is a multipart form
[FileUploadOperation(typeof(SwaggerFileItem))] // used to define the Swagger schema
[RequestSizeLimit(31457280)] // 30MB
[RequestFormLimits(MultipartBodyLengthLimit = 31457280)]
public async Task<IActionResult> PostAsync([FromRoute] int customerId)
{
// place holders
var uploadLocation = string.Empty;
var trustedFileNameForDisplay = string.Empty;
// this is using a nuget package that does the hard work on reading the multipart form-data.... using UploadStream;
var model = await this.StreamFiles<FileItem>(async x =>
{
// never trust the client
trustedFileNameForDisplay = WebUtility.HtmlEncode(Path.GetFileName(x.FileName));
// determien the quarantine location
uploadLocation = GetUploadLocation(trustedFileNameForDisplay);
// stream the input stream to the file stream
// importantly this should never load the file into memory
// it should be a straight pass through to disk
await using var fs = System.IO.File.Create(uploadLocation, BufSize);
// --> How do I extract the file signature? I.e. a copy of the header bytes as it is being streamed??? <--
await x.OpenReadStream().CopyToAsync(fs);
});
// The model state can now be checked
if (!ModelState.IsValid)
{
// delete the file
DeleteFileIfExists(uploadLocation);
// return a bad request
ThrowProblemDetails(ModelState, StatusCodes.Status400BadRequest);
}
// map as much as we can
var request = _mapper.Map<CreateAttachmentRequest>(model);
// map the remaining properties
request.CustomerId = customerId;
request.UploadServer = Environment.MachineName;
request.uploadLocation = uploadLocation;
request.FileName = trustedFileNameForDisplay;
// call mediator with this request to send it over WCF to Pulse Core.
var result = await _mediator.Send(request);
// build response
var response = new FileResponse { Id = result.FileId, CustomerId = customerId, ExternalId = request.ExternalId };
// return the 201 with the appropriate response
return CreatedAtAction(nameof(GetFile), new { fileId = response.Id, customerId = response.customerId }, response);
}
The section I'm stuck on is around the line await x.OpenReadStream().CopyToAsync(fs);. I would like to pull out the file header here as the stream is being copied to the FileStream. Is there a way to add some kind of inspector? I don't want to read the entire stream again, just the header.
Update
Based on the answer given by #Ackdari I have successfully switched the code to extract the header from the uploaded file stream. I don't know if this could be made any more efficient, but it does work:
//...... removed for clarity
var model = await this.StreamFiles<FileItem>(async x =>
{
trustedFileNameForDisplay = WebUtility.HtmlEncode(Path.GetFileName(x.FileName));
quarantineLocation = QuarantineLocation(trustedFileNameForDisplay);
await using (var fs = System.IO.File.Create(quarantineLocation, BufSize))
{
await x.OpenReadStream().CopyToAsync(fs);
fileFormat = await FileHelpers.GetFileFormatFromFileHeader(fs);
}
});
//...... removed for clarity
and
// using https://github.com/AJMitev/FileTypeChecker
public static async Task<IFileType> GetFileFormatFromFileHeader(FileStream fs)
{
IFileType fileFormat = null;
fs.Position = 0;
var headerData = new byte[40];
var bytesRead = await fs.ReadAsync(headerData, 0, 40);
if (bytesRead > 0)
{
await using (var ms = new MemoryStream(headerData))
{
if (!FileTypeValidator.IsTypeRecognizable(ms))
{
return null;
}
fileFormat = FileTypeValidator.GetFileType(ms);
}
}
return fileFormat;
}

You may want to consider reading the header yourself dependent on which file type is expected
int n = 4; // length of header
var headerData = new byte[n];
var bytesRead = 0;
while (bytesRead < n)
bytesRead += await x.ReadAsync(headerData.AsMemory(bytesRead));
CheckHeader(headerData);
await fs.WriteAsync(headerData.AsMemory());
await x.CopyToAsync(fs);

ASP.NET MVC 5 best approach for outputting and downloading file progress

I must to create a function which allow an user click a button in browser and request to server to get information from database. The information will be output to a .csv file, after that client able to download it.
I intend to use one of two method of System.Web.Mvc.Controller
protected internal virtual FileStreamResult File(Stream fileStream, string contentType, string fileDownloadName)
or
protected internal virtual FileContentResult File(byte[] fileContents, string contentType, string fileDownloadName);
But the output file size can be up to 4 gigabytes, so I worry the buffering size of a file in Stream or byte[] will cause server memory leaks.
I split my progress into 2 steps
Step 1: output csv file with memory leaks consideration
public CsvOutputHelper(List<string> csvHeader)
{
this._csvHeader = csvHeader;
}
public void OutputFile(List<List<string>> data, string filePath)
{
// Create the new file and write the header text
File.WriteAllText(filePath, this._csvHeader.ConvertToCsvRecord());
// 'StringBuilder' for output file
var sb = new StringBuilder();
sb.AppendLine();
// Line counter
var lineCounterValue = 1;
for (var i = 0; i < data.Count; i++)
{
// Create line content of csv file and append it to buffer string
sb.AppendLine(data[i].ConvertToCsvRecord());
// Increase value of line counter
lineCounterValue++;
// If buffer string is reach to 100 lines or the loop go to the end of data list,
// output text to file and reset value of buffer string and value of line counter
if (lineCounterValue == MaxCountOfOutputLine || i == data.Count - 1)
{
// Output buffer string
File.AppendAllText(filePath, sb.ToString());
sb = new StringBuilder(); // Re-create a new instance of 'StringBuilder'
lineCounterValue = 1; // Reset line counter value to 1
}
}
}
Step 2: return output filePath in server (relative path) to browser and browser request to path to download file.
Is my solution is a good way to implement in that case? Is using Stream or bytes will to be caused server memory leaks or not? Please explain for me.

Merging files with different extensions into another file bytes

I'm trying to figure out how to attach a few files whose paths can be found at a ListBox into another file bytes. The files can have different extensions. For example one may be text document, one picture and one executable. So, when you open the main file, they all should execute in same time with the main file. I have a Server/Client project where I send the main file bytes to the server and he does his job.
byte[] mainFileBytes = File.ReadAllBytes(MainFilePath);
int mainFileLength = mainFileBytes.Length;
byte[] mainFileLengthBuffer = BitConverter.GetBytes(mainFileLength);
Globals.GlobalBuffer = Methods.CombineBytes(mainFileLengthBuffer, mainFileBytes);
client.GetStream().Write(Globals.GlobalBuffer, 0, Globals.GlobalBuffer.Length);
This is how I send the main file to the server. Somehow, I have to attach those files to the main file.
Edit: Thinking of sending each of those files to the server, but the problem is that I don't know how to distinguish them.
Send code:
var requestBytes = new byte[26];
var requestByteCount = client.GetStream().Read(requestBytes, 0, requestBytes.Length);
if (Encoding.Unicode.GetString(requestBytes, 0, requestByteCount).Contains("ATTACHEDFILES"))
{
if (chMFBListBox.Items.Count > 0)
{
foreach (string file in chMFBListBox.Items)
{
var fileBytes = File.ReadAllBytes(file);
var fileLength = fileBytes.Length;
var fileLengthBuffer = BitConverter.GetBytes(fileLength);
Globals.GlobalBuffer = Methods.CombineBytes(fileLengthBuffer, fileBytes);
client.GetStream().Write(Globals.GlobalBuffer, 0, Globals.GlobalBuffer.Length);
}
}
else
{
// Notifying the server there aren't attached files
Globals.GlobalBuffer = Encoding.Unicode.GetBytes("NO");
client.GetStream().Write(Globals.GlobalBuffer, 0, Globals.GlobalBuffer.Length);
}
}

Attach custom (hidden) data to an image in Microsoft Word Add-in

I'm attempting to write an Office Add-in for Microsoft Word (2007 and higher) using VSTO on Visual Studio 2010 and C#.
The add-in will insert images into the document that are downloaded from the internet (from a custom CGI script). These images update daily, so I have added a button to 'refresh' all the images from the server upon request. However, I'm not certain how to 'store' the original identifier for the image along with the image embedded in the document to know which image to get from the server. The identifier can be anywhere from a few characters to several hundred (~200) characters long, but are an ASCII identifier.
Currently I insert the picture like so:
public void InsertPictureFromIdentifier(String identifier)
{
Document vstoDocument = Globals.Factory.GetVstoObject(this.Application.ActiveDocument);
Word.Selection selection = this.Application.Selection;
if (selection != null && selection.Range != null)
{
// Insert the picture control
var picture = vstoDocument.Controls.AddPictureContentControl(selection.Range, "mypic");
// And update the image
UpdatePicture(picture, identifier);
}
}
Then UpdatePicture is called both on the initial insert, and upon refresh, to update the image:
public void UpdatePicture(PictureContentControl picture, string identifier)
{
const int BytesToRead = 1000000;
// Download the image from the scrip
var request = WebRequest.Create("http://my.server.com/graph.cgi?"+identifier);
var response = request.GetResponse();
var responseStream = response.GetResponseStream();
var reader = new BinaryReader(responseStream);
var memoryStream = new MemoryStream();
Byte[] byteBuffer = new byte[BytesToRead];
// Transfer to a memory stream
var bytesRead = reader.Read(byteBuffer, 0, BytesToRead);
while (bytesRead > 0)
{
memoryStream.Write(byteBuffer, 0, bytesRead);
bytesRead = reader.Read(byteBuffer, 0, BytesToRead);
}
// Set the image from the memory stream
picture.Image = new System.Drawing.Bitmap(memoryStream);
}
As you can see I pass the identifier into update - but the question is how to get that identifier back from the 'picture' I've inserted the next day on a refresh. I've tried using the picture's Tag, but that is limited to 64 characters. I've even tried using the picture's Title property, but that seems to silently fail on some other maximum limit.
The identifier will need to be persisted between save / loads, move with the image in the document, and be invisible (I can't just add text after the image with the identifier).