Given the following method to read asynchronously from a Stream (simplified):
private async void WaitForIncomingMessagesAsync()
{
var responseStream = WebRequest.Create("some url").GetResponse().GetResponseStream(); // onle here in the example
var buffer = new byte[8192];
var stringBuilder = new StringBuilder();
while (IsRunning /*set by other parts of the program*/)
{
int receivedDataCount;
do
{
receivedDataCount = await responseStream.ReadAsync(buffer, 0, buffer.Length);
if (receivedDataCount != 0)
{
stringBuilder.Append(Encoding.ASCII.GetString(buffer, 0, receivedDataCount));
}
} while (receivedDataCount > 0);
var receivedDataJson = stringBuilder.ToString();
Array.Clear(buffer, 0, buffer.Length);
stringBuilder.Clear();
}
}
This works fine until I receive the first 'message' over the stream.
Since then, it will no longer stop at await because it always reads 0.
The stream itself must be kept open all the time.
What am I missing? Do I have to reset the stream, flush it or set the position back to 0?
responseStream.Position = 0; or responseStream.Seek(0, SeekOrigin.Begin)
throws NotSupportedException
Based on your code I suppose that you want to read new messages each time your outer loops runs. If so then your GetResponse should be inside the outer loop otherwise you will never read new data once your internal loop finished iteration over the stream.
private async void WaitForIncomingMessagesAsync()
{
var buffer = new byte[8192];
var stringBuilder = new StringBuilder();
while (IsRunning)
{
using (var responseStream = WebRequest.Create("some url").GetResponse().GetResponseStream())
{
int receivedDataCount;
do
{
receivedDataCount = await responseStream.ReadAsync(buffer, 0, buffer.Length);
if (receivedDataCount != 0)
{
stringBuilder.Append(Encoding.ASCII.GetString(buffer, 0, receivedDataCount));
}
} while (receivedDataCount > 0);
}
var receivedDataJson = stringBuilder.ToString();
Array.Clear(buffer, 0, buffer.Length);
stringBuilder.Clear();
}
}
I also wrapped your response stream into using (..) {} construction so you don't have to worry about closing and disposing it manually.
Related
I am reading a .bin file and writing it into a Stream. Later, I am reading that stream object and then writing it into a Network Stream. Code is as following:
public async Task<bool> UploadFirmware(Stream _stream)
{
bool success = false;
try
{
_tcpclient = new TcpClient();
_tcpclient.Connect(_deviceip, port);
_stream.Seek(0, SeekOrigin.Begin);
m_NetworkStream = _tcpclient.GetStream();
byte[] buffer = new byte[1024];
m_ReadBuffer = new byte[1024];
int readcount = 0;
m_NetworkStream.BeginRead(m_ReadBuffer, 0, m_ReadBuffer.Length, new AsyncCallback(EndReceive), null);
await Task.Run(() =>
{
while ((readcount = _stream.Read(buffer, 0, buffer.Length)) > 0)
{
m_NetworkStream.Write(buffer, 0, readcount);
m_NetworkStream.Flush();
}
});
success = true;
}
catch (Exception ex)
{}
return success;
}
Normally, this code works fine, but sometimes on an IP Address, the code gets stuck at m_NetworkStream.Write(buffer, 0, readcount);. The thing is, I am updating the status in UI based on success value, but the code gets hanged at above mentioned line and doesn't come out at all. No exception is thrown at all to identify the issue. So, in UI the status is not updated, and unexpected result is produced. I am not able to identify the issue. Help of any kind will be highly appreciated.
EDIT:
Also, I have to do an operation in parallel. The code for EndReceive is as follows:
private void EndReceive(IAsyncResult ar)
{
try
{
int nBytes;
nBytes = m_NetworkStream.EndRead(ar);
if (nBytes > 0)
{
string res = Encoding.UTF8.GetString(m_ReadBuffer, 0, nBytes);
DeviceStatus status = new DeviceStatus();
string[] readlines = res.Split(new string[] { CRLF }, StringSplitOptions.RemoveEmptyEntries);
foreach (string readline in readlines)
{
if (readline.StartsWith("CODE"))
{
status.code = Convert.ToInt32(readline.Replace("CODE=", ""));
break;
}
}
status.deviceip = this._deviceip;
status.port = this.port;
status.DeviceID = this._DeviceID;
status.FirmwareID = this._FirmwareID;
status.FilePath = this._Path;
StatusUpdate(status);
m_ReadBuffer = new byte[1024];
}
}
catch (ObjectDisposedException ods)
{
return;
}
if (_tcpclient.Connected)
m_NetworkStream.BeginRead(m_ReadBuffer, 0, m_ReadBuffer.Length, new AsyncCallback(EndReceive), null);
}
I'm not sure you should have a while loop in order to write bytes read from a file (from disk) to a network stream ... you can just read all the bytes and write to the stream and flush in one move.
You can also add a write timeout to specify how much time can pass before the stream write operation fails, to prevent any possibility of 'hanging'.
With these modifications, the code would look something like this:
// make the tcp connection to the remote endpoint
_tcpclient = new TcpClient();
_tcpclient.Connect(_deviceip, port);
// read the file bytes in one operation
var allBytes = File.ReadAllBytes(fileNameOnDisk);
// get the network stream
m_NetworkStream = _tcpclient.GetStream();
// wait a max of 500ms for the write operation to happen
m_NetworkStream.WriteTimeout = 500;
// write the file bytes to the stream and flush without while/stream/seek etc.
m_NetworkStream.Write(allBytes, 0, allBytes.Length);
m_NetworkStream.Flush();
And when you've finished with the stream:
m_NetworkStream.Close();
m_NetworkStream.Dispose();
It seem odd that you starting to read from network stream (m_NetworkStream.BeginRead(...)) and right away in another thread starting to write into same stream (m_NetworkStream.Write(...)). I would suggest to finish reading first and then start writing. Also you could use Stream.CopyTo to copy data between streams.
public async Task<bool> UploadFirmware(Stream fileStream, IPEndPoint deviceEP)
{
bool success = false;
try
{
TcpClient client = new TcpClient();
client.Connect(deviceEP);
NetworkStream networkStream = client.GetStream();
BeginReadFromDevice(networkStream);
// send bin data to device
await fileStream.CopyToAsync(networkStream);
success = true;
}
catch (Exception)
{
}
return success;
}
private void BeginReadFromDevice(Stream networkStream)
{
byte[] buffer = new byte[1024];
networkStream.BeginRead(buffer, 0, buffer.Length, new AsyncCallback(EndReceive), null);
}
I am trying to read all data present in the buffer of the Machine connected through TCP/IP but i don't know why i am not getting all data ,some data is getting Missed.
Here is the code that i am using ..
using (NetworkStream stream = client.GetStream())
{
byte[] data = new byte[1024];
int numBytesRead = stream.Read(data, 0, data.Length);
if (numBytesRead > 0)
{
string str= Encoding.ASCII.GetString(data, 0, numBytesRead);
}
}
Please tell me what i am missing to get all the data from the machine.
Thanks in advance..
The problem with your code is that you will not get all the data if the data size is bigger than the buffer size (1024 bytes in your case) so you have to Read the stream inside the loop. Then you can Write all the data inside a MemoryStream until the end of the NetworkStream.
string str;
using (NetworkStream stream = client.GetStream())
{
byte[] data = new byte[1024];
using (MemoryStream ms = new MemoryStream())
{
int numBytesRead ;
while ((numBytesRead = stream.Read(data, 0, data.Length)) > 0)
{
ms.Write(data, 0, numBytesRead);
}
str = Encoding.ASCII.GetString(ms.ToArray(), 0, (int)ms.Length);
}
}
This example from MSDN: NetworkStream.DataAvailable shows how you can use that property to do so:
// Examples for CanRead, Read, and DataAvailable.
// Check to see if this NetworkStream is readable.
if(myNetworkStream.CanRead)
{
byte[] myReadBuffer = new byte[1024];
StringBuilder myCompleteMessage = new StringBuilder();
int numberOfBytesRead = 0;
// Incoming message may be larger than the buffer size.
do{
numberOfBytesRead = myNetworkStream.Read(myReadBuffer, 0, myReadBuffer.Length);
myCompleteMessage.AppendFormat("{0}", Encoding.ASCII.GetString(myReadBuffer, 0, numberOfBytesRead));
}
while(myNetworkStream.DataAvailable);
// Print out the received message to the console.
Console.WriteLine("You received the following message : " +
myCompleteMessage);
}
else
{
Console.WriteLine("Sorry. You cannot read from this NetworkStream.");
}
Try this:
private string GetResponse(NetworkStream stream)
{
byte[] data = new byte[1024];
using (MemoryStream memoryStream = new MemoryStream())
{
do
{
stream.Read(data, 0, data.Length);
memoryStream.Write(data, 0, data.Length);
} while (stream.DataAvailable);
return Encoding.ASCII.GetString(memoryStream.ToArray(), 0, (int)memoryStream.Length);
}
}
Try this code:
using (NetworkStream stream = client.GetStream())
{
while (!stream.DataAvailable)
{
Thread.Sleep(20);
}
if (stream.DataAvailable && stream.CanRead)
{
Byte[] data = new Byte[1024];
List<byte> allData = new List<byte>();
do
{
int numBytesRead = stream.Read(data,0,data.Length);
if (numBytesRead == data.Length)
{
allData.AddRange(data);
}
else if (numBytesRead > 0)
{
allData.AddRange(data.Take(numBytesRead));
}
} while (stream.DataAvailable);
}
}
Hope this helps, it should prevent that you miss any data sended to you.
The synchronous method sometimes does not display the request body. Using the asynchronous method stably displays request body.
string request = default(string);
StringBuilder sb = new StringBuilder();
byte[] buffer = new byte[client.ReceiveBufferSize];
int bytesCount;
if (client.GetStream().CanRead)
{
do
{
bytesCount = client.GetStream().ReadAsync(buffer, 0, buffer.Length).Result;
sb.Append(Encoding.UTF8.GetString(buffer, 0, bytesCount));
}
while(client.GetStream().DataAvailable);
request = sb.ToString();
}
TCP itself does not have any ways to define "end of data" condition. This is responsibility of application level portocol.
For instance see HTTP request description:
A client request (consisting in this case of the request line and only one header field) is followed by a blank line, so that the request ends with a double newline
So, for request end of data is determined by two newline sequences. And for response:
Content-Type specifies the Internet media type of the data conveyed by the HTTP message, while Content-Length indicates its length in bytes.
The response content size is specified in header before data.
So, it's up to you how to encode amount of data transferred at once - it can be just first 2 or 4 bytes in the beginning of the data holding total size to read or more complex ways if needed.
for my scenario, the message itself was telling the length of subsequent message. here is the code
int lengthOfMessage=1024;
string message = "";
using (MemoryStream ms = new MemoryStream())
{
int numBytesRead;
while ((numBytesRead = memStream.Read(MessageBytes, 0, lengthOfMessage)) > 0)
{
lengthOfMessage = lengthOfMessage - numBytesRead;
ms.Write(MessageBytes, 0, numBytesRead);
}
message = Encoding.ASCII.GetString(ms.ToArray(), 0, (int)ms.Length);
}
#George Chondrompilas answer is correct but instead of writing it by yourself you can use CopyTo function which does the same :
https://stackoverflow.com/a/65188160/4120180
I was writing a light-weight proxy in c#. When I was decoding the gzip contentEncoding I noted that if I use a small buffer-size(4096) the stream is decoded partially depending the size of the input. Is it a bug in my code or something which is needed to make it work? I set the buffer to 10 MB, and it works okay but defeats my purpose of writing a light-weight proxy.
response = webEx.Response as HttpWebResponse;
Stream input = response.GetResponseStream();
//some other operations on response header
//calling DecompressGzip here
private static string DecompressGzip(Stream input, Encoding e)
{
StringBuilder sb = new StringBuilder();
using (Ionic.Zlib.GZipStream decompressor = new Ionic.Zlib.GZipStream(input, Ionic.Zlib.CompressionMode.Decompress))
{
// works okay for [1024*1024*8];
byte[] buffer = new byte[4096];
int n = 0;
do
{
n = decompressor.Read(buffer, 0, buffer.Length);
if (n > 0)
{
sb.Append(e.GetString(buffer));
}
} while (n > 0);
}
return sb.ToString();
}
Actually, I figured it out. I guess using the string builder causes the problem; instead, I used a memory stream and it works well.
private static string DecompressGzip(Stream input, Encoding e)
{
using (Ionic.Zlib.GZipStream decompressor = new Ionic.Zlib.GZipStream(input, Ionic.Zlib.CompressionMode.Decompress))
{
int read = 0;
var buffer = new byte[4096];
using (MemoryStream output = new MemoryStream())
{
while ((read = decompressor.Read(buffer, 0, buffer.Length)) > 0)
{
output.Write(buffer, 0, read);
}
return e.GetString(output.ToArray());
}
}
}
I want to save the incoming stream data to a WAV file on my hard disk drive. How can I change the code below to be able to record the stream into a valid WAV file?
From the demo here:
private void StreamMP3(object state)
{
this.fullyDownloaded = false;
string url = (string)state;
webRequest = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse resp = null;
try
{
resp = (HttpWebResponse)webRequest.GetResponse();
}
catch(WebException e)
{
if (e.Status != WebExceptionStatus.RequestCanceled)
{
ShowError(e.Message);
}
return;
}
byte[] buffer = new byte[16384 * 4]; // Needs to be big enough to hold a decompressed frame
IMp3FrameDecompressor decompressor = null;
try
{
using (var responseStream = resp.GetResponseStream())
{
var readFullyStream = new ReadFullyStream(responseStream);
do
{
if (bufferedWaveProvider != null &&
bufferedWaveProvider.BufferLength - bufferedWaveProvider.BufferedBytes <
bufferedWaveProvider.WaveFormat.AverageBytesPerSecond / 4)
{
Debug.WriteLine("Buffer getting full, taking a break");
Thread.Sleep(500);
}
else
{
Mp3Frame frame = null;
try
{
frame = Mp3Frame.LoadFromStream(readFullyStream);
}
catch (EndOfStreamException)
{
this.fullyDownloaded = true;
// Reached the end of the MP3 file / stream
break;
}
catch (WebException)
{
// Probably we have aborted download from the GUI thread
break;
}
if (decompressor == null)
{
// I don't think these details matter too much - just help ACM select the right codec.
// However, the buffered provider doesn't know what sample rate it is working at
// until we have a frame.
WaveFormat waveFormat = new Mp3WaveFormat(
frame.SampleRate,
frame.ChannelMode == ChannelMode.Mono ? 1 : 2,
frame.FrameLength,
frame.BitRate);
decompressor = new AcmMp3FrameDecompressor(waveFormat);
this.bufferedWaveProvider = new BufferedWaveProvider(decompressor.OutputFormat);
this.bufferedWaveProvider.BufferDuration = TimeSpan.FromSeconds(20); // Allow us to get well ahead of ourselves
//this.bufferedWaveProvider.BufferedDuration = 250;
}
int decompressed = decompressor.DecompressFrame(frame, buffer, 0);
//Debug.WriteLine(String.Format("Decompressed a frame {0}", decompressed));
bufferedWaveProvider.AddSamples(buffer, 0, decompressed);
}
} while (playbackState != StreamingPlaybackState.Stopped);
Debug.WriteLine("Exiting");
// I was doing this in a finally block, but for some reason
// we are hanging on response stream .Dispose, so we never get there.
decompressor.Dispose();
}
}
finally
{
if (decompressor != null)
{
decompressor.Dispose();
}
}
}
I wouldn't take that particular approach to saving to disk. It's a bit too hands-on, because it has to deal with playing back at the right rate. Just buffer up the response, and then wrap it in an Mp3FileReader stream and use WaveFileWriter to write the WAV file:
MemoryStream mp3Buffered = new MemoryStream();
using (var responseStream = resp.GetResponseStream())
{
byte[] buffer = new byte[65536];
int bytesRead = responseStream.Read(buffer, 0, buffer.Length);
while (bytesRead > 0)
{
mp3Buffered.Write(buffer, 0, bytesRead);
bytesRead = responseStream.Read(buffer, 0, buffer.Length);
}
}
mp3Buffered.Position = 0;
using (var mp3Stream = new Mp3FileReader(mp3Buffered))
{
WaveFileWriter.CreateWaveFile("file.wav", mp3Stream);
}
That does, of course, assume that your MP3 file's wave format is compatible with WAV and in particular, your WAV player. If it isn't, you'll need to inject and add a WaveFormatConversion stream as well.
You can use following line to save to MemoryStream :
mp3Buffered.Write(frame.RawData, 0, frame.RawData.Length);
Saving stream to file is described in MattW's answer.
As part of an upcoming project at my university, I need to write a client that downloads a media file from a server and writes it to the local disk. Since these files can be very large, I need to implement partial download and serialization in order to avoid excessive memory use.
What I came up with:
namespace PartialDownloadTester
{
using System;
using System.Diagnostics.Contracts;
using System.IO;
using System.Net;
using System.Text;
public class DownloadClient
{
public static void Main(string[] args)
{
var dlc = new DownloadClient(args[0], args[1], args[2]);
dlc.DownloadAndSaveToDisk();
Console.ReadLine();
}
private WebRequest request;
// directory of file
private string dir;
// full file identifier
private string filePath;
public DownloadClient(string uri, string fileName, string fileType)
{
this.request = WebRequest.Create(uri);
this.request.Method = "GET";
var sb = new StringBuilder();
sb.Append("C:\\testdata\\DownloadedData\\");
this.dir = sb.ToString();
sb.Append(fileName + "." + fileType);
this.filePath = sb.ToString();
}
public void DownloadAndSaveToDisk()
{
// make sure directory exists
this.CreateDir();
var response = (HttpWebResponse)request.GetResponse();
Console.WriteLine("Content length: " + response.ContentLength);
var rStream = response.GetResponseStream();
int bytesRead = -1;
do
{
var buf = new byte[2048];
bytesRead = rStream.Read(buf, 0, buf.Length);
rStream.Flush();
this.SerializeFileChunk(buf);
}
while (bytesRead != 0);
}
private void CreateDir()
{
if (!Directory.Exists(dir))
{
Directory.CreateDirectory(dir);
}
}
private void SerializeFileChunk(byte[] bytes)
{
Contract.Requires(!Object.ReferenceEquals(bytes, null));
FileStream fs = File.Open(filePath, FileMode.Append);
fs.Write(bytes, 0, bytes.Length);
fs.Flush();
fs.Close();
}
}
}
For testing purposes, I've used the following parameters:
"http://itu.dk/people/janv/mufc_abc.jpg" "mufc_abc" "jpg"
However, the picture is incomplete (only the first ~10% look right) even though the content length prints 63780 which is the actual size of the image.
So my questions are:
Is this the right way to go for partial download and serialization or is there a better/easier approach?
Is the full content of the response stream stored in client memory? If this is the case, do I need to use HttpWebRequest.AddRange to partially download data from the server in order to conserve my client's memory?
How come the serialization fails and I get a broken image?
Do I introduce a lot of overhead when I use the FileMode.Append? (msdn states that this option "seeks to the end of the file")
Thanks in advance
You could definitely simplify your code using a WebClient:
class Program
{
static void Main()
{
DownloadClient("http://itu.dk/people/janv/mufc_abc.jpg", "mufc_abc.jpg");
}
public static void DownloadClient(string uri, string fileName)
{
using (var client = new WebClient())
{
using (var stream = client.OpenRead(uri))
{
// work with chunks of 2KB => adjust if necessary
const int chunkSize = 2048;
var buffer = new byte[chunkSize];
using (var output = File.OpenWrite(fileName))
{
int bytesRead;
while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) > 0)
{
output.Write(buffer, 0, bytesRead);
}
}
}
}
}
}
Notice how I am writing only the number of bytes I have actually read from the socket to the output file and not the entire 2KB buffer.
I don't know if this is the source of the problem, however I would change the loop like this
const int ChunkSize = 2048;
var buf = new byte[ChunkSize];
var rStream = response.GetResponseStream();
do {
int bytesRead = rStream.Read(buf, 0, ChunkSize);
if (bytesRead > 0) {
this.SerializeFileChunk(buf, bytesRead);
}
} while (bytesRead == ChunkSize);
The serialize method would get an additional argument
private void SerializeFileChunk(byte[] bytes, int numBytes)
and then write the right number of bytes
fs.Write(bytes, 0, numBytes);
UPDATE:
I do not see the need for closing and reopening the file each time. I also would use the using statement, which closes the resources, even if an exception should occur. The using statement calls the Dispose() method of the resource at the end, which in turn calls Close() in the case of file streams. using can be applied to all types implementing IDisposable.
var buf = new byte[2048];
using (var rStream = response.GetResponseStream()) {
using (FileStream fs = File.Open(filePath, FileMode.Append)) {
do {
bytesRead = rStream.Read(buf, 0, buf.Length);
fs.Write(bytes, 0, bytesRead);
} while (...);
}
}
The using statement does something like this
{
var rStream = response.GetResponseStream();
try
{
// do some work with rStream here.
} finally {
if (rStream != null) {
rStream.Dispose();
}
}
}
Here is the solution from Microsoft: http://support.microsoft.com/kb/812406
Updated 2021-03-16: seems the original article is not available now. Here is the archived one: https://mskb.pkisolutions.com/kb/812406