Develop Reference

gRPC client streaming taking much longer than server streaming, despite the client and server running on the same machine

I've implemented file uploading/downloading via gRPC, where the client is a console application and the server is an ASP.NET Core application. Uploading is implemented as a client-streaming RPC and downloading is implemented as a server-streaming RPC. The problem I'm having is that whenever I run both the client and the server on the same machine, it takes roughly 10 times as long to upload file x to the server file system as it does to download file x from the server file system (note that all of this is taking place on the hard drive of my machine). The client upload (download) code is symmetric to the server download (upload) code. Here is the relevant snippet of the client upload code:
var clientStreamingCall = client.Upload();
// Now open the file-to-upload for reading
using (FileStream source = File.OpenRead(Path.Combine(baseDir + #"\UploadFiles", fileName)))
{
byte[] buffer = new byte[1024];
int bytesRead;
// write the file chunks to the requestStream
while ((bytesRead = await source.ReadAsync(buffer, 0, buffer.Length)) > 0)
{
await clientStreamingCall.RequestStream.WriteAsync(new UploadRequest()
{
ChunkData = Google.Protobuf.ByteString.CopyFrom(buffer),
});
}
}
Here is the server download code, which mirrors the above client upload code:
public override async Task Download(DownloadRequest request, IServerStreamWriter<DownloadResponse> responseStream, ServerCallContext context)
{
using (FileStream source = File.OpenRead(request.FilePath))
{
byte[] buffer = new byte[1024];
int bytesRead;
// write the file chunks to the responseStream
while ((bytesRead = await source.ReadAsync(buffer, 0, buffer.Length)) > 0)
{
await responseStream.WriteAsync(new DownloadResponse()
{
ChunkData = Google.Protobuf.ByteString.CopyFrom(buffer),
});
}
}
}
Likewise, the client download code mirrors the server upload code; here is the client download code, which writes a chunk of the file-to-download in the response stream to the destination file (on the hard drive of my machine):
while (await serverStreamingCall.ResponseStream.MoveNext(new CancellationToken()))
{
dest.Write(serverStreamingCall.ResponseStream.Current.ChunkData.ToByteArray(), 0, serverStreamingCall.ResponseStream.Current.ChunkData.Length);
}
Here is the server upload code, which writes a chunk of the file-to-upload to the destination file (on the hard drive of my machine):
while (await requestStream.MoveNext())
{
dest.Write(requestStream.Current.ChunkData.ToByteArray(), 0, requestStream.Current.ChunkData.Length);
}
Any idea as to why uploading takes so much longer than downloading, in spite of this symmetry?

Redirecting an Audio Stream to avoid CORS - C#

I need to redirect an audio stream to avoid CORS in order to use AudioContext() for visualizations.
Instead of using the original stream url, I'd point the player to "mysite.com/stream/", which would intercept the stream and feed it to the player.
I really don't know what I'm doing in this case, which should be obvious from my attempt below. Thanks to anyone for some help.
System.Net.HttpWebRequest web = (System.Net.HttpWebRequest)System.Net.WebRequest.Create("https://ssl.geckohost.nz/proxy/caitlinssl?mp=/stream");
Response.ContentType = "audio/aac";
char[] buffer = new char[8192];
byte[] buffer_bytes = new byte[8192];
using (System.Net.HttpWebResponse web_resp = (System.Net.HttpWebResponse)web.GetResponse())
{
using (System.IO.Stream stream_web = web_resp.GetResponseStream())
{
//stream_web.CopyTo(Response.OutputStream);
using (System.IO.StreamReader stream_rdr = new System.IO.StreamReader(stream_web))
{
for (var i = 0; i < 100; i++)
{
stream_rdr.Read(buffer, 0, 8192);
Response.Write(buffer);
Response.Flush();
}
}
}
}

The solution I found was originally created for use with NAudio to play a remote MP3 file, but I was able to repurpose it for my needs. There are probably better ways, but this works.
Play audio from a stream using C#
Edit: It seems that the connection is dropped server to server, but I'm not positive. IIS will not send "Connection: keep-alive" to the client, however it is being sent to the client by the original stream. Tried adding KeepAlive = true to the WebRequest, and changing the protocol version to HTTP 1.0 to keep the server to server connection alive with no joy.
string url = "https://ssl.geckohost.nz/proxy/caitlinssl?mp=/stream";
Response.ContentType = "audio/aac";
using (Stream stream = WebRequest.Create(url).GetResponse().GetResponseStream())
{
byte[] buffer = new byte[32768];
int read;
while ((read = stream.Read(buffer, 0, buffer.Length)) > 0)
{
Response.OutputStream.Write(buffer, 0, read);
Response.Flush();
}
}
Conclusion: It seems that in this case it is not possible with IIS to persist the connection server to server, as far as I was able to find. I'm not worried about scalability as this is a personal project for a user of 1. The solution I ended up with is to start capturing the stream with ffmpeg on click, transcode it to flac/mkv, and use a variation of the above to read the file while it's being written and feed it to the player (WPF WebView [IE/Edge]).

Sending large files to the server

I'm trying to send large files to Asp.net server from my Xamarin Android application. I found the best way to do it would be to send the file in chunks rather than sending it whole. I'm following the tutorial here which usesHttpWebRequest to send the request to the server. In the following lines of code in the tutorial, the file is sent in chunks to the server:
/ Send the file binaries over to the server, in 1024 bytes chunk
FileStream fileStream = new FileStream(fileUrl, FileMode.Open,
FileAccess.Read);
byte[] buffer = new byte[1024];
int bytesRead = 0;
while ((bytesRead = fileStream.Read(buffer, 0, buffer.Length)) != 0)
{
s.Write(buffer, 0, bytesRead);
} // end while
fileStream.Close();
I know that s.Write(buffer, 0, bytesRead) writes the byte on the network socket but how does the back and forth communication work? Does the client wait for the server to respond and send the next chunk to the server or all chunks are written on the network stream and sent together?
Right now the server code I have receives the request which is the full file rather than it's chunks. My server side code looks like this:
public async Task<HttpResponseMessage> UploadFile()
{
var httpRequest = HttpContext.Current.Request;
if (httpRequest.Files.Count > 0)
{
foreach (string file in httpRequest.Files)
{
var postedFile = httpRequest.Files[file];
var filePath = HttpContext.Current.Server.MapPath("~/" + postedFile.FileName);
postedFile.SaveAs(filePath);
// NOTE: To store in memory use postedFile.InputStream
}
return Request.CreateResponse(HttpStatusCode.Created);
}
return Request.CreateResponse(HttpStatusCode.BadRequest);
}
How can I make sure that the connection doesn't time out while the file is sending and is this the most efficient way of sending large file to the server? How can I resume the upload if their a blimp in the netowrk connection on the client side?

Streaming files from amazon s3

I have a problem trying to stream files from amazon s3. Basically, I have files stored on amazom s3, I can't provide direct access to these files as users need to be authenticated. I'm trying to find a way to stream files without downloading each file from amazon onto my server and then from my server to the end client. I just want to be able to stream the file direct, but it seems most of the stream reader classes in .NET are not cable of doing this. Please correct me if I am wrong.

You can stream the file from Amazon S3 to the client through your server without downloading the file to your server, by opening a stream to the Amazon S3 file then read from it and write on the client stream (buffer by buffer).
Sample Code:
byte[] buffer = new byte[BUFFER_SIZE];
GetObjectRequest getObjRequest = new GetObjectRequest().WithBucketName(Bucket_Name).WithKey(Object_Key);
using (GetObjectResponse getObjRespone = amazonS3Client.GetObject(getObjRequest))
using (Stream amazonStream = getObjRespone.ResponseStream)
{
int bytesReaded = 0;
Response.AddHeader("Content-Length", getObjRespone.ContentLength.ToString());
while ((bytesReaded = amazonStream.Read(buffer, 0, buffer.Length)) > 0 && Response.IsClientConnected)
{
Response.OutputStream.Write(buffer, 0, bytesReaded);
Response.OutputStream.Flush();
buffer = new byte[BUFFER_SIZE];
}
}

using python to send an image over TCP to c# server

I am working on a project that uses a python script to send an image taken with a webcam to a c# webserver using sockets. I am currently sending strings to the server from the python client using code like:
info = bytearray("Text to send", "UTF-8")
socket.send(info)
to send data to the c# server and it functions perfectly for text and numbers. I run into an issue when trying to encode the data read from the .bmp into "UTF-8", as trying this returns an error of not being able to encode certain characters into UTF-8.
I was wondering if anyone had any idea of a way to encode this that c# will be able to recognize, or, if there is a better way of trying to implement this process, I am all ears.
A couple of options I have come up with would be to 1 - use something like google drive to save the image to, or an FTP server and then have the c# server retrieve it from there or 2 - create a packet system containing the RGB values and recreating the image from those pixel values on the server.
Thanks for your help.
EDIT: I have tried sending the file this way
data = bytearray("123456789","UTF-8")
file = open("image.bmp", "rb")
data += file.read()
socket.send(data)
and was able to successfully retreive the string "123456789", but the data after this is garbage. I have also implemented sending the size of the file before sending the data and that size number is retrieved fine, but the img data saves as a black bmp.
Edit 2 :
Here is the server and client code I am using to try and recreate the image using a memory stream. I the client code is using the process mentioned by hcalves.
Client
if __name__ == "__main__":
sock = socket.socket()
sock.connect(("localhost", 50839))
with open("image.bmp", "rb") as fd:
buf = fd.read(1024)
while (buf):
sock.send(buf)
buf = fd.read(1024)
sock.close()
Server
Socket client = server.AcceptSocket();
NetworkStream stream = new NetworkStream(client);
byte[] imgData = new byte[1024];
MemoryStream memstream = new MemoryStream();
stream.Read(imgData,0, 1024);
int counter = 0;
while (stream.DataAvailable)
{
memstream.Write(imgData, 0, 1024);
stream.Read(imgData, 0, 1024);
counter = counter + 1024;
}
memstream.Seek(0, SeekOrigin.Begin);
using (Stream file = File.OpenWrite("img.bmp"))
{
byte[] buffer = new byte[8*1024];
int len;
while ((len = memstream.Read(buffer, 0, buffer.Length)) > 0)
{
file.Write(buffer,0,len);
}
}

You shouldn't need more than this recipe:
import socket
if __name__ == "__main__":
sock = socket.socket()
sock.connect(("localhost", 50839))
with open("data.bin", "rb") as fd:
buf = fd.read(1024)
while (buf):
sock.send(buf)
buf = fd.read(1024)
sock.close()
For practical reasons, you can treat str objects (the result of fd.read) as raw data, you don't need any further crazy encoding. Just iterate the file and send over the socket. Test by running this server which just echoes to stdout with python server.py > data2.bin:
import socket
import sys
if __name__ == "__main__":
sock = socket.socket()
sock.bind(("localhost", 50839))
sock.listen(1)
client, address = sock.accept()
buf = client.recv(1024)
while (buf):
sys.stdout.write(buf)
buf = client.recv(1024)
client.close()
sock.close()
A checksum shows the binary data is sent correctly:
% md5 data.bin data2.bin
MD5 (data.bin) = 8b3280072275badf3e53a6f7aae0b8be
MD5 (data2.bin) = 8b3280072275badf3e53a6f7aae0b8be
Your C# server should be able to accept this data as is. If it doesn't work, it's because your server is expecting something in particular, not just raw data.