Develop Reference

c# JSON REST response via 3 different approaches (WebRequest, RESTSharp, HttpClient) is empty, but Postman and browser works

When calling a specific endpoint in C# which works without issues in Postman (or via Firefox), I'm getting an empty response.
The url I'm calling is returning a collection of data. In the url parameters I can specify how much of said data I want.
I've inspected the response size in Postman, and when I limit the amount of data requested in my C# call such that the response is around 700kb, then I get a JSON response back.
However, if I exceed this size in the C# call, then the response is empty '{ }' and the ContentLength returned = -1 (the statusCode returned is 200, so this seems fine at least). This same request which fails in C# works fine within Postman and Firefox however...
I somehow suspect this occurs because either the deserializer's buffer is not big enough OR because the response is still in transit and the code somehow continues executing before it has read the whole response body...
See below for the 3 implementations which I've tested:
1:
var httpClient = new HttpClient();
var responseMessage = await httpClient.GetAsync(requestUrl, HttpCompletionOption.ResponseHeadersRead);
if (responseMessage.StatusCode == System.Net.HttpStatusCode.OK)
{
using (var httpStream = await responseMessage.Content.ReadAsStreamAsync())
{
using (var sr = new StreamReader(httpStream))
{
Info(await sr.ReadToEndAsync()); //Info logs the string to a file
}
}
}
2 (RESTSharp):
var client = new RestClient();
var request = new RestRequest(requestUrl, Method.GET, DataFormat.Json);
Info(request.Content); //Info logs the string to a file
3:
var httpWebRequest = (HttpWebRequest)WebRequest.Create(requestUrl);
httpWebRequest.ContentType = "application/json; charset-utf8";
var httpWebResponse = httpWebRequest.GetResponse() as HttpWebResponse;
var binReader = new BinaryReader(responseStream);
const int bufferSize = 4096;
byte[] responseBytes;
using (MemoryStream ms = new MemoryStream())
{
byte[] buffer = new byte[bufferSize];
int count;
while ((count = binReader.Read(buffer, 0, buffer.Length)) != 0)
{
ms.Write(buffer, 0, count);
}
responseBytes = ms.ToArray();
}
Info(Encoding.UTF8.GetString(responseBytes, 0, responseBytes.Length)); //Info logs the string to a file
I'm not modifying the HttpClient.MaxResponseContentBufferSize property, but for good measure I've also tried changing this value, to no avail.
How can I resolve this?

I've found the issue, it was being caused by the backend service to which I was connecting. Thank you again #Panagiotis Kanavos

How to Download a File from Firebase Storage using a Valid downloadURL in C#

I have a valid download url for a file located in google firebase storage and I'm trying to download the file into my application written which is in c# via a HTTP Get request. However, the request fails with the error "WebException: The remote server returned an error: (400) Bad Request." I would really appreciate it if you could point me towards what I'm doing wrong. Thank you in advance for your help! -- Here is reference to the google firebase documentation https://firebase.google.com/docs/storage/web/download-files
Below is the code I am using to download the file:
private void downloadFrame()
{
//Extract
try
{
//construct HTTP get request
HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create(**link address**);
httpRequest.Method = "GET";
httpRequest.ContentType = "text/xml; encoding='utf-8'";
//send the http request and get the http response from webserver
HttpWebResponse httpResponse = (HttpWebResponse)httpRequest.GetResponse();
Stream httpResponseStream = httpResponse.GetResponseStream();
// Define buffer and buffer size
int bufferSize = 1024;
byte[] buffer = new byte[bufferSize];
int bytesRead = 0;
// Read from response and write to file
FileStream fileStream = File.Create("frame.pcm");
while ((bytesRead = httpResponseStream.Read(buffer, 0, bufferSize)) != 0)
{
fileStream.Write(buffer, 0, bytesRead);
} // end while
}
catch (WebException we)
{
Debug.Log(we.Response.ToString());
}
}

I realize this is a very old question, but I hit it when I had the same issue and it took me quite a while to find an answer.
The download link I had from Firebase was for a PDF file and contained an escaped "/" character as "%2F". This was being changed back into a "/" on the fly, causing the error.
The answer I required was to alter this behaviour by adding an entry to the web.config file as described here: https://stackoverflow.com/a/12170132
I could then use a simple WebClient to download the file like this:
string targetFileName = #"C:\Temp\Target.pdf"
using (WebClient client = new WebClient())
{
Uri downloadURI = new Uri(<Firebase download URL>);
client.DownloadFile(downloadURI, targetFileName);
}

Note: By default, Cloud Storage buckets require Firebase
Authentication to download files. You can change your Firebase
Security Rules for Cloud Storage to allow unauthenticated access
https://firebase.google.com/docs/storage/web/download-files
Change this and try again.

What sort of message does Snapd's API expect?

Snapd has documentation on a REST API.
I'm able to connect to the socket from C# using the following
var snapSocket = new Socket(AddressFamily.Unix, SocketType.Stream, ProtocolType.IP);
var snapEndpoint = new UnixEndPoint("/run/snapd.socket");
snapSocket.Connect(snapEndpoint);
var req = Encoding.ASCII.GetBytes("GET /v2/system-info HTTP/1.1");
snapSocket.Send(req, req.Length, 0);
var bytesReceived = new byte[256];
var bytes = 0;
var response = "";
do
{
bytes = snapSocket.Receive(bytesReceived, bytesReceived.Length, 0);
response = response + Encoding.ASCII.GetString(bytesReceived, 0, bytes);
} while (bytes > 0);
Console.WriteLine(response);
But everything halts at snapSocket.Receive - a response is never sent. I suspect that there's something wrong with the message that I'm sending it.

It turns out that it expects a standard HTTP request, which means a Host: line, a Connection: Close line, and two \ns at the very end are required.
The documentation's following claim...
While it is expected to allow clients to connect using HTTPS over a TCP socket, at this point only a UNIX socket is supported.
... is meant only to imply that HTTPS and TCP do not work yet - HTTP is currently the valid request format even when using the UNIX Socket.

I am not fluent in C# at all, but maybe this python snippet can help lead into a solution:
import requests_unixsocket
session = requests_unixsocket.Session()
r = session.get('http+unix://%2Frun%2Fsnapd.socket/v2/snaps')
r.raise_for_status()
r.json()

Proxy won't work in chrome and only partly in firefox

I've written a proxy in c#. It works by getting the http request from the browser sending the request to the site and sending back the site's response to the client. It works in firefox but some of the pages are cut like if it did not send all the response and in chrome it gives blank pages and for google.co.uk in chrome the browser gives "no data received". Can you see a mistake in my code which might be causing all this?
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.Net.Sockets;
using System.Threading;
using System.Windows.Forms;
namespace LexProxy
{
class ProxyServer
{
private TcpListener tcpListener;
public ProxyServer()
{
this.tcpListener = new TcpListener(IPAddress.Any, 3000);
this.tcpListener.Start();
while (true)
{
Console.Write("Waiting for a connection... ");
TcpClient client = tcpListener.AcceptTcpClient();
Thread thread = new Thread(delegate()
{
Serve(client);
});
thread.Start();
}
}
private void Serve(TcpClient client)
{
Console.WriteLine("Connected!");
NetworkStream stream = client.GetStream();
byte[] request = GetBytesFromStream(stream, client.ReceiveBufferSize);
if (request != null)
{
string requestString = System.Text.Encoding.UTF8.GetString(request);
string[] requestParts = requestString.Split(' ');
if (requestParts.Length >= 2)
{
string method = requestParts[0];
if (!requestParts[1].Contains("http://") && !requestParts[1].Contains("https://"))
requestParts[1] = "http://" + requestParts[1];
Uri uri = new Uri(requestParts[1], UriKind.RelativeOrAbsolute);
string host = StringUtils.ReplaceFirst(uri.Host, "www.", "");
int port = uri.Port;
byte[] response = getResponse(host, port, request);
if (response != null)
stream.Write(response, 0, response.Length);
client.Close();
}
}
}
private byte[] getResponse(string host, int port, byte[] request)
{
TcpClient client = new TcpClient(host, port);
NetworkStream stream = client.GetStream();
stream.Write(request, 0, request.Length);
byte[] response = GetBytesFromStream(stream, client.ReceiveBufferSize);
return response;
}
private byte[] GetBytesFromStream(NetworkStream stream, int bufferSize)
{
Byte[] bytes = new Byte[bufferSize];
int i;
while ((i = stream.Read(bytes, 0, bytes.Length)) != 0)
{
return bytes;
}
return null;
}
}
}

One minor error is that you're using UTF8 to read first line of request. HTTP request line and headers are ASCII (and body may be UTF8, but doesn't have to be, and may not even be a string at all). The reason it works is b/c for english charset, UTF8 and ASCII happen to be encoded using the same bytes. But that's the minor issue.
Most likely primary issue is hiding in your GetBytesFromStream. You only call .Read once, but that doesn't guarantee that the whole message is returned. It may return as little as just 1 byte... so you need to continue getting more data. How do you know how much more? It's dictated by HTTP protocol which you'd need to properly parse and examine. (At a minimum you're reading until you reach \r\n\r\n byte sequence indicating the end of request headers.
However, this is not enough as there may be a request body, length of which will be specified via an HTTP header, Content-Length: (IIRC), or possibly using chunked encoding. I don't see any code beyond URI examination, so most likely you do not handle HTTP messaging protocol itself, and so it is unlikely to work properly (unless you somehow manage to force browser to use HTTP/0.9 or HTTP/1.0 since those do not reuse connection and send one message per connection, at which point you could just blindly read everything the socket has to give you until browser signals the end of stream by closing its write endpoint of the connection).
But your main problem is that GetBytesFromStream as written won't give you "all" bytes.

How to send an image to a browser

I am building a simplified web server, I was able to handle sending the HTML pages properly
but when I get a request for an image, my code doesn't give the browser the image
FileStream fstream = new FileStream(tempSplitArray[1],FileMode.Open,FileAccess.Read);
//The tempSplitArray //recieves the request from the browser
byte[] ar = new byte[(long)fstream.Length];
for (int i = 0; i < ar.Length; i++)
{
ar[i] = (byte)fstream.ReadByte();
}
string byteLine = "Content-Type: image/JPEG\n" + BitConverter.ToString(ar);
sw.WriteLine(byteLine);//This is the network stream writer
sw.Flush();
fstream.Close();
Excuse my ignorance and if there are any questions, or my question is not clear enough, let me know.

Basically you want your response to look like:
HTTP/1.1 200 OK
Content-Type: image/jpeg
Content-Length: *length of image*
Binary Image Data goes here
I'm assuming sw is a StreamWriter, but you need to write the raw bytes of the image.
So how about:
byte[] ar;
using(FileStream fstream = new FileStream(tempSplitArray[1],FileMode.Open,FileAccess.Read);)
{
//The tempSplitArray //recieves the request from the browser
ar = new byte[(long)fstream.Length];
fstream.read(ar, 0, fstream.Length);
}
sw.WriteLine("Content-Type: image/jpeg");
sw.WriteLine("Content-Length: {0}", ar.Length); //Let's
sw.WriteLine();
sw.BaseStream.Write(ar, 0, ar.Length);
It really helps to use a tool like fiddler to view the communications between browsers and a (real) webserver and try to replicate that.