I am having a bit of an issue is reading data posted from Client to the Server.
To explain the server code
I have created a TCP listener and it keeps listening for any client a while loop.
ie., via the below code
listener = new TcpListener("127.0.0.1",3148);
TcpClient client = listener.AcceptTcpClient();//This code is called\hit when HttpWebrequest.GetrequestStream is called.
But When I do Client.GetStream() - I do not get any data in it.
i.e., NetworkStream stream = Client.GetStream() - No dataavailable in network stream
Here is the Client Code :
Webrequest request = (HttpWebrequest)Webrequest.Create("http://127.0.0.1:3148/MovieData") ;
NetworkStream stream = request.GetRequestStream()
string Header = "\r\n Content-Type:MovieData \r\n";
byte[] headerbyte = Encoding.UTF8.GetBytes(header);
stream.Write(headerbyte,0,headerbyte.Length);
stream.close();
Am I missing something here , Is there anything conceptually missing in my implementation.
The Server and client are connect to same port and host, but the data written in the networkstream in client side could not be read by server.
Unless these are custom classes change the spelling like this.
From this:
Webrequest request = (HttpWebrequest)Webrequest.Create("http://127.0.0.1:3148/MovieData")
To this:
WebRequest request = (HttpWebRequest)WebRequest.Create("http://127.0.0.1:3148/MovieData")
If you are tying to communicate over tcp between a client and a server try only using tcpClient and not WebRequest. Here and Here are some examples on how to implement that.
It looks like you have a tcp Socket open and are trying to connect to it using Http. Instead try connecting to your tcp Socket using tcp like this:
TcpClient tcpclnt = new TcpClient();
tcpclnt.Connect("http://127.0.0.1", 3148);
You would then use tcpclnt to send and receive data between the client and server
Here is a link on how tcpclnt.Connect() works
I have encountered a very strange issue - HttpClient's SendAsync never returns when request to the specific web server takes 5 minutes or longer.
This is a sample WebApi controller method that I try to get the response from
[HttpGet]
[Route("api/Entity/Ping")]
public async Task<HttpResponseMessage> Ping([FromUri] int time)
{
await Task.Delay(TimeSpan.FromMinutes(time));
var bytes = Enumerable.Repeat((byte)42, 100_000_000).ToArray();
HttpResponseMessage response = new HttpResponseMessage(HttpStatusCode.OK);
response.Content = new ByteArrayContent(bytes);
response.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment");
response.Content.Headers.ContentDisposition.FileName = "result.bin";
response.Content.Headers.ContentType = new MediaTypeHeaderValue("application/pdf");
return response;
}
And this is a code for sending a request
using (var client = HttpClientFactory.Create(handler))
{
client.Timeout = TimeSpan.FromMinutes(10);
var url = "http://problem-server/WebApp/api/Entity/Ping?time=5";
var request = new HttpRequestMessage
{
Method = HttpMethod.Get,
RequestUri = new Uri(url)
};
var response = await client.SendAsync(
request,
HttpCompletionOption.ResponseHeadersRead,
default);
var stream = await response.Content.ReadAsStreamAsync();
if (response.IsSuccessStatusCode)
return stream;
return default;
}
As you can see, everything is pretty simple and should work without issues. But it doesn't and SendAsync call just hangs forever (for 10 minutes).
In the same time it works when [time] parameter is less then 5.
Additionally, when you open the URL in a browser it successfully downloads a result.bin file after 5 minutes of processing, so method works.
Firstly I thought this is due to a deadlock.
But synchronous request using old WebRequest class to the same URL also hangs
var url = "http://problem-server/WebApp/api/Entity/Ping?time=5";
var request = WebRequest.Create(url);
request.Timeout = (int)TimeSpan.FromMinutes(10).TotalMilliseconds;
var response = request.GetResponse();
var stream = response.GetResponseStream();
if (stream != null)
return stream;
return default;
Next, I copied the WebApp folder to another server, lets call it ok-server.
Modified the URLs in http client and web request methods.
And, magically, everything works - the response is received after [time] minutes.
So the issue is with the problem-server.
But how to debug \ investigate it - IIS request tracing or logs "say" that the request has completed successfully after [time] minutes and the response was sent.
Both machines, problem-server and ok-server, have IIS 8.5 and Windows Server 2012 R2.
Web Api uses .NET Framework 4.5.
(I have also tried to use .NET Core 3.1 with ASP.NET Core hosted on IIS for the Web Api - the result is the same)
Can you help me find the reason for this issue?
Perhaps, I need to look into global machine configs or maybe network setting.
I am truly lost right now.
UPDATE
problem_server and ok_server are in different network segments.
problem_server IP is 192.168.114.100 and ok_server IP is 192.150.0.15.
To diagnose possible network misconfigurations I decided to send a request to the problem_server from the machine in its IP segment.
Here is the result when executing the test client from 192.168.114.125 machine
My workstation is yet in another IP segment - 192.135.9/24. Perhaps there are some router settings between 192.150.0/24 and 192.135.9/24 segments that allow the request to the ok_server to succeed.
I would really recommend that you not execute a five minute delay in your API controller. It will give you more grief than it's worth. For example, when IIS restarts your AppPool, it will wait up to 90 seconds for requests to process. During these autonomous restarts, this request will be aborted.
The problem server may have TCP KeepAlive set to Microsoft's recommended (but not default) value of 5 minutes. Because the HttpClient doesn't implement TCP keepalives by default, the problem server OS is likely disconnecting the TCP socket before the response is sent to the client because the client fails to respond to the keepalive being sent by the problem server OS.
You could adjust the TCP KeepAlive setting at the OS level on the problem server by editing the HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\ subkey as described here.
Alternatively, you can configure the client to support TCP keepalive before sending the request by configuring the ServicePoint. If there is a network device, such as a stateful firewall, between the client and server, a high frequency keep-alive setting may help keep the connection open.
var sp = ServicePointManager.FindServicePoint(new Uri(url));
sp.SetTcpKeepAlive(true, 6000, 3000);
I have create an httplistener. So i need when client will send me data to read them. The problem is that i dont know how client should send the data
HttpListener listener = new HttpListener();
listener.Prefixes.Add("http://192.168.1.26:8282/");
listener.Prefixes.Add("http://localhost:8282/");
listener.Prefixes.Add("http://127.0.0.1:8282/");
listener.Start();
new Thread(() =>
{
Thread.CurrentThread.IsBackground = true;
for (;;)
{
Console.WriteLine("Listening...");
// Note: The GetContext method blocks while waiting for a request.
HttpListenerContext context = listener.GetContext();
HttpListenerRequest request = context.Request;
string text;
using (var reader = new StreamReader(request.InputStream,
request.ContentEncoding))
{
text = reader.ReadToEnd();
MessageBox.Show(text);
}
// Obtain a response object.
HttpListenerResponse response = context.Response;
// Construct a response.
string responseString = "HelloWorld";
byte[] buffer = System.Text.Encoding.UTF8.GetBytes(responseString);
// Get a response stream and write the response to it.
response.ContentLength64 = buffer.Length;
System.IO.Stream output = response.OutputStream;
output.Write(buffer, 0, buffer.Length);
// You must close the output stream.
output.Close();
}
}).Start();
}
So from client i send this command:
GET / 192.168.1.26:8282 HTTP/1.0
But i'm getting this message
Recv 34 bytes
SEND OK
+IPD,1,518:HTTP/1.1 400 Bad Request
Content-Type: text/html; charset=us-ascii
Server: Microsoft-HTTPAPI/2.0
Date: Wed, 13 Jun 2018 13:16:03 GMT
Connection: close
Content-Length: 339
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Bad Request</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Bad Request - Invalid Header</h2>
<hr><p>HTTP Error 400. The request has an invalid header name.</p>
</BODY></HTML>
1,CLOSED
I cant understand what is wrong. Also in my code i set to get a message box every time a request will happen. But it never runs
This s what mozilla is sending
You are not attempting to invoke the service correctly. Here is your client request:
GET / 192.168.1.26:8282 HTTP/1.0
What you should be doing is first establishing a socket connection to host 192.168.1.26 over port 8282. Then you must issue a HTTP request in a valid format:
GET / HTTP/1.0
Don't forget to add some newlines after the request (ie: \r\n\r\n). Then your web server should respond to the HTTP request.
Quick example in Telnet:
telnet 192.168.1.26 8282
GET / HTTP/1.0
Quick example with netcat:
nc 192.168.1.26 8282
GET / HTTP/1.0
Note that these quick examples are provided just to help you ensure that your web service is accessible and functioning correctly. Ideally, you would likely use a more robust HTTP client that is customized for whatever your particular needs are. The process is still the same:
Establish a connection to your host IP address over the listening port
Issue a HTTP request in a valid format: (HTTP_VERB PATH HTTP_VERSION)
*) Maybe check out the developer tools in your browser of choice (F12 -> Network) to see how HTTP headers are sent as well.
Parse the response in some meaningful way.
"Also in my code i set to get a message box every time a request will happen." - You should try putting in a manual message to the message box, instead of reading from the input stream. This is a good debugging technique. In a HTTP GET request you generally are not sending data except in the form of optional query string parameters. I have a feeling that you are not getting the results you are expecting because you are reading from input that isn't there. Before reading from the stream input, first make sure that the connection is successful.
When I use the following code to send HTTP GET request using sockets in C#
IPEndPoint RHost = new IPEndPoint(IPAddress.Parse("xxx.xxx.xxx.xxx"), 80);
Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
socket.Connect(RHost);
String req = "GET / HTTP/1.1\r\nHost: awebsite.com\r\n\r\n";
socket.Send(Encoding.ASCII.GetBytes(req), SocketFlags.None);
int bytes = 0;
byte[] buffer = new byte[256];
var result = new StringBuilder();
do
{
bytes = socket.Receive(buffer, 0, buffer.Length, SocketFlags.None);
result.Append(Encoding.ASCII.GetString(buffer, 0, bytes));
}
while (bytes > 0);
I get
System.Net.Sockets.SocketException: 'An existing connection was
forcibly closed by the remote host'
and when I add Connection: Close header to the request, it is working without any problem.
But using Repeater tool in the Burp Suite I am able to send and receive a response from the server without setting Connection: Close header.
Notes:
I have to use socket.
I am facing this problem just with few websites, not every website
You are sending a HTTP/1.1 request. Without an explicit Connection: close there is an implicit Connection: keep-alive with HTTP/1.1 (different to HTTP/1.0). This means that the server might wait for new requests within the same TCP connection after the response is done and close the connection sometimes later if no new requests arrive. This later close might be done with a RST which results in the error you see.
But, your code expects the server to behave differently: it expects that the server closes the connection once the request is done and not to wait for more requests and not to close the idle connection with RST.
To fix this you have either to adapt your request so that the server behavior matches your expectations or to adjust the expectations.
The first can be done by either explicitly adding Connection: close header or by simply using HTTP/1.0 instead of HTTP/1.1. The latter one is recommended because this also results in simpler responses (no chunked response).
If you instead want to adjust your expectations you have to properly parse the servers response: first read the header, then check for Transfer-Encoding: chunked or Content-length and then read the response body based on what these headers say. For details see the HTTP standard.
TL;DR version
When a transfer error occurs while writing to the request stream, I can't access the response, even though the server sends it.
Full version
I have a .NET application that uploads files to a Tomcat server, using HttpWebRequest. In some cases, the server closes the request stream prematurely (because it refuses the file for one reason or another, e.g. an invalid filename), and sends a 400 response with a custom header to indicate the cause of the error.
The problem is that if the uploaded file is large, the request stream is closed before I finish writing the request body, and I get an IOException:
Message: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
InnerException: SocketException: An existing connection was forcibly closed by the remote host
I can catch this exception, but then, when I call GetResponse, I get a WebException with the previous IOException as its inner exception, and a null Response property. So I can never get the response, even though the server sends it (checked with WireShark).
Since I can't get the response, I don't know what the actual problem is. From my application point of view, it looks like the connection was interrupted, so I treat it as a network-related error and retry the upload... which, of course, fails again.
How can I work around this issue and retrieve the actual response from the server? Is it even possible? To me, the current behavior looks like a bug in HttpWebRequest, or at least a severe design issue...
Here's the code I used to reproduce the problem:
var request = HttpWebRequest.CreateHttp(uri);
request.Method = "POST";
string filename = "foo\u00A0bar.dat"; // Invalid characters in filename, the server will refuse it
request.Headers["Content-Disposition"] = string.Format("attachment; filename*=utf-8''{0}", Uri.EscapeDataString(filename));
request.AllowWriteStreamBuffering = false;
request.ContentType = "application/octet-stream";
request.ContentLength = 100 * 1024 * 1024;
// Upload the "file" (just random data in this case)
try
{
using (var stream = request.GetRequestStream())
{
byte[] buffer = new byte[1024 * 1024];
new Random().NextBytes(buffer);
for (int i = 0; i < 100; i++)
{
stream.Write(buffer, 0, buffer.Length);
}
}
}
catch(Exception ex)
{
// here I get an IOException; InnerException is a SocketException
Console.WriteLine("Error writing to stream: {0}", ex);
}
// Now try to read the response
try
{
using (var response = (HttpWebResponse)request.GetResponse())
{
Console.WriteLine("{0} - {1}", (int)response.StatusCode, response.StatusDescription);
}
}
catch(Exception ex)
{
// here I get a WebException; InnerException is the IOException from the previous catch
Console.WriteLine("Error getting the response: {0}", ex);
var webEx = ex as WebException;
if (webEx != null)
{
Console.WriteLine(webEx.Status); // SendFailure
var response = (HttpWebResponse)webEx.Response;
if (response != null)
{
Console.WriteLine("{0} - {1}", (int)response.StatusCode, response.StatusDescription);
}
else
{
Console.WriteLine("No response");
}
}
}
Additional notes:
If I correctly understand the role of the 100 Continue status, the server shouldn't send it to me if it's going to refuse the file. However, it seems that this status is controlled directly by Tomcat, and can't be controlled by the application. Ideally, I'd like the server not to send me 100 Continue in this case, but according to my colleagues in charge of the back-end, there is no easy way to do it. So I'm looking for a client-side solution for now; but if you happen to know how to solve the problem on the server side, it would also be appreciated.
The app in which I encounter the issue targets .NET 4.0, but I also reproduced it with 4.5.
I'm not timing out. The exception is thrown long before the timeout.
I tried an async request. It doesn't change anything.
I tried setting the request protocol version to HTTP 1.0, with the same result.
Someone else has already filed a bug on Connect for this issue: https://connect.microsoft.com/VisualStudio/feedback/details/779622/unable-to-get-servers-error-response-when-uploading-file-with-httpwebrequest
I am out of ideas as to what can be a client side solution to your problem. But I still think the server side solution of using a custom tomcat valve can help here. I currently doesn`t have a tomcat setup where I can test this but I think a server side solution here would be along the following lines :
RFC section 8.2.3 clearly states :
Requirements for HTTP/1.1 origin servers:
- Upon receiving a request which includes an Expect request-header
field with the "100-continue" expectation, an origin server MUST
either respond with 100 (Continue) status and continue to read
from the input stream, or respond with a final status code. The
origin server MUST NOT wait for the request body before sending
the 100 (Continue) response. If it responds with a final status
code, it MAY close the transport connection or it MAY continue
to read and discard the rest of the request. It MUST NOT
perform the requested method if it returns a final status code.
So assuming tomcat confirms to the RFC, while in the custom valve you would have recieved the HTTP request header, but the request body would not be sent since the control is not yet in the servlet that reads the body.
So you can probably implement a custom valve, something similar to :
import org.apache.catalina.connector.Request;
import org.apache.catalina.connector.Response;
import org.apache.catalina.valves.ErrorReportValve;
public class CustomUploadHandlerValve extends ValveBase {
#Override
public void invoke(Request request, Response response) throws IOException, ServletException {
HttpServletRequest httpRequest = (HttpServletRequest) request;
String fileName = httpRequest.getHeader("Filename"); // get the filename or whatever other parameters required as per your code
bool validationSuccess = Validate(); // perform filename check or anyother validation here
if(!validationSuccess)
{
response = CreateResponse(); //create your custom 400 response here
request.SetResponse(response);
// return the response here
}
else
{
getNext().invoke(request, response); // to pass to the next valve/ servlet in the chain
}
}
...
}
DISCLAIMER : Again I haven`t tried this to success, need sometime and a tomcat setup to try it out ;).
Thought it might be a starting point for you.
I had the same problem. The server sends a response before the client end of the transmission of the request body, when I try to do async request. After a series of experiments, I found a workaround.
After the request stream has been received, I use reflection to check the private field _CoreResponse of the HttpWebRequest. If it is an object of class CoreResponseData, I take his private fields (using reflection): m_StatusCode, m_StatusDescription, m_ResponseHeaders, m_ContentLength. They contain information about the server's response!
In most cases, this hack works!
What are you getting in the status code and response of the second exception not the internal exception?
If a WebException is thrown, use the Response and Status properties of the exception to determine the response from the server.
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.getresponse(v=vs.110).aspx
You are not saying what exactly version of Tomcat 7 you are using...
checked with WireShark
What do you actually see with WireShark?
Do you see the status line of response?
Do you see the complete status line, up to CR-LF characters at its end?
Is Tomcat asking for authentication credentials (401), or it is refusing file upload for some other reason (first acknowledging it with 100 but then aborting it mid-flight)?
The problem is that if the uploaded file is large, the request stream
is closed before I finish writing the request body, and I get an IOException:
If you do not want the connection to be closed but all the data transferred over the wire and swallowed at the server side, on Tomcat 7.0.55 and later it is possible to configure maxSwallowSize attribute on HTTP connector, e.g. maxSwallowSize="-1".
http://tomcat.apache.org/tomcat-7.0-doc/config/http.html
If you want to discuss Tomcat side of connection handling, you would better ask on the Tomcat users' mailing list,
http://tomcat.apache.org/lists.html#tomcat-users
At .Net side:
Is it possible to perform stream.Write() and request.GetResponse() simultaneously, from different threads?
Is it possible to performs some checks at the client side before actually uploading the file?
hmmm... i don't get it - that is EXACTLY why in many real-life scenarios large files are uploaded in chunks (and not as a single large file)
by the way: many internet servers have size limitations. for instance in tomcat that is representad by maxPostSize (as seen in this link: http://tomcat.apache.org/tomcat-5.5-doc/config/http.html)
so tweaking the server configurations seems like the easy way, but i do think that the right way is to split the file to several requests
EDIT: replace Uri.EscapeDataString with HttpServerUtility.UrlEncode
Uri.EscapeDataString(filename) // a problematic .net implementation
HttpServerUtility.UrlEncode(filename) // the proper way to do it
I am experience a pretty similar problem currently also with Tomcat and a Java client. The Tomcat REST service sends a HTTP returncode with response body before reading the whole request body. The client however fails with IOException. I inserted a HTTP Proxy on the client to sniff the protocol and actually the HTTP response is sent to the client eventually. Most likly the Tomcat closed the request input stream before sending the response.
One solution is to use a different HTTP server like Jetty which does not have this problem. The other solution is a add a Apache HTTP server with AJP in front of Tomcat. Apache HTTP server has a different handling of streams and with that the problem goes away.