Develop Reference

C# python http post to C# httpwebrequest

i try to figure it out the black box of the server side that code by python for sending http post. In C# i trying to use httpwebrequest and httpwebresponse to send and receive message back from the server side.
Below is the python code:
**# Request login to server**
import httplib, urllib, requests
import sys, time
from requests.adapters import HTTPAdapter
import select, socket
s = requests.Session()
url = 'http://192.168.1.1/request_login'
data = 'user.login(test,test)' #login account and password set here
r = s.post(url, data=data)
if r.status_code != 200:
print "Failed loging in (%d)" % (r.status_code)
sys.exit(-1)
**# send key file to server**
url = 'http://192.168.1.1/request_sendfile'
files = {'file': (data_xml, open(data_xml, 'rb'), 'application/x-binary', {'Expires': '0'})} # **Don't understand this format**
r = s.post(url, files=files, timeout=30.0)
if r.status_code != 200:
print "Failed sending data file (%d, %s)" % (r.status_code, r.reason)
url = 'http://192.168.1.1/request_sendfile'
data = 'reader.view_log(4)'
r = s.post(url, data=data)
if r.status_code != 200:
print "Failed loging in (%d)" % (r.status_code)
sys.exit(-3)
print "Logs: "
print r.content
sys.exit(-4)
Above is the code for login and sending file to server in one session, i try to use C# do the same way.
I set two http request at the same time, however for the login is success while try to get second http request i get an error of "400" doesn't understand context of verb.
Below is the C# code:
private void httpost()
{
try
{
string Url = "http://192.168.1.1/request_login";
string Urls = "http://192.168.1.1/request_sendfile";
HttpWebRequest request = HttpWebRequest.Create(Url) as HttpWebRequest;
HttpWebRequest requests = HttpWebRequest.Create(Urls) as HttpWebRequest;
string result = null;
request.Method = "POST";
request.KeepAlive = true;
request.ContentType = "login";
string param = "user.login(test,test)";
byte[] bs = Encoding.ASCII.GetBytes(param);
using (Stream reqStream = request.GetRequestStream())
{
reqStream.Write(bs, 0, bs.Length);
}
using (WebResponse response = request.GetResponse())
{
StreamReader sr = new StreamReader(response.GetResponseStream());
HttpWebResponse resp = (HttpWebResponse)request.GetResponse();
if(resp.StatusCode == HttpStatusCode.OK)
{
richTextBox1.AppendText("Login Response Status:" + resp.StatusCode + "\n");
}
else
{
richTextBox1.AppendText("Login Response Failed: " + resp.StatusCode + "\n");
}
request.KeepAlive = true;
result = sr.ReadToEnd();
sr.Close();
}
using (Stream importFile = requests.GetRequestStream())
{
FileStream fs = File.OpenRead(#"data.xml");
byte[] bytes = ReadWholeArray(fs); // turn xml file to byte function
requests.ContentType = "application/x-binary";
requests.Method = "POST";
importFile.Write(bytes, 0, bytes.Length);
}
using (WebResponse responses = requests.GetResponse())
{
StreamReader srs = new StreamReader(responses.GetResponseStream());
result = srs.ReadToEnd();
srs.Close();
}
}
catch (Exception ex)
{
richTextBox1.AppendText("Exception Throw:" + ex.Message);
}
}
I try the python code it request login and send file at the same time.
And below is the code that i don't understand.
files = {'file': (data_xml, open(data_xml, 'rb'), 'application/x-binary', {'Expires': '0'})}

GetRequestStream() timeout an error after second time

I have some problems with code. I'm using HttpWebRequest class to do operations
Code as bellow:
ServicePointManager.SecurityProtocol = (SecurityProtocolType)3072 | (SecurityProtocolType)768;
ServicePointManager.Expect100Continue = true;
//ServicePointManager.DefaultConnectionLimit = 5;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(serverName + authorizePath);
request.Abort();
byte[] bearerMatch = UTF8Encoding.UTF8.GetBytes("username:password");
//request.Proxy = null; //<TNI_20180808
request.Method = "POST";
request.Accept = "application/json";
request.ContentType = "application/json";
request.Timeout = 100;
//request.Headers["Authorization"] = "Bearer " + Convert.ToBase64String(bearerMatch);
string content = String.Format(authorizeBody, login, password);
content = "{" + content + "}";
try
{
using (var writer = new StreamWriter(request.GetRequestStream()))
{
writer.Write(content);
}
using (WebResponse response = request.GetResponse())
{
using (var reader = new StreamReader(response.GetResponseStream()))
{
var responseContent = reader.ReadToEnd();
token = responseContent.Replace('"', ' ').Trim();
}
//response.Close();
//request.Abort();
}
}
catch (Exception ex)
{
throw ex;
}
Until third operation everything is okey, unfortunately i don't know what to do. When application is trying to create request third time there is an timeout error:
at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
I think that there is an opportunity to solve this with closing request and response in a proper way but i don't know how to do this. After increasing ServicePointManager.DefaultConnectionLimit operations were getting completed correctly.
Can you help me please? How should i abort or close request/response in a good way?
Thank you!

C# Restful client - connection timeout

I'm using WinForms and accessing my Restful webservice. For some reason the code after a while breaks, by giving the timeout error while connecting to the server.
The problem might also be due to my code design.
Here's my Restful client class
public class Restful
{
public string auth = "Basic " + Convert.ToBase64String(Encoding.Default.GetBytes("MyUserName:MyPassword"));
public string POST(string parameters)
{
var request = (HttpWebRequest)WebRequest.Create("http://myserverdomain.com/api/webservice/someMethod");
byte[] byteArray = Encoding.UTF8.GetBytes(parameters);
request.Method = WebRequestMethods.Http.Post;
request.Headers["Authorization"] = this.auth;
request.ContentLength = byteArray.Length;
request.ContentType = "application/x-www-form-urlencoded";
Stream postStream = null;
try
{
// ERROR IS IN THIS LINE
postStream = request.GetRequestStream();
}
catch (WebException ex)
{
// I'm kind of creating an hack here..which isn't good..
if (ex.Status.ToString() == "ConnectFailure")
{
System.Threading.Thread.Sleep(1000);
this.POST(parameters);
}
}
if (postStream == null)
return string.Empty;
postStream.Write(byteArray, 0, byteArray.Length);
postStream.Close();
using (var response = (HttpWebResponse)request.GetResponse())
{
var responseValue = string.Empty;
if (response.StatusCode != HttpStatusCode.OK)
return responseValue;
using (var responseStream = response.GetResponseStream())
if (responseStream != null)
using (var reader = new StreamReader(responseStream))
responseValue = reader.ReadToEnd();
return responseValue;
}
}
}
I'm receiving the error (after a few successfully send items):
Unable to connect to remote server
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond MyServerIpAddress
Sometimes I send thousands of items to the server, sometimes I only send 2 or 4. So I call this POST function within a loop.
try
{
foreach (app.views.Documents doc in DocumentsView)
{
var parameters = "key=" + doc.key;
parameters += "&param1=" + doc.param1 + "&param2=" + doc.param2;
/*
* Validates the response of Restful service.
* The response is always in JSON format.
*/
if (response(Restful.POST(parameters)) == false)
{
MessageBox.Show("Error while sending the ID: " + doc.id.ToString());
break;
};
}
}
catch (Exception ex)
{
MessageBox.Show("Error while sending documents: " + ex.Message);
}

You can change the default timeout of your HttpWebRequest to be some thing larger than the default, for example:
request.Timeout = 120000;
I think the default is 100 seconds.
You can also check this adjusting-httpwebrequest-connection-timeout-in-c-sharp

taking the failure at face value, it says that the remote machine didnt respond. Most likely causes
wrong name or ip address
windows firewall is on on the remote machine

WebException when loading rss feed

I am attempting to load a page I've received from an RSS feed and I receive the following WebException:
Cannot handle redirect from HTTP/HTTPS protocols to other dissimilar ones.
with an inner exception:
Invalid URI: The hostname could not be parsed.
I wrote a code that would attempt loading the url via an HttpWebRequest. Due to some suggestions I received, when the HttpWebRequest fails I then set the AllowAutoRedirect to false and basically manually loop through the iterations of redirect until I find out what ultimately fails. Here's the code I'm using, please forgive the gratuitous Console.Write/Writeline calls:
Uri url = new Uri(val);
bool result = true;
System.Net.HttpWebRequest req = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create(url);
string source = String.Empty;
Uri responseURI;
try
{
using (System.Net.WebResponse webResponse = req.GetResponse())
{
using (HttpWebResponse httpWebResponse = webResponse as HttpWebResponse)
{
responseURI = httpWebResponse.ResponseUri;
StreamReader reader;
if (httpWebResponse.ContentEncoding.ToLower().Contains("gzip"))
{
reader = new StreamReader(new GZipStream(httpWebResponse.GetResponseStream(), CompressionMode.Decompress));
}
else if (httpWebResponse.ContentEncoding.ToLower().Contains("deflate"))
{
reader = new StreamReader(new DeflateStream(httpWebResponse.GetResponseStream(), CompressionMode.Decompress));
}
else
{
reader = new StreamReader(httpWebResponse.GetResponseStream());
}
source = reader.ReadToEnd();
reader.Close();
}
}
req.Abort();
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(source);
result = true;
}
catch (ArgumentException ae)
{
Console.WriteLine(url + "\n--\n" + ae.Message);
result = false;
}
catch (WebException we)
{
Console.WriteLine(url + "\n--\n" + we.Message);
result = false;
string urlValue = url.ToString();
try
{
bool cont = true;
int count = 0;
do
{
req = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create(urlValue);
req.Headers.Add("Accept-Language", "en-us,en;q=0.5");
req.AllowAutoRedirect = false;
using (System.Net.WebResponse webResponse = req.GetResponse())
{
using (HttpWebResponse httpWebResponse = webResponse as HttpWebResponse)
{
responseURI = httpWebResponse.ResponseUri;
StreamReader reader;
if (httpWebResponse.ContentEncoding.ToLower().Contains("gzip"))
{
reader = new StreamReader(new GZipStream(httpWebResponse.GetResponseStream(), CompressionMode.Decompress));
}
else if (httpWebResponse.ContentEncoding.ToLower().Contains("deflate"))
{
reader = new StreamReader(new DeflateStream(httpWebResponse.GetResponseStream(), CompressionMode.Decompress));
}
else
{
reader = new StreamReader(httpWebResponse.GetResponseStream());
}
source = reader.ReadToEnd();
if (string.IsNullOrEmpty(source))
{
urlValue = httpWebResponse.Headers["Location"].ToString();
count++;
reader.Close();
}
else
{
cont = false;
}
}
}
} while (cont);
}
catch (UriFormatException uriEx)
{
Console.WriteLine(urlValue + "\n--\n" + uriEx.Message + "\r\n");
result = false;
}
catch (WebException innerWE)
{
Console.WriteLine(urlValue + "\n--\n" + innerWE.Message+"\r\n");
result = false;
}
}
if (result)
Console.WriteLine("testing successful");
else
Console.WriteLine("testing unsuccessful");
Since this is currently just test code I hardcode val as http://rss.nytimes.com/c/34625/f/642557/s/3d072012/sc/38/l/0Lartsbeat0Bblogs0Bnytimes0N0C20A140C0A70C30A0Csarah0Ekane0Eplay0Eamong0Eofferings0Eat0Est0Eanns0Ewarehouse0C0Dpartner0Frss0Gemc0Frss/story01.htm
the ending url that gives the UriFormatException is: http:////www-nc.nytimes.com/2014/07/30/sarah-kane-play-among-offerings-at-st-anns-warehouse/?=_php=true&_type=blogs&_php=true&_type=blogs&_php=true&_type=blogs&_php=true&_type=blogs&_php=true&_type=blogs&_php=true&_type=blogs&_php=true&_type=blogs&partner=rss&emc=rss&_r=6&
Now I'm sure if I'm missing something or if I'm doing the looping wrong, but if I take val and just put that into a browser the page loads fine, and if I take the url that causes the exception and put it in a browser I get taken to an account login for nytimes.
I have a number of these rss feed urls that are resulting in this problem. I also have a large number of these rss feed urls that have no problem loading at all. Let me know if there is any more information needed to help resolve this. Any help with this would be greatly appreciated.
Could it be that I need to have some sort of cookie capability enabled?

You need to keep track of the cookies while doing all your requests. You can use an instance of the CookieContainer class to achieve that.
At the top of your method I made the following changes:
Uri url = new Uri(val);
bool result = true;
// keep all our cookies for the duration of our calls
var cookies = new CookieContainer();
System.Net.HttpWebRequest req = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create(url);
// assign our CookieContainer to the new request
req.CookieContainer = cookies;
string source = String.Empty;
Uri responseURI;
try
{
And in the exception handler where you create a new HttpWebRequest, you do the assignment from our CookieContainer again:
do
{
req = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create(urlValue);
// reuse our cookies!
req.CookieContainer = cookies;
req.Headers.Add("Accept-Language", "en-us,en;q=0.5");
req.AllowAutoRedirect = false;
using (System.Net.WebResponse webResponse = req.GetResponse())
{
This makes sure that on each successive call the already present cookies are resend again in the next request. If you leave this out, no cookies are sent and therefore the site you try to visit assumes you are a fresh/new/unseen user and gives you a kind of authentication path.
If you want to store/keep cookies beyond this method you could move the cookie instance variable to a static public property so you can use all those cookies program-wide like so:
public static class Cookies
{
static readonly CookieContainer _cookies = new CookieContainer();
public static CookieContainer All
{
get
{
return _cookies;
}
}
}
And to use it in a WebRequest:
var req = (System.Net.HttpWebRequest) WebRequest.Create(url);
req.CookieContainer = Cookies.All;

HttpWebRequest: Unable to connect to the remote server

I develop client app which connects to the server and fetches different information from it. It is multithreaded app. When I start it with a few threads it works perfect.
Until it start throwing an exception with the following message:
"Unable to connect to the remote server"
I've used TCPView and cannot find anything about my client app. So when it starts returning "Unable to Connect" it doesn't even OPEN any http connections...
How can I figure out why it doesn't open connection?
Thanks,
EDIT:
Here is the code I'm using in Multiple threads to fecth page content:
HttpWebResponse response = null;
Stream resStream = null;
StreamReader reader = null;
string res = "";
try
{
HttpWebRequest request = (HttpWebRequest)
WebRequest.Create(requestUrl);
if (cookies_ == null)
{
cookies_ = new CookieContainer();
}
request.Timeout = timeoutVal_;
request.ReadWriteTimeout = timeoutVal_ * 2;
request.KeepAlive = false;
if (bUseCookies)
{
request.CookieContainer = cookies_;
}
// execute the request
response = (HttpWebResponse)
request.GetResponse();
resStream = response.GetResponseStream();
reader = new StreamReader(resStream);
res = reader.ReadToEnd();
}
catch (Exception ex)
{
throw new Exception(siteToken + " " + ex.Message);
}
finally
{
if (response != null)
response.Close();
if (resStream != null)
resStream.Close();
if (reader != null)
{
reader.Close();
}
}
return res;
After a couple minutes threads getting into cycle with Timeout Exception or Unable to connect to Server.
The strange thing that if I start Fiddler, connections get reactivated and threads continue working for some time. How does Fiddler fix that problem?

You may be connecting internet through a proxy check your IE lan settings. from c# you need to add proxy settings.
var request = (HttpWebRequest)WebRequest.CreateHttp(url);
WebProxy proxy = new WebProxy("http://127.0.0.1:8888", true);
proxy.Credentials = new NetworkCredential("user", "pwd", "ADomain");
request.Proxy = proxy;
request.Credentials = System.Net.CredentialCache.DefaultNetworkCredentials;
request.Timeout = 1000 * 60 * 5;
request.Method = method;
request.Headers.Add("DAUTH", dauth);request.GetResponse();