I have a small tool I use for testing a webservice.
It can either call the webservice using POST or using GET.
The code for using POST is
public void PerformRequest()
{
WebRequest webRequest = WebRequest.Create(_uri);
webRequest.ContentType = "application/ocsp-request";
webRequest.Method = "POST";
webRequest.Credentials = _credentials;
webRequest.ContentLength = _request.Length;
((HttpWebRequest)webRequest).KeepAlive = false;
using (Stream st = webRequest.GetRequestStream())
st.Write(_request, 0, _request.Length);
using (HttpWebResponse httpWebResponse = (HttpWebResponse)webRequest.GetResponse())
using (Stream responseStream = httpWebResponse.GetResponseStream())
using (BufferedStream bufferedStream = new BufferedStream(responseStream))
using (BinaryReader reader = new BinaryReader(bufferedStream))
{
if (httpWebResponse.StatusCode != HttpStatusCode.OK)
throw new WebException("Got response status code: " + httpWebResponse.StatusCode);
byte[] response = reader.ReadBytes((int)httpWebResponse.ContentLength);
httpWebResponse.Close();
}
}
The code for using GET is:
protected override void PerformRequest()
{
WebRequest webRequest = WebRequest.Create(_uri + "/" + Convert.ToBase64String(_request));
webRequest.Method = "GET";
webRequest.Credentials = _credentials;
((HttpWebRequest)webRequest).KeepAlive = false;
using (HttpWebResponse httpWebResponse = (HttpWebResponse)webRequest.GetResponse())
using (Stream responseStream = httpWebResponse.GetResponseStream())
using (BufferedStream bufferedStream = new BufferedStream(responseStream))
using (BinaryReader reader = new BinaryReader(bufferedStream))
{
if (httpWebResponse.StatusCode != HttpStatusCode.OK)
throw new WebException("Got response status code: " + httpWebResponse.StatusCode);
byte[] response = reader.ReadBytes((int)httpWebResponse.ContentLength);
httpWebResponse.Close();
}
}
As you can see the code is quite similar. If anything, I would expect the GET-method to be slightly slower, as it has to encode and transmit the data in Base64.
But when I run it, I see that the POST-method uses far more processing power than the GET-method. On my machine, I can run 80 threads of the GET-method using approximately 5% CPU, while 80 threads of the POST-method uses 95% CPU.
Is there something inherently more expensive about using POST? Is there anything I can do to optimize the POST-method? I cannot reuse the connection, as I want to simulate requests from different clients.
dotTrace reports that 65% of the processing-time is spent in webRequest.GetResponse() when using POST.
The underlying webservice uses Digest-Authentication if that makes any difference.
Well, depending on the complexity of the final uri, it might be that the "GET" requests are being cached? "POST" is not cached by default, but "GET" often is (as it should be idempotent). Have you tried sniffing to see if there is any difference here?
Also - you might find WebClient easier to use - something like:
using (WebClient wc = new WebClient())
{
byte[] fromGet = wc.DownloadData(uriWithData);
byte[] fromPost = wc.UploadData(uri, data);
}
Related
I have several Http Web Requests in various loops etc. The web requests get data from a variety of APIs.
These seem to work some of the time, but most of the time (recently) I am getting Timeout exception errors (Operation has timed out) and am not sure why.
I accept that every so often or once in a while you will get a time out error, but this is happening too often.
Here are two of my WebRequest codes:
public static EventList getEvents()
{
Uri myURI = new Uri("http://feeds.betway.com/events?key=XXX&keywords=horse-racing,uk-and-ireland&and=true");
WebRequest webRequest = WebRequest.Create(myURI);
webRequest.Timeout = 3000;
using (WebResponse webResponse = webRequest.GetResponse())
{
using (Stream stream = webResponse.GetResponseStream())
{
using (var reader = XmlReader.Create(stream))
{
XmlSerializer serializer = new XmlSerializer(typeof(EventList));
EventList data = (EventList)serializer.Deserialize(reader);
return data;
}
}
}
}
public static List<WilliamHillData.Event> GetAllCompetitionEvents(string compid)
{
string res = "";
Uri myURI = new Uri("https://gw.whapi.com/v2/sportsdata/competitions/" + compid + "/events/?&sort=startDateTime");
WebRequest webRequest = WebRequest.Create(myURI);
webRequest.Headers.Add("Content-Type", "application/json");
webRequest.Headers.Add("apiKey", "xxx");
webRequest.Timeout = 2000;
using (WebResponse webResponse = webRequest.GetResponse())
{
using (Stream stream = webResponse.GetResponseStream())
{
StreamReader reader = new StreamReader(stream, Encoding.UTF8);
res = reader.ReadToEnd();
}
}
JObject jobject = JObject.Parse(res);
List<WilliamHillData.Event> list = jobject["events"].ToObject<List<WilliamHillData.Event>>();
return list;
}
I cannot see anything wrong with my code as I am disposing objects correctly and have set the Timeout. Does the timeout need increasing or am I missing something?
Could this also possibly be a network issue on our end?
I am trying to change the price of an article using the websites API
the documentation for it is at https://www.mkmapi.eu/ws/documentation/API_1.1:Stock
When run the class I get an error 417 Expectation Failed, which is described from the documentation as:
Typically you get a 417 Expectation Failed HTTP status code, when your request has an XML body without the corresponding header and/or the body not sent as text, but its byte representation. Another possible reason for a 417 is, when you send body data with more than 1.024 bytes without adding the header Expect: to your request.
Any help would be appreciated. I should also say that the authentication is not the problem I can download my article prices.
public void UpdateMarketPrice(string MarketID, string NewPrice)
{
// https://www.mkmapi.eu/ws/documentation/API_1.1:Stock
String finalResult;
String method = "PUT";
String url = "https://www.mkmapi.eu/ws/v1.1/stock";
HttpWebRequest request = WebRequest.CreateHttp(url) as HttpWebRequest;
OAuthHeader header = new OAuthHeader();
request.Headers.Add(HttpRequestHeader.Authorization, header.getAuthorizationHeader(method, url));
request.Method = method;
request.ContentType = "text/xml; encoding='utf-8'";
XElement xmlDoc =
new XElement("request",
new XElement("article",
new XElement("idArticle", MarketID),
new XElement("idLanguage", 1),
new XElement("comments", "Edited through the API"),
new XElement("count", 7),
new XElement("price", 11),
new XElement("condition", "NM"),
new XElement("isFoil", false),
new XElement("isSigned", false),
new XElement("isPlayset", false)
)
);
String finalXML = "<?xml version=\"1.0\" encoding=\"UTF-8\" ?>\n" + xmlDoc.ToString();
MessageBox.Show(finalXML);
byte[] bytes = Encoding.ASCII.GetBytes(finalXML);
request.ContentLength = bytes.Length;
using (Stream putStream = request.GetRequestStream())
{
putStream.Write(bytes, 0, bytes.Length);
}
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
finalResult = reader.ReadToEnd();
}
MessageBox.Show(finalResult);
}
I have read that HttpWebRequest adds an "expect 100 continue" header to requests unless you turn it off. There are servers that possibly don't support this header. And will produce this 417 Expectation Failed message.
You could try setting it to false:
System.Net.ServicePointManager.Expect100Continue = false;
So the header isn't sent.
I've seen this suggested sollution to other similar questions also.
Maybe use StreamWriter ?
using (Stream putStream = request.GetRequestStream())
{
using (var writeStream = new StreamWriter(putStream, Encoding.ASCII))
{
writeStream.Write(finalXML);
}
request.ContentLength = putStream.Length; // I am not sure about that
}
I have an application that starts multiple threads, and each thread sends requests to a remote server. After profiling, I noticed that the most time is taken in sending & processing the request. This is the function:
private string GetRequest(string url, string postFields)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Proxy = proxy;
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
request.Host = Constants.URLDomain;
request.KeepAlive = true;
request.Accept = Constants.Accept;
request.ContentType = "application/json";
request.Headers.Add("MyHeader1", "true");
request.Headers.Add("MyHeader2", "header2value");
request.Headers.Add("MyHeader3", "header3value");
request.Method = "POST";
byte[] postBytes = Encoding.UTF8.GetBytes(postFields);
request.ContentLength = postBytes.Length;
using (Stream postStream = request.GetRequestStream())
{
postStream.Write(postBytes, 0, postBytes.Length);
}
string text = string.Empty;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream answer = response.GetResponseStream())
using (StreamReader reader = new StreamReader(answer, System.Text.Encoding.UTF8))
{
text = reader.ReadToEnd();
}
}
return text;
}
I was thinking of changing this code to use sockets so I could improve performance. Will there be actually an improvement if I change to sockets, and more importantly, how do will a sockets implementation of the above code look like?
I want to execute a QueryString using WebClient but using the POST Method
That is what I got so far
CODE:
using (var client = new WebClient())
{
client.QueryString.Add("somedata", "value");
client.DownloadString("uri");
}
It is working but unfortunately it is using GET not POST and the reason I want it to use POST is that I am doing a web scraping and this is how the request is made as I see in WireShark. [IT USES POST AS A METHOD BUT NO POST DATA ONLY THE QUERY STRING.]
In answer to your specific question:
client.Headers.Add("Content-Type", "application/x-www-form-urlencoded");
byte[] response = client.UploadData("your url", "POST", new byte[] { });
//get the response as a string and do something with it...
string s = System.Text.Encoding.Default.GetString(response);
But using WebClient can be a PITA since it doesn't accept cookies nor allow you to set a timeout.
this will help you, use WebRequest instead of WebClient.
using System;
using System.Net;
using System.Threading;
using System.IO;
using System.Text;
class ThreadTest
{
static void Main()
{
WebRequest req = WebRequest.Create("http://www.yourDomain.com/search");
req.Proxy = null;
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
string reqString = "searchtextbox=webclient&searchmode=simple";
byte[] reqData = Encoding.UTF8.GetBytes(reqString);
req.ContentLength = reqData.Length;
using (Stream reqStream = req.GetRequestStream())
reqStream.Write(reqData, 0, reqData.Length);
using (WebResponse res = req.GetResponse())
using (Stream resSteam = res.GetResponseStream())
using (StreamReader sr = new StreamReader(resSteam))
File.WriteAllText("SearchResults.html", sr.ReadToEnd());
System.Diagnostics.Process.Start("SearchResults.html");
}
}
how can i upload a large string (in my case XML with BLOB) with POST without getting Timeout with GetResponse?
Changing the timeout helps, but this isn't really a solution.
If the Server is really death or the POST was interrupted i have to wait for the extrem large timeout.
Any Idea?
HttpWebRequest webRequest = null;
string response = "";
byte[] bytes = Encoding.UTF8.GetBytes(xml);
try
{
webRequest = (HttpWebRequest)WebRequest.Create("http://" + this.host + ":" + this.port);
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.Method = "POST";
webRequest.Timeout = 5000;
webRequest.ContentLength = bytes.Length;
using (Stream requeststream = webRequest.GetRequestStream())
{
requeststream.Write(bytes, 0, bytes.Length);
requeststream.Close();
}
using (HttpWebResponse webResponse = (HttpWebResponse)webRequest.GetResponse())
{
using (StreamReader sr = new StreamReader(webResponse.GetResponseStream()))
{
response = sr.ReadToEnd().Trim();
sr.Close();
}
webResponse.Close();
}
}
catch(Exception ex)
{
MessageBox.Show(ex.ToString());
}
return response;
Yes, this is pretty much expected http behaviour.
Options:
have a large timeout (you've already done this), and accept that it could take a long time to legitimately time out (as opposed to taking a while because of bandwidth)
maybe you can apply gzip on the request (and tell the server you're sending it compressed); I honestly don't know if this is supported automatically, but it could certainly be done by the api explicitly checking for a particular header and applying gzip decompression on the payload
change the api to perform a number of small uploads, and a completion message
live with it