Develop Reference

GDax API GET request always returns Error 400

I'm trying to get the order book from GDAX (link to documentation of the call) but when doing it from the c# executable I always get Error 400 - Bad request.
When taking the actual URL and pasting it into my browser, it works fine.
String URL = "https://api.gdax.com/products/BTC-USD/book?level=2";
WebRequest request = WebRequest.Create(URL);
WebResponse response = request.GetResponse();

The actual issue with your API call is , the API is expecting a user-agent string while making the request: Below is the code in working condition:
try
{
String URL = "http://api.gdax.com/products/BTC-USD/book?level=2";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(URL);
request.UserAgent = ".NET Framework Test Client";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
var encoding = ASCIIEncoding.ASCII;
using (var reader = new System.IO.StreamReader(response.GetResponseStream(), encoding))
{
string responseText = reader.ReadToEnd();
}
}
catch(WebException ex)
{
HttpWebResponse xyz = ex.Response as HttpWebResponse;
var encoding = ASCIIEncoding.ASCII;
using (var reader = new System.IO.StreamReader(xyz.GetResponseStream(), encoding))
{
string responseText = reader.ReadToEnd();
}
}
Basically ProtocolError indicates that you have received the response but there is an error related to protocol, which you can observe, when you read the response content from exception. I have added catch to handle the exception and read ex.Response (which is HttpWebResponse) and could see that the API is asking for user-agent to be suppllied while making the call. I got to see the error as "{"message":"User-Agent header is required."}"
You can ignore the code inside the exception block, I used it only to see what is the actual response message, which contains actual error details
Note: I have boxed WebRequest to HttpWebRequest to have additional http protocol related properties and most importantly "UserAgent" property which is not available with the WebRequest class.

You need to Accept the certificarte, Google for access to a https webrequest.
Like this

Is there a way to extract the message from WebException?

Is there a way to extract the text that is being sent back as part of the WebException that occurs can occur with a HttpWebResponse? I can get all of the header information but there is a custom message being returned with the 400 or 401 response that I would like to get if possible. I am currently handling the exception in my test like this:
var ex = Assert.Throws<WebException>(() =>
{
HttpWebResponse response = Utils.GetRawResponse(url);
});
Assert.Contains("401", ex.Message);
Here is how am getting the response:
public static HttpWebResponse GetRawResponse(string requestURL)
{
HttpWebRequest request = WebRequest.Create(requestURL) as HttpWebRequest;
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
return response;
}
And this works, but does not have the custom message.
Top hopefully be a little more clear am am referring to the message text in the bottom of the screenshot.:

With Gusmans reminder I created a method to extract the response from the WebException:
public static string ParseExceptionRespose(WebException exception)
{
string responseContents;
Stream descrption = ((HttpWebResponse)exception.Response).GetResponseStream();
using (StreamReader readStream = new StreamReader(descrption))
{
responseContents = readStream.ReadToEnd();
}
return responseContents;
}

When handling the exception, try something like:
if (ex.Response.ContentLength > 0)
{
string ResponseBody = new StreamReader(ex.Response.GetResponseStream()).ReadToEnd();
// Do whatever you want with ResponseBody
}
Similar to examples in http://msdn.microsoft.com/en-us/library/system.net.httpwebresponse.contentlength(v=vs.110).aspx)
The key point being that WebException had a Response property: http://msdn.microsoft.com/en-us/library/system.net.webexception.response(v=vs.110).aspx

HttpWebRequest-The remote server returned an error: (400) Bad Request

I am getting The remote server returned an error: (400) Bad Request error while running the following code.
I am trying to upload xml file on the http server.
My xml file contains tag for the username,password and domain and when i am trying to connect is manually i am able to connect it,but using same credentials when i am trying to connect it through this code, i am getting 400 Bad Request error.
Please suggest me how to overcome this issue.
Thanks
`
public static void UploadHttp(string xml)
{
string txtResults = string.Empty;
try
{
string url = "http://my.server.com/upload.aspx ";
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.KeepAlive = false;
request.SendChunked = true;
request.AllowAutoRedirect = true;
request.Method = "Post";
request.ContentType = "text/xml";
var encoder = new UTF8Encoding();
var data = encoder.GetBytes(xml);
request.ContentLength = data.Length;
var reqStream = request.GetRequestStream();
reqStream.Write(data, 0, data.Length);
reqStream.Close();
WebResponse response = null;
response = request.GetResponse();
var reader = new StreamReader(response.GetResponseStream());
var str = reader.ReadToEnd();
}
catch (WebException ex)
{
if (ex.Status == WebExceptionStatus.ProtocolError)
{
HttpWebResponse err = ex.Response as HttpWebResponse;
if (err != null)
{
string htmlResponse = new StreamReader(err.GetResponseStream()).ReadToEnd();
txtResults = string.Format("{0} {1}", err.StatusDescription, htmlResponse);
}
}
else
{
}
}
catch (Exception ex)
{
txtResults = ex.ToString();
}
}`

Are you sure you should be using POST not PUT?
POST is usually used with application/x-www-urlencoded formats. If you are using a REST API, you should maybe be using PUT? If you are uploading a file you probably need to use multipart/form-data. Not always, but usually, that is the right thing to do..
Also you don't seem to be using the credentials to log in - you need to use the Credentials property of the HttpWebRequest object to send the username and password.

400 Bad request Error will be thrown due to incorrect authentication entries.
Check if your API URL is correct or wrong. Don't append or prepend spaces.
Verify that your username and password are valid. Please check any spelling mistake(s) while entering.
Note: Mostly due to Incorrect authentication entries due to spell changes will occur 400 Bad request.

What type of authentication do you use?
Send the credentials using the properties Ben said before and setup a cookie handler.
You already allow redirection, check your webserver if any redirection occurs (NTLM auth does for sure). If there is a redirection you need to store the session which is mostly stored in a session cookie.

//use "ASCII" or try with another encoding scheme instead of "UTF8".
using (StreamWriter postStream = new StreamWriter(request.GetRequestStream(), System.Text.Encoding.UTF8))
{
postStream.Write(postData);
postStream.Close();
}

using LINQ to filter a List<string> without changing the variable type

I am writing a web crawler in c#. Within the method to get all of the links on a page, i want to return the list of links, but 'filter' it with LINQ so that the list only contains urls that exist. I have a helper method written called RemoteFileExists that returns a boolean value. At the end of the method, I wrote the following LINQ line:
//Links is a List<string> that hasn't been filtered
return (from link in Links
where RemoteFileExists(link)
select link).ToList<string>();
For some reason, when I do this, the List is returned empty.
RemoteFileExists:
static bool RemoteFileExists(string url)
{
try
{
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
request.Method = "HEAD";
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
return (response.StatusCode == HttpStatusCode.OK);
}
catch
{
return false;
}

I guess either you links are not correct or your sites don't support HEAD. Since this code works
List<string> Links = new List<string>() {"http://www.google.com"};
var res = ( from link in Links
where RemoteFileExists(link)
select link).ToList<string>();

I have been using the RemoteFileExists method in my code. Sometimes the program hangs up because the request is not closed. Right now I am using the following code:
static bool RemoteFileExists(string url)
{
try
{
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
request.Method = "HEAD";
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
response.Close();
return (response.StatusCode == HttpStatusCode.OK);
}
catch
{
return false;
}
}
Also, the above code does not detect redirects. This is important to crawlers because you need to know when to advance to another page, instead of following redirects to the same page.

What would be the correct way to use the keyword USING in the following C# code?

I'm trying to get this correct as I feel I'm missing something. I want to use the keyword using whenever I have an IDisposable object. Please note that the code works, I just want to optimize it.
I have two questions here:
1) For this code:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
What does it mean to add (HttpWebRequest) like that? Am I converting WebRequest to HttpWebRequest?
Why can't I do this?
HttpWebRequest rq = new HttpWebRequest();
rq.Create(url);
2) In the functional code below, how would I go about using the keyword using where applicable?
public static int UploadFileToixLibrary(string url, string file)
{
NetworkCredential credentials = new NetworkCredential();
credentials.UserName = AppVars.Username;
credentials.Password = AppVars.Password;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Credentials = credentials;
request.Method = "POST";
request.ContentType = "image/tiff";
request.Headers.Add("X-Object-Key", Path.GetFileName(file));
byte[] bytes = File.ReadAllBytes(file);
Stream st = null;
try
{
request.ContentLength = bytes.Length;
st = request.GetRequestStream();
st.Write(bytes, 0, bytes.Length);
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
return 1;
}
finally
{
if (st != null)
{
st.Close();
}
}
try
{
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader sr = new StreamReader(response.GetResponseStream());
response.Close();
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
return 1;
}
return 0;
}

Question 1:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
The reason for the cast, is that the static Create method on WebRequest returns an instance of WebRequest which is most appropriate for the scheme you supply in the url (ie, http:// address will return an HttpWebRequest, ftp:// will return FtpWebRequest etc). As you know your url is http, you know you'll get back an HttpWebRequest, so you can explicitly cast to the right type to get access to the extra functionality HttpWebRequest over the abstract WebRequest.
Now, WebRequest is not IDisposable, so you cannot use it inside a using statement!
Question 2: In your functional code, the only place you could use a using statement is around the Stream access:
try
{
request.ContentLength = bytes.Length;
st = request.GetRequestStream();
st.Write(bytes, 0, bytes.Length);
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
return 1;
}
finally
{
if (st != null)
{
st.Close();
}
}
Could be re-written:
request.ContentLength = bytes.Length;
using(var st = request.GetRequestStream())
{
st.Write(bytes, 0, bytes.Length);
st.Close();
}

WebRequest is a abstract class, it does not expose most of the properties/methods that you're going to use. You must cast it to the appropriate type to use it. The static method Create will return the specialized WebRequest object according to your URL. As you know that this URL uses the HTTP protocol (http://), it's safe to cast it to HttpWebRequest. It could, however, be a FtpWebRequest.
You can put the GetRequestStream initilization inside using(...) statements. Looks like WebRequest (and its children) does not implement IDisposable

1.
The Webrequest.Create method returns a WebRequest reference, but the actual object has different types depending on what kind of request you want to do. In your case you are doing a HTTP request, so the actual object is an HttpWebRequest object. You cast the reference to the actual type of the object to get access to the members that are specific for the HttpWebRequest class.
2.
The HttpWebRequest is not disposable. Use a using block for the HttpWebResponse object:
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse()) {
...
}