Develop Reference

How do I make a case request from the Pacer.gov API?

I am trying to make a request to an API called Pacer.gov. I'm expecting a file to be returned, but I'm not getting it. Can someone help me with what I'm missing?
So my C# Rest call looks like this:
(The variable PacerSession is the authentication cookie I got (with help from #jonathon-reinhart); read more about that here: How do I use RestSharp to POST a login and password to an API?)
var client = new RestClient("https://pcl.uscourts.gov/dquery");
client.CookieContainer = new System.Net.CookieContainer();
//var request = new RestRequest("/dquery", Method.POST);
var request = new RestRequest(Method.POST);
request.AddParameter("download", "1");
request.AddParameter("dl_fmt", "xml");
request.AddParameter("party", "Moncrief");
request.AddHeader("user-agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36");
request.AddHeader("content-type", "text/plain; charset=utf-8");
request.AddHeader("accept", "*/*");
request.AddHeader("accept-encoding", "gzip, deflate, sdch");
request.AddHeader("accept-language", "en-US,en;q=0.8");
request.AddHeader("cookie", "PacerSession=" + PacerSession);
IRestResponse response = client.Execute(request);
If I just type the URL https://pcl.uscourts.gov/dquery?download=1&dl_fmt=xml&party=Moncrief into Chrome, I get back an XML file. When I look at the IRestResponse, I don't see anything that looks like a file. Is there something wrong with my request or am I getting the file back and just need to know how to retrieve it?
Here's part of the file I get back if I use the URL directly in the browser:
Here's what I see in VS when I debug it and look at the IRestResponse variable:
UPDATE - 6/3/16
Received this response from Pacer tech support:
In the Advanced REST Client, you will see a HTTP 302 response (a redirect to another page). In a normal browser, the redirect is automatically followed without the user seeing anything (even on the URL in the browser).
The ARC does not automatically follow that redirect to the target page.
You can see in the header of the response the target URL that has the results.
If you manually cut and paste this URL to the ARC as a HTTP GET request, you will get the XML results. I have never used C#, but there is usually a property associated with web clients that will force the client to follow the redirect.
I tried adding this:
client.FollowRedirects = true;
but I'm still not seeing an xml file when I debug this code:
IRestResponse response = client.Execute(request);
How do I get the file? Is there something I have to do to get the file from the URL it's being redirected to?

There's one major problem with your code. You're only carrying one of the three cookies that checp-pacer-passwd.pl returns. You need to preserve all three. The following code is a possible implementation of this, with some notes afterwards.
public class PacerClient
{
private CookieContainer m_Cookies = new CookieContainer();
public string Username { get; set; }
public string Password { get; set; }
public PacerClient(string username, string password)
{
this.Username = username;
this.Password = password;
}
public void Connect()
{
var client = new RestClient("https://pacer.login.uscourts.gov");
client.CookieContainer = this.m_Cookies;
RestRequest request = new RestRequest("/cgi-bin/check-pacer-passwd.pl", Method.POST);
request.AddParameter("loginid", this.Username);
request.AddParameter("passwd", this.Password);
IRestResponse response = client.Execute(request);
if (response.Cookies.Count < 1)
{
throw new WebException("No cookies returned.");
}
}
public XmlDocument SearchParty(string partyName)
{
string requestUri = $"/dquery?download=1&dl_fmt=xml&party={partyName}";
var client = new RestClient("https://pcl.uscourts.gov");
client.CookieContainer = this.m_Cookies;
var request = new RestRequest(requestUri);
IRestResponse response = client.Execute(request);
if (!String.IsNullOrEmpty(response.Content))
{
XmlDocument result = new XmlDocument();
result.LoadXml(response.Content);
return result;
}
else return null;
}
}
It's easiest to just keep a hold of the CookieContainer throughout the entire time you're working with Pacer. I wrapped the functionality into a class, just to make it a little easier to package up with this answer, but you can implement it however you want. I didn't put in any real error checking, so you probably want to check that response.ResponseUri is actually the search page and not the logon page, and that the content is actually well-formed XML.
I've tested this using my own Pacer account, like so:
PacerClient client = new PacerClient(Username, Password);
client.Connect();
var document = client.SearchParty("Moncrief");

Httpclinet getasync resetting cookies

I'm working on a cross platform app targeting windows phone8, windows 8, android and Ios. The app needs to get some data from the internet and reuse the data accordingly. So I have a project with few classes inside. One class to Sign in to the website and get the required cookies for further use. Below is the code from this class:-
public static CookieContainer Cookiejar { get; private set; }
public static async Task<CookieContainer> GetCookies()
{
var uri = new Uri("https://mywebsite.com");
Cookiejar = new CookieContainer();
var handler = new HttpClientHandler
{
CookieContainer = Cookiejar,
UseCookies = true,
UseDefaultCredentials = false
};
var client = new HttpClient(handler);
client.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/x-www-form-urlencoded"));
HttpContent content = new FormUrlEncodedContent(new[]
{
new KeyValuePair<string,string>("username","user"),
new KeyValuePair<string,string>("password","user1234"),
new KeyValuePair<string,string>("customer","AAAA"),
new KeyValuePair<string,string>("doLogin","Logga+in"),
new KeyValuePair<string,string>("language","se")
});
await client.PostAsync(uri, content);
return Cookiejar;
}
Another class is used to get the actual data. I'm passing the CookieContainer from the first class which has CookieHandler as name to the other class which has DataHandler as name. The problem I'm facing is when I use this code in windows phone 8 project and whenever I call httpclient getasync it acts as if I didn't pass the cookies and reset it to xxxxx and the response will be only the log-in page instead of my desired page. Below is the code for DataHandler class:-
public string Response { get; private set; }
public async Task<string> GetResponse(Uri uri)
{
HttpClientHandler handler = new HttpClientHandler();
handler.UseCookies = true;
handler.CookieContainer = CookieHandler.Cookiejar;
HttpClient client = new HttpClient(handler);
client.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/xhtml+xml"));
var response = await client.GetAsync(uri);
string webresponse = null;
if (response.IsSuccessStatusCode)
{
var resp = response.Content.ReadAsByteArrayAsync().Result;
Encoding encode = Encoding.GetEncoding("iso-8859-1");
var respString = encode.GetString(resp, 0, resp.Length - 1);
webresponse = respString;
}
handler.Dispose();
client.Dispose();
response.Dispose();
return Response = webresponse;
}
Been looking into this problem for few hours now and tested every possible tips I found here and on the net, but nothings work. OBS!!! the cookies are not HtmlOnly and not secure. What really gets me confused is the fact that if I use this exact same code in a windows 8 project it works perfectly but not in windows phone 8 project. Worth noting i'm using the latest BCL system.net.http lib.
Any help and suggestions are more than welcome.

Just wanted to answer this question which might be bothering other developers around the World.
First i would like to thank TheESJ for heading me to the right direction.
Now lets start with what i changed/added to my code to workaround this problem.
I just added the following code lines to my initial POST method:-
var wwwUri = new Uri("https://www.mywebsite.com");
Cookiejar.SetCookies(baseUri, Cookiejar.GetCookieHeader(wwwUri));
It appears that in Windows Phone there is '.' in the cookie header (description below), the above code will make a copy of your cookies with the correct header.
Before I applied the above code lines and the CookieContainer in debugger and look at the m_domainTable it had one item with a value {[.mywebsite.com, System.Net.PathList]} eventhough the domain property value is ‘mywebsite.com’ without ‘.’, while in Windows store app it hade one item with a value {[mywebsite.com, System.Net.PathList]} and the domain property value is ‘mywebsite.com’.
After I applied the above code and it worked I debugged and check CookieContainer, the m_domainTable have now two items, one with this value {[.mywebsite.com, System.Net.PathList]} with the domain property value is ‘mywebsite.com’ and the other with this value {[mywebsite.com, System.Net.PathList]} and the same domain property value. That is why it’s sending the right cookie now as it is sending the 2nd cookie {[mywebsite.com, System.Net.PathList]}.

RestSharp - Retrieving Authorization token from POSTed response

I am trying to pass username and password to the following URL :
https://maxcvservices.dnb.com/rest/Authentication
According to the documentation the user_id and password must be passed as headers with the keys: x-dnb-user, x-dnb-pwd respectively.
I thus far have the following code which seems to work but I am unable to retrieve the auth token returned by the response object:
public static void Main (string[] args)
{
var client = new RestClient ("https://maxcvservices.dnb.com/rest/Authentication");
var request = new RestRequest (Method.POST);
request.AddHeader("x-dnb-user", myEmail);
request.AddHeader("x-dnb-pwd", myPassword);
IRestResponse resp = client.Execute(request);
var content = resp.Content;
Console.WriteLine (resp.StatusDescription);
Console.WriteLine (resp.StatusCode);
}
When I try printing the content I get a blank line but what I am actually expecting is the auth token that is returned by the service. A couple of things I think I am doing in the code (but not sure), is passing the userid and password as headers in the POST request which is what is required. The token is returned as the value of the 'Authorization' field in the response object. I was wondering how I might print the token. Also the statusDescription,statusCode both print OK which tells me I have the correct request but am unable to locate the auth token in the response. Any help would be much appreciated in guiding me on how to access the auth token in the Authorization field of the returned POST response.

So you're trying to get the HttpHeader values for Authorization from the IRestResponse object?
You could use e.g. use LINQ for that:
var authroizationHeaderFromResponse = resp.Headers.FirstOrDefault(h => h.Name == "Authorization");
if (authroizationHeaderFromResponse != null)
{
Console.WriteLine(authroizationHeaderFromResponse.Value);
}
Which yields
INVALID CREDENTIALS
You assume that if the response status code is 200 - OK, then there must be a response body accompanying it.
Does the documentation specifically state that you should expect a token in the response body in return?
The D&B developers could send a 200 - OK response with no response body if they want, or they can add their serialized token (JSON, XML etc) elsewhere, e.g. in a header field.
An example of this can be seen in this code from an ASP.NET Web API returning a response from a successful PUT
if (result.Success)
{
var dto = Mapper.Map<TEntity, TDto>(result.Data as TEntity);
var response = Request.CreateResponse(HttpStatusCode.Created, dto);
var uri = Url.Link("DefaultApi", new {id = dto.Id});
response.Headers.Location = new Uri(uri);
return response;
}
This would return a 200 - OK with a serialized object (result.Data) in the response body, but there's nothing wrong with me changing the following
var response = Request.CreateResponse(HttpStatusCode.Created, dto);
To something like
var response = Request.CreateResponse(HttpStatusCode.Created);
That way you would still get a 200 - OK response, but without a response body. This of course is against the recommendations of the HTTP/1.1 Standard for PUT verbs, but it would still work.
I could even do this for giggles
throw new HttpResponseException(HttpStatusCode.Created);
And you would still get a 200 - OK response. Somewhat evil, but possible.
I would suggest trying to fetching data from another resource with the x-dnb-user and x-dnb-pwd header fields set, and check if a response body is returned then. Perhaps D&B was inspired by Basic Authentication when implementing these header fields, and as such require them to be present in every request?
It's worth a try.
Let me know how that works out.

Look in the Headers collection of IRestResponse. It will probably be there rather than the content.
Hth
Oli

It could be that the AUTHTOKEN comes back with the cookies, as this is a common approach.
In this case, you'll need to attach a CookieContanier to your IRestClient, then this container will store the cookies. Provided you use the same client for subsequent requests, that auth cookie will let you in.
private CookieContainer _cookieJar;
...
_cookieJar = new CookieContainer();
_client.CookieContainer = _cookieJar;
You can then inspect the container after a request
_client.PostAsync(MyRequest, (r, h) =>
{
r.Cookies... // inspect em

Error Forbidden 403 simulating request via C#

Scope:
I am developing a C# aplication to simulate queries into this site. I am quite familiar with simulating web requests for achieving the same human steps, but using code instead.
If you want to try yourself, just type this number into the CNPJ box:
08775724000119 and write the captcha and click on Confirmar
I've dealed with the captcha already, so it's not a problem anymore.
Problem:
As soon as i execute the POST request for a "CNPJ", a exception is thrown:
The remote server returned an error: (403) Forbidden.
Fiddler Debugger Output:
Link for Fiddler Download
This is the request generated by my browser, not by my code
POST https://www.sefaz.rr.gov.br/sintegra/servlet/hwsintco HTTP/1.1
Host: www.sefaz.rr.gov.br
Connection: keep-alive
Content-Length: 208
Cache-Control: max-age=0
Origin: https://www.sefaz.rr.gov.br
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11
Content-Type: application/x-www-form-urlencoded
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Referer: https://www.sefaz.rr.gov.br/sintegra/servlet/hwsintco
Accept-Encoding: gzip,deflate,sdch
Accept-Language: pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: GX_SESSION_ID=gGUYxyut5XRAijm0Fx9ou7WnXbVGuUYoYTIKtnDydVM%3D; JSESSIONID=OVuuMFCgQv9k2b3fGyHjSZ9a.undefined
// PostData :
_EventName=E%27CONFIRMAR%27.&_EventGridId=&_EventRowId=&_MSG=&_CONINSEST=&_CONINSESTG=08775724000119&cfield=rice&_VALIDATIONRESULT=1&BUTTON1=Confirmar&sCallerURL=http%3A%2F%2Fwww.sintegra.gov.br%2Fnew_bv.html
Code samples and References used:
I'm using a self developed library to handle/wrap the Post and Get requests.
The request object has the same parameters (Host,Origin, Referer, Cookies..) as the one issued by the browser (logged my fiddler up here).
I've also managed to set the ServicePointValidator of certificates by using:
ServicePointManager.ServerCertificateValidationCallback =
new RemoteCertificateValidationCallback (delegate { return true; });
After all that configuration, i stil getting the forbidden exception.
Here is how i simulate the request and the exception is thrown
try
{
this.Referer = Consts.REFERER;
// PARAMETERS: URL, POST DATA, ThrownException (bool)
response = Post (Consts.QUERYURL, postData, true);
}
catch (Exception ex)
{
string s = ex.Message;
}
Thanks in advance for any help / solution to my problem
Update 1:
I was missing the request for the homepage, which generates cookies (Thanks #W0lf for pointing me that out)
Now there's another weird thing. Fiddler is not showing my Cookies on the request, but here they are :

I made a successful request using the browser and recorded it in Fiddler.
The only things that differ from your request are:
my browser sent no value for the sCallerURL parameter (I have sCallerURL= instead of sCallerURL=http%3A%2F%2Fwww....)
the session ids are different (obviously)
I have other Accept-Language: values (I'm pretty sure this is not important)
the Content-Length is different (obviously)
Update
OK, I thought the Fiddler trace was from your application. In case you are not setting cookies on your request, do this:
before posting data, do a GET request to https://www.sefaz.rr.gov.br/sintegra/servlet/hwsintco. If you examine the response, you'll notice the website sends two session cookies.
when you do the POST request, make sure to attach the cookies you got at the previous step
If you don't know how to store the cookies and use them in the other request, take a look here.
Update 2
The problems
OK, I managed to reproduce the 403, figured out what caused it, and found a fix.
What happens in the POST request is that:
the server responds with status 302 (temporary redirect) and the redirect location
the browser redirects (basically does a GET request) to that location, also posting the two cookies.
.NET's HttpWebRequest attempts to do this redirect seamlessly, but in this case there are two issues (that I would consider bugs in the .NET implementation):
the GET request after the POST(redirect) has the same content-type as the POST request (application/x-www-form-urlencoded). For GET requests this shouldn't be specified
cookie handling issue (the most important issue) - The website sends two cookies: GX_SESSION_ID and JSESSIONID. The second has a path specified (/sintegra), while the first does not.
Here's the difference: the browser assigns by default a path of /(root) to the first cookie, while .NET assigns it the request url path (/sintegra/servlet/hwsintco).
Due to this, the last GET request (after redirect) to /sintegra/servlet/hwsintpe... does not get the first cookie passed in, as its path does not correspond.
The fixes
For the redirect problem (GET with content-type), the fix is to do the redirect manually, instead of relying on .NET for this.
To do this, tell it to not follow redirects:
postRequest.AllowAutoRedirect = false
and then read the redirect location from the POST response and manually do a GET request on it.
The cookie problem (that has happened to others as well)
For this, the fix I found was to take the misplaced cookie from the CookieContainer, set it's path correctly and add it back to the container in the correct location.
This is the code to do it:
private void FixMisplacedCookie(CookieContainer cookieContainer)
{
var misplacedCookie = cookieContainer.GetCookies(new Uri(Url))[0];
misplacedCookie.Path = "/"; // instead of "/sintegra/servlet/hwsintco"
//place the cookie in thee right place...
cookieContainer.SetCookies(
new Uri("https://www.sefaz.rr.gov.br/"),
misplacedCookie.ToString());
}
Here's all the code to make it work:
using System;
using System.IO;
using System.Net;
using System.Text;
namespace XYZ
{
public class Crawler
{
const string Url = "https://www.sefaz.rr.gov.br/sintegra/servlet/hwsintco";
public void Crawl()
{
var cookieContainer = new CookieContainer();
/* initial GET Request */
var getRequest = (HttpWebRequest)WebRequest.Create(Url);
getRequest.CookieContainer = cookieContainer;
ReadResponse(getRequest); // nothing to do with this, because captcha is f##%ing dumb :)
/* POST Request */
var postRequest = (HttpWebRequest)WebRequest.Create(Url);
postRequest.AllowAutoRedirect = false; // we'll do the redirect manually; .NET does it badly
postRequest.CookieContainer = cookieContainer;
postRequest.Method = "POST";
postRequest.ContentType = "application/x-www-form-urlencoded";
var postParameters =
"_EventName=E%27CONFIRMAR%27.&_EventGridId=&_EventRowId=&_MSG=&_CONINSEST=&" +
"_CONINSESTG=08775724000119&cfield=much&_VALIDATIONRESULT=1&BUTTON1=Confirmar&" +
"sCallerURL=";
var bytes = Encoding.UTF8.GetBytes(postParameters);
postRequest.ContentLength = bytes.Length;
using (var requestStream = postRequest.GetRequestStream())
requestStream.Write(bytes, 0, bytes.Length);
var webResponse = postRequest.GetResponse();
ReadResponse(postRequest); // not interested in this either
var redirectLocation = webResponse.Headers[HttpResponseHeader.Location];
var finalGetRequest = (HttpWebRequest)WebRequest.Create(redirectLocation);
/* Apply fix for the cookie */
FixMisplacedCookie(cookieContainer);
/* do the final request using the correct cookies. */
finalGetRequest.CookieContainer = cookieContainer;
var responseText = ReadResponse(finalGetRequest);
Console.WriteLine(responseText); // Hooray!
}
private static string ReadResponse(HttpWebRequest getRequest)
{
using (var responseStream = getRequest.GetResponse().GetResponseStream())
using (var sr = new StreamReader(responseStream, Encoding.UTF8))
{
return sr.ReadToEnd();
}
}
private void FixMisplacedCookie(CookieContainer cookieContainer)
{
var misplacedCookie = cookieContainer.GetCookies(new Uri(Url))[0];
misplacedCookie.Path = "/"; // instead of "/sintegra/servlet/hwsintco"
//place the cookie in thee right place...
cookieContainer.SetCookies(
new Uri("https://www.sefaz.rr.gov.br/"),
misplacedCookie.ToString());
}
}
}

Sometimes HttpWebRequest needs proxy initialization:
request.Proxy = new WebProxy();//in my case it doesn't need parameters, but you can set it to your proxy address

Problems authenticating to website from code

I am trying to write code that will authenticate to the website wallbase.cc. I've looked at what it does using Firfebug/Chrome Developer tools and it seems fairly easy:
Post "usrname=$USER&pass=$PASS&nopass_email=Type+in+your+e-mail+and+press+enter&nopass=0" to the webpage "http://wallbase.cc/user/login", store the returned cookies and use them on all future requests.
Here is my code:
private CookieContainer _cookies = new CookieContainer();
//......
HttpPost("http://wallbase.cc/user/login", string.Format("usrname={0}&pass={1}&nopass_email=Type+in+your+e-mail+and+press+enter&nopass=0", Username, assword));
//......
private string HttpPost(string url, string parameters)
{
try
{
System.Net.WebRequest req = System.Net.WebRequest.Create(url);
//Add these, as we're doing a POST
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
((HttpWebRequest)req).Referer = "http://wallbase.cc/home/";
((HttpWebRequest)req).CookieContainer = _cookies;
//We need to count how many bytes we're sending. Post'ed Faked Forms should be name=value&
byte[] bytes = System.Text.Encoding.ASCII.GetBytes(parameters);
req.ContentLength = bytes.Length;
System.IO.Stream os = req.GetRequestStream();
os.Write(bytes, 0, bytes.Length); //Push it out there
os.Close();
//get response
using (System.Net.WebResponse resp = req.GetResponse())
{
if (resp == null) return null;
using (Stream st = resp.GetResponseStream())
{
System.IO.StreamReader sr = new System.IO.StreamReader(st);
return sr.ReadToEnd().Trim();
}
}
}
catch (Exception)
{
return null;
}
}
After calling HttpPost with my login parameters I would expect all future calls using this same method to be authenticated (assuming a valid username/password). I do get a session cookie in my cookie collection but for some reason I'm not authenticated. I get a session cookie in my cookie collection regardless of which page I visit so I tried loading the home page first to get the initial session cookie and then logging in but there was no change.
To my knowledge this Python version works: https://github.com/sevensins/Wallbase-Downloader/blob/master/wallbase.sh (line 336)
Any ideas on how to get authentication working?
Update #1
When using a correct user/password pair the response automatically redirects to the referrer but when an incorrect user/pass pair is received it does not redirect and returns a bad user/pass pair. Based on this it seems as though authentication is happening, but maybe not all the key pieces of information are being saved??
Update #2
I am using .NET 3.5. When I tried the above code in .NET 4, with the added line of System.Net.ServicePointManager.Expect100Continue = false (which was in my code, just not shown here) it works, no changes necessary. The problem seems to stem directly from some pre-.Net 4 issue.

This is based on code from one of my projects, as well as code found from various answers here on stackoverflow.
First we need to set up a Cookie aware WebClient that is going to use HTML 1.0.
public class CookieAwareWebClient : WebClient
{
private CookieContainer cookie = new CookieContainer();
protected override WebRequest GetWebRequest(Uri address)
{
HttpWebRequest request = (HttpWebRequest)base.GetWebRequest(address);
request.ProtocolVersion = HttpVersion.Version10;
if (request is HttpWebRequest)
{
(request as HttpWebRequest).CookieContainer = cookie;
}
return request;
}
}
Next we set up the code that handles the Authentication and then finally loads the response.
var client = new CookieAwareWebClient();
client.UseDefaultCredentials = true;
client.BaseAddress = #"http://wallbase.cc";
var loginData = new NameValueCollection();
loginData.Add("usrname", "test");
loginData.Add("pass", "123");
loginData.Add("nopass_email", "Type in your e-mail and press enter");
loginData.Add("nopass", "0");
var result = client.UploadValues(#"http://wallbase.cc/user/login", "POST", loginData);
string response = System.Text.Encoding.UTF8.GetString(result);
We can try this out using the HTML Visualizer inbuilt into Visual Studio while staying in debug mode and use that to confirm that we were able to authenticate and load the Home page while staying authenticated.
The key here is to set up a CookieContainer and use HTTP 1.0, instead of 1.1. I am not entirely sure why forcing it to use 1.0 allows you to authenticate and load the page successfully, but part of the solution is based on this answer.
https://stackoverflow.com/a/10916014/408182
I used Fiddler to make sure that the response sent by the C# Client was the same as with my web browser Chrome. It also allows me to confirm if the C# client is being redirect correctly. In this case we can see that with HTML 1.0 we are getting the HTTP/1.0 302 Found and then redirects us to the home page as intended. If we switch back to HTML 1.1 we will get an HTTP/1.1 417 Expectation Failed message instead.
There is some information on this error message available in this stackoverflow thread.
HTTP POST Returns Error: 417 "Expectation Failed."
Edit: Hack/Fix for .NET 3.5
I have spent a lot of time trying to figure out the difference between 3.5 and 4.0, but I seriously have no clue. It looks like 3.5 is creating a new cookie after the authentication and the only way I found around this was to authenticate the user twice.
I also had to make some changes on the WebClient based on information from this post.
http://dot-net-expertise.blogspot.fr/2009/10/cookiecontainer-domain-handling-bug-fix.html
public class CookieAwareWebClient : WebClient
{
public CookieContainer cookies = new CookieContainer();
protected override WebRequest GetWebRequest(Uri address)
{
var request = base.GetWebRequest(address);
var httpRequest = request as HttpWebRequest;
if (httpRequest != null)
{
httpRequest.ProtocolVersion = HttpVersion.Version10;
httpRequest.CookieContainer = cookies;
var table = (Hashtable)cookies.GetType().InvokeMember("m_domainTable", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.GetField | System.Reflection.BindingFlags.Instance, null, cookies, new object[] { });
var keys = new ArrayList(table.Keys);
foreach (var key in keys)
{
var newKey = (key as string).Substring(1);
table[newKey] = table[key];
}
}
return request;
}
}
var client = new CookieAwareWebClient();
var loginData = new NameValueCollection();
loginData.Add("usrname", "test");
loginData.Add("pass", "123");
loginData.Add("nopass_email", "Type in your e-mail and press enter");
loginData.Add("nopass", "0");
// Hack: Authenticate the user twice!
client.UploadValues(#"http://wallbase.cc/user/login", "POST", loginData);
var result = client.UploadValues(#"http://wallbase.cc/user/login", "POST", loginData);
string response = System.Text.Encoding.UTF8.GetString(result);

You may need to add the following:
//get response
using (System.Net.WebResponse resp = req.GetResponse())
{
foreach (Cookie c in resp.Cookies)
_cookies.Add(c);
// Do other stuff with response....
}
Another thing that you might have to do is, if the server responds with a 302 (redirect) the .Net web request will automatically follow it and in the process you might lose the cookie you're after. You can turn off this behavior with the following code:
req.AllowAutoRedirect = false;

The Python you reference uses a different referrer (http://wallbase.cc/start/). It is also followed by another post to (http://wallbase.cc/user/adult_confirm/1). Try the other referrer and followup with this POST.
I think you are authenticating correctly, but that the site needs more info/assertions from you before proceeding.