Capture all requests with FiddlerCore - c#

I am trying to monitor traffic using FiddlerCore and WebBrowser Controller, I have below code to capture web requests in C#
private void button1_Click(object sender, EventArgs e)
{
List<Fiddler.Session> oAllSessions = new List<Fiddler.Session>();
URLMonInterop.SetProxyInProcess("127.0.0.1:8888", "<-loopback>");
webBrowser1.ScriptErrorsSuppressed = true;
WebProxy myProxy = new WebProxy();
Uri newUri = new Uri("http://localhost:8888");
myProxy.Address = newUri;
Fiddler.FiddlerApplication.Startup(8888, FiddlerCoreStartupFlags.Default);
Fiddler.FiddlerApplication.AfterSessionComplete += delegate(Fiddler.Session oS)
{
Monitor.Enter(oAllSessions);
oAllSessions.Add(oS);
Monitor.Exit(oAllSessions);
};
webBrowser1.Navigate("http://www.test.com/");
while (webBrowser1.ReadyState != WebBrowserReadyState.Complete)
{
System.Windows.Forms.Application.DoEvents();
}
var message = string.Join(Environment.NewLine, oAllSessions);
textBox1.Text = textBox1.Text + message;
Fiddler.FiddlerApplication.Shutdown();
}
private void Form1_FormClosing(object sender, FormClosingEventArgs e)
{
Fiddler.FiddlerApplication.Shutdown();
URLMonInterop.ResetProxyInProcessToDefault();
}
It is only returning just one request response (given url in webBroser.Navigate), I can not see requests for images, css and other loaded files on example site. I could not find any info on this, Can someone please help me to understand on how I can capture all GET POST requests when webBroswer.Navigate to given URL?
updated:
delegate void updateUI();
public Form1()
{
InitializeComponent();
}
private void Form1_Load(object sender, EventArgs e)
{
Fiddler.FiddlerApplication.AfterSessionComplete += FiddlerApplication_AfterSessionComplete;
Fiddler.FiddlerApplication.Startup(8888, FiddlerCoreStartupFlags.Default);
webBrowser1.ScriptErrorsSuppressed = true;
WebProxy myProxy = new WebProxy();
Uri newUri = new Uri("http://localhost:8888");
myProxy.Address = newUri;
string[] urls = new string[] { "http://localhost/test/page1",
"http://localhost/test/page2 "
};
foreach (string url in urls)
{
webBrowser1.Navigate(url);
// Capture root url
listBox1.Invoke(new updateUI(() =>
{
listBox1.Items.Add(url);
}));
while (webBrowser1.ReadyState != WebBrowserReadyState.Complete)
{
System.Windows.Forms.Application.DoEvents();
}
// Hack as I am not sure what to do here so wait 10 second for webBrowser to load all requests otherwise I only get last url data in listbox
for (int i = 0; i <= 10; i++)
{
System.Windows.Forms.Application.DoEvents();
Thread.Sleep(1000);
}
}
}
void FiddlerApplication_AfterSessionComplete(Session oSession)
{
var regex = new Regex("keywords-in-url-to-match");
// If my desired keyword match then grab request POST body
if (regex.IsMatch(oSession.fullUrl.ToString()))
{
string requestBody = oSession.GetRequestBodyAsString();
// Capture url and request body. This url is not root url
listBox1.Invoke(new updateUI(() =>
{
listBox1.Items.Add(oSession.fullUrl);
listBox1.Items.Add(System.Web.HttpUtility.UrlDecode(requestBody));
}));
}
}
private void Form1_FormClosing(object sender, FormClosingEventArgs e)
{
Fiddler.FiddlerApplication.Shutdown();
}

What makes you think that the other items aren't being pulled from the cache?
You shouldn't use FiddlerCoreStartupFlags.Default if you're going to use SetProxyInProcess; the former sets the proxy for every process on the system while the latter sets the proxy only for the current process.

Related

How to make a Save-file-as dialog box after downloading file

Description:
So I have this following script to download a file from the internet with a progress bar showing what percentage the download is at and custom message boxes. Now I have the files being saved to the users %TEMP% path. And it uses events to prevent people from clicking on the button again and starting a new download.
Problem:
I want to give the user a choice of where to save the file, but show his temp path as the default location. (Like a Save-file-dialog Box)
I'm still fairly new to coding and don't know where to exactly start.
What I tried:
I didn't try any new code, but I did go around google and try to find a solution. Here are some websites that I found that might be useful:
https://learn.microsoft.com/en-us/dotnet/framework/winforms/controls/how-to-save-files-using-the-savefiledialog-component
https://www.c-sharpcorner.com/UploadFile/mahesh/savefiledialog-in-C-Sharp/
They explain it really well. But I don't know how to incorporate it into this script.
And I dont want to write a brand new script. Any Help would be Appreciated!
private bool _isBusy = false;
private void button1_Click(object sender, EventArgs e)
=> DownloadFile("someurl1", "somefilename1.exe");
private void button2_Click(object sender, EventArgs e)
=> DownloadFile("someurl2", "somefilename2.exe");
private void button3_Click(object sender, EventArgs e)
=> DownloadFile("someurl3", "somefilename3.exe");
private void button4_Click(object sender, EventArgs e)
=> DownloadFile("someurl4", "somefilename4.exe");
private void DownloadFile(string url, string fileName)
{
if(_isBusy) return;
_isBusy = true;
var output = Path.Combine(Path.GetTempPath(), fileName);
MessageBox.Show($"{fileName} will start downloading from {url}");
using (WebClient client = new WebClient())
{
client.DownloadFileCompleted += (sender, args) =>
{
MessageBox.Show($"{fileName} Complete!");
Process.Start(output);
_isBusy = false;
};
client.DownloadProgressChanged += (sender, args) => progressBar1.Value = args.ProgressPercentage;
client.DownloadFileAsync(new Uri(url), output);
}
}
Maybe something like?
private SaveFileDialog save = new SaveFileDialog();
private void DownloadFile(string url, string fileName)
{
if (_isBusy) return;
save.InitialDirectory = Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments);
save.FileName = fileName;
if (save.ShowDialog() == DialogResult.OK)
{
_isBusy = true;
var output = save.FileName;
MessageBox.Show($"{fileName} will start downloading from {url}");
using (WebClient client = new WebClient())
{
client.DownloadFileCompleted += (sender, args) =>
{
MessageBox.Show($"{fileName} Complete!");
Process.Start(output);
_isBusy = false;
};
client.DownloadProgressChanged += (sender, args) => progressBar1.Value = args.ProgressPercentage;
client.DownloadFileAsync(new Uri(url), output);
}
}
}

how to capture real time stream data in csv or .data file in a windows form Application

I implemented a function in a windows form application to capture and read needed tabular data from a file (sourcedata.data) and save it in another file (result.data ).
How i and by using the application can capture a real time stream data like such available here :https://data.sparkfun.com/streams in csv or .data file to use it.
Or are there any direct waya to read the stream data directly from the website source periodically ?
private void button5_Click(object sender, EventArgs e)
{
List<string[]> rows = new List<string[]>();
int[] indexes = { 0, 1, 3, 5, 6, 7, 8, 9 };
using (var reader = new StreamReader(#"sourcedata.data"))
{
using (StreamWriter writetext = new StreamWriter("result.data"))
{
while (!reader.EndOfStream)
{
var line = reader.ReadLine();
if (line.IndexOf(',') == -1)
continue;
string[] values = line.Split(',');
string[] row = new string[indexes.Length];
int insertIndex = 0;
for (int i = 0; i < values.Length; i++)
{
string val = values[i];
if (val.Trim() == "?")
goto BREAK;
if (indexes.Contains(i))
row[insertIndex++] = val;
}
rows.Add(row);
writetext.WriteLine(String.Join(",", row));
BREAK:;
}
}
}
}
You have two split your problem into two separated sub problems:
Write a method public static string DownloadData(...) which will download the data from the source. This can be done by any HTTP client or library you can find like System.Net.Http.HttpClient or System.Net.WebClient.
See How to download a file from a URL in C#?
Add/start a timer which calls this method periodically. You can use classes like System.Windows.Forms.Timer or System.Timers.Timer.
See What is the best way to implement a "timer"?
#Progman
It is the code
public partial class Download : Form
{
public Download()
{
InitializeComponent();
}
WebClient client;
private void btnDownload_Click(object sender, EventArgs e)
{
string url = txtUrl.Text;
if (!string.IsNullOrEmpty(url))
{
Thread thread = new Thread(() =>
{
Uri uri = new Uri(url);
string filename = System.IO.Path.GetFileName(uri.AbsolutePath);
client.DownloadFileAsync(uri, Application.StartupPath + "/" + filename);
});
thread.Start();
}
}
private void Download_Load(object sender, EventArgs e)
{
client = new WebClient();
client.DownloadProgressChanged += Client_DownloadProgressChanged;
client.DownloadFileCompleted += Client_DownloadFileCompleted;
}
private void Client_DownloadFileCompleted(object sender, AsyncCompletedEventArgs e)
{
MessageBox.Show("Download Completed.", "Message", MessageBoxButtons.OK, MessageBoxIcon.Information);
}
private void Client_DownloadProgressChanged(object sender, DownloadProgressChangedEventArgs e)
{
Invoke(new MethodInvoker(delegate ()
{
progressBar.Minimum = 0;
double recieve = double.Parse(e.BytesReceived.ToString());
double total = double.Parse(e.TotalBytesToReceive.ToString());
double percentage = recieve / total * 100;
lblStatus.Text = $"Download {string.Format("{0:0.##}", percentage)}%";
progressBar.Value = int.Parse(Math.Truncate(percentage).ToString());
}));
}
}

c# webclient not timing out

Im trying to download files using extended WebClient with set timeout and I have a problem with the timeout (or what I think should cause timeout).
When I start the download with WebClient and receive some data, then disconnect wifi - my program hangs on the download without throwing any exception. How can I fix this?
EDIT: It actually throws exception but way later than it should (5 minutes vs 1 second which i set) - that is what Im trying to fix.
If you find anything else wrong with my code, please let me know too. Thank you for help
This is my extended class
class WebClientWithTimeout : WebClient
{
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest w = base.GetWebRequest(address);
w.Timeout = 1000;
return w;
}
}
This is the download
using (WebClientWithTimeout wct = new WebClientWithTimeout())
{
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
try
{
wct.DownloadFile("https://example.com", file);
}
catch (Exception e)
{
Console.WriteLine("Download: {0} failed with exception:{1} {2}", file, Environment.NewLine, e);
}
}
Try this, you can avoid UI blocking by this. Coming the WiFi when device connects to WiFi the download resumes.
//declare globally
DateTime lastDownloaded = DateTime.Now;
Timer t = new Timer();
WebClient wc = new WebClient();
//declarewherever you initiate download my case button click
private void button1_Click(object sender, EventArgs e)
{
wc.DownloadProgressChanged += Wc_DownloadProgressChanged;
wc.DownloadFileCompleted += Wc_DownloadFileCompleted;
lastDownloaded = DateTime.Now;
t.Interval = 1000;
t.Tick += T_Tick;
wc.DownloadFileAsync(new Uri("https://github.com/google/google-api-dotnet-client/archive/master.zip"), #"C:\Users\chkri\AppData\Local\Temp\master.zip");
}
private void T_Tick(object sender, EventArgs e)
{
if ((DateTime.Now - lastDownloaded).TotalMilliseconds > 1000)
{
wc.CancelAsync();
}
}
private void Wc_DownloadFileCompleted(object sender, AsyncCompletedEventArgs e)
{
if (e.Error != null)
{
lblProgress.Text = e.Error.Message;
}
}
private void Wc_DownloadProgressChanged(object sender, DownloadProgressChangedEventArgs e)
{
lastDownloaded = DateTime.Now;
lblProgress.Text = e.BytesReceived + "/" + e.TotalBytesToReceive;
}

WebBrowser.Document is null on return from thread

I'm currently trying to do some webscraping using this code:
public static User registerUser()
{
User toreturn = new User();
string csrf;
WebBrowser webcontrol = new WebBrowser();
webcontrol.AllowNavigation = true;
webcontrol.ScriptErrorsSuppressed = true;
webcontrol.Navigate("https://example.com/signup");
webcontrol.DocumentCompleted += new WebBrowserDocumentCompletedEventHandler(webcontrol_DocumentCompleted);
HtmlElementCollection forms = webcontrol.Document.GetElementById("csrf_token").GetElementsByTagName("value");
string tosend = forms[0].InnerText;
toreturn.apikey = tosend;
return toreturn;
}
private static void webcontrol_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
}
However, the entire variable webcontrol.Document is null, both before and after the event.
Any idea as to why this is? Hard to scrape when blind.
EDIT: Worth noting, obviously I need to use the values I get
OK
So now I have got it to load thanks to help below, but I can't get the threads to return the value..
public static User registerUser()
{
Uri test = new Uri("https://www.example.com/signup");
HtmlDocument testdoc = runBrowserThread(test);
string tosend = "test";
User user = new User();
user.apikey = tosend;
return user;
}
public static HtmlDocument runBrowserThread(Uri url)
{
HtmlDocument value = null;
var th = new Thread(() =>
{
var br = new WebBrowser();
br.DocumentCompleted += browser_DocumentCompleted;
br.Navigate(url);
value = br.Document;
Application.Run();
});
th.SetApartmentState(ApartmentState.STA);
th.Start();
th.Join(8000);
return value;
}
static void browser_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
var br = sender as WebBrowser;
if (br.Url == e.Url)
{
Console.WriteLine("Natigated to {0}", e.Url);
Console.WriteLine(br.Document.Body.InnerHtml);
System.Console.ReadLine();
Application.ExitThread(); // Stops the thread
}
}
The System.Console.WriteLine works - I see the HTML! Joy! (Although its cloudflare, but I should get whitelisted)
But the thread returns null..

Trouble with WebClient.DownloadDataAsync(). Not downloading the data?

public string[] SearchForMovie(string SearchParameter)
{
WebClientX.DownloadDataCompleted += new
DownloadDataCompletedEventHandler(WebClientX_DownloadDataCompleted);
WebClientX.DownloadDataAsync(new Uri(
"http://www.imdb.com/find?s=all&q=ironman+&x=0&y=0"));
string sitesearchSource = Encoding.ASCII.GetString(Buffer);
}
void WebClientX_DownloadDataCompleted(object sender,
DownloadDataCompletedEventArgs e)
{
Buffer = e.Result;
throw new NotImplementedException();
}
I get this exception:
The matrix cannot be null. Refering to my byte[] variable Buffer.
So, I can conclude that the DownloadDataAsync isn't really downloading anything. What is causing this problem?
PS. How can I easily format my code so it appear properly indented here. Why can't I just copy past the code from Visual C# express and maintain the indentation here? Thanks! :D
The key word here is "async"; when you call DownloadDataAsync, it only starts the download; it isn't complete yet. You need to process the data in the callback (WebClientX_DownloadDataCompleted).
public string[] SearchForMovie(string SearchParameter)
{
WebClientX.DownloadDataCompleted += WebClientX_DownloadDataCompleted;
WebClientX.DownloadDataAsync(new Uri(uri));
}
void WebClientX_DownloadDataCompleted(object sender,
DownloadDataCompletedEventArgs e)
{
Buffer = e.Result;
string sitesearchSource = Encoding.ASCII.GetString(Buffer);
}
Also - don't assume ASCII; WebClientX.Encoding would be better; or just DownloadStringAsync:
static void Main()
{
var client = new WebClient();
client.DownloadStringCompleted += DownloadStringCompleted;
client.DownloadStringAsync(new Uri("http://google.com"));
Console.ReadLine();
}
static void DownloadStringCompleted(object sender,
DownloadStringCompletedEventArgs e)
{
if (e.Error == null && !e.Cancelled)
{
Console.WriteLine(e.Result);
}
}

Categories

Resources