Develop Reference

Problem with multithreading and File options c#

my goal is to create a output file which parse all url in result.
My problem :
I use Paralelism and he can't be associated with File because it's secured,
i try to put a lock, and i think i don't put it correctly
message error is :
The process cannot access because it is being used by another process the file
What i want :
result is saved in a txt file
Code :
//VARIABLE
List<string> dorklist = File.ReadAllLines("dorks.txt").ToList();
int numdork = File.ReadAllLines("dorks.txt").Length;
int totalurls = File.ReadAllLines("Queue.txt").Length;
int chekeddork = 0;
int errors = 0;
File.Create("Queue.txt");
//TITLE
Task.Factory.StartNew(() =>
{
while (true) {
Console.Title = ("PARSER | " + chekeddork + "/" + numdork + " URLS : " + totalurls);
Thread.Sleep(100);
}
});
//BOUCLE
Parallel.ForEach(dorklist, new ParallelOptions { MaxDegreeOfParallelism = 100 }, dorklist => {
try
{
using (HttpRequest req = new HttpRequest())
{
//HEADERS
req.AddHeader(HttpHeader.UserAgent, "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:65.0) Gecko/20100101 Firefox/65.0");
//REQUESTS
var content = req.Get("https://www.google.com/search?q="+dorklist);
if (Convert.ToString(content.StatusCode) == "OK")
{
chekeddork++;
Console.WriteLine("Dork used : " + dorklist);
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(content.ToString());
var links = doc.DocumentNode.SelectNodes("//div[#class='r']//a");
foreach (var link in links)
{
var href = link.Attributes["href"];
var result = href.Value.ToString();
using (StreamWriter streamaze = new StreamWriter("Queue.txt", true))
{
streamaze.WriteLine(results);
streamaze.Close();
}
}
}
else
{ Console.WriteLine("Bad");
errors++;
}
}
}
catch (HttpException)
{
Console.WriteLine("Banned");
errors++;
}
});
}
}
}
AND the problem is here : `
using (StreamWriter streamaze = new StreamWriter("Queue.txt", true))
{
streamaze.WriteLine(results);
streamaze.Close();
Because there is Parallel.ForEach and the maxdegree
Thanks to all help me

you cannot be writing to one file from several threads at once via separate opens.
Open separate files per thread and merge afterwards.
Or open the file once before the parallel foreach and , under lock, write to the one open file.
Or put that using inside a lock
lock(dorkList) // or create a lock object
{
using (StreamWriter streamaze = new StreamWriter("Queue.txt", true))
{
streamaze.WriteLine(results);
streamaze.Close();
}
}
or use a specifc lock objecct like this
int chekeddork = 0;
int errors = 0;
File.Create("Queue.txt");
object locker = new Object();
....
lock(locker) // or create a lock object
{
using (StreamWriter streamaze = new StreamWriter("Queue.txt", true))
{
streamaze.WriteLine(results);
streamaze.Close();
}
}

How to filter SOAP results client side in C#?

I make SOAP requests to an e-commerce API to know how many orders were made for a specified product, in a given time frame.
My method takes a product ID and a number of days to create a time frame. Then it constructs a SOAP request and returns the amount sold products.
public static async Task<int?> GetOrdersFromApi(int providedId, int days) {
var dateFrom = DateTime.Now.AddDays(days * -1).ToString("yyyy-MM-dd") + " 00:00:00";
var dateTo = DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss");
float orderedCount = 0;
int orderedCountToPass = 0;
int i = 0;
var binding = new BasicHttpBinding {
MaxReceivedMessageSize = 40000000
};
var address = new EndpointAddress("http://example.com/api/get/");
using (var client = new ApiOrdersPortTypeClient(binding, address)) {
try {
while (true) {
// request parameters
var request = new ApiOrdersGetAsync.requestType {
authenticate = new ApiOrdersGetAsync.authenticateType {
userLogin = "Username",
authenticateKey = "Key",
}
};
request.#params = new ApiOrdersGetAsync.paramsType {
ordersStatuses = new string[] { "finished" },
products = new productType[1],
};
request.#params.products[0] = new productType {
productIdSpecified = true,
productId = providedId,
};
request.#params.ordersRange = new ordersRangeType {
ordersDateRange = new ordersDateRangeType()
};
request.#params.ordersRange.ordersDateRange.ordersDateTypeSpecified = true;
request.#params.ordersRange.ordersDateRange.ordersDateType = ordersDateTypeType.dispatch;
request.#params.ordersRange.ordersDateRange.ordersDateBegin = dateFrom;
request.#params.ordersRange.ordersDateRange.ordersDateEnd = dateTo;
request.#params.resultsPageSpecified = true;
request.#params.resultsPage = i;
// processing the result
var results = await client.getAsync(request);
foreach (var result in results.Results) {
int productsResultsPage = 0;
foreach (var product in result.orderDetails.productsResults) {
try {
if (result.orderDetails.productsResults[productsResultsPage ].productId == providedId) {
orderedCount += result.orderDetails.productsResults[y].productQuantity;
productsResultsPage++;
}
} catch (IndexOutOfRangeException ex) {
// error is thrown to escape loop - sloppy, I know
Console.WriteLine(ex);
};
}
};
orderedCountToPass = (int)orderedCount;
orderedCount = 0;
i++;
};
} catch (NullReferenceException) {
// do nothing, we just want to exit while loop when i is higher than page count
}
return orderedCountToPass;
};
}
The result often should be in hundreds, but regardless how well a product sells, it returns something from 0 to 4.
Here is a sample response:
For example, I'm only interested with productId == 479, but an order was made with other products as well that don't interest me - I need to filter them.
I'm doing something wrong with how I try to filter results. How do I do it properly? I'm certain the request is correct and response does contain all possible orders.

You are getting xml results so use Net Xml library :
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Xml;
using System.Xml.Linq;
namespace ConsoleApplication1
{
class Program
{
const string URL = #"Enter URL Here";
static void Main(string[] args)
{
XDocument doc = XDocument.Load(URL);
List<XElement> items = doc.Descendants("item").ToList();
}
}
}

Named Pipes unexpected result

Have a simple named pipes server:
static void Main(string[] args)
{
StartServer();
Console.Read();
}
static void StartServer()
{
Task.Factory.StartNew(() =>
{
var server = new NamedPipeServerStream("TestPipes");
server.WaitForConnection();
StreamReader reader = new StreamReader(server);
StreamWriter writer = new StreamWriter(server);
while (true)
{
var line = reader.ReadLine();
if (line == "Y")
{
for (int i = 0; i < 5; i++)
writer.WriteLine(i.ToString());
writer.Flush();
}
if (line=="N")
{
for (int i = 10; i < 15; i++)
writer.WriteLine(i.ToString());
writer.Flush();
}
}
});
}
and very simple client:
static void Main(string[] args)
{
//Client
var client = new NamedPipeClientStream(Environment.MachineName, "TestPipes");
client.Connect();
Console.WriteLine($"Connection esteblished at {DateTime.Now}, you may continue");
StreamReader reader = new StreamReader(client);
StreamWriter writer = new StreamWriter(client);
while (true)
{
string input = Console.ReadLine();
if (String.IsNullOrEmpty(input)) continue;
writer.WriteLine(input);
writer.Flush();
string serverString;
while (reader.Peek() >= 0)
{
serverString = reader.ReadLine();
Console.WriteLine(serverString);
}
}
}
but for some reason only first command is being completed.
for example if I enter 'Y' getting output 'Y' and then when 'N' is entered nothing comes from the server.
Need to make it work continuously.
Thank you.

That's because in your client you're defining StreamReader reader = new StreamReader(client); outside of the while loop, so on the first iteration when the reader hits the last line, the underlying stream never gets reset, so reader.Peek() >= 0 yields false for subsequent calls.
Move the declaration of the client reader object inside the while loop:
var client = new NamedPipeClientStream(Environment.MachineName, "TestPipes");
client.Connect();
Console.WriteLine($"Connection esteblished at {DateTime.Now}, you may continue");
StreamWriter writer = new StreamWriter(client);
while (true)
{
StreamReader reader = new StreamReader(client);
string input = Console.ReadLine();
if (String.IsNullOrEmpty(input)) continue;
writer.WriteLine(input);
writer.Flush();
string serverString;
while (reader.Peek() >= 0)
{
serverString = reader.ReadLine();
Console.WriteLine(serverString);
}
}

Using Peek is very rarely good idea, and it's not a good idea here either. For example, try to simulate server delay like this:
var line = reader.ReadLine();
if (line == "Y") {
for (int i = 0; i < 5; i++) {
writer.WriteLine(i.ToString());
writer.Flush();
// delay
Thread.Sleep(100);
}
}
And you will see that your code with Peek (including accepted answer) will fail and just read one line (and on subsequent inputs, like "N", will again display nothing, just like it does now). So your code with Peek is not reliable and will surprisingly fail at the most inappropriate moment.
Instead, let server explicitly mark end of data it sends. For example, with empty line:
if (line == "Y") {
for (int i = 0; i < 5; i++) {
writer.WriteLine(i.ToString());
writer.Flush();
// simulate delay
Thread.Sleep(100);
}
// empty line
writer.WriteLine();
writer.Flush();
}
And on client:
string serverString;
while (true)
{
serverString = reader.ReadLine();
if (!String.IsNullOrWhiteSpace(serverString))
Console.WriteLine(serverString);
else break;
};
This code will work reliably
Moving StreamReader to inner while loop as other answer suggests is also not needed, it looks like it "fixes" your code, but really it doesn't fix anything and just puts your problem under the carpet.

updating data via sockets in c#

i've got a server with all the logic of a four on row game of which i would like to display the state after each player turn i've got several class and i would like to send the game with the position played each turn,i succeded to send the first move to the client but the client just shoot down after that. the client and the server are both console application
in my main server side i've got this
var game = Game.CreateConsoleGame(DifficultyLevel.Hard, ActivePlayer.User, false);
game.Play();
Console.ReadKey();
those line are going to use other class Board,Game,Iodevice,judge and player
making all the logic of the game
in class player i've got this
and once i've got the position played ,i draw a String Builder
with this method
public static StringBuilder DrawStringBuilder(int nk)
{
cells = new CellStates[6, 7];
var builder = new StringBuilder();
var header = " 0 1 2 3 4 5 6";
var divisor = "-------------------------------------------------------------------";
builder.AppendLine(header);
builder.AppendLine(divisor);
int p = 0;
for (int i = 0; i < cells.GetLength(0); i++)
{
for (int j = 0; j < cells.GetLength(1); j++)
{
int counter1 = 0;
int counter2 = 1;
cells[5, nk] = CellStates.Player;
//cells[5, 0] = CellStates.User;
var str = cells[i, j] == CellStates.Empty ? "| ······· " : (cells[i, j] == CellStates.User ? "| 0 " : "| x ");
builder.Append(str);
counter1 = counter1 + 2;
counter2++;
}
builder.Append('|');
builder.AppendLine();
builder.AppendLine(divisor);
}
return builder;
}
i am just using this method
to send it to the client
public static void SendingPacket(StringBuilder packet)
{
IPAddress ipLocal;
IPEndPoint ipeLocal;
TcpListener List;
TcpClient tc = null;
NetworkStream ns = null;
string s2;
int n = 0;
try
{
ipLocal = IPAddress.Parse("127.0.0.1");
ipeLocal = new IPEndPoint(ipLocal, 8888);
List = new TcpListener(ipeLocal);
List.Start();
tc = List.AcceptTcpClient();
ns = tc.GetStream();
StreamReader reader = new StreamReader(ns);
StreamWriter write = new StreamWriter(ns);
string response = null;
//Console.WriteLine(response);
response = reader.ReadLine();
Console.WriteLine(response);
Thread.Sleep(200);
write.WriteLine(packet);
write.Flush();
n++;
Console.WriteLine("j'envoie");
Console.WriteLine("de player");
Console.ReadKey();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
if ((tc != null) && (ns != null))
{
tc.Close();
ns.Close();
}
}
}
for the client side i've got receveingdata method that i call in my main
why i can't receive at least twice my stringbuilder sent i'am getting an error
atthe second turn saying unable to read data from the transport connection an existing connection was closed by the distant host
public static void RecevingData()
{
IPAddress ipServeur;
IPEndPoint ipeServeur;
TcpClient tc = null;
NetworkStream ns = null;
try
{
ipServeur = IPAddress.Parse("127.0.0.1");
ipeServeur = new IPEndPoint(ipServeur, 8888);
tc = new TcpClient();
tc.Connect(ipeServeur);
ns = tc.GetStream();
string s = null;
string s1 = "ping";
string reponse = null;
StreamWriter write = new StreamWriter(ns);
StreamReader reader = new StreamReader(ns);
//Console.WriteLine("un string svp");
//s = Console.ReadLine();
write.Flush();
write.WriteLine(s1);
write.Flush();
do
{
reponse = reader.ReadLine();
Console.WriteLine(reponse);
} while (true);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Console.ReadKey();
}
}
i am a beginner so maybe my question is too stupid but i am just driven by getting better so please try to be indulgent,my idea was to send the stringbuilder the same way after each player's turn

Multithreading issue ,Maybe a DeadLock using Foreach

Parallel.ForEach keeps on running and my program does not end. I am unable to trace where it goes after the first iteration. My guess is that gets a deadlock and keeps on doing context switching.
private void ReadInputFile()
{
var collection = new ConcurrentBag<PropertyRecord>();
var lines = System.IO.File.ReadLines(InputFileName);
int i = 0;
int RecordsCount = lines.Count();
Parallel.ForEach(lines, line =>
{
if (string.IsNullOrWhiteSpace(line))
{
return;
}
var tokens = line.Split(',');
var postalCode = tokens[0];
var country = tokens.Length > 1 ? tokens[1] : "england";
SetLabelNotifyTwoText(
string.Format(
"Reading PostCode {0} out of {1}"
i,
lines.Length));
var tempRecord = GetAllAddesses(postalCode, country);
if (tempRecord != null)
{
foreach (PropertyRecord r in tempRecord)
{
collection.Add(r);
}
}
});
}
private List<PropertyRecord> GetAllAddesses(
string postalCode,
string country = "england")
{
SetLabelNotifyText("");
progressBar1.Value = 0;
progressBar1.Update();
var records = new List<PropertyRecord>();
using (WebClient w = new WebClient())
{
var url = CreateUrl(postalCode, country);
var document = w.DownloadString(url);
var pagesCount = GetPagesCount(document);
if (pagesCount == null)
{
return null;
}
for (int i = 0; i < pagesCount; i++)
{
SetLabelNotifyText(
string.Format(
"Reading Page {0} out of {1}",
i,
pagesCount - 1));
url = CreateUrl(postalcode,country, i);
document = w.DownloadString(url);
var collection = Regex.Matches(
document,
"<div class=\"soldDetails\">(.|\\n|\\r)*?class=" +
"\"soldAddress\".*?>(?<address>.*?)(</a>|</div>)" +
"(.|\\n|\\r)*?class=\\\"noBed\\\">(?<noBed>.*?)" +
"</td>|</tbody>");
foreach (var match in collection)
{
var r = new PropertyRecord();
var bedroomCount = match.Groups["noBed"].Value;
if(!string.IsNullOrEmpty(bedroomCount))
{
r.BedroomCount = bedroomCount;
}
else
{
r.BedroomCount = "-1";
}
r.address = match.Groups["address"].Value;
var line = string.Format(
"\"{0}\",{1}",
r.address
r.BedroomCount);
OutputLines.Add(line);
Records.Add(r);
}
}
}
return Records;
}
It runs fine without Parallel.ForEach, but using Parallel.ForEach is in requirements.
I have debugged it and after returning from GetAllAdresses-method first time, Step Next button halts and it just keep on debugging in the background. It doesn't come back on any bookmark I have placed.

As you said in comments, your SetLabelNotifyText and SetLabelNotifyTwoText methods calls Control.Invoke.
For Control.Invoke to work, Main thread has to be free, but in your case you seem to block the main thread by invoking Parallel.ForEach in it.
Here is a minimal reproduction:
private void button1_Click(object sender, EventArgs e)
{
Parallel.ForEach(Enumerable.Range(1, 100), (i) =>
{
Thread.Sleep(10);//Simulate some work
this.Invoke(new Action(() => SetText(i)));
});
}
private void SetText(int i)
{
textBox1.Text = i.ToString();
}
Main thread waits for Parallel.ForEach and worker threads waits for Main thread, and thus results in deadlock.
How to fix: Don't use Invoke simply use BeginInvoke or don't block the MainThread.
If this isn't the case post sscce, that will be helpful for us

Change your code like this, to use async and await. This is the modern alternative to using BeginInvoke and other asynchronous code models.
private async Task ReadInputFile()
{
var collection = new ConcurrentBag<PropertyRecord>();
var lines = System.IO.File.ReadLines(InputFileName);
int i = 0;
int RecordsCount = lines.Count();
Parallel.ForEach(lines, line =>
{
if (string.IsNullOrWhiteSpace(line))
{
return;
}
var tokens = line.Split(',');
var postalCode = tokens[0];
var country = tokens.Length > 1 ? tokens[1] : "england";
SetLabelNotifyTwoText(
string.Format(
"Reading PostCode {0} out of {1}"
i,
lines.Length));
var tempRecord = await GetAllAddesses(postalCode, country);
if (tempRecord != null)
{
foreach (PropertyRecord r in tempRecord)
{
collection.Add(r);
}
}
});
}
private async Task<List<PropertyRecord>> GetAllAddesses(
string postalCode,
string country = "england")
{
SetLabelNotifyText("");
progressBar1.Value = 0;
progressBar1.Update();
var records = new List<PropertyRecord>();
using (WebClient w = new WebClient())
{
var url = CreateUrl(postalCode, country);
var document = await w.DownloadStringTaskAsync(url);
var pagesCount = GetPagesCount(document);
if (pagesCount == null)
{
return null;
}
for (int i = 0; i < pagesCount; i++)
{
SetLabelNotifyText(
string.Format(
"Reading Page {0} out of {1}",
i,
pagesCount - 1));
url = CreateUrl(postalcode,country, i);
document = await w.DownloadStringTaskAsync(url);
var collection = Regex.Matches(
document,
"<div class=\"soldDetails\">(.|\\n|\\r)*?class=" +
"\"soldAddress\".*?>(?<address>.*?)(</a>|</div>)" +
"(.|\\n|\\r)*?class=\\\"noBed\\\">(?<noBed>.*?)" +
"</td>|</tbody>");
foreach (var match in collection)
{
var r = new PropertyRecord();
var bedroomCount = match.Groups["noBed"].Value;
if(!string.IsNullOrEmpty(bedroomCount))
{
r.BedroomCount = bedroomCount;
}
else
{
r.BedroomCount = "-1";
}
r.address = match.Groups["address"].Value;
var line = string.Format(
"\"{0}\",{1}",
r.address
r.BedroomCount);
OutputLines.Add(line);
Records.Add(r);
}
}
}
return Records;
}
Then call it like this
ReadInputFile.Wait();
or, even better, is the caller is async,
await ReadInputFile();