Develop Reference

MongoDB - Error connecting to 192.xx.xx.xx : Too many open files

I've setup my mongodb (v4.2.8) sharded cluster on ubuntu 18.04 and using c#.net for my application. I have 2 mongoS in my cluster and below is my code to connect to my mongodb shard
public sealed class MongoDBConnection
{
private static MongoClient client1 = null;
public static MongoClient Connection1
{
get
{
if (client1 == null)
{
lock (InstanceLock)
{
if (client1 == null)
{
TimeSpan connectionTime = new TimeSpan(1, 0, 0);
MongoClientSettings settingShard = new MongoClientSettings
{
MaxConnectionPoolSize = 60000,
MinConnectionPoolSize = 30000,
ConnectTimeout = connectionTime,
Server = new MongoServerAddress("xxx.xxx.xx.xx", 27017)
};
client1 = new MongoClient(settingShard);
}
}
}
return client1;
}
}
}
this class contains all settings to my connection and below is my class to perform DB operatoins
public static class MongoDBOperations
{
private static IMongoClient consDB1 = MongoDBConnection.Connection1;
private static IMongoDatabase MyDB1 = consDB1.GetDatabase("DBEmp");
private static IMongoCollection<EmpDetails> Mycollection1= MyDB1.GetCollection<EmpDetails>("tblEmployees");
public static Queue<string> returnEmployeeList(int limit)
{
Queue<string> empQ = new Queue<string>();
try
{
var filter = Builders<EmpDetails>.Filter.Eq("InfoUpdated", 0);
var emp1 = Mycollection1.Find(filter).Limit(limit).ToList();
if (emp1.Count > 0)
{
for (int i = 0; i < emp1.Count; i++)
{
var _Emp = Builders<EmpDetails>.Filter.Eq("EmailAddress", emp1[i].EmailAddress);
var UpdateDef = Builders<EmpDetails>.Update.Set(s => s.InfoUpdated, 1).Set(s => s.InfoUpdatedOn, DateTime.Now.Date);
var result = Mycollection1.UpdateManyAsync(_Emp, UpdateDef);
empQ.Enqueue(emp[i].EmailAddress.ToString());
}
}
else
{
Console.ForegroundColor = ConsoleColor.DarkRed;
Console.WriteLine("No Emp found to update : ");
Console.ForegroundColor = ConsoleColor.White;
}
}
catch (Exception ex)
{
Console.WriteLine("Error in Getting emp from DB: " + ex.Message);
}
return empQ;
}
}
so what I am actually doing is, I am returning a queue from my class and do some stuff on that list. As soon as the records are fetched from database, I update there status "InfoUpdated" to 1 so they are not picked agian.
I am doing this in my multithreaded application and below is the code for that
class Program
{
static Dictionary<string, Queue<string>> Qs = new Dictionary<string, Queue<string>>();
static int elementsInQueue = 100;
static int totalQueues = 200;
static void Main(string[] args)
{
new Program().FillUpQueuesOnStart();
}
void FillUpQueuesOnStart()
{
for (int i = 1; i <= totalQueues; i++)
{
if (!Qs.ContainsKey("q" + i))
Qs.Add("q" + i, new Queue<string>());
Qs["q" + i] = MongoDBOperations.returnEmployeeList(elementsInQueue);
Console.WriteLine("q" + i + " filled on start");
}
}
}
when I run my program, it works fine initially and fills up to 35 to 30 queues, but then gives the following error
Error connecting to xxx.xxx.xx.xxx:27017 :: caused by :: Too many open files
where xxx.xxx.xx.xxx is the IP of one of my shards. I see this error on one of my mongoS' terminal in my cluster. I've searched a lot on this to no help. I even changed the ulimit -n in all of my nodes in my cluster, still to no luck.
Any help would really be greatly appreciated.

Azure Devops - Get release definitions by agent pool ID

I'm trying to find all the builds and releases that are configured to use a specific agent pool, using the .NET Client Libraries.
Assuming agentPoolId, I can get all the build definitions like this:
// _connection is of type VssConnection
using (var buildClient = _connection.GetClient<BuildHttpClient>())
{
List<BuildDefinitionReference> allBuilds = await buildClient.GetDefinitionsAsync(projectName, top: 1000, queryOrder: DefinitionQueryOrder.DefinitionNameAscending);
List<BuildDefinitionReference> builds = allBuilds.Where(x => HasAgentPoolId(x, agentPoolId)).ToList();
}
private bool HasAgentPoolId(BuildDefinitionReference buildDefinition, int agentPoolId)
{
TaskAgentPoolReference pool = buildDefinition?.Queue?.Pool;
if (pool == null)
{
return false;
}
return pool.Id.Equals(agentPoolId);
}
But I couldn't find a way to find the release definitions that have one or more environments configured to use a particular agent. Any suggestion?

I was manged to get all releases by Agent Pool ID via Rest Api and not via NET Client Libraries.Hope that helps.
C# Code snippet:
public class ReleaseResponse
{
[JsonProperty("value")]
public List<ReleaseItem> Value { get; set; }
}
public class ReleaseItem
{
[JsonProperty("name")]
public string Name { get; set; }
[JsonProperty("Id")]
public int Id { get; set; }
}
static void Main(string[] args)
{
string tfsURL = "TFS URL";
string releaseDefurl = $"{tfsURL}/_apis/release/definitions?$expand=artifacts&api-version=3.2-preview.3";
const int agentPoolID = "AGENT Pool ID";
List<string> relevantReleases = new List<string>();
WebClient client = new WebClient();
client.UseDefaultCredentials = true;
client.Headers.Add("Content-Type", "application/json");
var releaseList = client.DownloadString(releaseDefurl);
var allReleases = JsonConvert.DeserializeObject<ReleaseResponse>(releaseList).Value;
foreach (var release in allReleases)
{
string releaseInfoApi = $"{tfsURL}/_apis/Release/definitions/{release.Id}";
var getReleseInfo = client.DownloadString(releaseInfoApi);
var releaseInfo = JsonConvert.DeserializeObject<TFSLogic.RootObject>(getReleseInfo);
var deploymentAgents = releaseInfo.environments.ToList().Where(e => e.deployPhases.FirstOrDefault().deploymentInput.queueId == agentPoolID).Count();
if (deploymentAgents > 0)
{
relevantReleases.Add(release.Name);
}
}
}
Find TFSLogic here : https://codebeautify.org/online-json-editor/cb7aa0d9
Powershell Code snippet:
$tfsUrl = "TFS URL"
$releaseDefurl = $tfsUrl + '/_apis/release/definitions?$expand=artifacts&api-version=3.2-preview.3'
$agentPoolID = "Agent Pool ID"
$relevantReleases = #();
$allReleasesID = (Invoke-RestMethod -Uri ($releaseDefurl) -Method Get -UseDefaultCredentials).value.id
function getReleaseByAgentPoolID($releaseID,$agentPoolID)
{
$ReleaseInfo = Invoke-RestMethod -Uri "$tfsUrl/_apis/Release/definitions/$releaseID" -Method Get -UseDefaultCredentials
$deploymentAgents = $ReleaseInfo.environments | % {$_.deployPhases.deploymentInput.queueId} | where {$_ -eq $agentPoolID}
if($deploymentAgents.Count -gt 0)
{
return $ReleaseInfo.name
}
}
foreach ($releaseID in $allReleasesID)
{
$relevantReleases += getReleaseByAgentPoolID -releaseID $releaseID -agentPoolID $agentPoolID
}
UPDATE :
It took me some time,But i was able to achieve that with azure-devops-dotnet-samples
I hope this example is finally what you are looking for.
using Microsoft.VisualStudio.Services.WebApi;
using System;
using System.Linq;
using Microsoft.VisualStudio.Services.ReleaseManagement.WebApi.Clients;
using Microsoft.VisualStudio.Services.ReleaseManagement.WebApi.Contracts;
using Microsoft.VisualStudio.Services.Common;
using System.Collections.Generic;
namespace FindReleaseByAgentPoolID
{
class Program
{
const int agentPoolID = 999;
static void Main(string[] args)
{
var relevantReleases = new List<string>();
VssCredentials c = new VssCredentials(new WindowsCredential(System.Net.CredentialCache.DefaultNetworkCredentials));
var tfsURL = new Uri("TFS URL");
var teamProjectName = "PROJECT";
using (var connection = new VssConnection(tfsURL, c))
using (var rmClient = connection.GetClient<ReleaseHttpClient2>())
{
var releases = rmClient
.GetReleaseDefinitionsAsync(teamProjectName, string.Empty, ReleaseDefinitionExpands.Environments)
.Result.ToArray();
foreach (var release in releases)
{
var r = rmClient.GetReleaseDefinitionAsync(teamProjectName, release.Id);
var deploymentAgents = r.Result.Environments.SelectMany(e =>
e.DeployPhases.Select(dp =>
dp.GetDeploymentInput()).Cast<DeploymentInput>()).Where(di =>
di.QueueId == agentPoolID).Count();
if (deploymentAgents > 0)
{
relevantReleases.Add(release.Name);
}
}
}
}
}
}

Found a solution, many thanks to #amit-baranes for pointing me in the right direction.
I've changed his code sample to use the await keyword instead of using .Result, and use .OfType<DeploymentInput>() instead of .Cast<DeploymentInput>() (it was throwing some exceptions).
Oh, and the most important thing I've learned: agent pool ID and queue ID are different things!!! If you intend to use the agent pool ID to get the release definitions you'll need to get the correspondent agent queue.
Code sample:
// set agent pool Id and project name
int agentPoolId = 123456;
string teamProjectName = ".....";
// _connection is of type VssConnection
using (var taskAgentClient = _connection.GetClient<TaskAgentHttpClient>())
using (var releaseClient = _connection.GetClient<ReleaseHttpClient2>())
{
// please note: agent pool Id != queue Id
// agent pool id is used to get the build definitions
// queue Id is used to get the release definitions
TaskAgentPool agentPool = await taskAgentClient.GetAgentPoolAsync(agentPoolId);
List<TaskAgentQueue> queues = await taskAgentClient.GetAgentQueuesByNamesAsync(teamProjectName, queueNames: new[] { agentPool.Name });
TaskAgentQueue queue = queues.FirstOrDefault();
List<ReleaseDefinition> definitions = await releaseClient.GetReleaseDefinitionsAsync(teamProjectName, string.Empty, ReleaseDefinitionExpands.Environments);
foreach (ReleaseDefinition definition in definitions)
{
var fullDefinition = await releaseClient.GetReleaseDefinitionAsync(teamProjectName, definition.Id);
bool hasReleasesWithPool = fullDefinition.Environments.SelectMany(GetDeploymentInputs)
.Any(di => di.QueueId == queue.Id);
if (hasReleasesWithPool)
{
Debug.WriteLine($"{definition.Name}");
}
}
}
private IEnumerable<DeploymentInput> GetDeploymentInputs(ReleaseDefinitionEnvironment environment)
{
return environment.DeployPhases.Select(dp => dp.GetDeploymentInput())
.OfType<DeploymentInput>();
}

How can I turn ClusterRouterPool from C# to HOCON configuration?

I'm learning akka.net and potentially will use it to replace part of our traditional message driven app.
Basically I'm trying to have X number of nodes joined to a cluster. It's peer to peer type, and I might run X number of actors (same actor) on a node.
If I have 10 jobs (say SendEmailActor), ideally, I would like each of the 10 jobs be executed on different nodes (spread the load evenly).
I have an extremely simple console app to demonstrate.
using System;
using System.Configuration;
using Akka;
using Akka.Actor;
using Akka.Cluster;
using Akka.Cluster.Routing;
using Akka.Configuration;
using Akka.Configuration.Hocon;
using Akka.Routing;
namespace Console1
{
class MainClass
{
public static void Main(string[] args)
{
Console.Write("Is this the seed node? (Y/n): ");
var port = Console.ReadLine().ToLowerInvariant() == "y" ? 9999 : 0;
var section = (AkkaConfigurationSection)ConfigurationManager.GetSection("akka");
var config =
ConfigurationFactory.ParseString("akka.remote.helios.tcp.port=" + port)
.WithFallback(section.AkkaConfig);
var cluster = ActorSystem.Create("MyCluster", config);
var worker = cluster.ActorOf(Props.Create<Worker>().WithRouter(
new ClusterRouterPool(
new RoundRobinPool(10),
new ClusterRouterPoolSettings(30, true, 5))), "worker");
while (true)
{
Console.Read();
var i = DateTime.Now.Millisecond;
Console.WriteLine("Announce: {0}", i);
worker.Tell(i);
}
}
}
public class Worker : UntypedActor
{
protected override void OnReceive(object message)
{
System.Threading.Thread.Sleep(new Random().Next(1000, 2000));
Console.WriteLine("WORKER ({0}) [{1}:{2}]", message, Context.Self.Path, Cluster.Get(Context.System).SelfUniqueAddress.Address.Port);
}
}
}
And my app.config looks like
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<configSections>
<section name="akka" type="Akka.Configuration.Hocon.AkkaConfigurationSection, Akka" />
</configSections>
<akka>
<hocon>
<![CDATA[
akka {
actor {
provider = "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
}
remote {
helios.tcp {
hostname = "127.0.0.1"
port = 0
}
}
cluster {
seed-nodes = ["akka.tcp://MyCluster#127.0.0.1:9999"]
}
}
]]>
</hocon>
</akka>
</configuration>
I want to use HOCON and setup akka.actor.deployment, and I couldn't get it to work. I don't quite understand the routees.paths, and its relationship with the the actor.deployment/worker and how will routees.paths map to the actors created in C#.
actor {
provider = "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
deployment {
/worker {
router = roundrobin-pool
routees.paths = ["/worker"] # what's this?
cluster {
enabled = on
max-nr-of-instances-per-node = 1
allow-local-routees = on
}
}
}
}
Another question: with aka.net.cluster is it possible to "mirror" the nodes to provide redundancy? Or GuaranteedDeliveryActor (I think it's renamed to AtLeastOnceDelivery) is the way to go?

Thanks to #RogerAlsing.
The C# line looks like this
var worker = cluster.ActorOf(Props.Create(() => new Worker()).WithRouter(FromConfig.Instance), "worker");
and the config looks like
akka {
actor {
provider = "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
deployment {
/worker {
router = round-robin-pool #broadcast-pool also works
nr-of-instances = 10
cluster {
enabled = on
max-nr-of-instances-per-node = 2
allow-local-routees = on
}
}
}
}
remote {
helios.tcp {
hostname = "127.0.0.1"
port = 0
}
}
cluster {
seed-nodes = ["akka.tcp://MyCluster#127.0.0.1:9999"]
}
}

Retrieve process network usage

How can I get a process send/receive bytes? the preferred way is doing it with C#.
I've searched this a lot and I didn't find any simple solution for this. Some solutions suggested to install the WinPCap on the machine and to work with this lib.
Like this guy asked: Need "Processes with Network Activity" functionality in managed code - Like resmon.exe does it
I don't want the overhead of the lib.
Is there a simple solution for this?
Actually I want the exactly data that the Resource Monitor of Windows gives under the "Processes with Network Activity" tab:
How does the Resource Monitor of Windows gets this information?
Any example?
Also, tried to use the counter method which is mentioned over here:
Missing network sent/received
but with no success - as not every process is shown under this counter.
And again I'm wondering how the Resource Monitor gets this information even without using this counter...

Resource monitor uses ETW - thankfully, Microsoft have created a nice nuget .net wrapper to make it easier to use.
I wrote something like this recently to report back my process's network IO:
using System;
using System.Diagnostics;
using System.Threading.Tasks;
using Microsoft.Diagnostics.Tracing.Parsers;
using Microsoft.Diagnostics.Tracing.Session;
namespace ProcessMonitoring
{
public sealed class NetworkPerformanceReporter : IDisposable
{
private DateTime m_EtwStartTime;
private TraceEventSession m_EtwSession;
private readonly Counters m_Counters = new Counters();
private class Counters
{
public long Received;
public long Sent;
}
private NetworkPerformanceReporter() { }
public static NetworkPerformanceReporter Create()
{
var networkPerformancePresenter = new NetworkPerformanceReporter();
networkPerformancePresenter.Initialise();
return networkPerformancePresenter;
}
private void Initialise()
{
// Note that the ETW class blocks processing messages, so should be run on a different thread if you want the application to remain responsive.
Task.Run(() => StartEtwSession());
}
private void StartEtwSession()
{
try
{
var processId = Process.GetCurrentProcess().Id;
ResetCounters();
using (m_EtwSession = new TraceEventSession("MyKernelAndClrEventsSession"))
{
m_EtwSession.EnableKernelProvider(KernelTraceEventParser.Keywords.NetworkTCPIP);
m_EtwSession.Source.Kernel.TcpIpRecv += data =>
{
if (data.ProcessID == processId)
{
lock (m_Counters)
{
m_Counters.Received += data.size;
}
}
};
m_EtwSession.Source.Kernel.TcpIpSend += data =>
{
if (data.ProcessID == processId)
{
lock (m_Counters)
{
m_Counters.Sent += data.size;
}
}
};
m_EtwSession.Source.Process();
}
}
catch
{
ResetCounters(); // Stop reporting figures
// Probably should log the exception
}
}
public NetworkPerformanceData GetNetworkPerformanceData()
{
var timeDifferenceInSeconds = (DateTime.Now - m_EtwStartTime).TotalSeconds;
NetworkPerformanceData networkData;
lock (m_Counters)
{
networkData = new NetworkPerformanceData
{
BytesReceived = Convert.ToInt64(m_Counters.Received / timeDifferenceInSeconds),
BytesSent = Convert.ToInt64(m_Counters.Sent / timeDifferenceInSeconds)
};
}
// Reset the counters to get a fresh reading for next time this is called.
ResetCounters();
return networkData;
}
private void ResetCounters()
{
lock (m_Counters)
{
m_Counters.Sent = 0;
m_Counters.Received = 0;
}
m_EtwStartTime = DateTime.Now;
}
public void Dispose()
{
m_EtwSession?.Dispose();
}
}
public sealed class NetworkPerformanceData
{
public long BytesReceived { get; set; }
public long BytesSent { get; set; }
}
}

You can use PerformanceCounter. Sample code:
//Define
string pn = "MyProcessName.exe";
var readOpSec = new PerformanceCounter("Process","IO Read Operations/sec", pn);
var writeOpSec = new PerformanceCounter("Process","IO Write Operations/sec", pn);
var dataOpSec = new PerformanceCounter("Process","IO Data Operations/sec", pn);
var readBytesSec = new PerformanceCounter("Process","IO Read Bytes/sec", pn);
var writeByteSec = new PerformanceCounter("Process","IO Write Bytes/sec", pn);
var dataBytesSec = new PerformanceCounter("Process","IO Data Bytes/sec", pn);
var counters = new List<PerformanceCounter>
{
readOpSec,
writeOpSec,
dataOpSec,
readBytesSec,
writeByteSec,
dataBytesSec
};
// get current value
foreach (PerformanceCounter counter in counters)
{
float rawValue = counter.NextValue();
// display the value
}
And this is to get performance counters for the Network card. Note it is not process specific
string cn = "get connection string from WMI";
var networkBytesSent = new PerformanceCounter("Network Interface", "Bytes Sent/sec", cn);
var networkBytesReceived = new PerformanceCounter("Network Interface", "Bytes Received/sec", cn);
var networkBytesTotal = new PerformanceCounter("Network Interface", "Bytes Total/sec", cn);
Counters.Add(networkBytesSent);
Counters.Add(networkBytesReceived);
Counters.Add(networkBytesTotal);

Have a look at the IP Helper API. There is an implementation in C# by Simon Mourier that sums transferred bytes per process: https://stackoverflow.com/a/25650933/385513
It would be interesting to know how this compares with Event Tracing for Windows (ETW)...

WCF With NetTCP across machines on the same network

I'm trying to implement some cross process communication that is between multiple computers and one server on the same network. What I'm trying right now is to use WCF with NetTcpBinding, hosted within my application which works on the same machine, but when I try to connect from another machine it throws a SSPI security error.
I've found lots of examples of doing this cross-machine, but all involve an app.config file which I would REALLY like to avoid. I want to be able to embed this functionality in a DLL that has not other dependencies (i.e. config files) for which I can just pass into it all of the necessary server addresses, etc and it will work. Is there anyway to setup this security (via the endpoints, etc) purely in code?
I'm testing this all out with the code below:
SERVER:
using System;
using System.ServiceModel;
namespace WCFServer
{
[ServiceContract]
public interface IStringReverser
{
[OperationContract]
string ReverseString(string value);
}
public class StringReverser : IStringReverser
{
public string ReverseString(string value)
{
char[] retVal = value.ToCharArray();
int idx = 0;
for (int i = value.Length - 1; i >= 0; i--)
retVal[idx++] = value[i];
string result = new string(retVal);
Console.WriteLine(value + " -> " + result);
return result;
}
}
class Program
{
static void Main(string[] args)
{
var uri = "net.tcp://" + System.Net.Dns.GetHostName() + ":9985";
Console.WriteLine("Opening connection on: " + uri);
using (ServiceHost host = new ServiceHost(
typeof(StringReverser),
new Uri[]{
new Uri("net.tcp://" + System.Net.Dns.GetHostName() + ":9985")
}))
{
host.AddServiceEndpoint(typeof(IStringReverser),
new NetTcpBinding(),
"TcpReverse");
host.Open();
Console.WriteLine("Service is available. " +
"Press <ENTER> to exit.");
Console.ReadLine();
host.Close();
}
}
}
}
CLIENT:
using System;
using System.ServiceModel;
using System.ServiceModel.Channels;
namespace WCFClient
{
[ServiceContract]
public interface IStringReverser
{
[OperationContract]
string ReverseString(string value);
}
class Program
{
static void Main(string[] args)
{
var ep = "net.tcp://SERVER:9985/TcpReverse";
ChannelFactory<IStringReverser> pipeFactory =
new ChannelFactory<IStringReverser>(
new NetTcpBinding(),
new EndpointAddress(
ep));
IStringReverser pipeProxy =
pipeFactory.CreateChannel();
Console.WriteLine("Connected to: " + ep);
while (true)
{
string str = Console.ReadLine();
Console.WriteLine("pipe: " +
pipeProxy.ReverseString(str));
}
}
}
}

Security is normally configured on the binding. You are using NetTcpBinding with its defaults which means that Transport security is enabled.
On both, server and client, you should assign the NetTcpBinding instance to a local variable so that you can change the security (and possibly other) settings, and then use that variable when calling AddServiceEndpoint or when creating the ChannelFactory.
Sample:
var binding = new NetTcpBinding();
// disable security:
binding.Security.Mode = SecurityMode.None;

This is probably an issue with the SPN that your service is running under. It's most likely a machine account instead of a domain account. There's more information in this thread.
UPDATE: There's information in there about setting the SPN programmatically, but it's buried a few clicks in... here's a direct link (see the last section of the page).