I am looking at the example from the Apache Ignite documentation about the use of distributed closures.
https://apacheignite-net.readme.io/docs/distributed-closures
Based on my understanding it is possible to define the closure in the client node by implementing the IComputeFunc interface. What is not completely clear to me is how the implementation of the interface is being delivered to the server nodes in the cluster. Is there any documentation of this mechanism?
Are there any limitations about the code that the implementation of the closure may contain? E.g. is it allowed to make http requests? Call code from external nuget packages?
I've tried looking at the source code of the project. It seems that the implementation of IComputFunc is being somehow binary serialized using the BinaryWriter.writeobjectdetached method. I didn't manage to get further than that in my investigation.
Below is the example that I was looking at:
async void Compute()
{
using (var ignite = Ignition.Start())
{
var funcs = "Count characters using compute func".Split(' ')
.Select(word => new ComputeFunc { Word = word });
ICollection<int> res = ignite.GetCompute().Call(funcs);
// Async mode
res = await ignite.GetCompute().CallAsync(funcs);
var sum = res.Sum();
Console.WriteLine(">>> Total number of characters in the phrase is '{0}'.", sum);
}
}
[Serializable]
class ComputeFunc : IComputeFunc<int>
{
public string Word { get; set; }
public int Invoke()
{
return Word.Length;
}
}
If this class is present on remote node, it gets used as is, after deserialization.
If this class is absent on remote node, then Peer Assembly Loading will kick in if enabled, it will send required assemblies to remote node.
Since CIL code is pretty well-defined it is a painless process: it might as well stream most of your dependencies to remote nodes on request, including libraries, etc. There is limitation to this mechanism in that it will not Peer Load caches' Key-Value classes. So if you need to operate on those, you need to distribute it between all nodes.
This code will work with same permissions as your generic Ignite code, so yes, there's no reason why it won't make HTTP calls. Note that it might still be better to explicitly add most libraries to all nodes and only Peer Load code that changes often.
Related
There are extension functions for submitting queries asynchronously in Gremlin.Net, some get string which is not recommended and others use RequestMessage which is not as readable as using GraphTraversal functions.
Is there a way to submit a query like the one below asynchronously, without submitting a string or RequestMessage?
var res = _graphTraversalSource.V().Has("name", "Armin").Out().Values<string>("name").ToList();
A little more context
I'm writing an API that queries AWS Neptune. Here's how I get the GraphTraversalSource in the constructor of a singleton service (not sure if I should make the RemoteConnection a singleton and generate a GraphTraversalSource for each query or this is the right approach):
private readonly GraphTraversalSource _graphTraversalSource;
public NeptuneHandler(string endpoint, int port)
{
var gremlinClient = new GremlinClient(new GremlinServer(endpoint, port));
var remoteConnection = new DriverRemoteConnection(gremlinClient);
_graphTraversalSource = AnonymousTraversalSource.Traversal().WithRemote(remoteConnection);
}
You can execute a traversal asynchronously with the Promise() terminator step
var names = await g.V().Has("name", "Armin").Out().Values<string>("name").Promise(t => t.ToList());
Promise() takes a callback as its argument that calls the usual terminator step you want to be executed for your traversal which is ToList in your case. If you however only want to get a single result back, then you can just replace ToList() with Next().
Note that I renamed the variable for the graph traversal source to g as that is the usual naming convention for Gremlin.
As I already mentioned in my comment, it is recommended to reuse this graph traversal source g across your application as it can contain configuration that applies to all traversals you want to execute. It also contains the DriverRemoteConnection which uses a connection pool for the communication with the server. By reusing g, you also use the same connection pool for all traversals in your application.
I just started to learn C# for a school project but I'm stuck on something.
I have a solution with 2 projects (and each project has a class), something like this:
Solution:
Server (project) (...) MyServerClass.cs, Program.cs
App (project) (...) MyAppClass.cs, Program.cs
In my "MyServerClass.cs", I have this:
class MyServerClass
{
...
public void SomeMethod()
{
Process.Start("App.exe", "MyAppClass");
}
}
How can I properly send, for example, an IP address and port? Would something like this work?
class MyServerClass
{
....
public void SomeMethod()
{
string ip = "127.0.0.1";
int port = 8888;
Process.Start("App.exe", "MyAppClass " + ip + " " + port);
}
}
Then in my "MyAppClass.cs", how can I receive that IP address and port?
EDIT:
The objective of this work is to practice processes/threads/sockets. The idea is having a server that receives emails and filter them, to know if they're spam or not. We got to have 4 or 5 filters. The idea was having them as separated projects (ex: Filter1.exe, Filter2.exe, ...), but I was trying to have only 1 project (ex: Filters.exe) and have the filters as classes (Filter1.cs, Filter2.cs, ...), and then create a new process for each different filter.
I guess I'll stick to a project for each filter!
Thanks!
There are a number of ways to achieve this, each with their own pros/cons.
Some possible solutions:
Pass the values in on the command line. Pros: Easy. Cons: Can only be passed in once on launch. Unidirectional (child process can't send info back). Doesn't scale well for complex structured data.
Create a webservice (either in the server or client). Connect to it and either pull/push the appropriate settings. Pros: Flexible, ongoing, potentially bi-directional with some form of polling and works if client/server are on different hosts. Cons: A little bit more complex, requires one app to be able to locate the web address of the other which is trivial locally and more involved over a network.
Use shared memory via a memory mapped file. This approach allows multiple processes to access the same chunk of memory. One process can write the required data and the others can read it. Pros: Efficient, bi-directional, can be disk-backed to persist state through restarts. Cons: Requires pointers and an understanding of how they work. Requires a little more manipulation of data to perform a read/write.
There are dozens more ways. Without knowing your situation in detail, it's hard to recommend one over another.
Edit Re: Updated requirements
Ok, command line is definitely a good choice here. A quick detour into some architecture...
There's no reason you can't do this with a single project.
First up, use an interface to make sure all your filters are interchangeable. Something like this...
public interface IFilter {
FilterResult Filter(string email);
void SetConfig(string config);
}
SetConfig() is optional but potentially useful to reconfigure a filter without a recompile.
You also need to decide what your IFilter's FilterResult is going to be. Is it a pass/fail? Or a score? Maybe some flags and other metrics.
If you wanted to do multiple projects, you'd put that interface in a "shared" or "common" project on its own and reference it from every other project. This also makes it easy for third parties to develop a filter.
Anyway, next up. Let's look at how the filter is hosted. You want something that's going to listen on the network but that's not the responsibility of the filter itself, so we need a network client. What you use here is up to you. WCF in one flavour or another seems to be a prime candidate. Your network client class should take in its constructor a network port to listen on and an instance of the filter...
public class NetworkClient {
private string endpoint;
private IFilter filter;
public NetworkClient(string Endpoint, IFilter Filter) {
this.filter = Filter;
this.endpoint = Endpoint;
this.Setup();
}
void Setup() {
// Set up your network client to listen on endpoint.
// When it receives a message, pass it to filter.Filter(msg);
}
}
Finally, we need an application to host everything. It's up to you whether you go for a console app or winforms/wpf. Depends if you want the process to have a GUI. If it's running as a service, the UI won't be visible on a user desktop anyway.
So, we'll have a process that takes the endpoint for the NetworkClient to listen on, a class name for the filter to use, and (optionally) a configuration string to be passed in to the filter before first use.
So, in your app's Main(), do something like this...
static void Main() {
try {
const string usage = "Usage: Filter.exe Endpoint FilterType [Config]";
var args = Environment.GetCommandLineArgs();
Type filterType;
IFilter filter;
string endpoint;
string config = null;
NetworkClient networkClient;
switch (args.Length) {
case 0:
throw new InvalidOperationException(String.Format("{0}. An endpoint and filter type are required", usage));
case 1:
throw new InvalidOperationException(String.Format("{0}. A filter type is required", usage));
case 2:
// We've been given an endpoint and type
break;
case 3:
// We've been given an endpoint, type and config.
config = args[3];
break;
default:
throw new InvalidOperationException(String.Format("{0}. Max three parameters supported. If your config contains spaces, ensure you are quoting/escaping as required.", usage));
}
endpoint = args[1];
filterType = Type.GetType(args[2]); //Look at the overloads here to control where you're searching
// Now actually create an instance of the filter
filter = (IFilter)Activator.CreateInstance(filterType);
if (config != null) {
// If required, set config
filter.SetConfig(config);
}
// Make a new NetworkClient and tell it where to listen and what to host.
networkClient = new NetworkClient(endpoint, filter);
// In a console, loop here until shutdown is requested, however you've implemented that.
// In winforms, the main UI loop will keep you alive.
} catch (Exception e) {
Console.WriteLine(e.ToString()); // Or display a dialog
}
}
You should then be able to invoke your process like this...
Filter.exe "127.0.0.1:8000" MyNamespace.MyFilterClass
or
Filter.exe "127.0.0.1:8000" MyNamespace.MyFilterClass "dictionary=en-gb;cutoff=0.5"
Of course, you can use a helper class to convert the config string into something your filter can use (like a dictionary).
When the network client gets a FilterResult back from the filter, it can pass the data back to the server / act accordingly.
I'd also suggest a little reading on Dependency Injection / Inversion of control and Unity. It makes a pluggable architecture much, much simpler. Instead of instantiating everything manually and tracking concrete instances, you can just do something like...
container.Resolve<IFilter>(filterType);
And the container will make sure that you get the appropriate instance for your thread/context.
Hope that helps
I have a DLL with some classes and methods. And two applications using it.
One admin-application that needs almost every method and a client-application that only needs parts of the stuff. But big parts of it are used by both of them. Now I want make a DLL with the admin stuff and one with the client stuff.
Duplicating the DLL and edit things manually everytime is horrible.
Maybe conditional compiling helps me but I dont know how to compile the DLL twice with different coditions in one solution with the three projects.
Is there a better approach for this issue than having two different DLLs and manually editing on every change?
In general, you probably don't want admin code exposed on the client side. Since it's a DLL, that code is just waiting to be exploited, because those methods are, by necessity, public. Not to mention decompiling a .NET DLL is trivial and may expose inner-workings of your admin program you really don't want a non-administrator to see.
The best, though not necessarily the "easiest" thing to do, if you want to minimize code duplication, is to have 3 DLLs:
A common library that contains ONLY functions that BOTH applications use
A library that ONLY the admin application will use (or else compile it straight into the application if nothing else uses those functions at all)
A library that ONLY the client application will use (with same caveat as above)
A project that consists of a server, client, and admin client should likely have 3-4 libraries:
Common library, used by all 3
Client library, used by client and server
Admin library, used by server and admin client
Server library, used only by server (or else compile the methods directly into the application)
Have you considered using dependency injection on the common library, some form of constructor injection to determine the rules that need to be applied during execution.
Here's a very simple example:
public interface IWorkerRule
{
string FormatText(string input);
}
internal class AdminRules : IWorkerRule
{
public string FormatText(string input)
{
return input.Replace("!", "?");
}
}
internal class UserRules : IWorkerRule
{
public string FormatText(string input)
{
return input.Replace("!", ".");
}
}
public class Worker
{
private IWorkerRule Rule { get; set; }
public Worker(IWorkerRule rule)
{
Rule = rule;
}
public string FormatText(string text)
{
//generic shared formatting applied to any consumer
text = text.Replace("#", "*");
//here we apply the injected logic
text = Rule.FormatText(text);
return text;
}
}
class Program
{
//injecting admin functions
static void Main()
{
const string sampleText = "This message is #Important# please do something about it!";
//inject the admin rules.
var worker = new Worker(new AdminRules());
Console.WriteLine(worker.FormatText(sampleText));
//inject the user rules
worker = new Worker(new UserRules());
Console.WriteLine(worker.FormatText(sampleText));
Console.ReadLine();
}
}
When run you'll produce this output.
This message is *Important* please do something about it?
This message is *Important* please do something about it.
I wanted to transfer (and execute) an Action or Func object from a C# client to a C# server application using WCF.
Here's my code:
[ServiceContract]
interface IRemoteExecuteServer
{
[OperationContract]
void Execute(Action action);
}
class RemoteExecuteServer : IRemoteExecuteServer
{
public void Execute(Action action)
{
action();
}
}
Servercode:
class Program
{
static void Main(string[] args)
{
using (ServiceHost host = new ServiceHost(typeof(RemoteExecuteServer), new Uri("net.tcp://localhost:8000")))
{
host.AddServiceEndpoint(typeof(IRemoteExecuteServer), new NetTcpBinding(), "RES");
host.Open();
Console.WriteLine("Server is running!");
Console.WriteLine("Press any key to exit...");
Console.ReadKey(true);
host.Close();
}
}
}
Clientcode:
class Program
{
static void Main(string[] args)
{
IRemoteExecuteServer server = new ChannelFactory<IRemoteExecuteServer>(new NetTcpBinding(), new EndpointAddress("net.tcp://localhost:8000/RES")).CreateChannel();
server.Execute(delegate()
{
Console.WriteLine("Hello server!");
});
}
}
When executing the line "server.Execute" I get a CommunicationException.
Does anyone know how to fix this error?
Thanks for your help!
I would think of two solutions, both being pretty crazy in their nature. Mainly because your request of sending code to server for execution is nothing people do every day (and I think noone ever did something like that).
DLL solution:
Compile your code into separate DLL. Send this DLL as stream to server. Load some class with interface using reflection on server from this DLL. Then you can run code in created class.
Code solution:
Basicaly same as first one, but instead of sending precompiled DLL, you just send your code as string and then use programatic C# compiler to compile and run that code.
But you still cant extract your code from any function. And remember Action is nothing more than delegate (reference) for hard-coded function in your assembly.
I want to be able to use a lambda expression to specify a range of values to return over a wcf service
is a similar question i asked.
I link you too it as eric lippert showed how such a solution would be possible.
However its a decent amount of work.
There may be existing solutions out there, I know not. (There are some pretty nifty run-time byte-code injection/manipulation tools available, so...)
But given sufficient permissions (and level of trust! -- that's a big one) one can compile-on-the-fly and/or send various self-evaluation expressions across the network (this is different than sending byte-code, per-se, although that could theoretically be done if the byte-code or an equivalent can be extracted).
The approaches above wouldn't be sending an Action<...>, which would require byte-code extraction for serialization (including capturing the context, if any), but rather something like Compiling C# on the Fly which you can follow to an interesting discussion and finally find CodeDOM.
On the other hand, LINQ "gets about" the issue of byte-code extraction/de-compilation with the use of Expression trees. Note that it doesn't send over byte-code, but rather uses this method to allow remote execution of expression trees (a common one being an SQL query).
This link http://msdn.microsoft.com/en-us/library/aa772153(VS.85).aspx says:
You can register up to five notification requests on a single LDAP connection. You must have a dedicated thread that waits for the notifications and processes them quickly. When you call the ldap_search_ext function to register a notification request, the function returns a message identifier that identifies that request. You then use the ldap_result function to wait for change notifications. When a change occurs, the server sends you an LDAP message that contains the message identifier for the notification request that generated the notification. This causes the ldap_result function to return with search results that identify the object that changed.
I cannot find a similar behavior looking through the .NET documentation. If anyone knows how to do this in C# I'd be very grateful to know. I'm looking to see when attributes change on all the users in the system so I can perform custom actions depending on what changed.
I've looked through stackoverflow and other sources with no luck.
Thanks.
I'm not sure it does what you need, but have a look at http://dunnry.com/blog/ImplementingChangeNotificationsInNET.aspx
Edit: Added text and code from the article:
There are three ways of figuring out things that have changed in Active Directory (or ADAM). These have been documented for some time over at MSDN in the aptly titled "Overview of Change Tracking Techniques". In summary: Polling for Changes using uSNChanged. This technique checks the 'highestCommittedUSN' value to start and then performs searches for 'uSNChanged' values that are higher subsequently. The 'uSNChanged' attribute is not replicated between domain controllers, so you must go back to the same domain controller each time for consistency. Essentially, you perform a search looking for the highest 'uSNChanged' value + 1 and then read in the results tracking them in any way you wish. Benefits This is the most compatible way. All languages and all versions of .NET support this way since it is a simple search. Disadvantages There is a lot here for the developer to take care of. You get the entire object back, and you must determine what has changed on the object (and if you care about that change). Dealing with deleted objects is a pain. This is a polling technique, so it is only as real-time as how often you query. This can be a good thing depending on the application. Note, intermediate values are not tracked here either. Polling for Changes Using the DirSync Control. This technique uses the ADS_SEARCHPREF_DIRSYNC option in ADSI and the LDAP_SERVER_DIRSYNC_OID control under the covers. Simply make an initial search, store the cookie, and then later search again and send the cookie. It will return only the objects that have changed. Benefits This is an easy model to follow. Both System.DirectoryServices and System.DirectoryServices.Protocols support this option. Filtering can reduce what you need to bother with. As an example, if my initial search is for all users "(objectClass=user)", I can subsequently filter on polling with "(sn=dunn)" and only get back the combination of both filters, instead of having to deal with everything from the intial filter. Windows 2003+ option removes the administrative limitation for using this option (object security). Windows 2003+ option will also give you the ability to return only the incremental values that have changed in large multi-valued attributes. This is a really nice feature. Deals well with deleted objects. Disadvantages This is .NET 2.0+ or later only option. Users of .NET 1.1 will need to use uSNChanged Tracking. Scripting languages cannot use this method. You can only scope the search to a partition. If you want to track only a particular OU or object, you must sort out those results yourself later. Using this with non-Windows 2003 mode domains comes with the restriction that you must have replication get changes permissions (default only admin) to use. This is a polling technique. It does not track intermediate values either. So, if an object you want to track changes between the searches multiple times, you will only get the last change. This can be an advantage depending on the application. Change Notifications in Active Directory. This technique registers a search on a separate thread that will receive notifications when any object changes that matches the filter. You can register up to 5 notifications per async connection. Benefits Instant notification. The other techniques require polling. Because this is a notification, you will get all changes, even the intermediate ones that would have been lost in the other two techniques. Disadvantages Relatively resource intensive. You don't want to do a whole ton of these as it could cause scalability issues with your controller. This only tells you if the object has changed, but it does not tell you what the change was. You need to figure out if the attribute you care about has changed or not. That being said, it is pretty easy to tell if the object has been deleted (easier than uSNChanged polling at least). You can only do this in unmanaged code or with System.DirectoryServices.Protocols. For the most part, I have found that DirSync has fit the bill for me in virtually every situation. I never bothered to try any of the other techniques. However, a reader asked if there was a way to do the change notifications in .NET. I figured it was possible using SDS.P, but had never tried it. Turns out, it is possible and actually not too hard to do. My first thought on writing this was to use the sample code found on MSDN (and referenced from option #3) and simply convert this to System.DirectoryServices.Protocols. This turned out to be a dead end. The way you do it in SDS.P and the way the sample code works are different enough that it is of no help. Here is the solution I came up with:
public class ChangeNotifier : IDisposable
{
LdapConnection _connection;
HashSet<IAsyncResult> _results = new HashSet<IAsyncResult>();
public ChangeNotifier(LdapConnection connection)
{
_connection = connection;
_connection.AutoBind = true;
}
public void Register(string dn, SearchScope scope)
{
SearchRequest request = new SearchRequest(
dn, //root the search here
"(objectClass=*)", //very inclusive
scope, //any scope works
null //we are interested in all attributes
);
//register our search
request.Controls.Add(new DirectoryNotificationControl());
//we will send this async and register our callback
//note how we would like to have partial results
IAsyncResult result = _connection.BeginSendRequest(
request,
TimeSpan.FromDays(1), //set timeout to a day...
PartialResultProcessing.ReturnPartialResultsAndNotifyCallback,
Notify,
request);
//store the hash for disposal later
_results.Add(result);
}
private void Notify(IAsyncResult result)
{
//since our search is long running, we don't want to use EndSendRequest
PartialResultsCollection prc = _connection.GetPartialResults(result);
foreach (SearchResultEntry entry in prc)
{
OnObjectChanged(new ObjectChangedEventArgs(entry));
}
}
private void OnObjectChanged(ObjectChangedEventArgs args)
{
if (ObjectChanged != null)
{
ObjectChanged(this, args);
}
}
public event EventHandler<ObjectChangedEventArgs> ObjectChanged;
#region IDisposable Members
public void Dispose()
{
foreach (var result in _results)
{
//end each async search
_connection.Abort(result);
}
}
#endregion
}
public class ObjectChangedEventArgs : EventArgs
{
public ObjectChangedEventArgs(SearchResultEntry entry)
{
Result = entry;
}
public SearchResultEntry Result { get; set;}
}
It is a relatively simple class that you can use to register searches. The trick is using the GetPartialResults method in the callback method to get only the change that has just occurred. I have also included the very simplified EventArgs class I am using to pass results back. Note, I am not doing anything about threading here and I don't have any error handling (this is just a sample). You can consume this class like so:
static void Main(string[] args)
{
using (LdapConnection connect = CreateConnection("localhost"))
{
using (ChangeNotifier notifier = new ChangeNotifier(connect))
{
//register some objects for notifications (limit 5)
notifier.Register("dc=dunnry,dc=net", SearchScope.OneLevel);
notifier.Register("cn=testuser1,ou=users,dc=dunnry,dc=net", SearchScope.Base);
notifier.ObjectChanged += new EventHandler<ObjectChangedEventArgs>(notifier_ObjectChanged);
Console.WriteLine("Waiting for changes...");
Console.WriteLine();
Console.ReadLine();
}
}
}
static void notifier_ObjectChanged(object sender, ObjectChangedEventArgs e)
{
Console.WriteLine(e.Result.DistinguishedName);
foreach (string attrib in e.Result.Attributes.AttributeNames)
{
foreach (var item in e.Result.Attributes[attrib].GetValues(typeof(string)))
{
Console.WriteLine("\t{0}: {1}", attrib, item);
}
}
Console.WriteLine();
Console.WriteLine("====================");
Console.WriteLine();
}