How to create new topic in c# by using kafka-net - c#

I have HomeController as below,
#region Properties
const string topic = "AnotherTestTopic";
const string host = "http://localhost:9092";
#endregion
[HttpPost]
public ActionResult Save(FormCollection form)
{
var kafkaOptions = new KafkaOptions(new Uri(host));
var brokerRouter = new BrokerRouter(kafkaOptions);
var producer = new Producer(brokerRouter);
producer.SendMessageAsync(topic, new[] { new Message("Test message") }).Wait();
return RedirectToAction("Index", "Home");
}
I am using kafka-net dll and my SendMessageAsync method as below
public async Task<List<ProduceResponse>> SendMessageAsync(string topic, IEnumerable<Message> messages, Int16 acks = 1,
TimeSpan? timeout = null, MessageCodec codec = MessageCodec.CodecNone)
{
if (_stopToken.IsCancellationRequested)
throw new ObjectDisposedException("Cannot send new documents as producer is disposing.");
if (timeout == null) timeout = TimeSpan.FromMilliseconds(DefaultAckTimeoutMS);
var batch = messages.Select(message => new TopicMessage
{
Acks = acks,
Codec = codec,
Timeout = timeout.Value,
Topic = topic,
Message = message
}).ToList();
_asyncCollection.AddRange(batch);
await Task.WhenAll(batch.Select(x => x.Tcs.Task));
return batch.Select(topicMessage => topicMessage.Tcs.Task.Result)
.Distinct()
.ToList();
}
Question:
I just started learning kafka.I really do not know how can i create topic from c# code.How can i add topic from c# ?
Any help will be appreciated.
Thanks.

You can set auto.create.topics.enable=true in your broker configuration to let Kafka create the topic for you when it is not previously created.
You may also want to set num.partitions and default.replication.factor to appropriate values in your broker configuration as well.

Related

Error when creating ServiceBus Queue using Azure.Messaging.ServiceBus.Administration

I am (trying) to use this code to create ServiceBus Queue:
using Azure.Messaging.ServiceBus;
using Azure.Messaging.ServiceBus.Administration;
...
class blabla
{
private string connectionString = "Endpoint=sb://XXXX.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=XXXYYY";
private string queueName = "testqueue";
...
public doit()
{
var adminClient = new ServiceBusAdministrationClient(connectionString);
bool queueExists = adminClient.QueueExistsAsync(queueName).Result;
if (!queueExists)
{
var options = new CreateQueueOptions(queueName)
{
DefaultMessageTimeToLive = TimeSpan.FromDays(2),
LockDuration = TimeSpan.FromSeconds(45),
MaxDeliveryCount = 8,
MaxSizeInMegabytes = 2048
};
options.AuthorizationRules.Add(new SharedAccessAuthorizationRule(
"allClaims",
new[] { AccessRights.Manage, AccessRights.Send, AccessRights.Listen }));
QueueProperties createdQueue = adminClient.CreateQueueAsync(options).Result;
}
}
}
but constantly getting this error:
System.AggregateException: One or more errors occurred. (SubCode=40900. Conflict. You're requesting an operation that isn't allowed in the resource's current state. To know more visit https://aka.ms/sbResourceMgrExceptions. . TrackingId:bc79fd98-73c8-4301-b6b9-05d0eae6ed6a_G17, SystemTracker:xxx.servicebus.windows.net:yyy, Timestamp:2021-05-09T00:24:57
Status: 409 (Conflict)
ErrorCode: 40900
Using old (NET) way with NamespaceManager from Microsoft.ServiceBus works with no problems.
var namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);
if (!namespaceManager.QueueExists(queueName))
{
namespaceManager.CreateQueue(queueName);
}
So, does anyone knows what am I doing wrong here?
*
Below is the updated working code, you need to make sure you have shared access policy with full access.
using Azure.Messaging.ServiceBus.Administration;
using System;
using System.Threading.Tasks;
namespace ServiceBusDemo
{
class Program
{
private static string connectionString = "Endpoint=sb://ns-servicebusshu.servicebus.windows.net/;SharedAccessKeyName=fullAccess;SharedAccessKey=oB+IsK8Aqp0/xfXnF9HCz6x9pqPIOysTXaJofSmHEYs=";
private static string queueName = "testqueue";
async static Task Main(string[] args)
{
await doit();
}
public static async Task doit()
{
var adminClient = new ServiceBusAdministrationClient(connectionString);
bool queueExists = await adminClient.QueueExistsAsync(queueName);
if (!queueExists)
{
var options = new CreateQueueOptions(queueName)
{
DefaultMessageTimeToLive = TimeSpan.FromDays(2),
LockDuration = TimeSpan.FromSeconds(45),
MaxDeliveryCount = 8,
MaxSizeInMegabytes = 2048
};
options.AuthorizationRules.Add(new SharedAccessAuthorizationRule("allClaims", new[] { AccessRights.Manage, AccessRights.Send, AccessRights.Listen }));
QueueProperties createdQueue = await adminClient.CreateQueueAsync(options);
}
}
}
}
Once you ran the application its successfully created the queue as below :
Maybe it's not your case... But if you have a TOPIC with the same name that you try to create your new QUEUE, QueueExistsAsync will return false, but you'll be spitted with this bizarre error at creation time. The fix is easy... changing the queue name or deleting the offending topic.
Sorry for the confusion.
My code (and Rahul Shukla as well) is working now (????).
I had to create a few new shared access policies with full access (????).
The third created started working (??).
The previous 2 I created are still not working (????).
There are no differences between the 3 policies created. Hence the question marks in my answer.
Posted question on MS NET SB forum about 1 out of 3 policies working. No answer/acknowledgment so far.

How to connect to a cluster of sentinels using StackExchange.Redis?

Does Stackexchange.Redis (C#) support connecting to a cluster of sentinels for high availability with the latest version ? I'm finding it a bit odd how such an important feature of redis is not properly documented or practically no examples at all. Would appreciate any help.
You can see how to connect to sentinels by examining the tests of StackExchange.Redis. Specifically this one https://github.com/StackExchange/StackExchange.Redis/blob/master/tests/StackExchange.Redis.Tests/Sentinel.cs
Essentially the code you need to look at is here:
public Sentinel(ITestOutputHelper output) : base(output)
{
ConnectionLog = new StringWriter();
Skip.IfNoConfig(nameof(TestConfig.Config.SentinelServer), TestConfig.Current.SentinelServer);
Skip.IfNoConfig(nameof(TestConfig.Config.SentinelSeviceName), TestConfig.Current.SentinelSeviceName);
var options = new ConfigurationOptions()
{
CommandMap = CommandMap.Sentinel,
EndPoints = { { TestConfig.Current.SentinelServer, TestConfig.Current.SentinelPort } },
AllowAdmin = true,
TieBreaker = "",
ServiceName = TestConfig.Current.SentinelSeviceName,
SyncTimeout = 5000
};
Conn = ConnectionMultiplexer.Connect(options, ConnectionLog);
Thread.Sleep(3000);
Assert.True(Conn.IsConnected);
Server = Conn.GetServer(TestConfig.Current.SentinelServer, TestConfig.Current.SentinelPort);
}
And then here:
[Fact]
public void SentinelGetMasterAddressByNameTest()
{
var endpoint = Server.SentinelGetMasterAddressByName(ServiceName);
Assert.NotNull(endpoint);
var ipEndPoint = endpoint as IPEndPoint;
Assert.NotNull(ipEndPoint);
Log("{0}:{1}", ipEndPoint.Address, ipEndPoint.Port);
}

Open new mail interaction window in Genesys Interaction Workspace

I got the task to show the "new outbound mail" dialog in Genesys IWS upon an external event from a webservice. I put my IWS extension in place and it loads and can provide a webservice interface.
My main problem now is that I don't understand how I can open the interactions window from my code. I tried to get an instance of it by using:
IInteractionsWindow interactionsView = Container.Resolve<IInteractionsWindow>();
interactionsView.Create();
interactionsView.ShowView();
This actually works only halfway, as I get a new window, but it's completely empty. Do I need to load every single region on its own? Is there a simpler way to achieve my goals in a fully integrated way?
UPDATE: I have now tried to achieve things using the Platform SDK although I have no idea if this really helps me in showing the "new mail" window to the agent. I tried with the following code:
interactionServerProtocol = new InteractionServerProtocol(new Endpoint(new Uri("tcp://ixnServer:7319")));
interactionServerProtocol.ClientName = "CRMIntegrationModule";
interactionServerProtocol.ClientType = InteractionClient.AgentApplication;
contactServerProtocol = new UniversalContactServerProtocol(new Endpoint(new Uri("tcp://ucsServer:5130")));
contactServerProtocol.ClientName = "CRMIntegrationModule";
interactionServerProtocol.Open();
contactServerProtocol.Open();
RequestSubmit request = RequestSubmit.Create();
request.InteractionType = "Outbound";
request.InteractionSubtype = "OutboundNew";
request.MediaType = "email";
request.Queue = "default";
EventAck response = interactionServerProtocol.Request(request) as EventAck;
if (response != null)
{
string id = Convert.ToString(response.Extension["InteractionId"]);
RequestInsertInteraction insertRequest = RequestInsertInteraction.Create();
insertRequest.InteractionAttributes = new InteractionAttributes
{
Id = id,
MediaTypeId = "email",
TypeId = "Outbound",
SubtypeId = "OutboundNew",
TenantId = 101,
Status = new NullableStatuses(Statuses.Pending),
Subject = "Testmail",
EntityTypeId = new NullableEntityTypes(EntityTypes.EmailOut)
};
insertRequest.EntityAttributes = new EmailOutEntityAttributes()
{
FromAddress = "dummy#gmx.net",
ToAddresses = "dummy#gmx.net",
};
insertRequest.InteractionContent = new InteractionContent()
{
Text = "This is the e-mail body."
};
contactServerProtocol.Send(insertRequest);
RequestPlaceInQueue queueRequest = RequestPlaceInQueue.Create();
queueRequest.InteractionId = id;
queueRequest.Queue = "default";
interactionServerProtocol.Send(queueRequest);
}
interactionServerProtocol.Close();
contactServerProtocol.Close();
The bad thing is the response from the interaction server which is:
attr_ref_id [int] = 2
attr_error_code [int] = 34
attr_error_desc [str] = "Client is not logged in"
I think this could be related to not being logged in correctly somehow but I have not a single clue how to achieve this. Any help?
UPDATE 2 I could send an e-mail using the Platform SDK, but this is not what I really want. The initial question is still valid, as I just want to invoke the interactions window and that's it. The other stuff is up to the user. Is it possible?
You need to use PlatformSDK. add Genesyslab.platform.webmedia.protocols.dll
After that you can use *webmedia.tserver.request, under that tab there is requestWeb or sth.
channelService.RegisterEvents(tServerChannel, new Action<Genesyslab.Enterprise.Model.Channel.IClientChannel>
In your main module(have Initialize method), need to registerevent like that. You can put a button or sth, then you can trigger event or you can use commandchain after logon, is up to you.
Good luck.
I made use of the given command chains:
public IObjectContainer Container { get; set; }
public void NewItem(string contactId, string emailAddress)
{
IAgent agent = Container.Resolve<IAgent>();
IRoutingBasedManager routingManager = Container.Resolve<IRoutingBasedManager>();
IDictionary<string, object> parameters = new Dictionary<string, object>();
parameters.Add("CommandParameter", agent.FirstMediaEmail);
parameters.Add("TargetId", contactId);
parameters.Add("OwnerId", agent.ConfPerson.EmailAddress);
parameters.Add("Destination", emailAddress);
parameters.Add("RecentIndex", contactId);
bool todo = routingManager.RequestToDo("CreateNewOutboundEmail", RoutingBasedTarget.Contact, parameters);
if (todo && parameters.ContainsKey("RoutingBaseCommand"))
{
IChainOfCommand chainOfCommand = parameters["RoutingBaseCommand"] as IChainOfCommand;
if (chainOfCommand != null)
{
chainOfCommand.Execute(parameters["RoutingBaseCommandParameters"]);
}
}
}

redis servicestack client List.Remove(item) does not work

I'm developing a "Task Control System" that will allow its users to enter task description information including when to execute the task and what environment (OS, browser, etc.) the task requires.
The 'controller' saves the description information and schedules the task. When the scheduled time arrives, the scheduler retrieves the task information and 'queues' the task for a remote machine that matches the required environment.
My first cut at this used a relational database to persist the task descriptions and enough history information to track problems (about 2 weeks worth). But this is not a 'big data' problem and the relationships are simple and I need better performance.
So I'm looking for something that offers more performance.
I'm trying to use redis for this, but I'm having some problems. I'm using ServiceStack.Redis version 3.9.71.0 for the client and Redis 2.8.4 is the server.
This sample code is taken from Dan Swain's tutorial. It's updated to work with ServiceStack.Redis client v 3.9.71.0. Much of it works, but 'currentShippers.Remove(lameShipper);' does NOT work.
Can anyone see why that might be?
Thanks
public void ShippersUseCase()
{
using (var redisClient = new RedisClient("localhost"))
{
//Create a 'strongly-typed' API that makes all Redis Value operations to apply against Shippers
var redis = redisClient.As<Shipper>();
//Redis lists implement IList<T> while Redis sets implement ICollection<T>
var currentShippers = redis.Lists["urn:shippers:current"];
var prospectiveShippers = redis.Lists["urn:shippers:prospective"];
currentShippers.Add(
new Shipper
{
Id = redis.GetNextSequence(),
CompanyName = "Trains R Us",
DateCreated = DateTime.UtcNow,
ShipperType = ShipperType.Trains,
UniqueRef = Guid.NewGuid()
});
currentShippers.Add(
new Shipper
{
Id = redis.GetNextSequence(),
CompanyName = "Planes R Us",
DateCreated = DateTime.UtcNow,
ShipperType = ShipperType.Planes,
UniqueRef = Guid.NewGuid()
});
var lameShipper = new Shipper
{
Id = redis.GetNextSequence(),
CompanyName = "We do everything!",
DateCreated = DateTime.UtcNow,
ShipperType = ShipperType.All,
UniqueRef = Guid.NewGuid()
};
currentShippers.Add(lameShipper);
Dump("ADDED 3 SHIPPERS:", currentShippers);
currentShippers.Remove(lameShipper);
.
.
.
}
}
Fixed the problem by adding these overrides to the 'Shipper' class:
public override bool Equals(object obj)
{
if (obj == null)
{
return false;
}
var input = obj as Shipper;
return input != null && Equals(input);
}
public bool Equals(Shipper other)
{
return other != null && (Id.Equals(other.Id));
}
public override int GetHashCode()
{
return (int)Id;
}
This working example shows how to implement List<>.Contains, List<>.Find, and List<>.Remove. Once applied to the 'Shipper' class the problem was solved!

Message Queue Error: cannot find a formatter capable of reading message

I'm writing messages to a Message Queue in C# as follows:
queue.Send(new Message("message"));
I'm trying to read the messages as follows:
Messages messages = queue.GetAllMessages();
foreach(Message m in messages)
{
String message = m.Body;
//do something with string
}
However I'm getting an error message which says: "Cannot find a formatter capable of reading this message."
What am I doing wrong?
I solved the problem by adding a formatter to each message. Adding a formatter to the queue didn't work.
Messages messages = queue.GetAllMessages();
foreach(Message m in messages)
{
m.Formatter = new XmlMessageFormatter(new String[] { "System.String,mscorlib" });
String message = m.Body;
//do something with string
}
Or you can use
message.Formatter =
new System.Messaging.XmlMessageFormatter(new Type[1] { typeof(string) });
you could try reading the bodystream of the message instead of the body, like this:
StreamReader sr = new StreamReader(m.BodyStream);
string messageBody = "";
while (sr.Peek() >= 0)
{
messageBody += sr.ReadLine();
}
Message recoverableMessage = new Message();
recoverableMessage.Body = "Sample Recoverable Message";
recoverableMessage.Formatter = new XmlMessageFormatter(new String[] {"System.String,mscorlib" });
MessageQueue myQueue = new MessageQueue(#".\private$\teste");
Queue must be set Formatter too.
myQueue.Formatter = new XmlMessageFormatter(new String[] { "System.String,mscorlib" });
Everyone here has done a fantastic job at providing solutions, and having just finished battling this problem myself I wanted to throw my own 2c in and show the solution I came up with that works very well.
Firstly when the queue is created I make sure I open up the permissions like so (I'm not concerned about queue security in the context of our application... this is a calculated decision):
queue.SetPermissions("Everyone", MessageQueueAccessRights.FullControl, AccessControlEntryType.Set);
Without that line I would receive all sorts of inaccessible errors and couldn't even browse the queue from the computer management screen. Incidentally if that happens to you and you're wondering how to kill the queue that you don't have access to just:
Stop the service "Message Queueing"
Goto "C:\Windows\System32\msmq\storage\lqs"
Open each file in notepad and look for your queue name (it will most likely be the file that was most recently modified)
Delete that file and restart the Messaging service
Create a base class for your queue message items and mark it [Serializable].
On application load cache a list of all your message types using something like this:
var types = typeof(QueueItemBase).Assembly
.GetTypes()
.Where(t => typeof(QueueItemBase).IsAssignableFrom(t) && t.IsAbstract == false)
.ToArray();
...
// Create and cache a message formatter instance
_messageFormatter = new XmlMessageFormatter(types);
Now you're ready to start receiving messages. My first instinct was to poll for messages, but the api doesn't really like working that way. So I create a background thread and call the blocking method Receive on the queue which will return once a message is available. From there decoding the message is as simple as:
var message = queue.Receive();
if (message == null)
continue;
// Tell the message about our formatter containing all our message types before we
// try and deserialise
message.Formatter = _messageFormatter;
var item = message.Body as QueueItemBase;
And that should be all you need to get nicely implemented, typesafe MSMQ integration!
This worked for me to read a private queue from a remote machine:
MessageQueue queue = new MessageQueue(#"FormatName:Direct=OS:MACHINENAME\private$\MyQueueName", QueueAccessMode.Peek);
Message msg = queue.Peek();
StreamReader sr = new StreamReader(msg.BodyStream);
string messageBody = sr.ReadToEnd();
Update 2019-11-29
Don't use Microsoft Message Queue (MSMQ). Just don't. It is both deprecated, and bottom of the pile in terms of anything useful, performant or even remotely well designed.
It seems that the serialization is only done when accessing the Body property of the Message class. As long as you access the Body property after you set on the message the right Formatter it works fine.
If you prefer not to create a Formatter for each message you can set the Formatter on the queue and for each message (before accessing the Body property) set the Formatter property from the Formatter of the queue.
_queue.Send(new Message() { Formatter = _queue.Formatter, Body = myData } );
var msg = _qeueu.Receive();
msg.Formatter = _queue.Formatter;
var myObject = (MyClass) msg.Body;
Adding formatter solved my issue:
public void ReceiveAsync<T>(MqReceived<T> mqReceived)
{
try
{
receiveEventHandler = (source, args) =>
{
var queue = (MessageQueue)source;
using (Message msg = queue.EndPeek(args.AsyncResult))
{
XmlMessageFormatter formatter = new XmlMessageFormatter(new Type[] { typeof(T) });
msg.Formatter = formatter;
queue.ReceiveById(msg.Id);
T tMsg = (T)msg.Body;
mqReceived(tMsg);
}
queue.BeginPeek();
};
messageQueu.PeekCompleted += receiveEventHandler;
messageQueu.BeginPeek();
}
catch (Exception e)
{
Console.WriteLine(e.Message);
}
}
You can see sample code and msmq library on github:
https://github.com/beyazc/MsmqInt
this works very fine:
static readonly XmlMessageFormatter f = new XmlMessageFormatter(new Type[] { typeof(String) });
private void Client()
{
var messageQueue = new MessageQueue(#".\Private$\SomeTestName");
foreach (Message message in messageQueue.GetAllMessages())
{
message.Formatter = f;
Console.WriteLine(message.Body);
}
messageQueue.Purge();
}

Categories

Resources