I'm using the following code to publish messages to a RabbitMQ queue:
ConnectionFactory factory = new ConnectionFactory {
HostName = hostName,
Port = port,
UserName = userName,
Password = password,
VirtualHost = "/",
Protocol = Protocols.DefaultProtocol
};
connection = factory.CreateConnection();
channel = connection.CreateModel();
channel.QueueDeclare(queue, true, false, false, null);
foreach (string message in messages) {
byte[] body = Encoding.UTF8.GetBytes(message);
channel.BasicPublish("", queue, null, body);
}
While publishing the messages to a local RabbitMQ server I get a message rate of up to 10,000 messages per second. The cpu load of the system (2x3.16 GHz) is at almost 100%. Is there a way to increase this message rate? My first idea was to use a bulk publish operation, but there doesn't seem to be anything like that in RabbitMQ. My second idea was using a Parallel.ForEach instead of the foreach, but that didn't change the message rate.
Defining a queue as "Durable" has the added benefit of surviving a rabbit or server restart. The downside is that to accomplish this it writes the data to disk, which is costly.
If your greatest concern is throughput and it's not a problem that you'd drop a few messages in the event of a failure than setting "Durable=false" would increase your throughput.
Note: Even with durable=false if the queue length becomes long enough it will dump them to disk (after all, there is only so much memory on the machine).
Related
I'm trying to write a high-performance TCP server (a LDAP server) using this tutorial by David Fowler as a base part of the MyServerListener.cs to handle incoming connections.
This is a simple .net 7 console app (with little changes) that I borrowed from David, it just accepts incoming clients, process the requests and writes hello to the response :
internal class Program
{
const int PORT = 389; // injecting from config
const int BACKLOG_LENGTH = 200; // max backlog size in windows server
static async Task Main(string[] args)
{
var listenSocket = new Socket(SocketType.Stream, ProtocolType.Tcp);
listenSocket.Bind(new IPEndPoint(IPAddress.Any, port));
Console.WriteLine("Listening on port " + port);
listenSocket.Listen(BACKLOG_LENGTH);
while (true)
{
var socket = await listenSocket.AcceptAsync();
_ = ProcessLinesAsync(socket);
}
}
private static async Task ProcessLinesAsync(Socket socket)
{
#if DEBUG
Console.WriteLine($"[{socket.RemoteEndPoint}]: connected");
#endif
// Create a PipeReader over the network stream
var stream = new NetworkStream(socket);
var reader = PipeReader.Create(stream);
var writer = PipeWriter.Create(stream);
while (true)
{
ReadResult result = await reader.ReadAsync();
ReadOnlySequence<byte> buffer = result.Buffer;
while (TryReadLine(ref buffer, out ReadOnlySequence<byte> line))
{
// Process the line.
ProcessLine(line);
try
{
// writing a sample message to the response
var helloBytes = Encoding.ASCII.GetBytes("hello\n");
await writer.WriteAsync(helloBytes);
}
catch (Exception ex)
{
throw;
}
}
// Tell the PipeReader how much of the buffer has been consumed.
reader.AdvanceTo(buffer.Start, buffer.End);
// Stop reading if there's no more data coming.
if (result.IsCompleted)
{
break;
}
}
// Mark the PipeReader as complete.
await reader.CompleteAsync();
#if DEBUG
Console.WriteLine($"[{socket.RemoteEndPoint}]: disconnected");
#endif
}
private static bool TryReadLine(ref ReadOnlySequence<byte> buffer, out ReadOnlySequence<byte> line)
{
// Look for a EOL in the buffer.
SequencePosition? position = buffer.PositionOf((byte)'\n');
if (position == null)
{
line = default;
return false;
}
// Skip the line + the \n.
line = buffer.Slice(0, position.Value);
buffer = buffer.Slice(buffer.GetPosition(1, position.Value));
return true;
}
private static void ProcessLine(in ReadOnlySequence<byte> buffer)
{
foreach (var segment in buffer)
{
// Doing some tasks
#if DEBUG
Console.Write(Encoding.UTF8.GetString(segment.Span));
Console.WriteLine();
#endif
}
}
}
This server listens on a port (389), processes the incoming request, doing some jobs and then writes a message to the response using PipeReader and PipeWriter.
I'm trying to do my best to a less memory/heap allocation code (using span<>, memory<>, ...) as I can, to keep my codebase so fast and optimize. But for now, I'm trying to test the production environment with the above code to examine the throughput; I mean: the server resources, my TCP server application itself, clients and the network;
I'm using Apache JMeter to test (load/stress test).
In some scenarios (sending more than 5000 request/sec) I get Connection refused error messages in JMeter logs, but I don't have any high pressure in the server or client's (JMeter[s]) resources (CPU/Memory).
I tried to optimize the server's configuration and changed some TCP related parameters (I googled about them) like MaxUserPort: 65534, TcpTimedWaitDelay: 30 or different backlog size, but no improvements.
So I'm almost sure that there is sth related to the network (packet dropping/rejecting or sth like this).
I also turned off firewall in the testing clients and the server, But I don't have any access to the network configurations (and I don't know what are they) like firewalls, ISA, TMG, etc.
_____________
Update 1:
I already increased our clients ephemeral ports to the maximum range using this PS script:
netsh int ipv4 set dynamic tcp start=5000 num=65535
and now we have this :
netsh int ipv4 show dynamicport tcp
Start Port : 1024
Number of Ports : 64511
And we also checked JMeter logs to see any error indicating this situation (Ephemeral ports exhaustion), at first we saw this message :
Non HTTP response code: java.net.BindException,Non HTTP response
message: Address already in use
But now, it's gone and we don't have large number of TIME_WAIT ports to worry about.
And we are also testing our scenario with SO_LINGER:0 and monitoring real times TIME_WAIT ports (using some tools), and we are sure that this isn't our concern right now.
_____________
So my question is, how can I find out why I can't send more traffic (threads/requests per seconds in JMeter clients) to the server to testing my TCP server application performance? Because for now, the server CPU doesn't increase more than ~10%.
At this point, is this a network related problem? How can I be sure about that? e.g: can I use some network analyzers (e.g: PRTG network monitor) to find out any dropped TCP packets? Or any other tips welcomed
Most probably TCP ports are not recycled fast enough, there is a network parameter which controls the time which connection can stay in TIME_WAIT state so you might also want to reduce TcpTimedWaitDelay
Also it might be a good idea to increase maximum number of TCP connections via TcpNumConnections parameter
And last but not the least it might be the case JMeter is not capable of sending the requests fast enough so you might need to play the same trick on the load generator side. In addition make sure to follow JMeter Best Practices and monitor CPU/RAM/Network/Disk/Swap usage on JMeter side as it might be the case you will need to switch to Distributed Testing if one machine is not capable of giving more than 5k requests per second.
BACKGROUND INFO
I have a queue (for emails) in RabbitMQ, and want to build a consumer for it. The queue is used by another .NET app for sending emails to customers. I wanted the emailing logic to sit outside of the .NET app, and also have the benefits of durability ...etc that RabbitMQ offers.
ISSUE
The .NET app is able to publish/push emails onto the queue, but I have difficulty building the consumer! Here's my code for the consumer:
// A console app that would be turned into a service via TopShelf
public void Start()
{
using (_connection = _connectionFactory.CreateConnection())
{
using (var model = _connection.CreateModel())
{
model.QueueDeclare(_queueName, true, false, false, null);
model.BasicQos(0, 1, false);
var consumer = new EventingBasicConsumer(model);
consumer.Received += (channelModel, ea) =>
{
var message = (Email) ea.Body.DeSerialize(typeof(Email));
Console.WriteLine("----- Email Processed {0} : {1}", message.To, message.Subject);
model.BasicAck(ea.DeliveryTag, false);
};
var consumerTag = model.BasicConsume(_queueName, false, consumer);
}
}
}
The code above should be able to grab messages off the queue and process them (according to this official guide), but this isn't happening.
The problem is premature connection disposal. People often think that BasicConsume is a blocking call, but it is not. It will return almost immediately, and the very next statement is disposing (closing) of channel and connection which of course will cancel your subscription. So to fix - store connection and model in private fields and dispose them only when you are done with queue consumption.
You said queue is used by another .Net app, is that another consumer? If that is another consumer then can you please confirm which exchange you are using? If you want multiple consumers to pick up the message then please go ahead with "FanOut" exchange
I'm using the current Apache.NMS 1.7.1 and Apache.NMS.ActiveMQ 1.7.2.
I'm using IndividualAcknowledge, so I'm trying to keep the number of loaded messages quite low, because it get's really slow if I have >>1000 messages loaded without Acking them (It's searching a linked list of all messages each time).
I have the following codesnippets:
BlockingCollection<IMessage> _collection = new BlockingCollection<IMessage>();
var factory = new ConnectionFactory("activemq:tcp://localhost:61616");
var _connection = (Connection) factory.CreateConnection();
_connection.PrefetchPolicy.All = 1000;
var session = (Session) _connection.CreateSession(AcknowledgementMode.IndividualAcknowledge);
var destination = SessionUtil.GetDestination(session, "queue://testQueue");
var messageConsumer = (MessageConsumer)session.CreateConsumer(destination);
messageConsumer.Listener += message => _collection.Add(message);
_connection.Start();
The queue testQueue contains >>20_000 messages. After waiting some seconds, _collection contains all the messages, without me acknowledging any of them.
If I understand the dokumentation right, I should get at most 1000 until I start acknowledging them.
Once the broker has dispatched a prefetch limit number of messages to a consumer it will not dispatch any more messages to that consumer until the consumer has acknowledged at least 50% of the prefetched messages, e.g., prefetch/2, that it received. When the broker has received said acknowledgements it will dispatch a further prefetch/2 number of messages to the consumer to 'top-up', as it were, its prefetch buffer.
I also tried some variations like only setting QueuePrefetch or setting the policy in the url:
activemq:tcp://localhost:61616?nms.prefetchPolicy.queuePrefetch=100
or in the queue:
queue://testQueue?consumer.prefetchSize=100
Regarding the slowness of the IndividualAcknowledge, I already tried several other options without much luck:
messageConsumer.OptimizeAcknowledge = true;
messageConsumer.OptimizeAcknowledgeTimeOut = 1000;
messageConsumer.OptimizedAckScheduledAckInterval = 500;
Though I'm not completely clear about the differences of the last to options.
Because you are using an asynchronous listener the broker will be given sending you everything as the client continues to grant credit to the broker on delivery of each message to your asynchronous event listener. To truly limit the amount of messages deliver to the client at any given time the client needs to use synchronous receive calls. Individual acknowledge is best paired with synchronous consumption such that you can control how many messages are read and acknowledge them at some point in time when ready.
The optimized acknowledge settings don't apply in individual acknowledge mode so that won't help with performance.
We have been using RabbitMQ as messaging service in the project. We will be pushing message into a queue and which will be received at the message consumer and will be trying to make entry into database. Once the values entered into the database we will be sending positive acknowledgement back to the RabbitMQ server if not we will be sending negative acknowledgement.
I have created Message Consumer as Windows service.Message has been successfully entered and well received by the message consumer(Made entry in table)but with an exception log "Shared Queue closed".
Please find the code block.
while (true)
{
try
{
if (!Connection.IsOpen || !Channel.IsOpen)
{
CreateConnection(existConnectionConfig, QueueName);
consumer = new QueueingBasicConsumer(Channel);
consumerTag=Channel.BasicConsume(QueueName,false,consumer);
}
BasicDeliverEventArgs e = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
IBasicProperties props = e.BasicProperties;
byte[] body = e.Body;
bool ack = onMessageReceived(body);
if (ack == true)
{
Channel.BasicAck(e.DeliveryTag, false);
}
else
Channel.BasicNack(e.DeliveryTag, false, true);
}
catch (Exception ex)
{
//Logged the exception in text file where i could see the
//message as "Shared queue closed"
}
}
I have surfed in net too but couldn't able to what the problem. It will be helpful if anyone able to help me out.
Thanks in advance,
Selva
In answer to your question, I have experienced the same problems when my Web Client has reset the connection due to App Pool recycling or some other underlying reason the connection has been dropped that appears beyond your scope. You may need to build in a retry mechanism to cope with this.
You might want to look at MassTransit. I have used this with RabbitMQ and it makes things a lot easier by effectively providing a management layer to RabbitMQ. MassTransit takes away the headache of retry mechanisms - see Connection management. It also provides a nice multi threaded concurrent consumer configuration.
This has the bonus of your implementation being more portable - you could easily change things to MSMQ should the requirement come along.
I have program where client peers communicate with each other via TCP-IP. When one client does something he will signal other clients one by one that this happened. Here is a code I use to send data across:
public static string SendDirect(string data, string hostName, int portNumber)
{
string responseData;
try
{
var client = new TcpClient(hostName, portNumber);
Stream s = client.GetStream();
var sw = new StreamWriter(s) { AutoFlush = true };
sw.WriteLine(data);
s.Close();
client.Close();
s.Dispose();
sw.Dispose();
responseData = "OK";
}
catch (SocketException ex)
{
responseData = ex.Message;
}
return responseData;
}
Line
var client = new TcpClient(hostName,
portNumber);
can be very slow at times for some machines. For example, in my home network it takes like 2 or 3 seconds. Can you see how it's real bad with 15 clients.
I was wondering how expensive or if even possible to not Close client every time and keep 30-40 of them open at all times? I assume some mechanism to check to make sure they alive and to make sure they all closed properly need to be coded but I wonder if idea itself is correct..
Thanks!
Nothing should keep or limit you from creating more than one client/connection at a time. Actually initiating and closing tons of connections might trigger different security stuff (trying to fend of a possible DDOS attack or whatever). You might as well speed up the process resolving host names before and caching those. It doesn't necessarily have to be the object creation that slows you down actually.
The OS might throttle the number of pending connections per second (think 10 per second under Windows) but other than that there shouldn't be any issues. You shouldn't open/close connections for single commands anyway in my opinion. You should think about keeping both open, the TcpClient as well as the StreamWriter. Just ensure you flush once you're done writing your packet. To improve performance you should think about manual flushing, especially if there's is more than one command/packet to be sent to each client as each packet will take the minimum TCP window size (usually something around 1492-1500 bytes).