Develop Reference

Kafka Producer can't connect to servers and will not throw any exception

I have an extremely simple setup for sending message to Kafka:
var producerConfig = new ProducerConfig
{
BootstrapServers = "www.example.com",
SecurityProtocol = SecurityProtocol.SaslSsl,
SaslMechanism = SaslMechanism.ScramSha512,
SaslUsername = _options.SaslUsername,
SaslPassword = _options.SaslPassword,
MessageTimeoutMs = 1
};
var producerBuilder = new ProducerBuilder<Null, string>(producerConfig);
using var producer = producerBuilder.Build();
producer.Produce("Some Topic", new Message<Null, string>()
{
Timestamp = Timestamp.Default,
Value = "hello"
});
Before, this code was working fine. Today it has decided to stop working and I'm trying to figure out why. I'm trying to get the Producer to throw an exception when failing to deliver a message, but it never seems to crash. Even when I fill in a wrong username and password, the producer still doesn't crash. Not even a logline in my local output window. How can I debug my Kafka connection when the producer never shows any problems?

You can add SetErrorHandler() to the ProducerBuilder. It would look like this:
var producerBuilder = new ProducerBuilder<Null, string>(producerConfig)
.SetErrorHandler(errorMessageString => .....);
Set a breakpoint in that lambda and you can break on errors.

Produce is asynchronous and not blocking, function signature is
void Produce(string topic, Message<TKey, TValue> message, Action<DeliveryReport<TKey, TValue>> deliveryHandler = null)
In order to verify that a message was delivered without error
you can add a delivery report handler function e.g.
private void DeliveryReportHandler(DeliveryReport<int, T> deliveryReport)
{
if (deliveryReport.Status == PersistenceStatus.NotPersisted)
{
_logger.LogError($"Failed message delivery: error reason:{deliveryReport.Error?.Reason}");
_messageWasNotDelivered = true;
}
}
_messageWasNotDelivered = false;
_producer.Produce(topic,
new Message<int, T>
{
Key = key,
Value = entity
},
DeliveryReportHandler)
_producer.Flush(); // Wait until all outstanding produce requests and delivery report callbacks are completed
if(_messageWasNotDelivered ){
// handle non delivery
}
This code can be trivially adjusted for batch producing like this
_messageWasNotDelivered = false;
foreach(var entity in entities){
_producer.Produce(topic,
new Message<int, T>
{
Key = entity.Id,
Value = entity
},
DeliveryReportHandler)
}
_producer.Flush(); // Wait until all outstanding produce requests and delivery report callbacks are completed
if(_messageWasNotDelivered ){
// handle non delivery
}

NServiceBus with AzureStorageQueues not removing poison messages from input queue due to changing message properties

I am experimenting with a new NServiceBus project utilizing Azure Storage Queues for message transport and JSON serialization using custom message unwrapping logic seen here:
var jsonSerializer = new Newtonsoft.Json.JsonSerializer();
transportExtensions.UnwrapMessagesWith(cloudQueueMessage =>
{
using (var stream = new MemoryStream(cloudQueueMessage.AsBytes))
using (var streamReader = new StreamReader(stream))
using (var textReader = new JsonTextReader(streamReader))
{
try
{
var jObject = JObject.Load(textReader);
using (var jsonReader = jObject.CreateReader())
{
// Try deserialize to a NServiceBus envelope first
var wrapper = jsonSerializer.Deserialize<MessageWrapper>(jsonReader);
if (wrapper.MessageIntent != default)
{
// This was a envelope message
return wrapper;
}
}
// Otherwise this was an EventGrid event
using (var jsonReader = jObject.CreateReader())
{
var #event = jsonSerializer.Deserialize<EventGridEvent>(jsonReader);
var wrapper = new MessageWrapper
{
Id = #event.Id,
Headers = new Dictionary<string, string>
{
{ "NServiceBus.EnclosedMessageTypes", #event.EventType },
{ "NServiceBus.MessageIntent", "Publish" },
{ "EventGrid.topic", #event.Topic },
{ "EventGrid.subject", #event.Subject },
{ "EventGrid.eventTime", #event.EventTime.ToString("u") },
{ "EventGrid.dataVersion", #event.DataVersion },
{ "EventGrid.metadataVersion", #event.MetadataVersion },
},
Body = Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(#event.Data)),
MessageIntent = MessageIntentEnum.Publish
};
return wrapper;
}
}
catch
{
logger.Error("Message deserialization failed, sending message to error queue");
throw;
}
}
});
The custom message unwrapping logic works correctly for properly formatted JSON messages and when an improperly formatted JSON message is put into the input queue the custom message unwrapping logic will error out on the first line inside the usings where I create the jObject which is the expected behavior. However, when the custom message unwrapping logic fails the error will get caught by the logic in the MessageRetrieved class which is part of the NServiceBus.Azure.Transports.WindowsAzureStorageQueues NuGet package (v8.2.0) seen below:
public async Task<MessageWrapper> Unwrap()
{
try
{
Logger.DebugFormat("Unwrapping message with native ID: '{0}'", rawMessage.Id);
return unwrapper.Unwrap(rawMessage);
}
catch (Exception ex)
{
await errorQueue.AddMessageAsync(rawMessage).ConfigureAwait(false);
await inputQueue.DeleteMessageAsync(rawMessage).ConfigureAwait(false);
throw new SerializationException($"Failed to deserialize message envelope for message with id {rawMessage.Id}. Make sure the configured serializer is used across all endpoints or configure the message wrapper serializer for this endpoint using the `SerializeMessageWrapperWith` extension on the transport configuration. Please refer to the Azure Storage Queue Transport configuration documentation for more details.", ex);
}
}
The first line of the try catch runs correctly adding the message to the configured error queue, however, when it does that, it appears to be changing the message ID and popreceipt of the raw message as seen here:
Initial Message Values
Updated Message Values
Then when the next line runs attempting to remove the original message from the input queue it is unable to find it as according to this article https://learn.microsoft.com/en-us/rest/api/storageservices/delete-message2#remarks it requires the original message ID and pop reciept which have now changed leading to the following error being thrown:
2020-04-20 14:17:58,603 WARN : Azure Storage Queue transport failed pushing a message through pipeline
Type: Microsoft.WindowsAzure.Storage.StorageException
Message: The remote server returned an error: (404) Not Found.
Source: Microsoft.WindowsAzure.Storage
StackTrace:
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 50
at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass7.<CreateCallbackVoid>b__5(IAsyncResult ar) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 121
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.Transport.AzureStorageQueues.MessageRetrieved.<Unwrap>d__3.MoveNext() in C:\BuildAgent\work\3c19e2a032c05076\src\Transport\MessageRetrieved.cs:line 40
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.Transport.AzureStorageQueues.MessagePump.<InnerReceive>d__7.MoveNext() in C:\BuildAgent\work\3c19e2a032c05076\src\Transport\MessagePump.cs:line 153
TargetSite: T EndExecuteAsync[T](System.IAsyncResult)
Is this an issue with the NServiceBus package logic, or is something in my custom message unwrapping logic causing these values to change?

This is a bug. When unwrapping is failing, the message is not yet going through the processing pipeline. As a result of that, the normal recoverability is not applicable. The CloudQueueMessage needs to be "cloned" and the clone to be sent to the error queue while the original message used to remove it from the input queue. I've raised a bug issue in GitHub and you can track the process there.

AggregateException "One or more errors occurred.An error occurred while sending the request."

I tried to post a file to an API rest from my WCF .Net framework 4.5. Here is my code:
public string CreateConclusion(string[] instanceUIDs)
{
var root = #"C:\";
string filename = "1.2.840.114257.1.9.1245.56421.52314.1119854.01248.dcm";
using (var client = new HttpClient())
{
var stream = new FileStream(root + filename, FileMode.Open);
using (var content =
new MultipartFormDataContent("Upload----" + DateTime.Now.ToString(CultureInfo.InvariantCulture)))
{
content.Add(new StreamContent(stream), "fileToUpload", filename);
using (var message = client.PostAsync("https://localhost:44343/api/ConclusionReports/UploadFile", content).Result)
{
var input = message.Content.ReadAsStringAsync();
return !string.IsNullOrWhiteSpace(input.Result) ? Regex.Match(input.Result, #"http://\w*\.directupload\.net/images/\d*/\w*\.[a-z]{3}").Value : null;
}
}
}
}
It doesn't work and throw an exception: "One or more errors occurred.An error occurred while sending the request."
Does anyone can help me to solve this problem? Thank you in advance

An aggregate exception can always be unwrapped to discover the real cause. Try to write your client call inside a try-catch like this:
try {
//Some risky client call that will call parallell code / async /TPL or in some way cause an AggregateException
}
catch (AggregateException err){
foreach (var errInner in err.InnerExceptions) {
Debug.WriteLine(errInner); //this will call ToString() on the inner execption and get you message, stacktrace and you could perhaps drill down further into the inner exception of it if necessary
}
}

Serialize SendGridMessage object

I am building a system that need to send some transactional mails, and to achieve this I am using Azure storage queues to store the message temporarily before it is picked up by a WebJob and sent off to the intended recipient.
My Code is as follows:
SendGridMessage message = new SendGridMessage();
//Populate message with details - omitted for brevity
var serializer = new JavaScriptSerializer();
var modelAsString = serializer.Serialize(message);
try
{
var setting = CloudConfigurationManager.GetSetting("AzureStorageConnectionString");
var account = CloudStorageAccount.Parse(setting);
var queueClient = account.CreateCloudQueueClient();
var queue = queueClient.GetQueueReference("FSPortalEmailQueue");
queue.CreateIfNotExists();
queue.AddMessage(new CloudQueueMessage(modelAsString));
}
catch (Exception ex)
{
//Something went wrong
}
Each time I try to execute the coder, an exception is thrown on the
var modelAsString = serializer.Serialize(message);
"Exception has been thrown by the target of an invocation."
The inner exception thrown was
{"Bad key path!"} from source "SendGrid.SmtpApi"
Please advise what I am doing wrong here.

After a bit more digging, it turns out that the message.header node was not being initialised. After adding
message.Header = new SendGrid.SmtpApi.Header();
message.Header.SetTo(new List<String> { enquiry.EnquiryCreatedBy.Email });
all started working pretty magically

StackExchange TimeoutException when trying to insert 750 items in 2 sets in redis

Initially I'm trying to insert some collection of items into 2 redis sets (maybe this is not good idea at all, but ...). amount of entries that I'm trying to add at once: 750+
For now I do receive timeout exception when trying to perform this action using StackExchange.redis client, the interesting thing that I'm able to complete similar action using "legacy" booksleeve client I've investigated previously.
So, I definitely wrong in something (even maybe in my intial bookSleeve implementation), just trying to figure out what exactly is wrong.
Below is sample of the code that I use with redis clients:
BookSleeve:
using (var tran = connection.CreateTransaction())
{
Task lastOpTask = null;
tran.SuspendFlush();
try
{
// perform required configurations/ actions
tran.Sets.Add(_redisConfiguration.DbNumber, CurrentIdsSetDbKey, stringIds);
tran.Sets.Add(_redisConfiguration.DbNumber, CurrentDetailsSetDbKey, stringDetails);
lastOpTask = tran.Execute();
isOperationSuccessful = true;
}
catch (TaskCanceledException ex)
{
...
}
catch (TimeoutException ex1)
{
...
}
finally
{
tran.ResumeFlush();
}
if (lastOpTask != null)
{
connection.Wait(lastOpTask);
...
}
}
StackExchange.redis implementation of the same code:
var tran = db.CreateTransaction();
// todo: do we need to add here any condition or PipeLining? any watch/unwatch verifications?
tran.SetAddAsync(CurrentIdsSetDbKey, stringIds);
tran.SetAddAsync(CurrentDetailsSetDbKey, stringDetails);
try
{
isOperationSuccessful = tran.Execute();
}
catch (TaskCanceledException ex)
{
...
}
catch (TimeoutException ex1)
{
...
}
After I start unit tests I receive next error for StackExchange client:
Message: Timeout performing EXEC, inst: 3, queue: 3, qu=0, qs=3, qc=0, wr=0/0
Source: in StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in c:\TeamCity\buildAgent\work\18a91a3757cef937\StackExchange.Redis\StackExchange\Redis\ConnectionMultiplexer.cs:line 1693
in StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in c:\TeamCity\buildAgent\work\18a91a3757cef937\StackExchange.Redis\StackExchange\Redis\RedisBase.cs:line 92
in StackExchange.Redis.RedisTransaction.Execute(CommandFlags flags) in c:\TeamCity\buildAgent\work\18a91a3757cef937\StackExchange.Redis\StackExchange\Redis\RedisTransaction.cs:line 51
in DFS.Cache.CacheManager.RedisStackExchange.RedisContestCacheManager.RegisterAvailableContests(IList1 contests) in d:\Projects\DFS\Code\DFS\DFS.Cache.CacheManager.RedisStackExchange\RedisContestCacheManager.cs:line 131
Just wondering, what exactly I'm doing wrong. Thank you in advance for any suggestions!.
P.S. For stackEchange.redis configuration I'm using config sample provided by Marc from github (available here: StackExchange/StackExchange.Redis/blob/master/Docs/Configuration.md)
P.S.
Please see current StackExchange client Config file:
var config = new ConfigurationOptions
{
EndPoints =
{
{"MasterIP", 6379},
{"SlaveIP", 6380}
},
CommandMap = CommandMap.Create(new HashSet<string>
{
// EXCLUDE a few commands (to work with data-flow-related mode only)
"INFO",
"CONFIG",
"CLUSTER",
"PING",
"ECHO",
"CLIENT"
}, available: false),
KeepAlive = 60, // 60 sec to ensure connection is alive
ConnectTimeout = 5000, // 5 sec
SyncTimeout = 5000, // 5 sec
ServiceName = "mymaster", // sentinel service name
DefaultVersion = new Version(2, 8, 8),
Password = "password"
};
Standard entry (used in sets) looks like:
{
"Id":"08e5ffdbced046cb8f55c50e4bab822d",
"Entries":0,
"State":"i",
"Name":"very dummy entry name value: autoGet 299",
"Summary": "entry summary details, some long string 299, some common info, some data: true, 8200"
"IsMultiple":true,
"IsPublic":true,
"MaxEntries":8200,
"IsEntryVisible":true,
"StartDate":"9/10/2014 12:00:00 AM"
}
P.S.
After Marc's response I ran couple of tests and got some other errors in Unit tests,
for example
{"Timeout performing SISMEMBER set:raw:Ids, inst: 1, queue: 6, qu=0, qs=6, qc=0, wr=0/0"} when set contains not more than 300 items.
So I agree that this is Connection issue and has nothing to do with current switch to the StackExchange.redis client.

The following passes just fine, and reports 10ms locally. I would be very interested if you could fill in the blanks a bit so I can do a representative test that reproduces the issue. Note that qu=0, qs=3 tells me that at the point that it times out, we're waiting for the redis server to respond. Obviously local bandwidth and latency would be of interest, but fundamentally, it should work. I'd also be interested in what your sync-timeout is set to.
using System.Diagnostics;
using System.Linq;
using NUnit.Framework;
namespace StackExchange.Redis.Tests.Issues
{
[TestFixture]
public class SO22786599 : TestBase
{
[Test]
public void Execute()
{
string CurrentIdsSetDbKey = Me() + ".x";
string CurrentDetailsSetDbKey = Me() + ".y";
RedisValue[] stringIds = Enumerable.Range(1, 750).Select(i => (RedisValue)(i + " id")).ToArray();
RedisValue[] stringDetails = Enumerable.Range(1, 750).Select(i => (RedisValue)(i + " detail")).ToArray();
using (var conn = Create())
{
var db = conn.GetDatabase();
var tran = db.CreateTransaction();
tran.SetAddAsync(CurrentIdsSetDbKey, stringIds);
tran.SetAddAsync(CurrentDetailsSetDbKey, stringDetails);
var watch = Stopwatch.StartNew();
var isOperationSuccessful = tran.Execute();
watch.Stop();
System.Console.WriteLine("{0}ms", watch.ElapsedMilliseconds);
Assert.IsTrue(isOperationSuccessful);
}
}
}
}