Develop Reference

Kafka consumer is not consuming message

I am new in Kafka. kafka consumer is not reading message from the given topic.
I am checking with kafka console as well. it is not working. i donot understand the problem. it was working fine earlier.
public string MessageConsumer(string brokerList, List<string> topics, CancellationToken cancellationToken)
{
//ConfigurationManager.AutoLoadAppSettings("", "", true);
string logKey = string.Format("ARIConsumer.StartPRoducer ==>Topics {0} Key{1} =>", "", string.Join(",", topics));
string message = string.Empty;
var conf = new ConsumerConfig
{
BootstrapServers = "localhost:9092",
GroupId = "23",
EnableAutoCommit = false,
AutoOffsetReset = AutoOffsetResetType.Latest,
};
using (var c = new Consumer<Ignore, string>(conf))
{
try
{
c.Subscribe(topics);
bool consuming = true;
// The client will automatically recover from non-fatal errors. You typically
// don't need to take any action unless an error is marked as fatal.
c.OnError += (_, e) => consuming = !e.IsFatal;
while (consuming)
{
try
{
TimeSpan timeSpan = new TimeSpan(0, 0, 5);
var cr = c.Consume(timeSpan);
// Thread.Sleep(5000);
if (cr != null)
{
message = cr.Value;
Console.WriteLine("Thread" + Thread.CurrentThread.ManagedThreadId + "Message : " + message);
CLogger.WriteLog(ELogLevel.INFO, $"Consumed message Partition '{cr.Partition}' at: '{cr.TopicPartitionOffset} thread: { Thread.CurrentThread.ManagedThreadId}'. Message: {message}");
//Console.WriteLine($"Consumed message Partition '{cr.Partition}' at: '{cr.TopicPartitionOffset}'. Topic: { cr.Topic} value :{cr.Value} Timestamp :{DateTime.UtcNow.ToString("yyyy-MM-dd HH:mm:ss.fff", CultureInfo.InvariantCulture)} GrpId: { conf.GroupId}");
c.Commit();
}
Console.WriteLine($"Calling the next Poll ");
}
catch (ConsumeException e)
{
CLogger.WriteLog(ELogLevel.ERROR, $"Error occured: {e.Error.Reason}");
Console.WriteLine($"Error occured: {e.Error.Reason}");
}
//consuming = false;
}
// Ensure the consumer leaves the group cleanly and final offsets are committed.
c.Close();
}
catch (Exception ex)
{
}
}
return message;
}
What is the issue with this code or there is installation issue with kafka

Is there a Producer actively sending data?
Your consumer is starting from the latest offsets based on the AutoOffsetReset, so it wouldn't read existing data in the topic
The console consumer also defaults to the latest offset
And if you haven't changed the GroupId, then your consumer might have worked once, then you consumed data, then commited the offsets for that group. When the consumer starts again in the same group, it will only resume from the end of the topic, or the offset of the last commit
You also have an empty catch (Exception ex), which might be hiding some other error

Try removing the timespan from consume method.

CRM transferring phonecall activity to a new Entity shows it on both Entities

I am trying to move the activities on some of our contacts/accounts to a new_private_contact_details entity. I have all of them but the phonecall working. The below code does seem to work, but it ends up showing the phonecall on the activity feed for both the new_private_contact_details entity as well as the existing Contact entity. Obviously this is a problem, since I'm trying to migrate those kinds of details to the private details, so having them still show up voids that process.
if (phoneCallsList != null && phoneCallsList.Count > 0)
{
foreach (PhoneCall pc in phoneCallsList)
{
if (pc.StatusCode.Value != 1)
{
int oldState = 0; //for the beginning statecode
int oldStatus = 0; //for the beginning statuscode
if (pc.StatusCode.Value == 3)
{
oldState = 2;
oldStatus = 3;
}
else
{
oldState = 1;
oldStatus = pc.StatusCode.Value;
}
//change status to open
SetStateRequest setStateRequest = new SetStateRequest()
{
EntityMoniker = new EntityReference
{
Id = pc.Id,
LogicalName = pc.LogicalName
},
State = new OptionSetValue(0),
Status = new OptionSetValue(1)
};
try
{
crm.Execute(setStateRequest);
}
catch (Exception ex)
{
throw new Exception("Error: " + ex);
}
pc.RegardingObjectId = pcd.ToEntityReference();
pc.StatusCode.Value = 1;
try
{
service.Update(pc);
}
catch (Exception ex)
{
throw new Exception("Error: " + ex);
}
//return status to closed
setStateRequest = new SetStateRequest()
{
EntityMoniker = new EntityReference
{
Id = pc.Id,
LogicalName = pc.LogicalName
},
State = new OptionSetValue(oldState),
Status = new OptionSetValue(oldStatus)
};
try
{
crm.Execute(setStateRequest);
}
catch (Exception ex)
{
throw new Exception("Error: " + ex);
}
}
else
{
pc.RegardingObjectId = pcd.ToEntityReference();
try
{
service.Update(pc);
}
catch (Exception ex)
{
throw new Exception("Error: " + ex);
}
}
}
}
I have already handled for when the phonecall is completed/closed. I'm updating the RegardingObjectId, but it doesn't remove it from the original entity, and deleting it in CRM from either entity, deletes it from both.
Again, I only seem to be having this issue with the phonecall entity. This code works perfectly for the others, i.e., appointments, tasks, letters, and emails

I figured out where the problem was occurring. It showed up on both entities, because for PhoneCalls there is also a PhoneCalls.To field that still referenced the old Contact entity. Unfortunately this field requires a PartyList, which, from what I've found, cannot be customized, so it must always point to a Contact, Lead, Opportunity or Account entity.

SqlException: do not abort a transaction

I have a code that adds data to two EntityFramework 6 DataContexts, like this:
using(var scope = new TransactionScope())
{
using(var requestsCtx = new RequestsContext())
{
using(var logsCtx = new LogsContext())
{
var req = new Request { Id = 1, Value = 2 };
requestsCtx.Requests.Add(req);
var log = new LogEntry { RequestId = 1, State = "OK" };
logsCtx.Logs.Add(log);
try
{
requestsCtx.SaveChanges();
}
catch(Exception ex)
{
log.State = "Error: " + ex.Message;
}
logsCtx.SaveChanges();
}
}
}
There is an insert trigger in Requests table that rejects some values using RAISEERROR. This situation is normal and should be handled by the try-catch block where the SaveChanges method is invoked. If the second SaveChanges method fails, however, the changes to both DataContexts must be reverted entirely - hence the transaction scope.
Here goes the error: when requestsCtx.SaveChanges() throws a exception, the whole Transaction.Current has its state set to Aborted and the latter logsCtx.SaveChanges() fails with the following:
TransactionException:
The operation is not valid for the state of the transaction.
Why is this happening and how do tell EF that the first exception is not critical?

Really not sure if this will work, but it might be worth trying.
private void SaveChanges()
{
using(var scope = new TransactionScope())
{
var log = CreateRequest();
bool saveLogSuccess = CreateLogEntry(log);
if (saveLogSuccess)
{
scope.Complete();
}
}
}
private LogEntry CreateRequest()
{
var req = new Request { Id = 1, Value = 2 };
var log = new LogEntry { RequestId = 1, State = "OK" };
using(var requestsCtx = new RequestsContext())
{
requestsCtx.Requests.Add(req);
try
{
requestsCtx.SaveChanges();
}
catch(Exception ex)
{
log.State = "Error: " + ex.Message;
}
finally
{
return log;
}
}
}
private bool CreateLogEntry(LogEntry log)
{
using(var logsCtx = new LogsContext())
{
try
{
logsCtx.Logs.Add(log);
logsCtx.SaveChanges();
}
catch (Exception)
{
return false;
}
return true;
}
}
from the documentation on transactionscope: http://msdn.microsoft.com/en-us/library/system.transactions.transactionscope%28v=vs.110%29.aspx
If no exception occurs within the transaction scope (that is, between
the initialization of the TransactionScope object and the calling of
its Dispose method), then the transaction in which the scope
participates is allowed to proceed. If an exception does occur within
the transaction scope, the transaction in which it participates will
be rolled back.
Basically as soon as an exception is encountered, the transaction is rolled back (as it seems you're aware) - I think this might work but am really not sure and can't test to confirm. It seems like this goes against the intended use of transaction scope, and I'm not familiar enough with exception handling/bubbling, but maybe it will help! :)

I think I finally figured it out. The trick was to use an isolated transaction for the first SaveChanges:
using(var requestsCtx = new RequestsContext())
using(var logsCtx = new LogsContext())
{
var req = new Request { Id = 1, Value = 2 };
requestsCtx.Requests.Add(req);
var log = new LogEntry { RequestId = 1, State = "OK" };
logsCtx.Logs.Add(log);
using(var outerScope = new TransactionScope())
{
using(var innerScope = new TransactionScope(TransactionScopeOption.RequiresNew))
{
try
{
requestsCtx.SaveChanges();
innerScope.Complete();
}
catch(Exception ex)
{
log.State = "Error: " + ex.Message;
}
}
logsCtx.SaveChanges();
outerScope.Complete();
}
}
Warning: most of the articles about RequiresNew mode discourage using it due to performance reasons. It works perfectly for my scenario, however if there are any side effects that I'm unaware of, please let me know.

Null values while updating DB in Parallel.Foreach

I use following script to get data from external service and store in dB. In certain rare cases less than 1% records gets updated with null values. In below code, the "re.status=fail" we see null. let us know if any thots.
public void ProcessEnquiries()
{
List<req> request = new List<req>();
var options = new ParallelOptions { MaxDegreeOfParallelism = Convert.ToInt32(System.Configuration.ConfigurationManager.AppSettings["MaxDegreeOfParallelism"]) };
try
{
Parallel.ForEach(request, options, currentRequest =>
{
ProcessedRequest processedRequest = null;
processedRequest = CommunicateToWS(currentRequest); // Here we call to webservice
});
}
catch (AggregateException exception)
{
foreach (Exception ex in exception.InnerExceptions)
{
// Handle Exception
}
}
}
public ProcessedRequest CommunicateToWS(req objReq)
{
ProcessedRequest re = new ProcessedRequest();
using (WebCall obj = new WebCall())
{
re.no = refnu;
try
{
retval = obj.getValue(inval);
objProxy.Close();
//get data
// parse and store to DB
}
catch (Exception e)
{
re.status = "fail";
//update DB that request has failed
//Handle Exception
obj.Close();
}
}
}

DataContext.Submit and TransactionScope

Please explain me pseudocode below.
My idea is: 3-nd SubmitChanges will commit o.Status and will not commit o.TransactionId, and my object will get corrupted in database (I mean it will not be consistent anymore).
XDataContext DB = .....;
XOrder o = DB.XOrders.Single(.......);
try
{
using (var t = new TransactionScope())
{
o.Status = XOrderStatus.Completed;
DB.SubmitChanges(); // 1
string s = null;
s.Trim(); // crash here;
o.TransactionId = ......; // some calculations here
DB.SubmitChanges(); // 2
t.Complete();
}
}
catch (Exception ex)
{
XEvent e = new XEvent();
e.Type = XEventType.Exception;
e.Data = .........; // some calculations here
DB.XEvents.InsertOnSubmit(e);
DB.SubmitChanges(); // 3
}
Is it any 'best practices' for my case?

Is it any 'best practices' for my case?
Yes. Use one DataContext instance per unit of work.
catch (Exception ex)
{
XEvent e = new XEvent();
e.Type = XEventType.Exception;
e.Data = .........; // some calculations here
using (XDataContext dc2 = new XDataContext())
{
dc2.XEvents.InsertOnSubmit(e);
dc2.SubmitChanges(); // 3
}
}