Develop Reference

How does the offset for a topic work in Kafka (Kafka_net)

I have a basic producer app and a consumer app. if I run both and have both start consuming on their respective topics, I have a great working system. My thought was that if I started the producer and sent a message that I would be able to then start the consumer and have it pick up that message. I was wrong.
Unless both are up and running, I lose messages (or they do not get consumed).
my consumer app looks like this for comsuming...
Uri uri = new Uri("http://localhost:9092");
KafkaOptions options = new KafkaOptions(uri);
BrokerRouter brokerRouter = new BrokerRouter(options);
Consumer consumer = new Consumer(new ConsumerOptions(receiveTopic, brokerRouter));
List<OffsetResponse> offset = consumer.GetTopicOffsetAsync(receiveTopic, 100000).Result;
IEnumerable<OffsetPosition> t = from x in offset select new OffsetPosition(x.PartitionId, x.Offsets.Max());
consumer.SetOffsetPosition(t.ToArray());
IEnumerable<KafkaNet.Protocol.Message> msgs = consumer.Consume();
foreach (KafkaNet.Protocol.Message msg in msgs)
{
do some stuff here based on the message received
}
unless I have the code between the lines, it starts at the beginning every time I start the application.
What is the proper way to manage topic offsets so messages are consumed after a disconnect happens?
If I run
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic chat-message-reply-XXX consumer-property fetch-size=40000000 --from-beginning
I can see the messages, but when I connect my application to that topic, the consumer.Consume() does not pick up the messages it has not already seen. I have tried this with and without runing the above bat file to see if that makes any difference. When I look at the consumer.SetOffsetPosition(t.ToArray()) call (t specifically) it shows that the offset is the count of all messages for the topic.
Please help,

Set auto.offset.reset configuration in your ConsumerOptions to earliest. When the consumer group starts the consume messages, it will consume from the latest offset because the default value for auto.offset.reset is latest.
But I looked at kafka-net API now, it does not have a AutoOffsetReset property, and it seems pretty insufficient with its configuration in consumers. It also lacks documentation with method summaries.
I would suggest you use Confluent .NET Kafka Nuget package because it is owned by Confluent itself.
Also, why are calling GetTopicOffsets and setting that offset back again in consumer. I think when you configure your consumer, you should just start reading messages with Consume().
Try this:
static void Main(string[] args)
{
var uri = new Uri("http://localhost:9092");
var kafkaOptions = new KafkaOptions(uri);
var brokerRouter = new BrokerRouter(kafkaOptions);
var consumerOptions = new ConsumerOptions(receivedTopic, brokerRouter);
var consumer = new Consumer(consumerOptions);
foreach (var msg in consumer.Consume())
{
var value = Encoding.UTF8.GetString(msg.Value);
// Process value here
}
}
In addition, enable logs in your KafkaOptions and ConsumerOptions, they will help you a lot:
var kafkaOptions = new KafkaOptions(uri)
{
Log = new ConsoleLog()
};
var consumerOptions = new ConsumerOptions(topic, brokerRouter)
{
Log = new ConsoleLog()
});

I switched over to use Confluent's C# .NET package and it now works.

Resolve 'configuration object is read only, because it has been committed by a call to ServerManager.CommitChanges()'?

I've written a custom action for an installer project that does the following:
Checks existing websites to see if any exist with the same name put
in by the user.
Creates the website in IIS if it doesn't exist.
Creates an application pool.
Assigns the application pool to the created website.
When it comes to assigning the application pool I get and error:
The configuration object is read only, because it has been committed
by a call to ServerManager.CommitChanges(). If write access is
required, use ServerManager to get a new reference.
This baffles me as it seems to suggest that I can't assign the newly created application pool with the ServerManager.CommitChanges() call. However, everything else works fine using this, which I wouldn't expect if this was an issue.
Here is my code:
I have a ServerManager instance created like so:
private ServerManager mgr = new ServerManager();
In my Install method I do the following:
Site site = CreateWebsite();
if (site != null)
{
CreateApplicationPool();
AssignAppPool(site);
}
Check existing websites - done in OnBeforeInstall method
private Site CheckWebsites()
{
SiteCollection sites = null;
Site site = null;
try
{
sites = mgr.Sites;
foreach (Site s in sites)
{
if (!string.IsNullOrEmpty(s.Name))
{
if (string.Compare(s.Name, targetSite, true) == 0) site = s;
}
}
}
catch{}
return site;
}
CreateWebSite method:
private Site CreateWebsite()
{
Site site = CheckWebsites();
if (site == null)
{
SiteCollection sites = mgr.Sites;
int port;
Int32.TryParse(targetPort, out port);
site = sites.Add(targetSite, targetDirectory, port);
mgr.CommitChanges();
}
else
{
//TO DO - if website already exists edit settings
}
return site;
}
Create App Pool
//non-relevant code...
ApplicationPool NewPool = mgr.ApplicationPools.Add(ApplicationPool);
NewPool.AutoStart = true;
NewPool.ManagedRuntimeVersion = "4.0";
NewPool.ManagedPipelineMode = ManagedPipelineMode.Classic;
mgr.CommitChanges();
Assign App Pool
private void AssignAppPool(Site site)
{
site.ApplicationDefaults.ApplicationPoolName = ApplicationPool; //ERRORS HERE
mgr.CommitChanges();
}
I can't see why a site could be created, an app pool created but then not assigned. Help.

I finally realised that the 'configuration object' referred to in the error was the 'site'. Seems obvious now, but basically I needed to re-get the site to then assign the app pool to it. I think this is allow the previous changes to take place and then pick them up. So I altered my code by removing the need to pass the Site into private void AssignAppPool() and just getting the site again like this:
Site site = mgr.Sites["TestWebApp"];

Why does counting messages on MSMQ returns 0 when using Powershell Method?

I'm developing a WCF service running on IIS and I need to count the messages i = for each private queue in MSMQ. The fastest way seems to be the powershell method.
The benchmark is here:
http://www.codeproject.com/Articles/346575/Message-Queue-Counting-Comparisions)
When debugging on Visual Studio 2012, it works great but when deployed on my local IIS 7.5 server, it returns 0.
Here is the method I'm using:
private int GetPowerShellCount()
{
return GetPowerShellCount(".\\private$\pingpong", Environment.MachineName, "", "");
}
private int GetPowerShellCount(string queuePath, string machine,string username, string password)
{
var path = string.Format(#"\\{0}\root\CIMv2", machine);
ManagementScope scope;
if (string.IsNullOrEmpty(username))
{
scope = new ManagementScope(path);
}
else
{
var options = new ConnectionOptions {Username = username, Password = password};
scope = new ManagementScope(path, options);
}
scope.Connect();
if (queuePath.StartsWith(".\\")) queuePath=queuePath.Replace(".\\",string.Format("{0}\\",machine));
string queryString = String.Format("SELECT * FROM Win32_PerfFormattedData_msmq_MSMQQueue");
var query = new ObjectQuery(queryString);
var searcher = new ManagementObjectSearcher(scope, query);
IEnumerable<int> messageCountEnumerable =
from ManagementObject queue in searcher.Get()
select (int)(UInt64)queue.GetPropertyValue("MessagesInQueue");
//IEnumerable<string> messageCountEnumerable =
// from ManagementObject queue in searcher.Get()
// select (string)queue.GetPropertyValue("Name");
var x = messageCountEnumerable.First();
return x;
}
please note that I'm not using the user/pass params so it's all local (WCF service and MSMQ on the same machine).
Why it's returning 0 when deployed to IIS?
What do you think I should try out?

Your problem might be related to IIS specific behavior.
Have a look at Tom Hollander's blog: MSMQ, WCF and IIS: Getting them to play nice (Part 1)
In general, message queues can be called whatever you want. However
when you are hosting your MSMQ-enabled service in IIS 7 WAS, the queue
name must match the URI of your service's .svc file. In this example
we'll be hosting the service in an application called MsmqService
with an .svc file called MsmqService.svc, so the queue must be
called MsmqService/MsmqService.svc.

Detect IIS website is suspended

I am currently able to detect whether a IIS Website is started/paused/stopped using the following code:
public int GetWebsiteStatus(string machineName, int websiteId)
{
DirectoryEntry root = new DirectoryEntry(
String.Format("IIS://{0}/W3SVC/{1}", machineName, websiteId));
PropertyValueCollection pvc = root.Properties["ServerState"];
return pvc.Value
// - 2: Website Started
// - 4: Website Stopped
// - 6: Website Paused
}
I also want to detect if a Website is suspended or not. If the Website is suspended the method above still returns 2 (which is correct) but not enough for me.
I cannot find any code which do the job for IIS6 and higher.

Ah, do you mean the App Pool as stopped because of the timeout configuration? This is a different state to the web site remember? Well, certainly, you could change the settings so it doesn't recycle, but you could also try using code like this;
First, add a reference to \Windows\System32\inetsrv\Microsoft.Web.Administration.dll, then;
using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.Web.Administration;
namespace MSWebAdmin_Application
{
class Program
{
static void Main(string[] args)
{
ServerManager serverManager = new ServerManager();
Site site = serverManager.Sites["Default Web Site"];
// get the app for this site
var appName = site.Applications[0].ApplicationPoolName;
ApplicationPool appPool = serverManager.ApplicationPools[appName];
Console.WriteLine("Site state is : {0}", site.State);
Console.WriteLine("App '{0}' state is : {1}", appName, appPool.State);
if (appPool.State == ObjectState.Stopped)
{
// do something because the web site is "suspended"
}
}
}
}
That code will independantly check the state of your appPool as opposed to your web site. It's possible for the web site to return "started" and the appPool to return "stopped".
See if it works in your case.

You might want to try using the following code, add your own logic and tidy up of course... but in essence you need to do the following and modify your code as you see fit.
Add the following enum
public enum ServerState
{
Unknown = 0,
Starting = 1,
Started = 2,
Stopping = 3,
Stopped = 4,
Pausing = 5,
Paused = 6,
Continuing = 7
}
Search for site and process it...
DirectoryEntry w3svc = new DirectoryEntry("IIS://" + "localhost" + "/W3SVC");
//check each site
foreach (DirectoryEntry site in w3svc.Children)
{
foreach (var s in site.Properties)
{
try
{
ServerState state =
(ServerState)
Enum.Parse(typeof (ServerState), site.Properties["ServerState"].Value.ToString());
if (state == ServerState.Paused)
{
//Do action
}
}
catch (Exception)
{
}
}
}
I hope this is useful for you as well...
http://csharp-tipsandtricks.blogspot.co.uk/2009_12_01_archive.html

How to start an Amazon EC2 instance programmatically in .NET

I have been attempting to start an instance of EC2 in C# without luck.
When passing in an instance id to start the instance I get an error that the instance cannot be found despite that I am passing in an instance ID that I have obtained from the object property.

Amazon made huge efforts to integrate its AWS Cloud .Net SDK To VS2008 & VS 2010
1 - Download and Install the AWS SDK msi
2 - Create an AWS Console project, enter your credentials (available from your AWS Console under your login name menu on the top right corner)
3 - Add the following code (see below images).
4 - Your're done. It's very straightforward. You can check the programmatic start/stop success by refreshing your AWS Console Screen.
AmazonEC2 ec2 = AWSClientFactory.CreateAmazonEC2Client();
//Start Your Instance
ec2.StartInstances(new StartInstancesRequest().WithInstanceId("i-00000000"));
//Stop it
ec2.StopInstances(new StopInstancesRequest().WithInstanceId("i-00000000"));
You just need to replace "i-00000000" by your instance Id (available in your AWS Management Console)
Hope this helps those googling this and stumbling upon this question (as I did myself) start off quickly. Following these simple steps via these wizards will spare you considerable headaches.

Try something like this with the AWSSDK to start new instances of an "image id":
RunInstancesResponse response = Client.RunInstances(new RunInstancesRequest()
.WithImageId(ami_id)
.WithInstanceType(instance_type)
.WithKeyName(YOUR_KEYPAIR_NAME)
.WithMinCount(1)
.WithMaxCount(max_number_of_instances)
.WithUserData(Convert.ToBase64String(Encoding.UTF8.GetBytes(bootScript.Replace("\r", ""))))
);
(Note: The .WithUserData() is optional and is used above to pass a short shell script.)
If the call is successful the response should contain a list of instances. You can use something like this to create a list of "instance ids":
if (response.IsSetRunInstancesResult() && response.RunInstancesResult.IsSetReservation() && response.RunInstancesResult.Reservation.IsSetRunningInstance())
{
List<string> instance_ids = new List<string>();
foreach (RunningInstance ri in response.RunInstancesResult.Reservation.RunningInstance)
{
instance_ids.Add(ri.InstanceId);
}
// do something with instance_ids
...
}

Be mindful that Amazon AWS instances exist only in one region. If your instance id i-12345 is in the EU-West-1 region, and you just make a new EC2Client and tell the client to start i-12345 it may well complain that it cannot find that instance, because the client started up in the us-east-1 region, which does not have i-12345 instance.
Your call that creates the client should specify the region, if it is not the default region (I've no idea which AWS region is default, so I specify every time):
AmazonEC2 ec2 = AWSClientFactory.CreateAmazonEC2Client(
new Amazon.EC2.AmazonEC2Config().WithServiceURL("https://eu-west-1.ec2.amazonaws.com")
);

Ok, this is the FULL, end-to-end instructions.
1. Install AWSSDK.Core and AWSSDK.EC2 using Nuget Package Manager.
2. Then copy this whole class to your project. AccessKey and Secret are obtained in AWS IAM. You will need to ensure the user you create has "AmazonEC2FullAccess" (You can probably use a lower-level permission policy, I am just lazy here :D). region is your AW S EC2 instance region. and Instance ID can be found in the EC2 dashboard list. Simple, works perfectly... You can also write extra code to manage the response object.
3. Be mindful if you are behind a proxy, you will have to configure it (I havent included code here).
public class AWSClass : IDisposable
{
Amazon.EC2.AmazonEC2Client _client;
public AWSClass(string region, string AccessKey, string Secret)
{
RegionEndpoint EndPoint = RegionEndpoint.GetBySystemName(region);
Amazon.Runtime.BasicAWSCredentials Credentials = new Amazon.Runtime.BasicAWSCredentials(AccessKey, Secret);
_client = new AmazonEC2Client(Credentials, EndPoint);
}
public void Dispose()
{
_client = null;
}
public void StopInstance(string InstanceID)
{
StopInstancesResponse response = _client.StopInstances(new StopInstancesRequest
{
InstanceIds = new List<string> {InstanceID }
});
//Can also do something with the response object too
}
public void StartInstance(string InstanceID)
{
StartInstancesResponse response = _client.StartInstances(new StartInstancesRequest
{
InstanceIds = new List<string> { InstanceID }
});
}
}

try this.
var startRequest = new StartInstancesRequest
{
InstanceIds = new List<string>() { instanceId }
};
bool isError = true;
StartInstancesResponse startInstancesResponse = null;
while (isError)
{
try
{
startInstancesResponse=amazonEc2client.StartInstances(startRequest);
isError = false;
}
catch (Exception ex)
{
Console.WriteLine(ex.Message + "\n" + ex.StackTrace);
isError = true;
}
}