Good afternoon,
I have only recently started working with Kafka and have a question about the producer in connection with the schema.
Initially I tried to build a simple producer without a schema in C#. This works so far, the code is also given in a shortened version.
Code of producer without schema:
var config = new ProducerConfig
{
BootstrapServers = "localhost:9092",
BrokerAddressFamily = BrokerAddressFamily.V4,
};
using (var producer = new ProducerBuilder<Null, string>(config).Build())
{
producer.Flush();
for(int i = 0; i < 3; i++)
{
producer.Produce("topic", new Message<Null, string> { Value = "Value: " + i + "..." });
}
producer.Flush();
};
But the schema causes me problems (see next section).
Suppose I have given a consumer, say in Python, who uses the following scheme, to receive integer numbers:
{"type": "record",
"name": "Sample",
"fields": [{"name": "Value", "type": ["int"]}]
}
I now want to create a C# producer that uses this scheme and send messages to the Python consumer. The message should contain only numbers, according to the scheme.
I tried to build a producer that uses the schema, but unfortunately in many tutorials the "schema registry url" is necessary to run the producer. This is what I have, but unfortunately I cannot avoid the use of "schema registry url"...
Code of producer that uses the schema
using Avro;
using Avro.Generic;
using Confluent.Kafka.SyncOverAsync;
using Confluent.SchemaRegistry.Serdes;
using Confluent.SchemaRegistry;
using System;
using System.Threading;
using System.Text;
using System.Threading.Tasks;
using System.Diagnostics;
using System.IO;
namespace producer
{
class Program
{
static async Task Main(string[] args)
{
//This is not further specified, nothing is set up here
string schemaRegistryUrl = "localhost:8081";
var schema = (RecordSchema)RecordSchema.Parse(
#"{
""type"": ""record"",
""name"": ""Sample"",
""fields"": [
{""name"": ""Value"", ""type"": [""int""]}
]
}"
);
var config = new Confluent.Kafka.ProducerConfig
{
BootstrapServers = "localhost:9092",
BrokerAddressFamily = BrokerAddressFamily.V4,
};
using (var schemaRegistry = new CachedSchemaRegistryClient(new SchemaRegistryConfig { Url = schemaRegistryUrl }))
using (var producer = new Confluent.Kafka.ProducerBuilder<Confluent.Kafka.Null, GenericRecord>(config)
.SetValueSerializer(new AvroSerializer<GenericRecord>(schemaRegistry))
.Build())
{
for(int i = 0; i < 3; i++)
{
var record = new GenericRecord(schema);
record.Add("Value", i);
producer.ProduceAsync("topic", new Confluent.Kafka.Message<Confluent.Kafka.Null, GenericRecord> { Value = record });
}
producer.Flush();
}
}
}
}
Here are two questions, how can one build a producer without the "schema registry url"? I have already found something like this (but unfortunately in Java). How would it look like in C#? I'm interested in a solution from a producer that sends numbers to the Python consumer using the scheme (from above), preferably without using the "schema registry url". However, I would also be interested in how to get the "schema registry url" to work.
Just as a hint: If the producer tries to send a message to the Python consumer without a schema, the consumer registers this. But the consumer cannot display the sent number, because the simple producer does not use a scheme. I refer to the code of the very first producer!
I hope that my question(s) are as far as understandable, I am looking forward to receiving answers!
The confluent_kafka Python library requires the data adheres to the Confluent Schema Registry wire format.
This means, you're required to have the producer match that contract as well, so you cannot bypass the registry without writing your own implementation of the new AvroSerializer you've referenced in the code
If the producer tries to send a message to the Python consumer without a schema, the consumer registers this
This doesn't happen in confluent_kafka library. The consumer fetches a schema from the registry based on the ID emedded in the message.
Therefore, unless you're using some other deserializer, you'll need to write your own implementation there, as well
Related
For a project I'm working on I have to get a Pull Request and Repository from an Azure DevOps WorkItem ID.
I'm using the Microsoft.TeamFoundationServer.Client NuGet-Package for this.
Now i also want to be able to check if a build pipeline ran successfully before moving on to further steps.
After trying to figure it out myself and not finding a single article on how to do that, I'm just gonna ask the question myself.
So, I already have:
the WorkItem object
the GitRepository object
the PullRequest object
And I want:
some form of pipeline object of that specific Pull Request/Commit
I hope there even is a way to get that.
Any help or references are apprechiated. Thanks!
I'm working on I have to get a Pull Request and Repository from an
Azure DevOps WorkItem ID.
For pull request that related to work item, I can write a C# code for you. But for the repository, I think there doesn't have a relationship between repository itself and work item in DevOps concept(link commit and work item is possible.).
Now i also want to be able to check if a build pipeline ran
successfully before moving on to further steps.
Do you mean you want the pipeline run status related to pull request? I checked the sdk definition, there doesn't have such definition, also no in the REST API. A possible solution is following the f12 to capture the API to get the build pipeline run id and it status.
Just a demo:
using System;
using System.Threading.Tasks;
using Microsoft.TeamFoundation.Build.WebApi;
using Microsoft.TeamFoundation.SourceControl.WebApi;
using Microsoft.TeamFoundation.WorkItemTracking.WebApi;
using Microsoft.TeamFoundation.WorkItemTracking.WebApi.Models;
using Microsoft.VisualStudio.Services.Common;
using Microsoft.VisualStudio.Services.WebApi;
namespace GetPipelineResults
{
class Program
{
static void Main(string[] args)
{
Console.WriteLine("Hello World!");
string url_string = "https://dev.azure.com/xxx/";
string personalAccessToken = "xxx";
Uri orgUrl = new Uri(url_string);
string project = "xxx";
int workitemId = 122;
var workitem = GetPullRequestAndRepositoryFromWorkItemId(orgUrl,personalAccessToken,workitemId);
var pullRequestUrl = workitem.Result.Relations[0].Url.ToString();
var pullRequestUrl2 = pullRequestUrl.Substring(pullRequestUrl.LastIndexOf('/') + 1);
string[] pullRequestUrl2Array = pullRequestUrl2.Split("%2F");
string pullRequestIdString = pullRequestUrl2Array[pullRequestUrl2Array.Length - 1];
Console.WriteLine(pullRequestIdString);
}
//Get Pull request from work item id
public static async Task<WorkItem> GetPullRequestAndRepositoryFromWorkItemId(Uri orgUrl, string personalAccessToken, int workItemId)
{
VssConnection connection = new VssConnection(orgUrl, new VssBasicCredential(string.Empty, personalAccessToken));
WorkItemTrackingHttpClient workItemTrackingHttpClient = connection.GetClient<WorkItemTrackingHttpClient>();
WorkItemExpand workItemExpand = WorkItemExpand.All;
var workItem = workItemTrackingHttpClient.GetWorkItemAsync(workItemId, expand: workItemExpand).Result;
return workItem;
}
}
}
I am trying to find the best way to see the last date a subscription in a topic was accessed via c# (SDK or otherwise) i.e. to purge the queue if not accessed in over x hours. I know there is that functionality built into the service bus explorer but have not been able to find any SDK functionality. If anyone could point me in the right direction it would be appreciated.
Please see the code below. It uses Azure.Messaging.ServiceBus SDK. The properties you're interested in is available in SubscriptionRuntimeProperties class.
using System;
using System.Threading.Tasks;
using Azure.Messaging.ServiceBus.Administration;
namespace ConsoleApp1
{
class Program
{
static async Task Main(string[] args)
{
string connectionString =
"connection-string";
string topicName = "topic-name";
string subscriptionName = "subscription-name";
ServiceBusAdministrationClient administrationClient = new ServiceBusAdministrationClient(connectionString);
var result = await administrationClient.GetSubscriptionRuntimePropertiesAsync(topicName, subscriptionName);
Console.WriteLine(result.Value.AccessedAt.ToString("yyyy-MM-ddTHH:mm:ss"));
}
}
}
If I send an Akka.NET actor a message which is an object containing an IActorRef, and then persist that message, the JSON written to the journal table looks like this:
{"$id":"1","$type":"LearningAkka.Program+BindReference, LearningAkka","Reference":{"$id":"2","$type":"Akka.Actor.ActorRefBase+Surrogate, Akka","Path":"akka://LearningAkka/user/$b#1222898859"}}
If I'm understanding this right, this is just a reference to an actor instance; the "Props" required to create it are not stored in this message.
Weirdly, I am seeing an object there after restarting the app. However, as expected, it is not as constructed before the restart. Where did this actor come from? Has Akka Persistence found an actor which is "similar enough" and used it instead?
The following C# test application creates an object and sends a message binding it to one of three others. After disposing of the actor system, that object is recreated from persistence (SQL Server) and the reference is checked.
My expected behaviour is any of the following (I'm not sure what's most appropriate):
The actor can't be created because one of its messages contains an unresolvable reference.
The actor reference is null because it cannot be resolved.
The actor reference points to dead letters or similar.
Console output:
[WARNING][27/05/2017 21:02:27][Thread 0001][ActorSystem(LearningAkka)] NewtonSoftJsonSerializer has been detected as a default serializer. It will be obsoleted in Akka.NET starting from version 1.5 in the favor of Hyperion (for more info visit: http://getakka.net/docs/Serialization#how-to-setup-hyperion-as-default-serializer ). If you want to suppress this message set HOCON `akka.suppress-json-serializer-warning` config flag to on.
From the first run B
[WARNING][27/05/2017 21:02:28][Thread 0001][ActorSystem(LearningAkka)] NewtonSoftJsonSerializer has been detected as a default serializer. It will be obsoleted in Akka.NET starting from version 1.5 in the favor of Hyperion (for more info visit: http://getakka.net/docs/Serialization#how-to-setup-hyperion-as-default-serializer ). If you want to suppress this message set HOCON `akka.suppress-json-serializer-warning` config flag to on.
From the second run B
C#:
using Akka.Actor;
using Akka.Event;
using Akka.Persistence;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace LearningAkka
{
class Program
{
static void Main(string[] args)
{
using (var actorSystem = ActorSystem.Create("LearningAkka"))
{
var referenceA = actorSystem.ActorOf(Props.Create(() => new TestReferencedActor("From the first run A")));
var referenceB = actorSystem.ActorOf(Props.Create(() => new TestReferencedActor("From the first run B")));
var referenceC = actorSystem.ActorOf(Props.Create(() => new TestReferencedActor("From the first run C")));
var actor = actorSystem.ActorOf(Props.Create(() => new TestActor()));
actor.Tell(new BindReference { Reference = referenceB });
actor.Tell(new CheckReference());
Console.ReadLine();
}
using (var actorSystem = ActorSystem.Create("LearningAkka"))
{
var referenceA = actorSystem.ActorOf(Props.Create(() => new TestReferencedActor("From the second run A")));
var referenceB = actorSystem.ActorOf(Props.Create(() => new TestReferencedActor("From the second run B")));
var referenceC = actorSystem.ActorOf(Props.Create(() => new TestReferencedActor("From the second run C")));
var actor = actorSystem.ActorOf(Props.Create(() => new TestActor()));
actor.Tell(new CheckReference());
Console.ReadLine();
}
}
public struct BindReference { public IActorRef Reference; }
public struct CheckReference { }
public sealed class TestActor : ReceivePersistentActor
{
public override string PersistenceId => "test hardcoded";
private IActorRef StoredFromMessage;
public TestActor()
{
Command<CheckReference>(m => StoredFromMessage.Tell(m));
Command<BindReference>(m => Persist(m, m2 => StoredFromMessage = m2.Reference));
Recover<BindReference>(m => StoredFromMessage = m.Reference);
}
}
public sealed class TestReferencedActor : ReceiveActor
{
public TestReferencedActor(string ourLabel)
{
Receive<CheckReference>(m => Console.WriteLine(ourLabel));
}
}
}
}
HOCON:
akka {
persistence {
journal {
plugin = "akka.persistence.journal.sql-server"
sql-server {
class = "Akka.Persistence.SqlServer.Journal.SqlServerJournal, Akka.Persistence.SqlServer"
connection-string = "Data Source=(localdb)\\MSSQLLocalDB;Initial Catalog=LearningAkka;Integrated Security=True;Connect Timeout=30;Encrypt=False;TrustServerCertificate=True;ApplicationIntent=ReadWrite;MultiSubnetFailover=False"
schema-name = dbo
table-name = Journal
auto-initialize = on
}
}
snapshot-store {
plugin = "akka.persistence.snapshot-store.sql-server"
sql-server {
class = "Akka.Persistence.SqlServer.Snapshot.SqlServerSnapshotStore, Akka.Persistence.SqlServer"
connection-string = "Data Source=(localdb)\\MSSQLLocalDB;Initial Catalog=LearningAkka;Integrated Security=True;Connect Timeout=30;Encrypt=False;TrustServerCertificate=True;ApplicationIntent=ReadWrite;MultiSubnetFailover=False"
schema-name = dbo
table-name = Snapshot
auto-initialize = on
}
}
}
}
Could someone please comment on the behaviour here? Thank you.
As you can see from serialization data - your IActorRef points to this address akka://LearningAkka/user/$b. Where $b is usually placed for unnamed actors. So it will always be the second unnamed actor you create in the actor system root (as far as I know).
So you are right - the system behavior is undefined here.
I am trying to connect mongolab to an application in Unity, but I have a problem in this line:
async static Task AsyncCrud(BsonDocument[] seedData)
The problem is here:
I wonder if someone can help me to solve this problem or suggest me another way to create the connection.
This is all the code:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using MongoDB.Bson;
using MongoDB.Driver;
/*
* Copyright (c) 2015 ObjectLabs Corporation
* Distributed under the MIT license - http://opensource.org/licenses/MIT
*
* Written with CSharpDriver-2.0.0
* Documentation: http://api.mongodb.org/csharp/
* A C# class connecting to a MongoDB database given a MongoDB Connection URI.
*/
namespace MongoLab
{
class Simple
{
// Extra helper code
static BsonDocument[] CreateSeedData()
{
BsonDocument seventies = new BsonDocument {
{ "Decade" , "1970s" },
{ "Artist" , "Debby Boone" },
{ "Title" , "You Light Up My Life" },
{ "WeeksAtOne" , 10 }
};
BsonDocument eighties = new BsonDocument {
{ "Decade" , "1980s" },
{ "Artist" , "Olivia Newton-John" },
{ "Title" , "Physical" },
{ "WeeksAtOne" , 10 }
};
BsonDocument nineties = new BsonDocument {
{ "Decade" , "1990s" },
{ "Artist" , "Mariah Carey" },
{ "Title" , "One Sweet Day" },
{ "WeeksAtOne" , 16 }
};
BsonDocument[] SeedData = { seventies, eighties, nineties };
return SeedData;
}//end create
async static Task AsyncCrud(BsonDocument[] seedData)
{
// Create seed data
BsonDocument[] songData = seedData;
// Standard URI format: mongodb://[dbuser:dbpassword#]host:port/dbname
String uri = "mongodb://usuario:pass#server.mongolab.com:puerto/basededatos";
var client = new MongoClient(uri);
var db = client.GetDatabase("learnygames");//debe de ser la base de datos existente
Console.WriteLine(db);
Console.WriteLine(client.GetDatabase("db"));
/*
* First we'll add a few songs. Nothing is required to create the
* songs collection; it is created automatically when we insert.
*/
var songs = db.GetCollection<BsonDocument>("songs");
// Use InsertOneAsync for single BsonDocument insertion.
await songs.InsertManyAsync(songData);
Console.WriteLine("songs ********************************************");
/*
* Then we need to give Boyz II Men credit for their contribution to
* the hit "One Sweet Day".
*/
var updateFilter = Builders<BsonDocument>.Filter.Eq("Title", "One Sweet Day");
var update = Builders<BsonDocument>.Update.Set("Artist", "Mariah Carey ft. Boyz II Men");
await songs.UpdateOneAsync(updateFilter, update);
/*
* Finally we run a query which returns all the hits that spent 10
* or more weeks at number 1.
*/
var filter = Builders<BsonDocument>.Filter.Gte("WeeksAtOne", 10);
var sort = Builders<BsonDocument>.Sort.Ascending("Decade");
await songs.Find(filter).Sort(sort).ForEachAsync(song =>
Console.WriteLine("In the {0}, {1} by {2} topped the charts for {3} straight weeks",
song["Decade"], song["Title"], song["Artist"], song["WeeksAtOne"])
);
// Since this is an example, we'll clean up after ourselves.
// await db.DropCollectionAsync("songs");
}
static void Main()
{
BsonDocument[] seedData = CreateSeedData();
AsyncCrud(seedData).Wait();
}
}//end class conexxion
}//end name space
Unity doesnt support the async keyword since it uses .Net 3.5.
Take a look here.
You can do async operations with coroutines
Edit: I have learnt that Async isn't supported in Unity as another user has an highlighted as an answer.
Update
I was also trying to implement an async call in unity which brought my search to this question. Further research I discovered that Async was not supported but I also had found a github project by Real Serious Games that implement promises which were their answer to implementing async in unity.
https://github.com/Real-Serious-Games/C-Sharp-Promise
I first discovered their promise library from the unity knight's blog
http://blog.theknightsofunity.com/coroutines-unity-encapsulating-promises-part-1/
They have a 3/4 part article going over the basics and there is also a detail blog by Real Serious Games about the library on their blog.
I am now doing async by attaching the promise object to my unity coroutines to for async functionality.
Original Answer
The error you've pictured is basically saying put the keyword static before async.
static async Task AsyncCrud(BsonDocument[] seedData){}
you may need to Template the Task type with your actual return type
static async Task<T> AsyncCrud(BsonDocument[] seedData){}
I Know I'm able to pipe out/in using simple Console.WriteLine and Console.ReadLine methods, but that way I'm passing a string between processes (which must be parsed to recreate the object).
What I'm wondering is if I would be able to pipe my own types, so that I could retrieve them easily in destiny process. What I expect is to do something like:
myProgram | get-member
And the output would something like MyNameSpace.MyType and the list of its members (currently it shows the typeName System.String)
Is that possible in a console app or could I only achieve this using cmdlets?
The easiest way to do this is to use serialization to turn the objects you wish to send from one to the other into a pipeable format to send them from one to the other. There are, however, a number of constraints on doing this:
First, the implementation of the types you're passing back and forth have to be available to all the apps that may handle them. (That's not a problem for PowerShell because all the cmdlets run inside the same process.) So the easiest way to do this is to create the types you're going to pipe around inside a class library that's referenced by all the console apps. This class, for example, I put in my sample shared library:
[Serializable]
public class TestClass
{
public string Test { get; set; }
public string TestAgain { get; set; }
public string Cheese { get; set; }
}
The [Serializable] attribute marks it as serializable, which is sufficient for simple classes. For more complex classes, more may be required - see MSDN, starting here: http://msdn.microsoft.com/en-us/library/4abbf6k0(v=VS.71).aspx
Then, in the program you're piping from, you serialize it to XML and write it out to console like this:
using System;
using System.IO;
using System.Text;
using System.Xml.Serialization;
using Shared;
namespace Out
{
class Program
{
static void Main(string[] args)
{
// Create the object.
TestClass test = new TestClass();
test.Test = "Monkey";
test.TestAgain = "Hat";
test.Cheese = "Fish";
// Serialize it.
XmlSerializer serializer = new XmlSerializer(typeof (TestClass));
StringBuilder sb = new StringBuilder();
using (var writer = new StringWriter(sb))
serializer.Serialize(writer, test);
// And write it to console.
Console.WriteLine(sb.ToString());
}
}
}
When run, this outputs the instance's properties encoded in XML, thus:
<?xml version="1.0" encoding="utf-16"?>
<TestClass xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http
://www.w3.org/2001/XMLSchema">
<Test>Monkey</Test>
<TestAgain>Hat</TestAgain>
<Cheese>Fish</Cheese>
</TestClass>
Then, in your receiving application, you reverse the process, reading from the console, thus:
using System;
using System.IO;
using System.Xml.Serialization;
using Shared;
namespace In
{
class Program
{
static void Main(string[] args)
{
// Read the input XML; until complete.
string input = Console.In.ReadToEnd();
TestClass passedIn;
// Deserialize it.
var serializer = new XmlSerializer(typeof (TestClass));
using (var reader = new StringReader(input))
passedIn = (TestClass) serializer.Deserialize(reader);
// Do something with the object.
Console.WriteLine("Test: {0}", passedIn.Test);
Console.WriteLine("TestAgain: {0}", passedIn.TestAgain);
Console.WriteLine("Cheese: {0}", passedIn.Cheese);
}
}
}
And voila!
C:\Working\PipeExample\In\bin\Debug>..\..\..\Out\bin\Debug\Out.exe | in
Test: Monkey
TestAgain: Hat
Cheese: Fish
You'll need some additional code, of course, to make sure that the receiving application knows what type(s) to expect - or can handle anything it gets - and since the intermediate XML is not very human-parsable, you'll need a way to make sure that the sending application knows when it's talking to a pipe and when it's talking to a human. In .NET 4.5, the Console.IsOutputRedirected() method will do that for you ( http://msdn.microsoft.com/en-us/library/system.console.isoutputredirected%28v=VS.110%29.aspx ), but in earlier versions, there's not an easy way to get at this information programmatically.
But this is the core of the thing, and looking at the documentation for and around XmlSerializer should give you the rest.
Why don't you write your own cmdlet instead of a console program?
A PowerShell module can be a binary module (a DLL assembly) composed by cmdlets writen in C#. Have a look to Installing the Windows PowerShell SDK.