Develop Reference

Azure Devops - Get release definitions by agent pool ID

I'm trying to find all the builds and releases that are configured to use a specific agent pool, using the .NET Client Libraries.
Assuming agentPoolId, I can get all the build definitions like this:
// _connection is of type VssConnection
using (var buildClient = _connection.GetClient<BuildHttpClient>())
{
List<BuildDefinitionReference> allBuilds = await buildClient.GetDefinitionsAsync(projectName, top: 1000, queryOrder: DefinitionQueryOrder.DefinitionNameAscending);
List<BuildDefinitionReference> builds = allBuilds.Where(x => HasAgentPoolId(x, agentPoolId)).ToList();
}
private bool HasAgentPoolId(BuildDefinitionReference buildDefinition, int agentPoolId)
{
TaskAgentPoolReference pool = buildDefinition?.Queue?.Pool;
if (pool == null)
{
return false;
}
return pool.Id.Equals(agentPoolId);
}
But I couldn't find a way to find the release definitions that have one or more environments configured to use a particular agent. Any suggestion?

I was manged to get all releases by Agent Pool ID via Rest Api and not via NET Client Libraries.Hope that helps.
C# Code snippet:
public class ReleaseResponse
{
[JsonProperty("value")]
public List<ReleaseItem> Value { get; set; }
}
public class ReleaseItem
{
[JsonProperty("name")]
public string Name { get; set; }
[JsonProperty("Id")]
public int Id { get; set; }
}
static void Main(string[] args)
{
string tfsURL = "TFS URL";
string releaseDefurl = $"{tfsURL}/_apis/release/definitions?$expand=artifacts&api-version=3.2-preview.3";
const int agentPoolID = "AGENT Pool ID";
List<string> relevantReleases = new List<string>();
WebClient client = new WebClient();
client.UseDefaultCredentials = true;
client.Headers.Add("Content-Type", "application/json");
var releaseList = client.DownloadString(releaseDefurl);
var allReleases = JsonConvert.DeserializeObject<ReleaseResponse>(releaseList).Value;
foreach (var release in allReleases)
{
string releaseInfoApi = $"{tfsURL}/_apis/Release/definitions/{release.Id}";
var getReleseInfo = client.DownloadString(releaseInfoApi);
var releaseInfo = JsonConvert.DeserializeObject<TFSLogic.RootObject>(getReleseInfo);
var deploymentAgents = releaseInfo.environments.ToList().Where(e => e.deployPhases.FirstOrDefault().deploymentInput.queueId == agentPoolID).Count();
if (deploymentAgents > 0)
{
relevantReleases.Add(release.Name);
}
}
}
Find TFSLogic here : https://codebeautify.org/online-json-editor/cb7aa0d9
Powershell Code snippet:
$tfsUrl = "TFS URL"
$releaseDefurl = $tfsUrl + '/_apis/release/definitions?$expand=artifacts&api-version=3.2-preview.3'
$agentPoolID = "Agent Pool ID"
$relevantReleases = #();
$allReleasesID = (Invoke-RestMethod -Uri ($releaseDefurl) -Method Get -UseDefaultCredentials).value.id
function getReleaseByAgentPoolID($releaseID,$agentPoolID)
{
$ReleaseInfo = Invoke-RestMethod -Uri "$tfsUrl/_apis/Release/definitions/$releaseID" -Method Get -UseDefaultCredentials
$deploymentAgents = $ReleaseInfo.environments | % {$_.deployPhases.deploymentInput.queueId} | where {$_ -eq $agentPoolID}
if($deploymentAgents.Count -gt 0)
{
return $ReleaseInfo.name
}
}
foreach ($releaseID in $allReleasesID)
{
$relevantReleases += getReleaseByAgentPoolID -releaseID $releaseID -agentPoolID $agentPoolID
}
UPDATE :
It took me some time,But i was able to achieve that with azure-devops-dotnet-samples
I hope this example is finally what you are looking for.
using Microsoft.VisualStudio.Services.WebApi;
using System;
using System.Linq;
using Microsoft.VisualStudio.Services.ReleaseManagement.WebApi.Clients;
using Microsoft.VisualStudio.Services.ReleaseManagement.WebApi.Contracts;
using Microsoft.VisualStudio.Services.Common;
using System.Collections.Generic;
namespace FindReleaseByAgentPoolID
{
class Program
{
const int agentPoolID = 999;
static void Main(string[] args)
{
var relevantReleases = new List<string>();
VssCredentials c = new VssCredentials(new WindowsCredential(System.Net.CredentialCache.DefaultNetworkCredentials));
var tfsURL = new Uri("TFS URL");
var teamProjectName = "PROJECT";
using (var connection = new VssConnection(tfsURL, c))
using (var rmClient = connection.GetClient<ReleaseHttpClient2>())
{
var releases = rmClient
.GetReleaseDefinitionsAsync(teamProjectName, string.Empty, ReleaseDefinitionExpands.Environments)
.Result.ToArray();
foreach (var release in releases)
{
var r = rmClient.GetReleaseDefinitionAsync(teamProjectName, release.Id);
var deploymentAgents = r.Result.Environments.SelectMany(e =>
e.DeployPhases.Select(dp =>
dp.GetDeploymentInput()).Cast<DeploymentInput>()).Where(di =>
di.QueueId == agentPoolID).Count();
if (deploymentAgents > 0)
{
relevantReleases.Add(release.Name);
}
}
}
}
}
}

Found a solution, many thanks to #amit-baranes for pointing me in the right direction.
I've changed his code sample to use the await keyword instead of using .Result, and use .OfType<DeploymentInput>() instead of .Cast<DeploymentInput>() (it was throwing some exceptions).
Oh, and the most important thing I've learned: agent pool ID and queue ID are different things!!! If you intend to use the agent pool ID to get the release definitions you'll need to get the correspondent agent queue.
Code sample:
// set agent pool Id and project name
int agentPoolId = 123456;
string teamProjectName = ".....";
// _connection is of type VssConnection
using (var taskAgentClient = _connection.GetClient<TaskAgentHttpClient>())
using (var releaseClient = _connection.GetClient<ReleaseHttpClient2>())
{
// please note: agent pool Id != queue Id
// agent pool id is used to get the build definitions
// queue Id is used to get the release definitions
TaskAgentPool agentPool = await taskAgentClient.GetAgentPoolAsync(agentPoolId);
List<TaskAgentQueue> queues = await taskAgentClient.GetAgentQueuesByNamesAsync(teamProjectName, queueNames: new[] { agentPool.Name });
TaskAgentQueue queue = queues.FirstOrDefault();
List<ReleaseDefinition> definitions = await releaseClient.GetReleaseDefinitionsAsync(teamProjectName, string.Empty, ReleaseDefinitionExpands.Environments);
foreach (ReleaseDefinition definition in definitions)
{
var fullDefinition = await releaseClient.GetReleaseDefinitionAsync(teamProjectName, definition.Id);
bool hasReleasesWithPool = fullDefinition.Environments.SelectMany(GetDeploymentInputs)
.Any(di => di.QueueId == queue.Id);
if (hasReleasesWithPool)
{
Debug.WriteLine($"{definition.Name}");
}
}
}
private IEnumerable<DeploymentInput> GetDeploymentInputs(ReleaseDefinitionEnvironment environment)
{
return environment.DeployPhases.Select(dp => dp.GetDeploymentInput())
.OfType<DeploymentInput>();
}

Renew cridentials for AmazonKinesisClient using AWSSDK

Im creating a on-prem application in .Net that consumes a kinesis stream.
How do I write a method that can renew the cridentials for my KinesisClient (They are only valid for 1 hour) without re-create the client itself. Is that possible? Or am I approching this the wrong way?
This is the factory class that im using to produce the client.
public static AmazonKinesisClient CreateKinesisClient(BasicAWSCredentials credentials, string roleArn, string roleSessionName, RegionEndpoint region, LogWriter logWriter)
{
try
{
var stsClient = new AmazonSecurityTokenServiceClient(credentials);
var ownerRoleReq = new AssumeRoleRequest
{
RoleArn = roleArn,
RoleSessionName = roleSessionName
};
var ownerRoleResp = stsClient.AssumeRoleAsync(ownerRoleReq).Result;
logWriter.LogInfo<AmazonKinesisClient>("Role taken:" + ownerRoleResp.AssumedRoleUser);
return new AmazonKinesisClient(ownerRoleResp.Credentials, region);
}
catch (Exception e)
{
logWriter.LogError<AmazonClientFactory>(e.Message,e);
throw;
}
}

I ended up with saving the latest record id in a concurrent dictionary and while re-creating the client injecting the dictionary containing the latest id for each shard in the KinesisStream.
private string CreateShardIteratorRequestWithSequenceNumber(Shard shard, string sequenceNumber)
{
var iteratorRequest = new GetShardIteratorRequest()
{
ShardId = shard.ShardId,
StreamName = _streamName,
ShardIteratorType = ShardIteratorType.AFTER_SEQUENCE_NUMBER,
StartingSequenceNumber = sequenceNumber
};
return GetShardIterator(iteratorRequest);
}

Nethereum C# automated ether transfer

I wish to automate the transfer of ether to a list of people.
Assume the list is in a csv.
I wrote some code to automate the process.
class Program
{
int nonce = 0;
static void Main(string[] args)
{
var account = SetupAccount();
var recipients = ReadCsv();
var web3Init = GetConnection();
nonce = web3.Eth.Transactions.GetTransactionCount.SendRequestAsync(account.Address).Result;
//var recipients = new List<Records>() { new Records() { Value = 10000000000000000, Address = "0x5CC494843e3f4AC175A5e730c300b011FAbF2cEa" } };
foreach (var recipient in recipients)
{
try
{
var web3 = GetConnection();
var receipt = SendEther(account, recipient, web3).Result;
}
catch (System.Exception)
{
MessageBox.Show("Failed");
}
Thread.Sleep(30000);
}
}
private static async Task<TransactionReceipt> SendEther(Account account, Records recipient, Web3 web3)
{
var transactionPolling = web3.TransactionManager.TransactionReceiptService;
//var currentBalance = await web3.Eth.GetBalance.SendRequestAsync(account.Address);
//assumed client is mining already
//when sending a transaction using an Account, a raw transaction is signed and send using the private key
return await transactionPolling.SendRequestAndWaitForReceiptAsync(() =>
{
var transactionInput = new TransactionInput
{
From = account.Address,
//Gas = new HexBigInteger(25000),
GasPrice = new HexBigInteger(10 ^ 10),
To = recipient.Address,
Value = new HexBigInteger(new BigInteger(recipient.Value)),
Nonce = nonce
};
var txSigned = new Nethereum.Signer.TransactionSigner();
var signedTx = txSigned.SignTransaction(account.PrivateKey, transactionInput.To, transactionInput.Value, transactionInput.Nonce);
var transaction = new Nethereum.RPC.Eth.Transactions.EthSendRawTransaction(web3.Client);
nonce++;
return transaction.SendRequestAsync(signedTx);
});
}
private static Web3 GetConnection()
{
return new Web3("https://mainnet.infura.io");
}
private static Account SetupAccount()
{
var password = "#Password";
var accountFilePath = #"filePath";
return Account.LoadFromKeyStoreFile(accountFilePath, password);
}
private static List<Records> ReadCsv()
{
string filePath = #"C:\Users\Potti\source\repos\ConversionFiles\XrcfRecipients.csv";
if (File.Exists(filePath))
{
using (StreamReader stream = new StreamReader(filePath))
{
CsvReader reader = new CsvReader(stream, new Configuration
{
TrimOptions = TrimOptions.Trim,
HasHeaderRecord = true,
HeaderValidated = null
});
reader.Configuration.RegisterClassMap<RecordMapper>();
return reader.GetRecords<Records>().ToList();
}
}
else
{
return null;
}
}
}
class Records
{
public string Address { get; set; }
public decimal Value { get; set; }
}
sealed class RecordMapper : ClassMap<Records>
{
public RecordMapper()
{
Map(x => x.Address).Name("Address");
Map(x => x.Value).Name("Value");
}
}
How do i modify the process to execute all the transactions at once instead of waiting for each to complete? (Fire and forget)
Also, are there any security considerations of doing this?

What you are currently doing is waiting for each transaction to be mined. What you can do is the following:
var account = new Account("privateKey"); // or load it from your keystore file as you are doing.
var web3 = new Web3(account, "https://mainnet.infura.io");
First create a web3 instance using the same Account object, because we are using an account with a private key, Nethereum will sign your transactions offline before sending them.
Now using the TransactionManager, you can then send a transaction per each recepient
var transactionHashes = new List<string>();
foreach(var recepient in recepients){
var transactionInput = new TransactionInput
{
From = account.Address,
GasPrice = Web3.Convert.ToWei(1.5, UnitConversion.EthUnit.Gwei);,
To = recipient.Address,
Value = new HexBigInteger(new BigInteger(recipient.Value)),
};
var transactionHash = web3.Eth.TransactionManager.SendTransactionAsync(transactionInput);
transanctionHashes.Add(transactionHash);
}
Note that when Nethereum uses the same instance of an Account and TransactionManager (or Web3 in this scenario) it creates a default NonceMemoryService, so you don't need to keep track of your nonces (Transaction number), to sign the transaction.
Also I have done a conversion for the GasPrice from Gwei to Wei, as an example of Unit conversions, I assume that you already have converted to Wei the Ether amounts you are going to send.
Finally, another note, to further simplify this, there is an upcoming EtherTransferService which allows you to input Ether amounts and Gwei price amounts to avoid doing conversions. Also the gas price will be calculated for you, if not passed any parameter.
web3.Eth.GetEtherTransferService().TransferEtherAsync("toAddress", EtherAmount);

How do I optimize calculating the hash of thousands of files?

I have a .Net program that runs through a directory containing tens of thousands of relatively small files (around 10MB each), calculates their MD5 hash and stores that data in an SQLite database. The whole process works fine, however it takes a relatively long time (1094353ms with around 60 thousand files) and I'm looking for ways to optimize it. Here are the solutions I've thought of:
Use additional threads and calculate the hash of more than one file simultaneously. Not sure how I/O speed would limit me with this one.
Use a better hashing algorithm. I've looked around and the one I'm currently using seems to be the fastest one (on C# at least).
Which would be the best approach, and are there any better ones?
Here's my current code:
private async Task<string> CalculateHash(string file, System.Security.Cryptography.MD5 md5) {
Task<string> MD5 = Task.Run(() =>
{
{
using (var stream = new BufferedStream(System.IO.File.OpenRead(file), 1200000))
{
var hash = md5.ComputeHash(stream);
var fileMD5 = string.Concat(Array.ConvertAll(hash, x => x.ToString("X2")));
return fileMD5;
}
};
});
return await MD5;
}
public async Main() {
using (var md5 = System.Security.Cryptography.MD5.Create()) {
foreach (var file in Directory.GetFiles(path)) {
var hash = await CalculateHash(file, md5);
// Adds `hash` to the database
}
}
}

Create a pipeline of work, the easiest way I know how to create a pipeline that uses both parts of the code that must be single threaded and parts that must be multi-threaded is to use TPL Dataflow
public static class Example
{
private class Dto
{
public Dto(string filePath, byte[] data)
{
FilePath = filePath;
Data = data;
}
public string FilePath { get; }
public byte[] Data { get; }
}
public static async Task ProcessFiles(string path)
{
var getFilesBlock = new TransformBlock<string, Dto>(filePath => new Dto(filePath, File.ReadAllBytes(filePath))); //Only lets one thread do this at a time.
var hashFilesBlock = new TransformBlock<Dto, Dto>(dto => HashFile(dto),
new ExecutionDataflowBlockOptions{MaxDegreeOfParallelism = Environment.ProcessorCount, //We can multi-thread this part.
BoundedCapacity = 50}); //Only allow 50 byte[]'s to be waiting in the queue. It will unblock getFilesBlock once there is room.
var writeToDatabaseBlock = new ActionBlock<Dto>(WriteToDatabase,
new ExecutionDataflowBlockOptions {BoundedCapacity = 50});//MaxDegreeOfParallelism defaults to 1 so we don't need to specifiy it.
//Link the blocks together.
getFilesBlock.LinkTo(hashFilesBlock, new DataflowLinkOptions {PropagateCompletion = true});
hashFilesBlock.LinkTo(writeToDatabaseBlock, new DataflowLinkOptions {PropagateCompletion = true});
//Queue the work for the first block.
foreach (var filePath in Directory.EnumerateFiles(path))
{
await getFilesBlock.SendAsync(filePath).ConfigureAwait(false);
}
//Tell the first block we are done adding files.
getFilesBlock.Complete();
//Wait for the last block to finish processing its last item.
await writeToDatabaseBlock.Completion.ConfigureAwait(false);
}
private static Dto HashFile(Dto dto)
{
using (var md5 = System.Security.Cryptography.MD5.Create())
{
return new Dto(dto.FilePath, md5.ComputeHash(dto.Data));
}
}
private static async Task WriteToDatabase(Dto arg)
{
//Write to the database here.
}
}
This creates a pipeline with 3 segments.
One that is single threaded that reads the files from the hard drive in to memory and stored as a byte[].
A second one that can use up to Enviorement.ProcessorCount threads to hash the files, it will only allow 50 items to be sitting on it's inbound queue, when the first block tries to add it will stop processing new items until the next block is ready to accept new items.
And a third one that is single threaded and adds the data to the database, it allows only 50 items in it's inbound queue at a time.
Because of the two 50 limits there will be at most 100 byte[] in memory (50 in hashFilesBlock queue, 50 in the writeToDatabaseBlock queue, items currently being processed count toward the BoundedCapacity limit.
Update: for fun I wrote a version that reports progress too, it's untested though and uses C# 7 features.
using System;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
using System.Threading.Tasks.Dataflow;
public static class Example
{
private class Dto
{
public Dto(string filePath, byte[] data)
{
FilePath = filePath;
Data = data;
}
public string FilePath { get; }
public byte[] Data { get; }
}
public static async Task ProcessFiles(string path, IProgress<ProgressReport> progress)
{
int totalFilesFound = 0;
int totalFilesRead = 0;
int totalFilesHashed = 0;
int totalFilesUploaded = 0;
DateTime lastReported = DateTime.UtcNow;
void ReportProgress()
{
if (DateTime.UtcNow - lastReported < TimeSpan.FromSeconds(1)) //Try to fire only once a second, but this code is not perfect so you may get a few rapid fire.
{
return;
}
lastReported = DateTime.UtcNow;
var report = new ProgressReport(totalFilesFound, totalFilesRead, totalFilesHashed, totalFilesUploaded);
progress.Report(report);
}
var getFilesBlock = new TransformBlock<string, Dto>(filePath =>
{
var dto = new Dto(filePath, File.ReadAllBytes(filePath));
totalFilesRead++; //safe because single threaded.
return dto;
});
var hashFilesBlock = new TransformBlock<Dto, Dto>(inDto =>
{
using (var md5 = System.Security.Cryptography.MD5.Create())
{
var outDto = new Dto(inDto.FilePath, md5.ComputeHash(inDto.Data));
Interlocked.Increment(ref totalFilesHashed); //Need the interlocked due to multithreaded.
ReportProgress();
return outDto;
}
},
new ExecutionDataflowBlockOptions{MaxDegreeOfParallelism = Environment.ProcessorCount, BoundedCapacity = 50});
var writeToDatabaseBlock = new ActionBlock<Dto>(arg =>
{
//Write to database here.
totalFilesUploaded++;
ReportProgress();
},
new ExecutionDataflowBlockOptions {BoundedCapacity = 50});
getFilesBlock.LinkTo(hashFilesBlock, new DataflowLinkOptions {PropagateCompletion = true});
hashFilesBlock.LinkTo(writeToDatabaseBlock, new DataflowLinkOptions {PropagateCompletion = true});
foreach (var filePath in Directory.EnumerateFiles(path))
{
await getFilesBlock.SendAsync(filePath).ConfigureAwait(false);
totalFilesFound++;
ReportProgress();
}
getFilesBlock.Complete();
await writeToDatabaseBlock.Completion.ConfigureAwait(false);
ReportProgress();
}
}
public class ProgressReport
{
public ProgressReport(int totalFilesFound, int totalFilesRead, int totalFilesHashed, int totalFilesUploaded)
{
TotalFilesFound = totalFilesFound;
TotalFilesRead = totalFilesRead;
TotalFilesHashed = totalFilesHashed;
TotalFilesUploaded = totalFilesUploaded;
}
public int TotalFilesFound { get; }
public int TotalFilesRead{ get; }
public int TotalFilesHashed{ get; }
public int TotalFilesUploaded{ get; }
}

As far as I understand, Task.Run will instantiate a new thread for every file you have there, which leads to lots of threads and context switching between them. The case like you describe, sounds like a good case for using Parallel.For or Parallel.Foreach, something like this:
public void CalcHashes(string path)
{
string GetFileHash(System.Security.Cryptography.MD5 md5, string fileName)
{
using (var stream = new BufferedStream(System.IO.File.OpenRead(fileName), 1200000))
{
var hash = md5.ComputeHash(stream);
var fileMD5 = string.Concat(Array.ConvertAll(hash, x => x.ToString("X2")));
return fileMD5;
}
}
ParallelOptions options = new ParallelOptions();
options.MaxDegreeOfParallelism = 8;
Parallel.ForEach(filenames, options, fileName =>
{
using (var md5 = System.Security.Cryptography.MD5.Create())
{
GetFileHash(md5, fileName);
}
});
}
EDIT: Seems Parallel.ForEach does not actually do the partitioning automatically. Added max degree of parallelism limit to 8. As a result:
107005 files
46628 ms

How to run an infinite thread?

I'm trying to run a Thread infinitely, but it's not working ...
Follow the code:
namespace BTCPrice
{
public static class Price
{
private static volatile bool _shouldStop = false;
public static void RequestStop()
{
_shouldStop = true;
}
public static void Refresh(out string value, int tipo = 0, string source = "https://www.mercadobitcoin.net/api/")
{
while (_shouldStop == false)
{
JavaScriptSerializer serializer = new System.Web.Script.Serialization.JavaScriptSerializer();
WebClient cliente = new WebClient();
string json = cliente.DownloadString(string.Format("{0}/{1}/{2}", source, "BTC", "ticker"));
JObject j = JObject.Parse(json);
switch (tipo)
{
//Get High Price
case 0:
value = j["ticker"]["high"].ToString();
break;
//Get Low Price
case 1:
value = j["ticker"]["low"].ToString();
break;
default:
value = "default";
break;
}
Thread.Sleep(1000);
}
value = "Stopped";
}
}
}
On Start:
string result = "";
Thread workerThread = new Thread(() => {
Price.Refresh(out result);
MessageBox.Show(result);
Invoke(textBox1, result);
Thread.Sleep(1000);
});
No exception occurs ... as long as I remove the While (_shouldStop == false) class the code works perfectly. However, I would like that, while the program is open, it executes the code and updates the textbox with the value that I get by the API.
result without While(_shouldStop == false) in class:
Expected Result with While

You really shouldn't be using threads these days when there are excellent alternatives that handle all of the mess for you.
I'd suggest using Microsoft's Reactive Framework (aka "Rx"). Just NuGet "System.Reactive", "System.Reactive.Windows.Forms" (Windows Forms), "System.Reactive.Windows.Threading" (WPF).
Then you can do this:
int tipo = 0;
string source = "https://www.mercadobitcoin.net/api/";
string url = string.Format("{0}/{1}/{2}", source, "BTC", "ticker");
IObservable<string> feed =
from n in Observable.Interval(TimeSpan.FromSeconds(1.0))
from json in Observable.Using<string, WebClient>(() => new WebClient(), cliente => cliente.DownloadStringTaskAsync(url).ToObservable())
let j = JObject.Parse(json)
let high = j["ticker"]["high"].ToString()
let low = j["ticker"]["low"].ToString()
select tipo == 0 ? high : (tipo == 1 ? low : "default");
IDisposable subscription =
feed
.ObserveOn(this); // for Windows Forms OR .ObservableOnDispatcher() for WPF
.Subscribe(value =>
{
/* Do something with `value` */
});
You'll now get a steady stream of the string value every second. A thread is started automatically and the results are automatically pasted to the UI thread.
When you want to stop the feed producing values just call subscription.Dispose();.
This code entirely replaces your Price class.

Change your while loop in Price.Refresh to inside the thread. Have Price.Refresh return a string instead.
Thread workerThread = new Thread(() => {
while (true)
{
String result = Price.Refresh();
MessageBox.Show(result);
Invoke(textBox1, result);
Thread.Sleep(1000);
});
I agree with Scott Chamberlain in that you should use a timer instead and rewrite this but this will work for you.