Develop Reference

Is it possible to get a batch of text content through Azure DevOps REST API?

I need to get (not download) the content from 10.000~ manifest files within a project in Azure DevOps, but I don't manage to achieve this. I have found several ways to retrieve the content from one file at a time, but in this context, it is neither an efficient nor sustainable solution. I have managed to retrieve all files of a particular file type by checking if the file path ends with the name of the file, then using the TfvcHttpClientBase.GetItemsBatch method. However, this method does not return the item's content.
Program.cs
using Microsoft.TeamFoundation.SourceControl.WebApi;
AzureRest azureRest = new AzureRest();
var tfvcItems = azureRest.GetTfvcItems();
List<TfvcItemDescriptor> itemDescriptorsList = new List<TfvcItemDescriptor>();
foreach(var item in tfvcItems)
{
//Example manifest file .NET
if (item.Path.EndsWith("packages.config"))
{
var itemDescriptor = new TfvcItemDescriptor()
{
Path = item.Path,
RecursionLevel = VersionControlRecursionType.None,
Version = "",
VersionOption = TfvcVersionOption.None,
VersionType = TfvcVersionType.Latest
};
itemDescriptorsList.Add(itemDescriptor);
}
}
TfvcItemDescriptor[] itemDescriptorsArray = itemDescriptorsList.ToArray();
var itemBatch = azureRest.GetTfvcItemsBatch(itemDescriptorsArray);
foreach(var itemList in itemBatch)
{
foreach(var itemListList in itemList)
{
Console.WriteLine("Content: " + itemListList.Content); //empty/null
Console.WriteLine("ContentMetadata: " + itemListList.ContentMetadata); //not empty/null
}
}
AzureRest.cs
using Microsoft.TeamFoundation.SourceControl.WebApi;
using Microsoft.VisualStudio.Services.Common;
using Microsoft.VisualStudio.Services.WebApi;
public class AzureRest
{
const string ORG_URL = "https://org/url/url";
const string PROJECT = "Project";
const string PAT = "PersonalAccessToken";
private string GetTokenConfig()
{
return PAT;
}
private string GetProjectNameConfig()
{
return PROJECT;
}
private VssConnection Authenticate()
{
string token = GetTokenConfig();
string projectName = GetProjectNameConfig();
var credentials = new VssBasicCredential(string.Empty, token);
var connection = new VssConnection(new Uri(ORG_URL), credentials);
return connection;
}
public List<TfvcItem> GetTfvcItems()
{
var connection = Authenticate();
using (TfvcHttpClient tfvcClient = connection.GetClient<TfvcHttpClient>())
{
var tfvcItems = tfvcClient.GetItemsAsync(scopePath: "/Path", recursionLevel: VersionControlRecursionType.Full, true).Result;
return tfvcItems;
}
}
public List<List<TfvcItem>> GetTfvcItemsBatch(TfvcItemDescriptor[] itemDescriptors)
{
TfvcItemRequestData requestData = new TfvcItemRequestData()
{
IncludeContentMetadata = true,
IncludeLinks = true,
ItemDescriptors = itemDescriptors
};
var connection = Authenticate();
using (TfvcHttpClient tfvcClient = connection.GetClient<TfvcHttpClient>())
{
var tfvcItems = tfvcClient.GetItemsBatchAsync(requestData).Result;
return tfvcItems;
}
}
}
}

For reference:
I have tested the codes you shared and when debugging at "itemDescriptorsList" and have found that there is no content specified in it, so that's why you cannot get the txt content.
You should first check and add the content property into the "itemDescriptorsList".

Nethereum C# automated ether transfer

I wish to automate the transfer of ether to a list of people.
Assume the list is in a csv.
I wrote some code to automate the process.
class Program
{
int nonce = 0;
static void Main(string[] args)
{
var account = SetupAccount();
var recipients = ReadCsv();
var web3Init = GetConnection();
nonce = web3.Eth.Transactions.GetTransactionCount.SendRequestAsync(account.Address).Result;
//var recipients = new List<Records>() { new Records() { Value = 10000000000000000, Address = "0x5CC494843e3f4AC175A5e730c300b011FAbF2cEa" } };
foreach (var recipient in recipients)
{
try
{
var web3 = GetConnection();
var receipt = SendEther(account, recipient, web3).Result;
}
catch (System.Exception)
{
MessageBox.Show("Failed");
}
Thread.Sleep(30000);
}
}
private static async Task<TransactionReceipt> SendEther(Account account, Records recipient, Web3 web3)
{
var transactionPolling = web3.TransactionManager.TransactionReceiptService;
//var currentBalance = await web3.Eth.GetBalance.SendRequestAsync(account.Address);
//assumed client is mining already
//when sending a transaction using an Account, a raw transaction is signed and send using the private key
return await transactionPolling.SendRequestAndWaitForReceiptAsync(() =>
{
var transactionInput = new TransactionInput
{
From = account.Address,
//Gas = new HexBigInteger(25000),
GasPrice = new HexBigInteger(10 ^ 10),
To = recipient.Address,
Value = new HexBigInteger(new BigInteger(recipient.Value)),
Nonce = nonce
};
var txSigned = new Nethereum.Signer.TransactionSigner();
var signedTx = txSigned.SignTransaction(account.PrivateKey, transactionInput.To, transactionInput.Value, transactionInput.Nonce);
var transaction = new Nethereum.RPC.Eth.Transactions.EthSendRawTransaction(web3.Client);
nonce++;
return transaction.SendRequestAsync(signedTx);
});
}
private static Web3 GetConnection()
{
return new Web3("https://mainnet.infura.io");
}
private static Account SetupAccount()
{
var password = "#Password";
var accountFilePath = #"filePath";
return Account.LoadFromKeyStoreFile(accountFilePath, password);
}
private static List<Records> ReadCsv()
{
string filePath = #"C:\Users\Potti\source\repos\ConversionFiles\XrcfRecipients.csv";
if (File.Exists(filePath))
{
using (StreamReader stream = new StreamReader(filePath))
{
CsvReader reader = new CsvReader(stream, new Configuration
{
TrimOptions = TrimOptions.Trim,
HasHeaderRecord = true,
HeaderValidated = null
});
reader.Configuration.RegisterClassMap<RecordMapper>();
return reader.GetRecords<Records>().ToList();
}
}
else
{
return null;
}
}
}
class Records
{
public string Address { get; set; }
public decimal Value { get; set; }
}
sealed class RecordMapper : ClassMap<Records>
{
public RecordMapper()
{
Map(x => x.Address).Name("Address");
Map(x => x.Value).Name("Value");
}
}
How do i modify the process to execute all the transactions at once instead of waiting for each to complete? (Fire and forget)
Also, are there any security considerations of doing this?

What you are currently doing is waiting for each transaction to be mined. What you can do is the following:
var account = new Account("privateKey"); // or load it from your keystore file as you are doing.
var web3 = new Web3(account, "https://mainnet.infura.io");
First create a web3 instance using the same Account object, because we are using an account with a private key, Nethereum will sign your transactions offline before sending them.
Now using the TransactionManager, you can then send a transaction per each recepient
var transactionHashes = new List<string>();
foreach(var recepient in recepients){
var transactionInput = new TransactionInput
{
From = account.Address,
GasPrice = Web3.Convert.ToWei(1.5, UnitConversion.EthUnit.Gwei);,
To = recipient.Address,
Value = new HexBigInteger(new BigInteger(recipient.Value)),
};
var transactionHash = web3.Eth.TransactionManager.SendTransactionAsync(transactionInput);
transanctionHashes.Add(transactionHash);
}
Note that when Nethereum uses the same instance of an Account and TransactionManager (or Web3 in this scenario) it creates a default NonceMemoryService, so you don't need to keep track of your nonces (Transaction number), to sign the transaction.
Also I have done a conversion for the GasPrice from Gwei to Wei, as an example of Unit conversions, I assume that you already have converted to Wei the Ether amounts you are going to send.
Finally, another note, to further simplify this, there is an upcoming EtherTransferService which allows you to input Ether amounts and Gwei price amounts to avoid doing conversions. Also the gas price will be calculated for you, if not passed any parameter.
web3.Eth.GetEtherTransferService().TransferEtherAsync("toAddress", EtherAmount);

ReliableCollections Service Fabric: Statemanager's GetOrAddAsync() returns invalid queue with different names

I have two reliable queues and they are being accessed by two guest executables and each of them access their own. Sometimes the function I use to access them doesn't update the reliable queue object in the function and the wrong request is sent to the wrong guest executable.
What happens is that the clientId is passed by the guest executable to this function in the Get request. Let us say that there are two clientId(s) called T1 and T2.
What happens is that the guest executable (client) T2 at times gets the request that was meant for T1. Even though I tried line by line debugging the parameters passed to this function are correct.
Here is my API's POST that is passed a json to be added to the queue for the clients to receive from the GET
[HttpPost("MarketInfo")]
public JObject GetMarketInfo([FromBody] JObject jObject)
{
List<JToken> clients = jObject.GetValue("clients").ToList();
string json;
JObject response = new JObject();
JArray jsonArray = new JArray();
try
{
foreach (JToken client in clients)
{
var id = Guid.NewGuid();
json = "{'name':'MarketInfo','id':'" + id.ToString() + "','mtClientId':'" + terminal["name"].ToString() + "','parameters':{'symbol':'" + terminal["symbol"].ToString() + "','property':24}}";
bool result = _requestsCollectionHandler.CreateRequestForClient(JObject.Parse(json));
JObject clientResponse = new JObject();
if (result==true)
{
clientResponse["name"] = client["name"].ToString();
clientResponse["guid"] = id.ToString();
jsonArray.Add(clientResponse);
}
else
{
clientResponse["name"] = terminal.Children()["name"].ToString();
clientResponse["guid"] = "ERROR";
jsonArray.Add(terminalResponse);
}
}
response["clients"] = jsonArray;
return response;
}
catch (Exception e)
{
Debug.Write(e.Message);
return null;
}
}
This is the json that we pass to this API
{"clients":[{"name":"T1","symbol":"SomeInfo"},{"name":"T2","symbol":"SomeInfo"}]}
The problem is always with the clients object that is passed first.
Before I explain further let me also share the code for the client's HttpGet
[HttpGet("{clientId}")]
public string Get([FromRoute] string clientId)
{
try
{
string request = _requestsCollectionHandler.GetRequestJsonFromQueue(clientId);
return request;
}
catch(Exception e)
{
return e.Message;
}
}
This is the function that creates an object that is to be added by another function in the reliable queue
public bool CreateRequestForClient(JObject jObject)
{
try
{
this._jObject = new JObject(jObject);
CreateKey();
AddToRequestToQueueAsync();
return true;
}
catch (Exception e)
{
Debug.Write(e.Message);
_exceptionMessage = e.Message;
return false;
}
}
private void CreateKey()
{
dynamic data = JObject.Parse(_jObject.ToString(Newtonsoft.Json.Formatting.None));
string name = data.name;
string id = data.id;
string clientId = data.clientId;
_key.id = id;
_key.name = name;
_key.clientId = clientId;
//key.timestamp = GetTimestamp();
_key.timestamp = GetTimestamp();
_key.requestJson = _jObject.ToString(Newtonsoft.Json.Formatting.None);
}
_key is a private variable in class a custom class
This is the function in my class of request handler that adds the requests to the queue
private void AddToRequestToQueueAsync()
{
var transaction = this._stateManager.CreateTransaction();
CancellationToken cancellationToken
= new CancellationToken(false);
try
{
string queue = _key.clientId;
IReliableConcurrentQueue<TerminalResponseKey> reliableQueue =
_stateManager.GetOrAddAsync<IReliableConcurrentQueue<TerminalResponseKey>>(queue).Result;
transaction = this._stateManager.CreateTransaction();
if (reliableQueue!=null)
{
long count = reliableQueue.Count;
reliableQueue.EnqueueAsync(transaction, _key);
count = reliableQueue.Count;
transaction.CommitAsync().Wait();
}
else
{
transaction.Abort();
}
}
catch
{
transaction.Abort();
throw;
}
}
This is function that is used by the client
public string GetRequestJsonFromQueue(string clientId)
{
string queue = clientId;
try
{
IReliableConcurrentQueue<TerminalResponseKey> reliableQueue =
this._stateManager.GetOrAddAsync<IReliableConcurrentQueue<TerminalResponseKey>>(queue).Result;
if(reliableQueue != null)
{
ConditionalValue<TerminalResponseKey> key =
reliableQueue.TryDequeueAsync(transaction).Result;
if(key.HasValue)
{
string request = key.Value.requestJson;
transaction.CommitAsync().Wait();
return request;
}
}
else
{
transaction.Abort();
}
return "NO QUEUE";
}
catch (Exception e)
{
Debug.WriteLine(e);
transaction.Abort();
return e.InnerException.Message;
}
}
As far as I have found out I think my problem is in this function above. Because I don't know how the client T2 or client T1 gets another client's queue because the parameters determining the queue are their IDs and are totally unique.
These Ids are also passed correctly to this:
IReliableConcurrentQueue<TerminalResponseKey> reliableQueue =
this._stateManager.GetOrAddAsync<IReliableConcurrentQueue<TerminalResponseKey>>(queue).Result;
As you can see that we have queue=clientId
I have tried adding proper timespans but it was of no use as there is no exception thrown for OperationTimedOut. Furthermore since I am new to ServiceFabric I maybe totally doing anything wrong.
PS: Sorry for maybe a lot of jumbled up and confused code and question AND SOME OF THE INFORMATION IS OBFUSCATED DUE TO CONFIDENTIALITY BUT NOTHING OBSTRUCTING THE UNDERSTANDING OF THIS IS HIDDEN (I Hope not an issue)
I hope this is not an issue maybe an error I am overlooking at my side

When you put the request in the queue, in AddToRequestToQueueAsync(), the name of the queue is set from _key.terminalId (and I don't see where you assign it), but when you read from it, in GetRequestJsonFromQueue(), the clientId
is used as the queue name.

Website crashes during high load when using Amazon Kinesis Client, issue with number of open connections

So my website crashes during high load when using the AmazonKinesisFireHoseClient to put records in kinesis.
I get no exceptions logged it just sort of dies and you need to restart the application to get it to work again, sometimes it just starts working by itself after a crash.
I know it has something to do with the number of connections that are open but I have no idea how to fix it.
this is my logger that I instantiate as a singleton
public class KinesisFirehoseLogger<T>
{
private string streamName;
private AmazonKinesisFirehoseClient client;
protected ILogger logger;
public KinesisFirehoseLogger(IKinesisLogConfig config, ILogger logger)
{
this.streamName = config.StreamName;
this.logger = logger;
this.PartitionKey = config.PartitionKey;
AWSCredentials credentials = new BasicAWSCredentials(config.AccessKey, config.SecretKey);
this.client = new AmazonKinesisFirehoseClient(credentials, config.Region);
}
public async Task<bool> WriteToKinesisAsync(T logObject)
{
using (var ms = new MemoryStream(Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(logObject) + "\n")))
{
PutRecordRequest putRecordRequest = new PutRecordRequest() { DeliveryStreamName = this.streamName, Record = new Record() { Data = ms } };
// Put record into the DeliveryStream
return (await client.PutRecordAsync(putRecordRequest)).HttpStatusCode == HttpStatusCode.OK;
}
}
}
Am I supposed to use the dispose() method of the AmazonKinesisFirehoseClient somewhere ? How do I know when to call it.

Assuming that the client.PutRecordAsync opens the connection then it might make sense to put it in a using block as well. For example:
using (var ms = new MemoryStream(Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(logObject) + "\n")))
{
PutRecordRequest putRecordRequest = new PutRecordRequest() { DeliveryStreamName = this.streamName, Record = new Record() { Data = ms } };
// Put record into the DeliveryStream
using (PutRecordResponse response = await client.PutRecordAsync(putRecordRequest))
{
return response.HttpStatusCode == HttpStatusCode.Ok;
}
}

Duplicate entries on server response .net

Scenario
One windows service polls a url every two minutes to retrieve certain data.
If any data has been added since the previous call, the data is retrieved and stored otherwise the loop carries on.
Issue
Sometimes a request takes more than two minutes to return a response.
When this happens, the next request is still made and finds new data, since the previous request hasn't return a response yet
This results in duplicate entries when the data is stored.
What I've tried
I tried to handle that by using a boolean like so:
Boolean InProgress = true;
foreach (var item in Lists)
{
\\Make a request and return new data (if any)
InProgress = false;
if (InProgress = false)
{
\\Store new data
}
}
This doesn't solve the issue. I believe I'm using the boolean in wrong place, but I'm not sure where it should.
This is the loop that makes the request and store the data
void serviceTimer_Elapsed(object sender, ElapsedEventArgs e)
{
try
{
Data getCredentials = new Data();
DataTable credentials = getCredentials.loadCredentials();
Boolean InProgress = true;
for (int i = 0; i < credentials.Rows.Count; i++)
{
if (credentials != null)
{
var PBranchID = (int)credentials.Rows[i]["PortalBranchID"];
var negRef = (int)credentials.Rows[i]["NegotiatorRef"];
var Username = credentials.Rows[i]["Username"].ToString();
var Password = credentials.Rows[i]["Password"].ToString();
var Domain = credentials.Rows[i]["Domain"].ToString();
var FooCompanyBaseUrl = "https://" + Domain + ".FooCompany.com/";
Data getCalls = new Data();
DataTable calls = getCalls.loadCalls(PBranchID);
//If it's not the first call
if (calls != null && calls.Rows.Count > 0)
{
//Makes a call
DateTime CreatedSince = DateTime.SpecifyKind((DateTime)calls.Rows[0]["LastSuccessOn"], DateTimeKind.Local);
string IssueListUrl = FooCompany.WebApi.V2.URLs.Issues(BaseUrl, null, CreatedSince.ToUniversalTime(), null);
FooCompany.WebApi.V2.DTO.PrevNextPagedList resultIssueList;
resultIssueList = FooCompany.WebApi.Client.Helper.Utils.Getter<Foocompany.WebApi.V2.DTO.PrevNextPagedList>(IssueListUrl, Username, Password);
InProgress = false;
if (InProgress == false)
{
if (resultIssueList.Items.Count > 0)
{
//If call returns new issues, save call
Data saveCalls = new Data();
saveCalls.saveCalls(PBranchID);
foreach (var item in resultIssueList.Items)
{
var Issue = FooCompany.WebApi.Client.Helper.Utils.Getter<FooCompany.WebApi.V2.DTO.Issue>(item, Username, Password);
string TenantSurname = Issue.Surname;
string TenantEmail = Issue.EmailAddress;
Data tenants = new Data();
int tenantPropRef = Convert.ToInt32(tenants.loadTenantPropRef(PBranchID, TenantSurname, TenantEmail));
Data Properties = new Data();
DataTable propAddress = Properties.loadPropAddress(PBranchID, tenantPropRef);
var Address1 = propAddress.Rows[0]["Address1"];
var Address2 = propAddress.Rows[0]["Address2"];
var AddressFolder = Address1 + "," + Address2;
if (!Directory.Exists("path"))
{
Directory.CreateDirectory("path");
}
string ReportPDFDestination = "path";
if (File.Exists(ReportPDFDestination))
{
File.Delete(ReportPDFDestination);
}
FooCompany.WebApi.Client.Helper.Utils.DownloadFileAuthenticated(FooCompany.WebApi.V2.URLs.IssueReport(BaseUrl, Issue.Id), Username, Password, ReportPDFDestination);
//Store data
}
IssueListUrl = resultIssueList.NextURL;
}
}
}
else
{
continue;
}
}
}
catch (Exception ex)
{
//write to log
}
}
Question
I'm sure there is a better way than a boolean.
Could anyone advice a different method to handle the issue properly?
Thanks.
Solution
I ended up using a combination of both Thomas and Mason suggestions. I wrapped a lock statement around the main function of my windows service and used a boolean inside the function section that makes the call to the remote server.
Tested many times and it's error free.

You seems to have a problem of synchronisation, just surround the code that iterate though the List with a lock, and you will be fine.
public class MyClass{
private readonly object internalLock= new object();
private bool AlreadyRunning { get; set; }
void serviceTimer_Elapsed(object sender, ElapsedEventArgs e)
{
if(AlreadyRunning){
return;
}
try{
lock(internalLock){
Thread.MemoryBarrier();
if(AlreadyRunning){
return;
}
AlreadyRunning = true;
...Do all the things...
}
}
catch(Exception e){
..Exception handling
}
finally
{
AlreadyRunning = false;
}
}

bool InProgress=false;
void serviceTimer_Elapsed(object sender, ElapsedEventArgs e)
{
if(!InProgress)
{
InProgress=true;
//retrieve data
InProgress=false;
}
}
Your InProgress variable needs to be declared outside the event handler. When you enter the method, check to see if it's already running. If it is, then we do nothing. If it's not running, then we say it's running, retrieve our data, then reset our flag to say we've finished running.
You'll probably need to add appropriate locks for thread safety, similar to Thomas's answer.