Develop Reference

Microsoft.Azure.Cosmos Most Efficient way to Export Large Data

I need to export thousands of files from a Cosmos DB, and I am wondering if there may be a more efficient way to get all these documents (but I haven't been able to figure one out by browsing the documentation and searching).
Right now I am using the FeedIterator to get my results:
Database database = m_cosmosClient.GetDatabase(m_databaseId);
DatabaseResponse databaseResponse = await database.ReadAsync();
// The response from Azure Cosmos
DatabaseProperties properties = databaseResponse;
Container container = databaseResponse.Database.GetContainer(m_cosmosDbContainer);
QueryDefinition query = new QueryDefinition(queryString);
QueryRequestOptions queryOptions = new QueryRequestOptions { MaxItemCount = 10000, MaxBufferedItemCount = 10000 };
List<Article> results = new List<Article>();
FeedIterator<Article> resultSetIterator = container.GetItemQueryIterator<Article>(query, null, queryOptions);
while (resultSetIterator.HasMoreResults)
{
FeedResponse<Article> response = await resultSetIterator.ReadNextAsync();
results.AddRange(response);
if (response.Diagnostics != null)
{
Console.WriteLine($"\nQueryWithSqlParameters Diagnostics: {response.Diagnostics.ToString()}");
}
}
I am worried that without some form of multi-tasking that I could run out of memory, and then again it is always nice to have a faster run time.

The Cosmos DB Data Migration Tool is a good (and simple) option if you want to run the extract from a workstation. It can be run interactively or automated using scripts.
Creating a job in Azure Data Factory is a bit more complex but also offers a lot more flexibility.
This article discusses the various options for data migration in and out of Cosmos DB.

Querying for agent running a build

I am trying to query which agents are currently running a build using the TFS Extended Client. This is for the new TFS 2015 (non-XAML) build system.
I can easily query currently running builds with the code below, but I can not see any way to access details of the agent running the build. Neither the properties of the Build type or any of the BuildHttpClient methods appear to expose this.
Is there any way to achieve this?
VssConnection connection = new VssConnection(new Uri("TfsUri"), new VssAadCredential());
var buildClient = connection.GetClient<BuildHttpClient>();
var builds = await buildClient.GetBuildsAsync(
statusFilter: BuildStatus.InProgress,
project:"project"
);
foreach (var build in builds)
{
var buildNumber = build.BuildNumber;
var buildQueue = build.Queue;
//How to query for agent executing build?
}

You can potentially use the workername property contained in the build's timeline record as an identification mechanism for the agent.
The agent's worker name typically matches the agent name when added to a queue.
To access this detail of the build via the api you're using; I tested against builds that have already completed and builds in progress, this detail was populated already at the point of execution, you may want to do additional object reference validity checking in case a worker has not yet been assigned to the specified build.
None the less here is the foreach loop from your snippet modified to include a lookup of the worker name using the build's timeline records.
foreach (var build in builds)
{
var buildNumber = build.BuildNumber;
var buildQueue = build.Queue;
//How to query for agent executing build? ... Perhaps use the the worker name
var buildDetail = await buildClient.GetBuildTimelineAsync(build.Project.Id, build.Id);
var workerName = (string.IsNullOrEmpty(buildDetail.Records.Select(x => x.WorkerName).First())) ? "WORKER NOT INDICATED" : buildDetail.Records.Select(x => x.WorkerName).First();
}

Search with SharePoint CSOM (portable) throws an exception

I am trying to use SharePoint client framework to execute a search, using the portable dll's from a Windows app.
Using Fiddler I can see that my search is executed, and returns a JSON collection of metadata and search results. This is identical to the result from the non-portable CSOM.
When CSOM tries to map the result to it's data objects I get the following exception:
Unable to cast object of type 'System.Collections.Generic.Dictionary`2[System.String,System.Object]' to type 'Microsoft.SharePoint.Client.Search.Query.ResultTableCollection'.
This exception occurs inside CSOM (portable). Non-portable CSOM runs without exception, and returns the expected result.
The code I am running to get this exception is:
var query = new KeywordQuery(ctx);
query.QueryText = "something";
var executor = new SearchExecutor(ctx);
var results = executor.ExecuteQuery(query);
await ctx.ExecuteQueryAsync();
In the above, ctx is a ClientContext that has already been authenticated. Other requests, such as getting a specific list, works as expected.
I am referencing the following dll's from c:\Program Files\Common Files\microsoft shared\Web Server Extensions\16\ISAPI:
Microsoft.SharePoint.Client.Portable.dll
Microsoft.SharePoint.Client.Runtime.Portable.dll
Microsoft.SharePoint.Client.Runtime.WindowsStore.dll
Microsoft.SharePoint.Client.Search.Portable.dll
My question is.
How do I solve this, so that I can use CSOM to run search queries from a Windows Store app?
UPDATE:
I added the following after authenticating the ClientContext:
ctx.ExecutingWebRequest += (s, e) =>
e.WebRequest.Headers["Accept-Encoding"] = "gzip, deflate";
This solved the immediate problem, but introduced a new one. I am now getting a System.FormatException:
Not well formatted JSON stream.
Since the JSON from portable and non-portable CSOM is the same, there should not be a parsing error in one CSOM and not the other.

What I can identify from your exception is that casting of execute query result creates problem here.
Use below code to cast execute query result
ResultTable rtSharePointSearchResult = new ResultTable();
KeywordQuery query = new KeywordQuery(clientContext);
query.QueryText = "Keywords";
query.TrimDuplicates = false;
SearchExecutor searchExecutor = new SearchExecutor(clientContext);
ClientResult<ResultTableCollection> results = searchExecutor.ExecuteQuery(query);
clientContext.ExecuteQuery();
rtSharePointSearchResult = results.Value[0];

Notice that the first post uses ctx.ExecuteQueryAsync but the "answer" uses ctx.ExecuteQuery.
The bug is in the portable class library (that the first post uses) but this works in the non-portable version (the second post).
Cheers,
Paul

Execute a `Hyperlinq`-like query in LinqPad programmatically

I use LinqPad with the MySQL IQ driver to query data from a Magento database. I not only use this for reporting on the database but for doing updates. I can use the standard SubmitChanges() method to update data, but that often ends up with unbearably slow updates that can literally take hours - one of my tables has 35,707 records that I recreate on a regular basis.
So instead I generate SQL statements in my LinqPad queries and then execute them in a separate tab after selecting "SQL" in the language drop-down.
For example, my output might be something like this:
UPDATE catalog_category_product SET position = 6040 WHERE (category_id = 156 AND product_id = 12648);
UPDATE catalog_product_entity_media_gallery_value SET label = 'Sandy Beach' WHERE ((store_id = 0) AND (value_id = 8791));
-- Done.
I have recently found that LinqPad has a nice class called Hyperlinq that allows me to write code like this:
(new Hyperlinq(QueryLanguage.SQL, myGeneratedSqlText, "Run Query")).Dump();
The result is that a hyperlinq is put in the output window that will run the query (in my example the contents of myGeneratedSqlText) in a new tab and execute the query.
This is very convenient.
However, I now want to be able to save a log of queries that are executed. There doesn't seem to be (an easy) built-in way to manually execute a "generated" query in LinqPad. I can certainly use Util.Run to execute an existing saved query, in fact I do something like this:
Util
.OnDemand("Run Query", () =>
{
var fn = createOutputQueryFileName(); // Timestamped query name
System.IO.File.WriteAllText(fn, myGeneratedSqlText);
var run = Util.Run(fn, QueryResultFormat.Text);
var result = run.AsString();
return result.StartsWith("[]") ? "Success" : result;
})
.Dump();
The only drama with this is that I have to prefix the text in myGeneratedSqlText with the following:
var preamble = #"<Query Kind=""SQL"">
<Connection>
<ID>ec026b74-8d58-4214-b603-6d3145e03d7e</ID>
<Driver Assembly=""IQDriver"" PublicKeyToken=""5b59726538a49684"">IQDriver.IQDriver</Driver>
<Provider>Devart.Data.MySql</Provider>
<CustomCxString>[DELETED]</CustomCxString>
<Server>127.0.0.1</Server>
<Database>prod_1_8</Database>
<Password>[DELETED]</Password>
<UserName>[DELETED]</UserName>
<NoPluralization>true</NoPluralization>
<NoCapitalization>true</NoCapitalization>
<DisplayName>Production Tunnel</DisplayName>
<EncryptCustomCxString>true</EncryptCustomCxString>
<Persist>true</Persist>
<DriverData>
<StripUnderscores>false</StripUnderscores>
<QuietenAllCaps>false</QuietenAllCaps>
<Port>6606</Port>
</DriverData>
</Connection>
</Query>
";
I would really like to avoid all of this preamble stuff and include a line like this in my Util.OnDemand(...) code:
var run = Util.Run(QueryLanguage.SQL, myGeneratedSqlText, QueryResultFormat.Text);
(But this method doesn't exist.)
The key requirement here is to display a hyperlinq in the LinqPad output window that, if clicked, will save the query to disk as a log and also execute the query.
Can anyone suggest a clean way for me to do it?

I hope I've understood you correctly. When you've selected a connection in the top bar, your UserQuery becomes a datacontext. For this reason, you can use ExecuteQuery and ExecuteCommand on this within an action based Hyperlinq.
new Hyperlinq(() => {
"do log work here".Dump();
this.ExecuteQuery<string>(generatedSelect).Dump("Results");
this.ExecuteCommand(generatedCommand).Dump("Results");
}, "Run Query").Dump();
Unfortunately this outputs to the current tab, but hopefully this will at least get you most of the way to done :)
Here's an image of it at work:
As you're using MySQL, you can go via the connection property on this:
new Hyperlinq(() => {
"do log work here".Dump();
using (var command = this.Connection.CreateCommand())
{
// Usual command logic here
}
}, "Run Query").Dump();

EzAPI OLE DB Destination

I've searched all over and I now have to ask SO. I'm trying to construct a simple dataflow using EzAPI. It's been anything but easy, but I'm committed to figuring this out. What I can't figure out is how to get the EzOleDBDestination working. Here's my complete code
var a = new Application();
// using a template since it's impossible to set up an ADO.NET connection to MySQL
// using EzAPI and potentially even with the raw SSIS API...
var pkg = new EzPackage(a.LoadPackage(#"C:\...\Package.dtsx", null));
pkg.Name = "Star";
var df = new EzDataFlow(pkg);
df.Name = "My DataFlow";
var src = new EzAdoNetSource(df);
src.Name = "Source Database";
src.SqlCommand = "SELECT * FROM enum_institution";
src.AccessMode = AccessMode.AM_SQLCOMMAND;
src.Connection = new EzConnectionManager(pkg, pkg.Connections["SourceDB"]);
src.ReinitializeMetaData();
var derived = new EzDerivedColumn(df);
derived.AttachTo(src);
derived.Name = "Prepare Dimension Attributes";
derived.LinkAllInputsToOutputs();
derived.Expression["SourceNumber"] = "id";
derived.Expression["Name"] = "(DT_STR,255,1252)description";
// EDIT: reordered the operation here and I no longer get an error, but
// I'm not getting any mappings or any input columns when I open the package in the designer
var dest = new EzOleDbDestination(df);
dest.AttachTo(derived, 0, 0);
dest.Name = "Target Database";
dest.AccessMode = 0;
dest.Table = "[dbo].[DimInstitution]";
dest.Connection = new EzConnectionManager(pkg, pkg.Connections["TargetDB"]);
// this comes from Yahia's link
var destInput = dest.Meta.InputCollection[0];
var destVirInput = destInput.GetVirtualInput();
var destInputCols = destInput.InputColumnCollection;
var destExtCols = destInput.ExternalMetadataColumnCollection;
var sourceColumns = derived.Meta.OutputCollection[0].OutputColumnCollection;
foreach(IDTSOutputColumn100 outputCol in sourceColumns) {
// Now getting COM Exception here...
var extCol = destExtCols[outputCol.Name];
if(extCol != null) {
// Create an input column from an output col of previous component.
destVirInput.SetUsageType(outputCol.ID, DTSUsageType.UT_READONLY);
var inputCol = destInputCols.GetInputColumnByLineageID(outputCol.ID);
if(inputCol != null) {
// map the input column with an external metadata column
dest.Comp.MapInputColumn(destInput.ID, inputCol.ID, extCol.ID);
}
}
}
Basically, anything that involves calls to ReinitializeMetadata() results in 0xC0090001, because that method is where the error happens. There's no real documentation to help me, so I have to rely on any gurus here.
I should mention that the source DB is MySQL and the target DB is SQL Server. Building packages like this using the SSIS designer works fine, so I know it's possible.
Feel free to tell me if I'm doing anything else wrong.
EDIT: here's a link to the base package I'm using as a template: http://www.filedropper.com/package_1 . I've redacted the connection details, but any MySQL and SQL Server database will do. The package will read from MySQL (using the MySQL ADO.NET Connector) and write to SQL Server.
The database schema is mostly irrelevant. For testing, just make a table in MySQL that has two columns: id (int) and description (varchar), with id being the primary key. Make equivalent columns in SQL Server. The goal here is simply to copy from one to the other. It may end up being more complex at some point, but I have to get past this hurdle first.

I can't test this now BUT I am rather sure that the following will help you get it working:
Calling ReinitializeMetadata() causes the component to fetch the table metadata. This should only be called after setting the AccessMode and related property. You are calling it before setting AccessMode...
Various samples including advice on debugging problems
define the derived column(s) directly in the SQL command instead of using a EzDerivedColumn
try to get it working with 2 SQL Server DBs first, some of the available MySQL ADO.NET provider have some shortcomings under some circumstances
UPDATE - as per comments some more information on debugging this and a link to a complete end-to-end sample with source:
http://blogs.msdn.com/b/mattm/archive/2009/08/03/looking-up-ssis-hresult-comexception-errorcode.aspx
http://blogs.msdn.com/b/mattm/archive/2009/08/03/debugging-a-comexception-during-package-generation.aspx
Complete working sample with source

I've had this exact same issue and been able to resolve it with a lot of experimentation. In short you must set the connection for both the source and destination, and then call the attachTo after both connections are set. You must call attachTo for every component.
I've written a blog about starting with an SSIS package as a template, and then manipulating it programmatically to produce a set of new packages.
The article explains the issue more.