Develop Reference

AMO get partitions where data are processed but not indexes

I am writing a script that return all unprocessed partitions within a measure group using the following command:
objMeasureGroup.Partitions.Cast<Partition>().Where(x => x.State != AnalysisState.Processed)
After doing some experiments, it looks like this property indicates if the data is processed and doesn't mention the indexes.
After searching for hours, i didn't find any method to list the partitions where data is processed but indexes are not.
Any suggestions?
Environment:
SQL Server 2014
SSAS multidimensional cube
Script are written within a SSIS package / Script task

First, ProcessIndexes is an incremental operation. So if you run it twice the second time will be pretty quick because there is nothing to do. So I would recommend just running it on the cube and not worrying about whether it was previously run. However if you do need to analyze the current state then read on.
The best way (only way I know of) to distinguish whether ProcessIndexes has been run on a partition is to study the DISCOVER_PARTITION_STAT and DISCOVER_PARTITION_DIMENSION_STAT DMVs as seen below.
The DISCOVER_PARTITION_STAT DMV returns one row per aggregation with the rowcount. The first row of that DMV has a blank aggregation name and represents the rowcount of the lowest level data processed in that partition.
The DISCOVER_PARTITION_DIMENSION_STAT DMV can tell you about whether indexes are processed and which range of values by each dimension attribute are in this partition (by internal IDs, so not super easy to interpret). We assume at least one dimension attribute is set to be optimized so it will be indexed.
You will need to add a reference to Microsoft.AnalysisServices.AdomdClient also to simplify running these DMVs:
string sDatabaseName = "YourDatabaseName";
string sCubeName = "YourCubeName";
string sMeasureGroupName = "YourMeasureGroupName";
Microsoft.AnalysisServices.Server s = new Microsoft.AnalysisServices.Server();
s.Connect("Data Source=localhost");
Microsoft.AnalysisServices.Database db = s.Databases.GetByName(sDatabaseName);
Microsoft.AnalysisServices.Cube c = db.Cubes.GetByName(sCubeName);
Microsoft.AnalysisServices.MeasureGroup mg = c.MeasureGroups.GetByName(sMeasureGroupName);
Microsoft.AnalysisServices.AdomdClient.AdomdConnection conn = new Microsoft.AnalysisServices.AdomdClient.AdomdConnection(s.ConnectionString);
conn.Open();
foreach (Microsoft.AnalysisServices.Partition p in mg.Partitions)
{
Console.Write(p.Name + " - " + p.State + " - ");
var restrictions = new Microsoft.AnalysisServices.AdomdClient.AdomdRestrictionCollection();
restrictions.Add("DATABASE_NAME", db.Name);
restrictions.Add("CUBE_NAME", c.Name);
restrictions.Add("MEASURE_GROUP_NAME", mg.Name);
restrictions.Add("PARTITION_NAME", p.Name);
var dsAggs = conn.GetSchemaDataSet("DISCOVER_PARTITION_STAT", restrictions);
var dsIndexes = conn.GetSchemaDataSet("DISCOVER_PARTITION_DIMENSION_STAT", restrictions);
if (dsAggs.Tables[0].Rows.Count == 0)
Console.WriteLine("ProcessData not run yet");
else if (dsAggs.Tables[0].Rows.Count > 1)
Console.WriteLine("aggs processed");
else if (p.AggregationDesign == null || p.AggregationDesign.Aggregations.Count == 0)
{
bool bIndexesBuilt = false;
foreach (System.Data.DataRow row in dsIndexes.Tables[0].Rows)
{
if (Convert.ToBoolean(row["ATTRIBUTE_INDEXED"]))
{
bIndexesBuilt = true;
break;
}
}
if (bIndexesBuilt)
Console.WriteLine("indexes have been processed. no aggs defined");
else
Console.WriteLine("no aggs defined. need to run ProcessIndexes on this partition to build indexes");
}
else
Console.WriteLine("need to run ProcessIndexes on this partition to process aggs and indexes");
}

I am posting this answer as additional information of #GregGalloway excellent answer
After searching for a while, the only way to know if partition are processed is using DISCOVER_PARTITION_STAT and DISCOVER_PARTITION_DIMENSION_STAT.
I found an article posted by Daren Gossbel describing the whole process:
SSAS: Are my Aggregations processed?
In the artcile above the author provided two methods:
using XMLA
One way in which you can find it out with an XMLA discover call to the DISCOVER_PARTITION_STAT rowset, but that returns the results in big lump of XML which is not as easy to read as a tabular result set.
example
<Discover xmlns="urn:schemas-microsoft-com:xml-analysis">
<RequestType>DISCOVER_PARTITION_STAT</RequestType>
<Restrictions>
<RestrictionList>
<DATABASE_NAME>Adventure Works DW</DATABASE_NAME>
<CUBE_NAME>Adventure Works</CUBE_NAME>
<MEASURE_GROUP_NAME>Internet Sales</MEASURE_GROUP_NAME>
<PARTITION_NAME>Internet_Sales_2003</PARTITION_NAME>
</RestrictionList>
</Restrictions>
<Properties>
<PropertyList>
</PropertyList>
</Properties>
</Discover>
using DMV queries
If you have SSAS 2008, you can use the new DMV feature to query this same rowset and return a tabular result.
example
SELECT *
FROM SystemRestrictSchema($system.discover_partition_stat
,DATABASE_NAME = 'Adventure Works DW 2008'
,CUBE_NAME = 'Adventure Works'
,MEASURE_GROUP_NAME = 'Internet Sales'
,PARTITION_NAME = 'Internet_Sales_2003')
Similar posts:
How to find out using AMO if aggregation exists on partition?
Detect aggregation processing state with AMO?

Calculated column in view performance issue

I've got a table with location data (Lat/Lngs) stored as geography type.
Now this location data represents a Journey, so I want to calculate the distance travelled during the Journey, distance per day and distance per month. So for this I need a property to calculate the distance from the next location in the Journey.
If I do this using a Select from the table with a Window Function, the performance is very good and the Execution Plan seems fine. Execution Plan
This is the Query -
Select
iVehicleMonitoringId,
iAssetId,
iDriverId,
dtUtcDatetime,
sptGeoLocaitonPoint,
fAngel,
sDirection,
fSpeedKPH,
eEventCode,
fDistanceTravelled = sptGeoLocaitonPoint.STDistance(LEAD(sptGeoLocaitonPoint) OVER(PARTITION BY iAssetId ORDER BY dtUTCDateTime)),
eIgnition,
eDoor,
eSeatbelt,
eIgnition2,
sAddress
From dbo.VehicleMonitoringLog
Where iAssetId = 1
AND dtUTCDateTime BETWEEN N'2017-12-27' AND N'2017-12-27 23:59:59.997'
Now because I need this Distance Calculated property in my code. I decided to create an SQL View with the fDistanceTravelled property.
CREATE VIEW [dbo].[TB5_xPT_VehicleMonitoringLogs]
AS
Select
iVehicleMonitoringId = ISNULL(iVehicleMonitoringId, -1),
iAssetId,
iDriverId,
dtUtcDatetime,
sptGeoLocaitonPoint,
fAngel,
sDirection,
fSpeedKPH,
eEventCode,
fDistanceTravelled = sptGeoLocaitonPoint.STDistance(LEAD(sptGeoLocaitonPoint) OVER(PARTITION BY iAssetId ORDER BY dtUTCDateTime)),
eIgnition,
eDoor,
eSeatbelt,
eIgnition2,
sAddress
From dbo.VehicleMonitoringLog
GO
But when I import this View into my Entity Framework, it gets a timeout. So I checked the Execution Plan and it doesn't match that of selecting from the Table.
View Execution Plan
This is the query -
Select
*
From TB5_xPT_VehicleMonitoringLogs
Where iAssetId = 1
AND dtUTCDateTime BETWEEN N'2017-12-27' AND N'2017-12-27 23:59:59.997'
To do this in code I need to loop through all the location logs in the list.
DbGeography prevPoint = null;
foreach (var historyLog in historyLogs)
{
var distanceFromPrevPoint = 0.0;
if (prevPoint != null)
{
var distance = prevPoint.Distance(historyLog.sptGeoLocaitonPoint);
if (distance != null)
distanceFromPrevPoint = Math.Round(((double)distance * .001), 2);
}
var locationLog = Mapper.Map<LocationLogDTO>(historyLog);
locationLog.DistanceTravelled = distanceFromPrevPoint;
prevPoint = historyLog.sptGeoLocaitonPoint;
historyDto.LocationLogs.Add(locationLog);
}
I don't want to do this. I need a better way of doing this.
EDIT
Image Explaining the fDistanceTravelled column calculation.

Get field data outside of reporting database using Encompass360 SDK

I'm trying to build a standalone application that creates a custom report for Encompass360 without needing to put certain fields into the reporting database.
So far I have only found one way to do it, but it is extremely slow. (Much slower than a normal report within encompass when retrieving data outside of the reporting database.) It takes almost 2 minutes to pull the data for 5 loans doing this:
int count = 5;
StringList fields = new StringList();
fields.Add("Fields.317");
fields.Add("Fields.3238");
fields.Add("Fields.313");
fields.Add("Fields.319");
fields.Add("Fields.2");
// lstLoans.Items contains the string location of the loans(i.e. "My Pipeline\Dave#6")
foreach (LoanIdentity loanID in lstLoans.Items)
{
string[] loanIdentifier = loanID.ToString().Split('\\');
Loan loan = Globals.Session.Loans.Folders[loanIdentifier[0]].OpenLoan(loanIdentifier[1]);
bool fundingPlus = true; // if milestone == funding || shipping || suspended || completion;
if (!fundingPlus)
continue;
bool oneIsChecked = false;
LogMilestoneEvents msEvents = loan.Log.MilestoneEvents;
DateTime date;
MilestoneEvent ms = null; // better way to do this probably
if (checkBox4.Checked)
{
ms = msEvents.GetEventForMilestone("Completion");
if (ms.Completed)
{
oneIsChecked = true;
}
}
else if (checkBox3.Checked)
{
ms = msEvents.GetEventForMilestone("Suspended");
if (ms.Completed)
{
oneIsChecked = true;
}
}
else if (checkBox2.Checked)
{
ms = msEvents.GetEventForMilestone("Shipping");
if (ms.Completed)
{
oneIsChecked = true;
}
}
else if (checkBox1.Checked)
{
ms = msEvents.GetEventForMilestone("Funding");
if (ms.Completed)
{
oneIsChecked = true;
}
}
if (!oneIsChecked)
continue;
string LO = loan.Fields["317"].FormattedValue;
string LOid = loan.Fields["3238"].FormattedValue;
string city = loan.Fields["313"].FormattedValue;
string address = loan.Fields["319"].FormattedValue;
string loanAmount = loan.Fields["2"].FormattedValue;
if (loanAmount == "")
{
Console.WriteLine(LO);
continue;
}
int numLoans = 1;
addLoanFieldToListView(LO, numLoans, city, address, loanAmount);
if (--count == 0)
break;
}
}
I haven't been able to figure out how to use any of the pipeline methods to retrieve data outside the reporting database, but when all of the fields I am looking for are in the reporting database, it hardly takes a couple seconds to retrieve the contents of hundreds of loans using these tools:
session.Reports.SelectReportingFieldsForLoans(loanGUIDs, fields);
session.Loans.QueryPipeline(selectedDate, PipelineSortOrder.None);
session.Loans.OpenPipeline(PipelineSortOrder.None);
What would really help me is if somebody provided a simple example for retrieving data outside of the reporting database by using the encompass sdk that doesn't take longer than it ought to for retrieving the data.
Note: I am aware I can add the fields to the reporting database that aren't in it currently, so this is not the answer I am looking for.
Note #2: Encompass360 doesn't have it's own tag, if somebody knows of better tags that can be added for the subject at hand, please add them.

I use the SelectFields method on Loans to retrieve loan field data that is not in the reporting database in Encompass. It is very performant compared to opening loans up one by one but the results are returned as strings so it requires some parsing to get the values in their native types. Below is the example from the documentation for using this method.
using System;
using System.IO;
using EllieMae.Encompass.Client;
using EllieMae.Encompass.BusinessObjects;
using EllieMae.Encompass.Query;
using EllieMae.Encompass.Collections;
using EllieMae.Encompass.BusinessObjects.Loans;
class LoanReader
{
public static void Main()
{
// Open the session to the remote server
Session session = new Session();
session.Start("myserver", "mary", "maryspwd");
// Build the query criterion for all loans that were opened this year
DateFieldCriterion dateCri = new DateFieldCriterion();
dateCri.FieldName = "Loan.DateFileOpened";
dateCri.Value = DateTime.Now;
dateCri.Precision = DateFieldMatchPrecision.Year;
// Perform the query to get the IDs of the loans
LoanIdentityList ids = session.Loans.Query(dateCri);
// Create a list of the specific fields we want to print from each loan.
// In this case, we'll select the Loan Amount and Interest Rate.
StringList fieldIds = new StringList();
fieldIds.Add("2"); // Loan Amount
fieldIds.Add("3"); // Rate
// For each loan, select the desired fields
foreach (LoanIdentity id in ids)
{
// Select the field values for the current loan
StringList fieldValues = session.Loans.SelectFields(id.Guid, fieldIds);
// Print out the returned values
Console.WriteLine("Fields for loan " + id.ToString());
Console.WriteLine("Amount: " + fieldValues[0]);
Console.WriteLine("Rate: " + fieldValues[1]);
}
// End the session to gracefully disconnect from the server
session.End();
}
}

You will highly benefit from adding these fields to the reporting DB and using RDB query instead. Internally, Encompass has to open / parse files when you read fields without RDB, which is a slow process. Yet it just does a SELECT query on fields in RDB which is a very fast process. This tool will allow you quickly checking / finding which fields are in RDB so that you can create a plan for your query as well as a plan to update RDB: https://www.encompdev.com/Products/FieldExplorer
You query RDB via Session.Loans.QueryPipeline() very similarly to your use of Loan Query. Here's a good example of source code (in VB): https://www.encompdev.com/Products/AlertCounterFieldPlugin

Optimize or Shard Large set of permutations

I have data set that I have generate every permutation, then check some properties on it to see if is an object that I want to keep and use. The number of permutations is staggering, in the quadrillions. Is there anything that you can see in the code below that I can use to speed this up? I suspect that I can't speed it up to a reasonable amount of time, so I'm also looking at possibly sharding it onto multiple servers to process, but I'm having a hard time deciding where to shard it.
Any opinions or ideas is appreciated.
var boats = _warMachineRepository.AllBoats();
var marines = _warMachineRepository.AllMarines();
var bombers = _warMachineRepository.AllBombers().ToList();
var carriers = _warMachineRepository.AllCarriers().ToList();
var tanks = _warMachineRepository.AllTanks().ToList();
var submarines = _warMachineRepository.AllSubmarines();
var armies = new List<Army>();
int processed = 0;
Console.WriteLine((long)boats.Count*marines.Count*bombers.Count*carriers.Count*tanks.Count*submarines.Count);
// 70k of these
Parallel.ForEach(boats, new ParallelOptions(){MaxDegreeOfParallelism = Environment.ProcessorCount},boat =>
{
// 7500 of these
foreach (var marine in marines)
{
// 200 of these
foreach (var bomber in bombers)
{
// 200 of these
foreach (var carrier in carriers)
{
// 400 of these
foreach (var tank in tanks)
{
// 50 of these
foreach (var submarine in submarines)
{
var lineup = new Army()
{
Tank = tank,
Submarine = submarine,
Carrier = carrier,
Marine = marine,
Bomber = bomber,
Boats = boat
};
if (army.Hitpoints > 50000)
{
lock (lockObject)
{
armies.Add(lineup);
}
}
processed++;
if (processed%10000000 == 0)
{
Console.WriteLine("Processed: {0}, valid: {1}, DateTime: {2}", processed, armies.Count, DateTime.Now);
}
}
}
}
}
}
});
return armies;

If this code is referring to a simulation you might want to add some optimizations by:
Mark an object as changed (put it in a list) when it changes so there is no need to search multiple times
Decrease/throttle/tune the object update frequency
Use other information available to filter objects: are objects close to one another so they might affect/hurt/heal each other -> only then investigate changes
Change the data structure; by putting all attributes of all objects in a smartly setup matrix you might be able to use simple matrix multiplication to have the object interact. You might even be able to offload the multiplication to the GPU
You might be asking too much: so scale out by using more nodes/machines.

How to retrieve keywords within thousands of records from SQL Server 2008 fast?

Using the query function of entity collection in C# and it takes a long time to load the related records back from SQL Server 2008. Is there any fast way to do this? This is the query function I use:
public void SearchProducts()
{
//Filter by search string array(searchArray)
List<string> prodId = new List<string>();
foreach (string src in searchArray)
{
StoreProductCollection prod = new StoreProductCollection();
prod.Query.Where(prod.Query.StptName.ToLower() == src.ToLower() && prod.Query.StptDeleted.IsNull());
prod.Query.Select(prod.Query.StptName, prod.Query.StptPrice, prod.Query.StptImage, prod.Query.StptStoreProductID);
// prod.Query.es.Top = 4;
prod.Query.Load();
if (prod.Count > 0)
{
foreach (StoreProduct stpt in prod)
{
if (!prodId.Contains(stpt.StptStoreProductID.ToString().Trim()))
{
prodId.Add(stpt.StptStoreProductID.ToString().Trim());
productObjectsList.Add(stpt);
}
}
}
}

You're hitting the database once per searchArray item, this is very wrong.
You might get better performance like this (have no way of testing it, give it a shot):
public void SearchProducts()
{
//Filter by search string array(searchArray)
List<string> prodId = new List<string>();
StoreProductCollection prod = new StoreProductCollection();
// Notice that your foreach() is gone
// replace this
// prod.Query.Where(prod.Query.StptName.ToLower() == src.ToLower() && prod.Query.StptDeleted.IsNull());
// with this (or something similar: point is, you should call .Load() exactly once)
prod.Query.where(prod.Query.StptDeleted.IsNull() && src.Any(srcArrayString => prod.Query.StptName.ToLower()==srcArrayString.ToLower());
prod.Query.Select(prod.Query.StptName, prod.Query.StptPrice, prod.Query.StptImage, prod.Query.StptStoreProductID);
// prod.Query.es.Top = 4;
prod.Query.Load();
// ... rest of your code follows.
}

Given List<string> searchArray containing lowered words :
public void SearchProducts()
{
//Filter by search string array(searchArray)
List<string> prodId = new List<string>();
StoreProductCollection prod = new StoreProductCollection();
prod.Query.Where(searchArray.Contains(prod.Query.StptName.ToLower()) && prod.Query.StptDeleted.IsNull());
prod.Query.Select(prod.Query.StptName, prod.Query.StptPrice, prod.Query.StptImage, prod.Query.StptStoreProductID);
// prod.Query.es.Top = 4;
prod.Query.Load();
if (prod.Count > 0)
{
foreach (StoreProduct stpt in prod)
{
if (!prodId.Contains(stpt.StptStoreProductID.ToString().Trim()))
{
prodId.Add(stpt.StptStoreProductID.ToString().Trim());
productObjectsList.Add(stpt);
}
}
}
}
This way you have only one query for all words.

First of all, put an index on StptName column.
Second, if you need even better performance, write a Stored Procedure in SQL, to do your querying, and map it with Entity Framework.
Let me know if you need explanation on how to do any of the above.
A couple more micro-optimizations you can do if you don't want to write a Stored Procedure:
Write src.ToLower() in a temporary varaible, and than compare prod.Query.StptName.ToLower() to it.
By default, SQL Server queries are case insensitive, so check if that's the case, and if so, you can get rid of the ToLower altogether. You can change case sensitivity through Collation.
EDIT:
To create an Index:
Open the table designer in SQL Server Managment Studio.
Right click anywhere and select Indexes/Keys.
Click Add.
Under Columns add StptName.
Under Is Unique specify whether StptName is unique or not.
Under type select "index".
That's all!
As for mapping stored procedures - here's a nice tutorial:
http://www.robbagby.com/entity-framework/entity-framework-modeling-select-stored-procedures/
(You can jump straight to the "Map in the Select Stored Procedure" Section).