Develop Reference

Google Analytics API pagination missing records

We have a system that queries GA API for a large amount of tokens, mostly for website visits and sessions data.
We have noticed lately that we're getting weird results while querying the API - specifically we're seeing missing records from the result set. Even more specifically, it looks like when we have a few pages of rows, when starting the next page the results would "skip" the beginning of the page.
This behavior is not consistent - each run a different set of sites/tokens would display this bug, and when I tried debugging the code manually I've never encountered this behavior.
At first I thought the problem was with our code, maybe some sort of race condition or shared memory, but it seems like the problem is with the API access itself - this is because I've checked the TotalResults property that is returned with the query, and when this bug happens it shows less rows in total than the amount I see when I query manually.
For example, we'd query for a site with date and country dimensions, and the rows logged would be:
domain | year | month | day | country | metrics
-----------------------------------------------
X.com 2017 09 22 IT ..... // metrics
// finished result page
X.com 2017 09 24 BW ..... // metrics
....
Total rows - 1295
when we ran the same code again, we got rows with 2017-09-23 values for this site, and total rows - 1368
Is this a bug in the API? Or maybe in the way we access it? I haven't found a mention of such a problem.
EDIT: I've added the API call method code we use.
private GaDataFlat GetDataV3(string type, string profileID,
List<Metric> v4metrics, List<MetricFilterClause> v4metricFilters,
List<Dimension> v4dimensions, List<DimensionFilterClause> v4dimensionFilters,
List<OrderBy> v4sorting, DateTime start, DateTime end, int maxResults)
{
List<string> metrics = (v4metrics == null ? null : v4metrics.Select(x => x.Expression).ToList());
List<string> dimensions = (v4dimensions == null ? null : v4dimensions.Select(x => x.Name).ToList());
List<string> sorting = (v4sorting == null ? null : v4sorting.Select(x => x.FieldName).ToList());
List<string> filters = (v4dimensionFilters == null ? null : v4dimensionFilters.Select(x => deconstructFilter(x)).ToList());
return ExponentialBackoff.Go(() =>
{
var gaData = new GaDataFlat { DataTable = new DataTable() };
DataResource.GaResource.GetRequest request = service.Data.Ga.Get("ga:" + profileID,
start.ToString("yyyy-MM-dd"), end.ToString("yyyy-MM-dd"), String.Join(",", metrics));
//Set the user Quota to not have concurrent limitiation
request.QuotaUser = profileID + Thread.CurrentThread.ManagedThreadId;
if (dimensions != null)
{
request.Dimensions = string.Join(",", dimensions);
}
if (filters != null)
{
request.Filters = string.Join(";", filters);
}
if (sorting != null)
{
request.Sort = "-" + string.Join(";-", sorting);
}
request.SamplingLevel = DataResource.GaResource.GetRequest.SamplingLevelEnum.HIGHERPRECISION;
bool hasNext;
int rowCount = 0;
int iteration = 0;
do
{
iteration++;
MetricsProvider.Counter("ga.iteration", 1, "type:" + type);
if (iteration > 100)
{
string error = "Too many iterations ";
LogFacade.Fatal(error);
throw new Exception(error);
}
if (!counter.IncrementAndCheckAvailablility(Constants.APIS.GA))
{
Console.WriteLine("Daily Limit Exceeded - counter");
throw new QuotaExceededException();
}
GaData DataList = request.Execute();
gaData.SampleSize = DataList.SampleSize;
gaData.SampleSpace = DataList.SampleSpace;
if (DataList.Rows != null)
{
if (gaData.DataTable.Columns.Count == 0)
{
for (int j = 0; j < DataList.ColumnHeaders.Count; j++)
{
gaData.DataTable.Columns.Add(new DataColumn
{
ColumnName = DataList.ColumnHeaders[j].Name
});
}
}
foreach (var row in DataList.Rows.ToList())
{
var reportRow = new List<object>();
for (int j = 0; j < DataList.ColumnHeaders.Count; j++)
{
reportRow.Add(row[j]);
}
Console.WriteLine(string.Join(":", v4dimensionFilters.SelectMany(f => f.Filters.SelectMany(inner => inner.Expressions))) + "," +
string.Join(",", reportRow.Select(cell => cell.ToString())));
gaData.DataTable.Rows.Add(reportRow.ToArray());
}
rowCount += DataList.Rows.Count;
request.StartIndex = rowCount;
Console.WriteLine(string.Join(":", v4dimensionFilters.SelectMany(f => f.Filters.SelectMany(inner => inner.Expressions))) + ", next page starts " + request.StartIndex);
hasNext = rowCount < DataList.TotalResults;
}
else
{
hasNext = false;
}
} while (hasNext && (maxResults == 0 || rowCount < maxResults));
return gaData;
}, type, "GetData " + profileID + " " + Thread.CurrentThread.ManagedThreadId);
}
EDIT: The filter we use is consistent - for example, we want to get desktop visits for website x.com, the filter would be:
ga:hostname=~x\.com(\/|)$;ga:deviceCategory==desktop

How to find duplicate values while importing excel and displays an error message on the same table where imported data displays?

I am using MVC with C#, I am trying to import an excel sheet and validate those sheet and displays the excel data along with error message on one column in a table. If got any error my selection checkbox will not available, so the user cannot select and save the particular record. my page is actually working. What is my issue right now is, I wanna add Unique id validation. which means I have one Id column, where I don't want duplicate value. I added validation to check with my table and if found any same id, it returns an error message. but, if let say, user upload duplicate values on the excel sheet when they upload means, how to find out while uploading and how to prevent them from adding duplicate id records.
my sample coding below.
if (theFile != null && theFile.ContentLength > 0)
{
try
{
string[,] data = ExportUtil.GetData(theFile);
int rowValue = data.GetLength(0);
int colValue = data.GetLength(1);
Info _I;
ViewModel _VM;
for (int i = 0; i < rowValue; i++)
{
_VM= new ViewModel();
// _VM.Id = i;
_VM.Id = data[i, 0].ToString() != null ? data[i, 0].ToString() : "";
_VM.Description = data[i, 1].ToString() != null ? data[i, 1].ToString() : "";
if (string.IsNullOrWhiteSpace(_VM.Id))
{
_VM.Message = "Id cannot be empty" + System.Environment.NewLine;
}
_ID = TagInfo.Where(a => !string.IsNullOrEmpty(_VM.Id) && a.Id.ToUpper() == _VM.Id.ToUpper()).FirstOrDefault();
if (_ID != null)
{
_VM.Message += "Duplicate ID" + System.Environment.NewLine;
_ID =null;
}
if (string.IsNullOrEmpty(_VM.Description))
{
_VM.Message += "Description cannot be empty" + System.Environment.NewLine;
}
if (!string.IsNullOrEmpty(_VM.Message))
{
_VM.Message = string.Format("{0}{1}", "Row Number " + (i + 1) + " has " + Environment.NewLine, _VM.Message);
}
listvm.Add(_VM);
}
TempData["ID_DOWNLOAD"] = listvm;
}
}
_ID I declared with table name TableID _ID; above the try block. kindly help.

Add your unique ID to a list of int then find duplicate ids in that list by linq
var duplicateKeys = list.GroupBy(x => x)
.Where(group => group.Count() > 1)
.Select(group => group.Key);
if duplicateKeys count is greater than 1 then you know there are some IDs duplicated.

How to display instant output in gridview for every loop in asp.net C#

I am collecting the number of system details and their availability in grid view, using the FOR loop for doing the same. Do I have any option of updating grid once each loop completes (every iteration of FOR loop)?
gvReport -> Grid View, which am using here
for (int i = 0; i < table.Rows.Count; i++)
{
RfcDestinationManager.RegisterDestinationConfiguration(sapcfg);
RfcDestination dest = RfcDestinationManager.GetDestination(table.Rows[i]["host"].ToString() + "," + table.Rows[i]["SID"].ToString() + "," + table.Rows[i]["Client"].ToString() + "," + table.Rows[i]["InsNo"].ToString());
if (dest != null)
{
result = true
dest.Ping();
}
if (result == true)
{
gvReport.Rows[i].Cells[1].BackColor = Color.Green;
}
}

How to set Countdown For Each Candidate using Jquery

I am working on online Q/A system,i have to show countdown for each candidate like 3 min,on expiring user will be redirect to Result.aspx page.
I am facing following
1.how to set counter for each candidate.
2.on page refresh counter set to default value.
i have following code
<div id="timer">
</div>
<script type="text/javascript">
function countdown(minutes) {
var seconds = 60;
var mins = minutes;
if (getCookie("minutes") && getCookie("seconds")) {
var seconds = getCookie("seconds");
var mins = getCookie("minutes");
}
function tick() {
var counter = document.getElementById("timer");
setCookie("minutes", mins, 10)
setCookie("seconds", seconds, 10)
var current_minutes = mins - 1
seconds--;
counter.innerHTML =
current_minutes.toString() + ":" + (seconds < 10 ? "0" : "") + String(seconds);
//save the time in cookie
//minutesSpan.innerHTML = current_minutes.toString();
//secondsSpan.innerHTML = (seconds < 10 ? "0" : "") + String(seconds);
if (seconds > 0) {
setTimeout(tick, 1000);
}
else {
if (mins > 1) {
// countdown(mins-1); never reach “00″ issue solved:Contributed by Victor Streithorst
setTimeout(function () { countdown(mins - 1); }, 1000);
}
}
}
tick();
}
function setCookie(cname, cvalue, exdays) {
var d = new Date();
d.setTime(d.getTime() + (exdays * 24 * 60 * 60 * 1000));
var expires = "expires=" + d.toGMTString();
document.cookie = cname + "=" + cvalue + "; " + expires;
}
function getCookie(cname) {
var name = cname + "=";
var ca = document.cookie.split(';');
for (var i = 0; i < ca.length; i++) {
var c = ca[i];
while (c.charAt(0) == ' ') c = c.substring(1);
if (c.indexOf(name) == 0) {
return c.substring(name.length, c.length);
}
}
return "";
}
countdown(3);
</script>
because multiple user will doing these test so i have handle each one differently,i have following code to assign to assign test for each candidate
private void NewCandidate()
{
var ClistCount = (from cl in ef.Candidate_Table where cl.Name == btnname.Text && cl.Status_Id==2 select cl).FirstOrDefault();
if (ClistCount != null)
{
string cnic = ClistCount.Cnic;
Session["token"] = cnic;
Response.Redirect("MainTestForm.aspx?id=" + cnic);
}
else
{
MsgLitteral.ShowNotification("No Candidate Is Here", true);
btnbrowse.Visible = false;
btnname.Visible = false;
}
}

There are two things which you need to do in order to make things working
1) Create perfectly working count down timer method
2) Solve the reload dependency
for reload thing just before reload/refresh, trigger a function that would store the current time elapsed from the predefined count down.
$(window).bind('beforeunload', function(){
//below function stores current elapsed time in cookie/local storage
callFunction();
return true;
});
for e.g if 3min countdown is set and the user refreshes or moves to
next question at 2min 40 secs then store 2min 40 sec in cookies or
html5 local storage
On every document ready event check the cookie value
if value present then
take this value and set countdown
else
with predefined value (for the first time case)
A simple countdown timer for reference

I need code for FOREACH loop or for loop for four variables

I have four arrays of equal length
I am using those arrays in foreach loops.
I am using four variables(i,j,k,l) to increment and to proceed
I am writing my code inside the four nested loops, that code should execute when i=0,j=0,k=,0,l=0
i=1,j=1,k=1,l=1
i=2,j=2,k=2,l=2
.....(depending on the array length)
Please suggest me the code for this required segment.
int i = 0, j = 0, k = 0, l = 0;
foreach (string fieldName in splitFieldnames)
{
i = 0;
foreach (string dataType in splitDatatypeNames)
{
j = 0;
foreach (string controlName in SplitControlNames)
{
k = 0;
foreach (string controlType in splitControlTypeNames)
{
if (i == j && j == k && k == l)
{
if (controlType == "textbox" && dataType == "string")
{
Response.Write("_Student." + fieldName + "= " + controlName + ".Text;");
l++;
break;
}
}
k++;
}
j++;
}
i++;
}
}

I think using LINQ would save you a lot of manual coding:
static void Main(string[] args)
{
var splitFieldnames = new string[] { "field1", "field2", "field3" };
var splitDatatypeNames = new string[] { "datatype1", "string", "string" };
var SplitControlNames = new string[] { "control1", "control2", "control3" };
var splitControlTypeNames = new string[] { "combobox", "textbox", "textbox"};
// this code can handle different sized arrays, but is based strictly
// on the size of the splitFieldnames array as the base.
var splitMerged = splitFieldnames.Select
((c, idx) =>
new
{
fieldName = c,
dataType = splitDatatypeNames.Length > idx ?
splitDatatypeNames[idx] : "",
controlName = SplitControlNames.Length > idx ?
SplitControlNames[idx] : "",
controlTypeName = splitControlTypeNames.Length > idx?
splitControlTypeNames[idx] : "",
});
foreach (var item in splitMerged
.Where(c => c.controlTypeName == "textbox" && c.dataType == "string"))
{
Response.Write("_Student." + item.fieldName + "= "
+ item.controlName + ".Text;");
}
The resulting output would be look like:
_Student.field2= control2.Text;
_Student.field3= control3.Text;
I hope that's what you're looking for, LOL...

I think that in this case a DataTable would be better than 4 arrays, and algorithm you require will be trivial with such data structure.

Use for instead of foreach
for (int i = 0; i < splitFieldnames.Length; i++)
{
string fieldName = splitFieldnames[i];
for (int j = 0; j < splitDatatypeNames.Length; j++)
{
string dataType = splitDatatypeNames[j];
for (int k = 0; k < SplitControlNames.Length; k++)
{
string controlName = SplitControlNames[k];
for (int l = 0; l < splitControlTypeNames.Length; l++)
{
string controlType = splitControlTypeNames[l];
if (i == j && j == k && k == l)
{
if (controlType == "textbox" && dataType == "string")
{
Response.Write("_Student." + fieldName + "= " + controlName + ".Text;");
break;
}
}
}
}
}
}
Note also that break will only break out of the innermost loop! Consider using a return statement instead.
UPDATE (in response to your edit):
The solution is simple, use only one index variable and only one for-loop instead of a lot of foreach-loops
for (int i = 0; i < splitFieldnames.Length; i++)
{
if (splitControlTypeNames[i] == "textbox" && splitDatatypeNames[i] == "string")
{
Response.Write("_Student." + splitFieldnames[i] + "= " + SplitControlNames[i] + ".Text;");
break;
}
}
(Assuming that you want to stop after the first match. If you want to output all string textboxes, drop the break statement.)

Ok, this code is really simple.. If you only need to show the data when i=j=k=l, then no need for any loop, or even for the existence of i,j,k,l.
If you can assure me that all this strings in whatever thing (a collection? an array? a dictionary?) are ordered, is ok.
If they are ordered, then throw away all the for each, and just access each one by position for all the controls you have in the collection that has the minimun.
If all this collection are not ordered, then this is totally uselles, since it will produce differents results on each run.
I will edit this answers with some code, once you can tell me if this is ordered or not.
EDIT:
First of all, you need to check what collection has the least items (since you can't go beyond that)... I don't know the types of this things (you didn't provide them), so let's assume they have a count property.
int minimun = splitFieldnames.count;
if (splitDatatypeNames.count < minimun)
minimun = splitDatatypeNames.count;
if (SplitControlNames.count < minimun)
minimun = SplitControlNames.count
if (splitControlTypeNames.count < minimun)
minimun = splitControlTypeNames.count
once you have the minimun value (since you can't go beyond that), just iterate on that and print whatever you want
for (int i = 0; i < minimun;i++)
{
if (splitControlTypeNames[i].tostring() == "textbox" && splitDatatypeNames[i].tostring() == "string")
{
//Response.Write("_Student." + fieldName + "= " + controlName + ".Text;");
//Also, a parametric string would be better ;)
string result = string.format("_Student.{0}= {1}.Text;",splitFieldnames[0].tostring(),SplitControlNames[0].tostring());
Response.Write(result);
l++;
}
}
I don't know the types, so I'm assuming that they have a tostring method an a count property

If what you are trying to accomplish is process the same index in the four arrays, just use one loop and use a counter to access the value in each array in that one loop:
foreach(string fieldName in SplitControlNames)
{
dataType = splitDatatypeNames[arrayPosition];
controlName = SplitControlNames[arrayPosition];
controlType = splitControlTypeNames[arrayPosition];
if (controlType == "textbox" && dataType == "string")
Response.Write("_Student." + fieldName + "= " + controlName + ".Text;");
arrayPosition++;
}
Or, create and populate a structure that has four values in it (fieldName, dataType, controlName, controlType) and have one array of that structure