I am using LinqToTwitter Api and context for getting all tweets from screen name but i get stuck around 3200 tweets. using max_id and since_id concept as well.
what changes i need to do to get all tweets from a profile using screen name?
Below is my code :`
List<Status> searchResults = new List<Status>();
try
{
var twitterCtx = new TwitterContext(auth);
int maxNumberToFind = 200;
ulong sinceID = 1;
ulong maxID;
var tweets = (from tweet in twitterCtx.Status
where tweet.Type == StatusType.User &&
tweet.ScreenName == screenName &&
tweet.Count == maxNumberToFind &&
tweet.SinceID == sinceID &&
tweet.Retweeted==false
select tweet).ToList();
if (tweets.Count > 0)
{
maxID = searchResults.Min(status => status.StatusID) - 1;
searchResults.AddRange(tweets);
}
do
{
tweets.Clear();
tweets = (from tweet in twitterCtx.Status
where tweet.Type == StatusType.User &&
tweet.ScreenName == screenName &&
tweet.Count == maxNumberToFind &&
tweet.MaxID == maxID &&
tweet.SinceID == sinceID &&
tweet.Retweeted == false
select tweet).ToList();
searchResults.AddRange(tweets);
maxID = tweets.Min(x => x.StatusID);
} while (searchResults.Count <= 5000);//tweets.Any()`
}
catch (Exception ex) { ex.ToString(); }
return searchResults;
}
You can not get more than 3200 tweets for a screen name. It is an API restriction :
https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline
To get more, you can try web scraping or you can buy data from Gnip.
Bests
Related
I have this below linq query of which 95% remains the same across 6 different scenarios.How can I refactor it so that I can write in one function and use it for all the cases?
var results = await (from r in Requests
join ra in RequestAuthorisers
on cr.ID equals ra._REQUEST_ID
where cr.FACILITY_ID == facilityId
&& (r._REQUEST_STATUS_ID == RequestSubmitted || r._REQUEST_STATUS_ID == RequestPartiallyAuthorised )//check against more status codes based on different conditions
&& ra.FACILITY_USER_ID == facilityUserId//don't check this if admin
&& ra.AUTHORIZATION_DATE != null
&& ra.REJECTION_DATE == null
select new
{
FacilityId = r.FACILITY_ID,
VersionId = r.VERSION_ID,
CreatedByTitle = r.CREATED_BY_USER_TITLE,
CreatedByFirstName = r.CREATED_BY_USER_FIRST_NAME,
CreatedByLastName = r.CREATED_BY_USER_LAST_NAME,
RequestDate = r.SUBMITTED_DATE,
RequestId = r.ID,
RequestType = r._TYPE_ID,
Status = r._REQUEST_STATUS_ID
}).ToListAsync();
var RequestResponse = results.Select(
r => new RequestResponseDto
{
FacilityId = r.FacilityId,
VersionId = r.VersionId,
CreatedByTitle = r.CreatedByTitle,
CreatedByFirstName = r.CreatedByFirstName,
CreatedByLastName = r.CreatedByLastName,
RequestDate = Convert.ToDateTime(r.RequestDate).ToString("s"),
RequestType = ((RequestType)r.RequestType).ToString(),
Status = ((RequestStatus)r.Status).ToString()
}).ToList();
As you can the scenarios/queries differ only by if isAdmin and check against more status codes(as commented above), rest of the part becomes same.
I am using LINQ to EF(DB First).
Thanks in advance.
Why not pass a boolean into the method:
public RequestResponseDto MyMethod(bool isAdmin, Status status)
{
...
&&
((status == Status.Status1 && (r._REQUEST_STATUS_ID == RequestSubmitted || r._REQUEST_STATUS_ID == RequestPartiallyAuthorised))
||
(status == Status.Status2 && (r._REQUEST_STATUS_ID == Something))
||
(status == Status.Status3 && (r._REQUEST_STATUS_ID == SomethingElse || r._REQUEST_STATUS_ID == AnotherThing)))
...
&& (isAdmin || ra.FACILITY_USER_ID == facilityUserId)
...
}
I have following method that registers a contact in database, but before register I check the contact exists or not:
bool RegisterContact(Contact contactInfo) {
bool entityExists =
_dbContext.Contacts.FirstOrDefault(
p => (p.FilesID.Equals(contactInfo.FilesID))
&& (p.EmailAddress ==
(string.IsNullOrEmpty(
contactInfo.EmailAddress)
? p.EmailAddress
: contactInfo.EmailAddress))
&&
(p.DisplayName ==
(string.IsNullOrEmpty(
contactInfo.DisplayName)
? p.DisplayName
: contactInfo.DisplayName)));
}
this query includes the fields that contain value (not null) in search condition (FilesID, EmailAddress, DisplayName)
this technique works fine in MSSQL, today i changed the database manager to PostgreSQL and use Npgsql.
All things work except above linq query, which raises an exception with message of : "could not determine data type of parameter $2"
I was forced to solve it in this way:
bool RegisterContact(Contact contactInfo)
{
Contact entityExists = null;
if (string.IsNullOrEmpty(contactInfo.EmailAddress) &&
(string.IsNullOrEmpty(contactInfo.DisplayName)))
entityExists =
_dbContext.Contacts.FirstOrDefault(
p => p.FilesID.Equals(contactInfo.FilesID));
if (!string.IsNullOrEmpty(contactInfo.EmailAddress) && string.IsNullOrEmpty(contactInfo.DisplayName))
entityExists =
_dbContext.Contacts.FirstOrDefault(
p =>
p.FilesID.Equals(contactInfo.FilesID) &&
p.EmailAddress == contactInfo.EmailAddress);
if (string.IsNullOrEmpty(contactInfo.EmailAddress) && !string.IsNullOrEmpty(contactInfo.DisplayName))
entityExists =
_dbContext.Contacts.FirstOrDefault(
p =>
p.FilesID.Equals(contactInfo.FilesID) &&
p.DisplayName == contactInfo.DisplayName);
if (!string.IsNullOrEmpty(contactInfo.EmailAddress) &&
!string.IsNullOrEmpty(contactInfo.DisplayName))
entityExists =
_dbContext.Contacts.FirstOrDefault(
p =>
p.FilesID.Equals(contactInfo.FilesID) &&
p.EmailAddress == contactInfo.EmailAddress &&
p.DisplayName == contactInfo.DisplayName);
}
Is this Npgsql bug or by design? any known solutions/workarounds for the problem?
I currently have the same cases. I think the problem is the lack of recognition, by NpgSQL, of string.IsNullOrEmpty.
I replaced the test with a check on empty string, always recognizing as not NULL the input parameter.
-- bad
var data = from art in _ctx.Set<Soleo.Model.DLAR>()
from iva in _ctx.Set<Soleo.Model.DLAI>().Where(k => k.DITTA == art.DITTA && k.COD == art.CIVA).DefaultIfEmpty()
from fam in _ctx.Set<Soleo.Model.DLFA>().Where(k => k.DITTA == art.DITTA && k.COD == art.FAM).DefaultIfEmpty()
from mar in _ctx.Set<Soleo.Model.DLMA>().Where(k => k.DITTA == art.DITTA && k.COD == art.MAR).DefaultIfEmpty()
from udm in _ctx.Set<Soleo.Model.DLUM>().Where(k => k.DITTA == art.DITTA && k.COD == art.UM).DefaultIfEmpty()
where art.DITTA == DLAUTH.Config.Current.DITTA && art.COD.Contains(sel_cod) && art.DES.Contains(sel_des)
&& (string.IsNullOrEmpty(sel_fam) || string.Compare(art.FAM, sel_fam, true) == 0)
&& (string.IsNullOrEmpty(sel_mar) || string.Compare(art.MAR, sel_mar, true) == 0)
&& (art.DIS >= sel_dis_da && art.DIS <= sel_dis_a)
select new
{
COD = art.COD,
DES = art.DES,
DES_UDM = udm.DES,
DES_MAR = mar.DES,
DES_FAM = fam.DES,
DES_CIVA = iva.DES,
MAG1 = art.MAG1,
MAG2 = art.MAG2,
DES_DIS = art.DIS == 1 ? "Si" : "No"
};
-- good:
var data = from art in _ctx.Set<Soleo.Model.DLAR>()
from iva in _ctx.Set<Soleo.Model.DLAI>().Where(k => k.DITTA == art.DITTA && k.COD == art.CIVA).DefaultIfEmpty()
from fam in _ctx.Set<Soleo.Model.DLFA>().Where(k => k.DITTA == art.DITTA && k.COD == art.FAM).DefaultIfEmpty()
from mar in _ctx.Set<Soleo.Model.DLMA>().Where(k => k.DITTA == art.DITTA && k.COD == art.MAR).DefaultIfEmpty()
from udm in _ctx.Set<Soleo.Model.DLUM>().Where(k => k.DITTA == art.DITTA && k.COD == art.UM).DefaultIfEmpty()
where art.DITTA == DLAUTH.Config.Current.DITTA && art.COD.Contains(sel_cod) && art.DES.Contains(sel_des)
&& (string.Compare(sel_fam, "", true) == 0 || string.Compare(art.FAM, sel_fam, true) == 0)
&& (string.Compare(sel_mar, "", true) == 0 || string.Compare(art.MAR, sel_mar, true) == 0)
&& (art.DIS >= sel_dis_da && art.DIS <= sel_dis_a)
select new
{
COD = art.COD,
DES = art.DES,
DES_UDM = udm.DES,
DES_MAR = mar.DES,
DES_FAM = fam.DES,
DES_CIVA = iva.DES,
MAG1 = art.MAG1,
MAG2 = art.MAG2,
DES_DIS = art.DIS == 1 ? "Si" : "No"
};
But I do not think this is the solution. I will report the case to NpgSQL.
It clearly state that it allows to get up to 3200 tweets per person (see documentation). But using this code:
var statusTweets =
from tweet in twitterCtx.Status
where tweet.Type == StatusType.User
&& tweet.ScreenName == "JoeMayo"
select tweet;
statusTweets.ToList().ForEach(
tweet => Console.WriteLine(
"Name: {0}, Tweet: {1}\n",
tweet.User.Name, tweet.Text));
it gives me the first 20 tweets only, also you can't apply cursor movement using this code. Does anyone have a LINQ To Twitter code that allows me to fetch as many tweets as possible?
To fetch tweets from Twitter we can get by 2 ways:
using Hastag and ScreenName
I created one custom method where you can get 3200 tweets in one request.
public void GetTwitterFeeds(dynamic settings, bool ishashtag, DateTime startDate, DateTime endDate)
{
string screenname = settings.socialFetchTerm.ToString();
var auth = new SingleUserAuthorizer
{
Credentials = new InMemoryCredentials
{
ConsumerKey = AppSettings.ConsumerKey,
ConsumerSecret = AppSettings.ConsumerSecret,
OAuthToken = AppSettings.AccessToken,
AccessToken = AppSettings.AccessTokenSecret
}
};
var twitterCtx = new TwitterContext(auth);
var ownTweets = new List<Status>();
ulong sinceId = 0;
ulong maxID = 0;
int lastStatusCount = 0;
bool flag = true;
var statusResponse = new List<Status>();
if (!ishashtag)
{
statusResponse = (from tweet in twitterCtx.Status
where tweet.Type == StatusType.User
&& tweet.ScreenName == screenname
&& tweet.Count == 200
&& (tweet.CreatedAt >= startDate && tweet.CreatedAt <= endDate)
select tweet).ToList();
}
else
{
statusResponse = (from search in twitterCtx.Search
where search.Type == SearchType.Search &&
search.Query == screenname
&& search.Count == 200
from Status status in search.Statuses
where (status.CreatedAt >= startDate && status.CreatedAt <= endDate)
select status
).ToList();
}
if (statusResponse.Count > 0)
{
maxID = ulong.Parse(statusResponse.First().StatusID);
ownTweets.AddRange(statusResponse);
}
do
{
int rateLimitStatus = twitterCtx.RateLimitRemaining;
if (rateLimitStatus != 0)
{
if (ishashtag)
{
statusResponse = (from search in twitterCtx.Search
where search.Type == SearchType.Search &&
search.Query == screenname
&& search.Count ==200
from Status status in search.Statuses
where
(status.CreatedAt >= startDate && status.CreatedAt <= endDate) &&
status.SinceID == sinceId && status.MaxID == maxID
select status
).ToList();
lastStatusCount = statusResponse.Count;
if (lastStatusCount != 0)
{
maxID = ulong.Parse(statusResponse.Last().StatusID) - 1;
ownTweets.AddRange(statusResponse);
}
else
{
flag = false;
}
}
else
{
statusResponse = (from tweet in twitterCtx.Status
where tweet.Type == StatusType.User
&& tweet.ScreenName == screenname
&& tweet.SinceID == sinceId && tweet.MaxID == maxID
&& tweet.Count == 200
&& (tweet.CreatedAt >= startDate && tweet.CreatedAt <= endDate)
select tweet).ToList();
lastStatusCount = statusResponse.Count;
if (lastStatusCount != 0)
{
maxID = ulong.Parse(statusResponse.Last().StatusID) - 1;
ownTweets.AddRange(statusResponse);
}
else
{
flag = false;
}
}
}
else
{
flag = false;
}
} while (flag);
foreach (var tweetStatus in ownTweets)
{
if (tweetStatus != null)
{
var socialSiteData = new SocialSitesData
{
// SocialType = SocialType.Twitter,
SocialType = settings.socialType,
SocialSubType = settings.socialSubType,
SocialFetchTerm = settings.socialFetchTerm,
PostId = tweetStatus.StatusID,
Post = tweetStatus.Text,
PostUrl = "https://twitter.com/" + tweetStatus.ScreenName + "/status/" + tweetStatus.StatusID,
ImageSource =
tweetStatus.Entities.MediaEntities.Count > 0
? tweetStatus.Entities.MediaEntities[0].MediaUrl
: "",
VideoSource =
tweetStatus.Entities.UrlEntities.Count > 0
? tweetStatus.Entities.UrlEntities[0].ExpandedUrl
: "",
PostTime = tweetStatus.CreatedAt,
};
if (!_socialHubCrudDal.IsSocialSiteDataExists(tweetStatus.StatusID))
_socialHubCrudDal.AddSocialSiteData(socialSiteData);
}
}
}
The documentation says "Gets the 20 most recent statuses for the authorized user".
You might have to use/send additional "Parameters/Filters" - count as mentioned in the document, which states "Number of tweets to return; max is 200." and which is an optional field, and hence only top 20 is returned in your case.
Recoded from #Rakesh Sadhula
public void GetTwitterFeeds(string screenName, DateTime startDate, DateTime endDate)
{
var twitterCtx = new TwitterContext(authorizer);
var ownTweets = new List<Status>();
ulong maxID = 0;
int lastStatusCount = 0;
bool flag = true;
var statusResponse = new List<Status>();
statusResponse = (from tweet in twitterCtx.Status
where tweet.Type == StatusType.User
&& tweet.ScreenName == screenName
&& tweet.Count == 200
&& (tweet.CreatedAt >= startDate && tweet.CreatedAt <= endDate)
select tweet).ToList();
if (statusResponse.Count > 0)
{
maxID = ulong.Parse(statusResponse.Last().StatusID.ToString());
ownTweets.AddRange(statusResponse);
}
if (ownTweets.Count == 200)
{
do
{
int rateLimitStatus = twitterCtx.RateLimitRemaining;
if (rateLimitStatus != 0)
{
statusResponse = (from tweet in twitterCtx.Status
where tweet.Type == StatusType.User
&& tweet.ScreenName == screenName
&& tweet.MaxID == maxID
&& tweet.Count == 200
&& (tweet.CreatedAt >= startDate && tweet.CreatedAt <= endDate)
select tweet).ToList();
lastStatusCount = statusResponse.Count;
if (lastStatusCount != 0)
{
maxID = ulong.Parse(statusResponse.Last().StatusID.ToString()) - 1;
ownTweets.AddRange(statusResponse);
}
else
{
flag = false;
}
}
else
{
flag = false;
}
} while (flag);
}
if (statusResponse.Count > 0)
{
foreach (Status tweet in ownTweets)
{
listTweets.Items.Add(tweet.Text);
}
lb_tUser.Text = "#" + screenName;
lb_tweeted.Text = statusResponse.Count.ToString();
}
}
I have the following code:
result = from i in _dbContext.Meetings
where i.UserInvitedID == CurrentUserID && i.MeetingStatus == null && //!i.IsTex &&
DbFunctions.AddMinutes(DbFunctions.AddHours(i.MeetingTime.Day, i.MeetingTime.Hour).Value, i.MeetingTime.Minute).Value > dateWithTime
//where i.UserInvitedID == CurrentUserID && i.MeetingStatus == null && DbFunctions.TruncateTime(i.AllowedTime.AllowedDate.Day) >= date
select new ITW2012Mobile.Core.DataTable.MeetingModel2Tmp()
{
Name = i.UserInviter.FirstName + " " + i.UserInviter.LastName,
Company = i.UserInviter.Company,
Company2 = i.UserInvited.Company,
MeetingID = i.MeetingID,
Time = DbFunctions.AddMinutes(DbFunctions.AddHours(i.MeetingTime.Day, i.MeetingTime.Hour).Value, i.MeetingTime.Minute).Value,
CreatedTime = i.dateCreated,
Image = i.UserInviter.ProfileImage,
TableNumber = i.TableNumber,
Username = i.UserInviter.aspnet_User.UserName,
Username2 = i.UserInvited.aspnet_User.UserName,
UsernameInviter = i.UserInviter.aspnet_User.UserName,
RequestText = i.RequestText,
NoteInviter = i.NoteInviter,
ResendInvitationCount = (i.ResendInvitationCount.HasValue) ? i.ResendInvitationCount.Value : 0,
NoteInvited = i.NoteInvited,
MeetingType = i.MeetingType.TypeName
};
I use many such extentions with different modifications, but each of them has the same part select new ITW2012Mobile.Core.DataTable.MeetingModel2Tmp().....
i.e.
result = from i in _dbContext.Meetings
where i.UserInviterID == CurrentUserID && i.MeetingStatus == null && !i.IsTex && DbFunctions.AddMinutes(DbFunctions.AddHours(i.MeetingTime.Day, i.MeetingTime.Hour).Value, i.MeetingTime.Minute).Value > dateWithTime
//where i.UserInviterID == CurrentUserID && i.MeetingStatus == null && DbFunctions.TruncateTime(i.AllowedTime.AllowedDate.Day) >= date
select new ITW2012Mobile.Core.DataTable.MeetingModel2Tmp()
{
Name = i.UserInvited.FirstName + " " + i.UserInvited.LastName,
//...
};
result = from i in _dbContext.Meetings
where (i.UserInviterID == CurrentUserID) && i.MeetingStatus == true && !i.IsTex && DbFunctions.AddMinutes(DbFunctions.AddHours(i.MeetingTime.Day, i.MeetingTime.Hour).Value, i.MeetingTime.Minute).Value > dateWithTime
//where (i.UserInviterID == CurrentUserID) && i.MeetingStatus == true && DbFunctions.TruncateTime(i.AllowedTime.AllowedDate.Day) >= date
select new ITW2012Mobile.Core.DataTable.MeetingModel2Tmp()
{
Name = i.UserInvited.FirstName + " " + i.UserInvited.LastName,
//...
};
Can I set this part to variable to use in all other extentions?
You'll need to give up on the LINQ query syntax, and call the Select method directly. That should be no big deal. If you do that, you can store the projection part of your query in a separate variable.
Expression<Func<Meeting, ITW2012Mobile.Core.DataTable.MeetingModel2Tmp>> projection = i =>
new ITW2012Mobile.Core.DataTable.MeetingModel2Tmp()
{
Name = i.UserInviter.FirstName + " " + i.UserInviter.LastName,
Company = i.UserInviter.Company,
Company2 = i.UserInvited.Company,
MeetingID = i.MeetingID,
Time = DbFunctions.AddMinutes(DbFunctions.AddHours(i.MeetingTime.Day, i.MeetingTime.Hour).Value, i.MeetingTime.Minute).Value,
CreatedTime = i.dateCreated,
Image = i.UserInviter.ProfileImage,
TableNumber = i.TableNumber,
Username = i.UserInviter.aspnet_User.UserName,
Username2 = i.UserInvited.aspnet_User.UserName,
UsernameInviter = i.UserInviter.aspnet_User.UserName,
RequestText = i.RequestText,
NoteInviter = i.NoteInviter,
ResendInvitationCount = (i.ResendInvitationCount.HasValue) ? i.ResendInvitationCount.Value : 0,
NoteInvited = i.NoteInvited,
MeetingType = i.MeetingType.TypeName
};
...
result =
(from i in _dbContext.Meetings
where i.UserInviterID == CurrentUserID
&& i.MeetingStatus == null
&& !i.IsTex
&& DbFunctions.AddMinutes(DbFunctions.AddHours(i.MeetingTime.Day, i.MeetingTime.Hour).Value, i.MeetingTime.Minute).Value > dateWithTime
select i).Select(projection);
This keeps the entire part of the query server-side, exactly as if you had written it out in each query.
Yes, you can create a method that returns a ITW2012Mobile.Core.DataTable.MeetingModel2Tmp for you and call that method in the select.
Something like
private ITW2012Mobile.Core.DataTable.MeetingModel2Tmp MethodName(TypeOfI i)
{
return new ITW2012Mobile.Core.DataTable.MeetingModel2Tmp() {
//same stuff as in your question's initializer
};
}
and call it with
result = from i in _dbContext.Meetings
where i.UserInvitedID == CurrentUserID && i.MeetingStatus == null && //!i.IsTex &&
DbFunctions.AddMinutes(DbFunctions.AddHours(i.MeetingTime.Day, i.MeetingTime.Hour).Value, i.MeetingTime.Minute).Value > dateWithTime
//where i.UserInvitedID == CurrentUserID && i.MeetingStatus == null && DbFunctions.TruncateTime(i.AllowedTime.AllowedDate.Day) >= date
select MethodName(i);
The query will get compiled down into LINQ extension methods anyway, and .Select() takes in a Func<TSource, TResult> as an argument. A Func is just a delegate (method) that has one input parameter (of type TSource) and returns a TResult object.
i try to use sum and Coalesce . How can i translate to linq?
SELECT #SumQtyOut=COALESCE(SUM(Qty),0) FROM dbo.StockMovement WHERE FromLocationType=#FromLocationType AND
* FromNo=#FromNo AND FromSeq=#FromSeq AND ItemTypeNo=#ItemTypeNo AND ItemID=#ItemID
i do sometihng :
using (StockProcedureDataContext stock = new StockProcedureDataContext())
{
SumQtyOut = from s in stock.StockMovements
where s.FromLocationType == FromLocationType &&
s.FromNo== FromNo &&
s.FromSeq == FromSeq &&
s.ItemTypeNo == ItemTypeNo &&
s.ItemID == ItemID select
}
This snippet should yield the result you are looking for.
using (StockProcedureDataContext stock = new StockProcedureDataContext())
{
var items = from s in stock.StockMovements
where s.FromLocationType == FromLocationType &&
s.FromNo== FromNo &&
s.FromSeq == FromSeq &&
s.ItemTypeNo == ItemTypeNo &&
s.ItemID == ItemID
select s.Qty ?? 0;
SumQtyOut = items.Sum(x => x);
}
select s.Qty ?? 0 returns 0 if s.Qty is null. items.Sum(x => x) summs up the quantities you have selected.