I have a huge database (sql server) of text documents (~ 500GB so far). I do full text search on them.
I want to use solr/elastic search for this purpose.
However, text documents are associated with roles in organization, ie: manager documents, or bosses' documents. And roles of people change now and then.
I dont have a problem writing a sql query for this purpose.
I can't think of doing the same with Solr/elastic search.
How would you solve this problem?
There is a good blog post about this topic "Custom security filtering in Solr". It shows how to implement the PostFilter Interface where you can do anything Java offers to check if a document maybe accessed or not. This is intended for the case that you have some remote system that holds the access information, like a corporate LDAP.
But as the author notes
It’s important to note that PostFilter is a last resort for implementing document filtering. Don’t make the solution more complicated than it needs to be. More often than not, even access control filtering can be implemented using plain ol’ search techniques, by indexing allowed users and groups onto documents and using the lucene (or another) query parser to do the trick. Only when the rules are too complicated, or external information is needed, does a custom PostFilter make sense.
That means: Would it not be possible to add fields to the documents that hold the access information? Something like
<fields>
<!-- your other fields -->
<field name="owner"
type="String" indexed="true" stored="true" multiValued="false" />
<field name="team"
type="String" indexed="true" stored="true" multiValued="false" />
<field name="team-lead"
type="String" indexed="true" stored="true" multiValued="false" />
<field name="roles"
type="String" indexed="true" stored="true" multiValued="true" />
</fields>
Then when searching you can add to the query
q=some+cool+query&fq=owner:username+OR+team:user's team name+OR+role:role1
Related
I would like to remove all xml nodes where name is not a number of values:
<Property Name="Operation" Type="String" Access="ReadWrite" Value="ProduceFile" />
<Property Name="BackOfficeType" Type="String" Access="ReadWrite" Value="growBusiness Solutions" />
<Property Name="module" Type="String" Access="ReadWrite" Value="Document" />
<Property Name="vti_pluggableparserversion" Type="String" Access="ReadOnly" Value="16.0.0.20405" />
<Property Name="_Author" Type="String" Access="ReadWrite" Value="hfhf fghfgh" />
<Property Name="modifiedBy" Type="String" Access="ReadWrite" Value="fghfghfghfg" />
<Property Name="vti_parserversion" Type="String" Access="ReadOnly" Value="16.0.0.20405" />
How do I remove all element above with regex where Name is not Operation or module?
I was thinking something like:
xml = Regex.Replace(xml, #"<Property Name=""(?!Operation |module)"".*?/>", "");
But this is not working.
I don't understand why, because " is not a special character in the C# Regex system, but removing the second quote and the space after Operation makes it work (this is without the necessary escaping):
<Property Name="(?!Operation|module).*?/>
I'll update this answer if I figure out what's going on with that second quote.
EDIT: Well I feel a fool for not noticing this myself. A friend of mine pointed out that by having Name="(?!Operation|module)" it essentially says "Only match on Name="". If you add the following example to your sample data you'll see that's what is happening:
<Property Name="" Type="String" Access="ReadOnly" Value="16.0.0.20405" />
So adding another wildcard inside the quotes will allow it to match on all the entries that don't have "Operation" or "module" in them:
<Property Name="(?!Operation|module).*".*?/>
However, this raises a new issue, which is now if you have Name="Operation Awesome" the filtering group will ignore it as well. So the negative lookahead would have to be changed somehow to specifically ignore exact words and not property names simply containing the words. So how do we do that?
<Property Name=(?!"Operation"|"module").*?/>
This ensures it only keep an exact match of "Operation" or "module". The only side effect present now is it will delete any malformed XML like PropertyName="Operation Type="string". You may consider this a negative, but if you want to be able to handle invalid XML you should be doing so with another method before this one.
I'm trying to create an initiator based on QuickFix/n.
My counterpart (server) demand the logon message to carry two custom fields.I already added those to the Data Dictionary, in the Fields as well as in the Logon message.
In the code i can manipulate the message to add the username and password, but i can't find how to load custom fields.
Here are some excerpts from what i've done so far:
TradeClientApp.cs
...
public void ToAdmin(Message message, SessionID sessionID)
{
QuickFix.SessionSettings settings = new QuickFix.SessionSettings("./initiator.cfg");
List<SessionID> sids = settings.GetSessions().ToList();
Dictionary settingsDict = settings.Get(sids.First());
var appName = settingsDict.GetString("ApplicationName");
var userType = settingsDict.GetString("UserType");
if (message.GetType() == typeof(QuickFix.FIX44.Logon))
{
message.SetField(new Username("USERNAME"));
message.SetField(new Password("PASSWORD"));
message.SetField(new QuickFix.Fields.ResetSeqNumFlag(true));
//tag 9933
message.SetField(new RawData(string.Format($"9933={appName}")));
//tag 20110
message.SetField(new RawData(string.Format($"20110={userType}")));
}
}
The initiator.cfg file:
[DEFAULT]
UseDataDictionary=Y
DataDictionary=./spec/FIX44.xml
FileStorePath=store
FileLogPath=log
ConnectionType=initiator
ReconnectInterval=60
[SESSION]
BeginString=FIX.4.4
SenderCompID=USERNAME
ResetSeqNumFlag=Y
Username=USERNAME
Password=PASSWORD
TargetCompID=TARGETCOMPID
StartTime=12:30:00
EndTime=23:30:00
HeartBtInt=10
ApplicationName=app-name
UserType=V
SocketConnectPort=446
SocketConnectHost=SERVERHOST
Inside the FIX44.XML
<message name="Logon" msgtype="A" msgcat="admin">
<field name="EncryptMethod" required="Y" />
<field name="HeartBtInt" required="Y" />
<field name="RawDataLength" required="N" />
<field name="RawData" required="N" />
<field name="ResetSeqNumFlag" required="N" />
<field name="NextExpectedMsgSeqNum" required="N" />
<field name="MaxMessageSize" required="N" />
<group name="NoMsgTypes" required="N">
<field name="RefMsgType" required="N" />
<field name="MsgDirection" required="N" />
</group>
<field name="TestMessageIndicator" required="N" />
<field name="Username" required="N" />
<field name="Password" required="N" />
<field name="ApplicationName" required="N" />
<field name="UserType" required="N" />
</message>
<fields>
...
<field number="9933" name="ApplicationName" type="STRING"/>
<field number="20110" name="UserType" type="STRING" />
</fields>
</fix>
When i try to connect i get this log:
<outgoing> 8=FIX.4.49=12035=A34=149=USERNAME52=20191008-21:19:41.49856=TARGETCOMPID96=20110=V98=0108=10141=Y553=USERNAME554=PASSWORD10=097
Using the RawData as shown in my example, the message carries 20110=V as RawData, which is tag 96, which doesn't help me.
I already tried inside ToAdmin:
message.Header.SetField(new StringField(QuickFix.Fields.Tags.UserType, ""));
or
message.SetField(new QuickFix.Fields.UserType(true));
but neither work.
How on earth do you add custom fields to the logon message?
I think you are getting the RawData data type wrong. That is literally meant for raw data, not for Strings that follow the default encoding.
From the spec:
string field containing raw data with no format or content
restrictions. Data fields are always immediately preceded by a length
field. The length field should specify the number of bytes of the
value of the data field (up to but not including the terminating SOH).
I am not so familiar with the C# implementation of QuickFIX but you should simply be able to add these fields with the specific tag number, e.g.
message.SetField(new StringField(20110, "V"));
I hope there are no syntax errors, but you should get the idea.
Edit: I see you are manually setting the ResetSeqNum field on the Logon message. This is discouraged. quickFIX/n should deal with this when you set ResetSeqNum=Y in the settings (you already have that setting).
Folks,
I have a database-first project in EF6 against an Oracle DB, and I have specified in the model that certain columns should be Int32s. However, when the Web API payload is returned, those properties are returned as Numbers, such as 58.0, 3.0, 486.0 and so on... which makes thr entire JSONDeserializer throw up royally, of course.
I presume I should be able to change the mapping within EF for it, but how exactly is eluding me at the moment. Any help would be much appreciated!
Editing for clarification:
Looking at the .edmx, the properties show the should-be integers as:
<Property Name="OrderId" Type="Int32" />
<Property Name="StatusId" Type="Int32" Nullable="false" />
On the Oracle side, I can confirm that they are Number.
As to why Web API is included in this OP: I'm just trying to provide some context as to how the requests are being returned. I'm not using a DTO but rather returning the model directly.
Happy to provide more info (as soon as I get to a laptop).
N
OK, looks like this will work, for anyone else who may be having the same difficulties.
If you're using the managed data access libraries from Oracle, add the following to your app/web.config:
<oracle.manageddataaccess.client>
<version number="*">
<edmMappings>
<edmMapping dataType="number">
<add name="bool" precision="1" />
<add name="byte" precision="2" />
<add name="int16" precision="5" />
<add name="int32" precision="38" />
<add name="int64" precision="38" />
</edmMapping>
</edmMappings>
</version>
</oracle.manageddataaccess.client>
Or adjust to your own taste. Do note, however, that you might have to map Int64 as well (even though I don't believe I have anything in my DB that would quantify as such, I would sometimes get an initializer exception.
You may have to delete your model from your .edmx file and re-add it.
But that will fix your issues, and your JSON will not be returned as decimals anymore.
Hope that helps! (And keep it classy, Oracle... ;) )
I tried the above solution but didn't work for me. I am not able to find web.config file, instead I have modified App.config file with above suggestion.
I have ELMAH set up for a webapp, logging exceptions to a SQL server.
I wish to have ELMAH send me an email too, but only when a specific exception is thrown (ie. MySpecialException).
ELMAH must still log all exceptions to SQL server.
I know you can do it programmatically in global.asax, but I'd prefer to use web.config.
So, how do I restrict ELMAH error mails to filter out everything but a specific exception type, using web.config?
UPDATE
The filter ended up looking like this:
<test>
<and>
<not>
<is-type binding="Exception" type="MyApp.MySpecialException" />
</not>
<regex binding="FilterSourceType.Name" pattern="mail" caseSensitive="false"/>
</and>
</test>
Its certainly possible to do. Check out the filtering documentation for elmah.
In particular look at the section Filtering By Source
<elmah>
...
<errorFilter>
<test>
<and>
<equal binding="HttpStatusCode" value="404" type="Int32" />
<regex binding="FilterSourceType.Name" pattern="mail" />
</and>
</test>
</errorFilter>
I want to Upload File on Sharepoint and while apploading only i want to add all properties of Uploaded Document.
We get ID field only when Document is uploaded on Sharepoint.
Is there any other way to Update List without passing ID Field.
Example:
<Batch OnError="Continue" ListVersion="1"
ViewName="270C0508-A54F-4387-8AD0-49686D685EB2">
<Method ID="1" Cmd="Update">
<Field Name="ID">4<Field>
<Field Name="Field_Name">Value</Field>
</Method>
<Method ID="2" Cmd="Update">
<Field Name="ID" >6</Field>
<Field Name="Field_Name">Value</Field>
</Method>
</Batch>
Refering Link
**** I am using Sharepoint Web Services.And Uploading Document in Chunks.****
This is not possible. First the file should complete uploading, then it will have properties that can be set! (otherwise properties may be set for non-existing files, in case that the file fails to upload.)