Is there an API or Web Service that can be used to read AIS data? Most links I read starting at Wikipedia (http://en.wikipedia.org/wiki/Automatic_Identification_System) say that AIS data is freely available but I'm having a hard time finding a provider of the data. A C# example or language agnostic web service would be helpful.
Building a project map for a clients website. Basically a world map based on the google maps api with pin's where they did their projects and if you click on a pin you get additional information about the project.
Most were just static addresses which was ok, but they did 6 project's on luxury yachts. So had the idea to base this marker on the current position of the yacht. Came across this service, they have a nice API for it.
https://www.marinetraffic.com
The down side to this, it's a bit pricey.
Cheapest option, checking daily position of 1 ship -> € 5,- a month.
So this would be € 30,- a month for a relative useless but awesome feature.
Cheaper alternative's are welcome.
I ended up using vesseltracker.com for this project. Unfortunately it's a "call us for a price" service so I'll continue looking for a provider with a flat/reasonable/free rate.
There is a feed from the San Fransisco Bay available for non-commercial use at hd-sf.com:9009.
I have used it to test my Java-based AIS decoder https://github.com/tbsalling/aismessages/wiki.
AIS data is freely available in the sense that you can freely receive it with the proper equipment, just by holding up an antenna in an area with shipping traffic.
Samples of received AIS data popped up quite a bit in my brief Google search, so I assume that your question is about where to get a real-time feed of AIS messages (that someone else is receiving). Whether you'd be able to get this at no cost is questionable; most organizations that would offer this seem to want you to either pay for the service or to share in kind.
There are a few places that offer a free stream, but none of them seem to offer any guarantees on availability in the short or long term.
So the answer to your question is "yes, and you should expect to pay something for it".
Related
I am in a bit of a crisis here. I would really appreciate your help on the matter.
My Final Year Project is a "Location Based Product Recommendation Service". Now, due to some communication gap, we got stuck with an extremely difficult algorithm. Here is how it went:
We had done some research about recommendation systems prior to the project defense. We knew there were two approaches, "Collaborative Filtering" and "Content Based Recommendation". We had planned on using whichever technique gave us the best results. So, in essence, we were more focused on the end product than the actual process. The HOD asked us what algorithms OUR product would use? But, my group members thought that he meant what are the algorithms that are used for "Content Based Recommendations". They answered with "Rule Mining, Classification and Clustering". He was astonished that we planned on using all these algorithms for our project. He told us that he would accept our project proposal if we use his algorithm in our project. He gave us his research paper, without any other resources such as data, simulations, samples, etc. The algorithm is named "Context Based Positive and Negative Spatio-Temporal Association Rule Mining" In the paper, this algorithm was used to recommend sites for hydrocarbon taps and mining with extremely accurate results. Now here are a few issues I face:
I am not sure how or IF this algorithm fits in our project scenario
I cannot find spatio-temporal data, MarketBaskets, documentation or indeed any helpful resource
I tried asking the HOD for the data he used for the paper, as a reference. He was unable to provide the data to me
I tried coding the algorithm myself, in an incremental fashion, but found I was completely out of my depth. I divided the algo in 3 phases. Positive Spatio-Temporal Association Rule Mining, Negative Spatio-Temporal Association Rule Mining and Context Based Adjustments. Alas! The code I write is not mature enough. I couldn't even generate frequent itemsets properly. I understand the theory quite well, but I am not able to translate it into efficient code.
When the algorithm has been coded, I need to develop a web service. We also need a client website to access the web service. But with the code not even 10% done, I really am panicking. The project submission is in a fortnight.
Our supervisor is an expert in Artificial Intelligence, but he cannot guide us in the algorithm development. He dictates the importance of reuse and utilizing open-source resources. But, I am unable to find anything of actual use.
My group members are waiting on me to deliver the algorithm, so they can deploy it as a web service. There are other adjustments than need to be done, but with the algorithm not available, there is nothing we can do.
I have found a data set of Market Baskets. It's a simple excel file, with about 9000 transactions. There is not spatial or temporal data in it and I fear adding artificial data would compromise the integrity of the data.
I would appreciate if somebody could guide me. I guess the best approach would be to use an open-source API to partially implement the algorithm and then build the service and client application. We need to demonstrate something on 17th of June. I am really looking forward to your help, guidance and constructive criticism. Some solutions that I have considered are:
Use "User Clustering" as a "Collaborate Filtering" technique. Then
recommend the products from similar users via an alternative "Rule
Mining" algorithm. I need all these algorithms to be openly available
either as source code or an API, if I have any chance of making this
project on time.
Drop the algorithm altogether and make a project that actually works
as we intended, using available resources. I am 60% certain that we
would fail or marked extremely low.
Pay a software house to develop the algorithm for us and then
over-fit it into our project. I am not inclined to do this because it
would be unethical to do this.
As you can clearly see, my situation is quite dire. I really do need extensive help and guidance if I am to complete this project properly, in time. The project needs to be completely deployed and operational. I really am in a loop here
"Collaborative Filtering", "Content Based Recommendation", "Rule Mining, Classification and Clustering"
None of these are algorithms. They are tasks or subtasks, for each of which several algorithms exist.
I think you had a bad start already by not really knowing well enough what you proposed... but granted, the advice from your advisor was also not at all helpful.
I want to design an efficient & flexible architecture to process Swaps based on a standard financial protocol - FPML (Financial products markup language).
I've researched on internet but didn't find much information. The definitions which I found are:
SWAP (definition):
Swap refers to an exchange of one financial instrument for another between the parties concerned. This exchange takes place at a predetermined time, as specified in the contract.
FPML:
FPML (Financial products Markup Language) is the open source XML
standard for electronic dealing and processing of OTC derivatives. It
establishes the industry protocol for sharing information on, and
dealing in, financial derivatives and structured products.
Seems you have got an assignment to complete. I had almost the same thing to build using c# and ms sql.
Few important links and mentioning which will help you.
Few References
http://www.fpml.org
http://www.investinganswers.com/financial-dictionary/optionsderivatives/interest-rate-swap-2252
http://www.investopedia.com/terms/s/swap.asp
http://en.wikipedia.org/wiki/Swap_(finance)
http://www.investopedia.com/terms/e/equityswap.asp
http://www.fpml.org/documents/FpML5-products-framework.pdf
http://www.investopedia.com/articles/optioninvestor/07/swaps.asp
Hope this will help you.
Have a look at the following stackoverflow posting: http://bit.ly/python-xml-swap
Today i come up with in situation where i need to implement system where i need to filter out few number when i call them. My requirement is i have Primary Rate Interface (Pri) network connected to 30 land line telephone sets. With my starting research i am able to find few solution's
Telephony Application Programming Interface (TAPI)
Microsoft Unified Communications Managed
Lync 2013
asterisk
I am able to achieve above motive by refereeing TAPI for single connection but still not have any idea about implementation on PRI lines. Asterisk is open source but to implement this we need to tools which are not free. As i am new with this, please give me directions to move on with.
The learning curve in this area can be wicked complicated, if you don't have T1/ISDN background you may be in for a long tough haul if you try to write PRI call control yourself in C#.
You should probably start by refining your requirements statement, keeping it as technology neutral as possible. You need to be prepared to explain the business problem you are trying to solve clearly. E.g. if you need PRI, is it for connecting to the telephone sets, to the telephone company, or both. What actions happen when you "filter out" telephone numbers? Focus first on a clear statement of what, rather than how.
As to how, there used to be several companies that produced this sort of hardware/software packages, but over the years one of them (Dialogics) seems to have bought out most of the others hardware vendors, so a lot of the software vendors partner with them. I'd recommend you start at http://www.dialogics.com and look into solutions partners to see if you can find a close match for your requirements.
Good luck!
I am working on my mapper and I need to get the full map of newegg.com
I could try to scrap NE directly (which kind of violates NE's policies), but they have many products that are not available via direct NE search, but only via google.com search; and I need those links too.
Here is the search string that returns 16mil of results:
https://www.google.com/search?as_q=&as_epq=.com%2FProduct%2FProduct.aspx%3FItem%3D&as_oq=&as_eq=&as_nlo=&as_nhi=&lr=&cr=&as_qdr=all&as_sitesearch=newegg.com&as_occt=url&safe=off&tbs=&as_filetype=&as_rights=
I want my scraper to go over all results and log hyperlinks to all these results.
I can scrap all the links from google search results, but google has limit of 100 pages for each query- 1,000 results and again, google is not happy with this approach. :)
I am new to this; Could you advise / point me in the right direction ? Are there any tools/methodology that could help me to achieve my goals?
I am new to this; Could you advise / point me in the right direction ?
Are there any tools/methodology that could help me to achieve my
goals?
Google takes a lot of steps to prevent you from crawling their pages and I'm not talking about merely asking you to abide by their robots.txt. I don't agree with their ethics, nor their T&C, not even the "simplified" version that they pushed out (but that's a separate issue).
If you want to be seen, then you have to let google crawl your page; however, if you want to crawl Google then you have to jump through some major hoops! Namely, you have to get a bunch of proxies so you can get past the rate limiting and the 302s + captcha pages that they post up any time they get suspicious about your "activity."
Despite being thoroughly aggravated about Google's T&C, I would NOT recommend that you violate it! However, if you absolutely need to get the data, then you can get a big list of proxies, load them in a queue and pull a proxy from the queue each time you want to get a page. If the proxy works, then put it back in the queue; otherwise, discard the proxy. Maybe even give a counter for each failed proxy and discard it if it exceeds some number of failures.
I've not tried it but you can use googles custom search API. Of course, its starts to cost money after 100 searches a day. I guess they must be running a business ;p
It might be a bit late but I think it is worth to mention that you can professionally scrape Google reliable and not cause problems with it.
Actually it is not of any threat I know about to scrape Google.
It is cahllenging if you are unexperienced but I am not aware about a single case of legal consequence and I am always following this topic.
Maybe one of the largest cases of scraping happened some years ago when Microsoft scraped Google to power Bing. Google was able to proof it by placing fake results which do not exist in real world and Bing suddenly took them up.
Google named and shamed them, that's all that happened as far as I remember.
Using the API is rarely ever a real use, it costs a lot of money to use it for even a small amount of results and the free amount is rather small (40 lookups per hour before ban).
The other downside is that the API does not mirror the real search results, in your case maybe less a problem but in most cases people want to get the real ranking positions.
Now if you do not accept Googles TOS or ignore it (they did not care about your TOS when they scraped you in their startup) you can go another route.
Mimic a real user and get the data directly from the SERPs.
The clue here is to send around 10 requests per hour (can be increased to 20) with each IP address (yes you use more than one IP). That amount has proven to cause no problem with Google over the past years.
Use caching, databases, ip rotation management to avoid hitting it more often than required.
The IP addresses need to be clean, unshared and if possible without abusive history.
The originally suggested proxy-list would complicate the topic a lot as you receive unstable, unreliable IPs with questionable absuive use, share and history.
There is an open source PHP project on http://scraping.compunect.com which contains all the features you need to start, I used it for my work which now runs for some years without troubles.
Thats a finished project which is mainly built to be used as customizable base of your project but runs standalone too.
Also PHP is not a bad choice, I originally was sceptical but I was running PHP (5) as background process for two years without a single interruption.
The performance is easily good enough for such a project so I would give it a shot.
Otherwise, PHP code is like C/JAVA .. you can see how things are done and repeat them in your own project.
I am thinking about writing a program to check some mileage claims (about 45,000 of them actually) made by Members of the British Parliament recently.
The data I have is quite course - the origin and destination of the trips is generally provided only at the Town level. What I am looking to do is basically look at the mileage figures they have submitted, look at the "calculated" mileage obtained from a web service, and then compare / contrast. (I'm thinking of perhaps being interested in cases where the claimed mileage exceeds 40 miles, and the claimed mileage exceeds the website figure by a factor of at least 1.4 for instance).
Code wise I'm fine, but what I don't have is a webservice I can use to get the distances. Google Directions is out of the question (has to be used with Google Maps) - and I am looking at writing this as a console app in C#.
Can anyone suggest any suitable service?
You should take a look at any routing services that are using Open Street Maps. Many services based off of OSM are provided free of charge and are less restrictive than Google Maps.
One that I can think of is Open Route Service, though I don't know if they have a permissive API that allows you to do what you're asking.
If you are working for a government entity, many of them have licenses for Esri ArcGIS which includes a street data, turn-by-turn routing directions, and an API.