Unable to access Bloomburg.com with webparser plugin

[Mordasius]

Bloomberg haven't got around to replying to my email yet so, as Brian's header additions have held up for the past few days, I've posted updated versions of the currency and market prices skins to deviant art. Let's see how long they hold up for.

Currency :-> https://www.deviantart.com/mordasius/art/Currency-Markets-453172734
Stocks:-> https://www.deviantart.com/mordasius/art/Market-Prices-Bloomberg-442922018

[Mordasius]

The Bloomberg.com customer support team pulled out all the stops to explain what part of the terms of service they considered were being violated in an email which said:

"Hello,
Scraping Bloomberg.com is a terms of violation.
Sincerely,
Bloomberg News Customer Support Team"

So that's clear then.

[Brian]

So that's clear then.

I personally don't think it's clear at all. Perhaps bloomberg doesn't quite understand what Rainmeter is actually doing with the "data" once downloaded. Traditionally, web scraping is done by some automated process to basically gather data for analysis, or to extract specific content to be used for other reason (like grabbing email addresses). While it is possible that someone could write a complicated script to save all the data WebParser is downloading and use that data in some other way, I think the overwhelming majority of the time, WebParser extracts a chunk of data to directly display on the desktop - just like a web browser.

I could understand "scraping" would be a problem if someone were to gather and distribute bloomberg data and give access to that data to multiple users publicly. However, in the context of Rainmeter, the only "scraping" that is happening is totally at the user level. In my opinion, it is no different than accessing bloomberg.com from a web browser. Web browser's download and parse the html to display content. Web browsers can display things quite differently compared to their counterparts, and there are various "extensions" that can basically edit the content being displayed.

Ultimately, no website has control over how their content is actually displayed to the user.

---

Having said all that, it is their data and they do have the right to restrict who has access to their data. They have clearly defined what they consider is a violation, and even if we disagree with the "meaning" of terms or how their data is being displayed (and not stored for analysis), they may still block access.

Even setting specific Headers, will violate the terms according to their TOS:

You shall not use or attempt to use any “scraper,” “robot,” “bot,” “spider,” “data mining,” “computer code,” or any other automate device, program, tool, algorithm, process or methodology to access, acquire, copy, or monitor any portion of the Service, any data or content found on or accessed through the Service, or any other Service information without the prior express written consent of BLP. You may not forge headers or otherwise manipulate identifiers in order to disguise the origin of any other content.

---

Going forward, I would suggest to not use bloomburg for stock quotes and find another source that will not violate any terms of data usage. We encourage our users to be good netizens and follow the rules/regulations/terms/licenses from Rainmeter and all sources of data used in a skin. Ultimately, we (the Rainmeter dev team) cannot stop people of accessing any publicly available website and grabbing data to display on their desktop. This is a one of the most fundamental parts of Rainmeter.

-Brian

[ncnmra]

Not sure if this is an option, but how about this site?

https://www.alphavantage.co/

[SilverAzide]

So that's clear then.

Even if Bloomberg figures out a way to block your skins or whatever, I hope you will consider leaving them available for downloading on DA. Over the years I've found your skins extremely helpful in learning Rainmeter skin development, especially tricky stuff with WebParser and Lua (like sun/moon calcs!).

(I'm still hoping one day you have some sort of weird freakout and build a humongous Rube Goldberg machine, such that I'll need a whole separate monitor and 6-core CPU just to run it....)

[jsmorley]

I worry about you Bevy. I worry a lot...

[Mordasius]

Going forward, I would suggest to not use bloomburg for stock quotes and find another source that will not violate any terms of data usage. We encourage our users to be good netizens and follow the rules/regulations/terms/licenses from Rainmeter and all sources of data used in a skin.

I hear you Brian and have taken the skins down from Deviant Art while I look for another source.

I've received a few pointers already but some ( https://iextrading.com/developer/ ) only cover a limited range of stocks whilst others ( https://www.alphavantage.co/ ) are only good for stocks in the US and another ( https://www.asx.com.au/ ) is only good for Australia.

Now of course, if anyone wanted to download the old .rmskins from DropBox and apply the fixes given above for educational purposes they would learn that....

Cheers

[dudebaker]

Maybe you can try cnbc.com as a new source.
I have used bloomberg for my skins too, so had the same issue.
After some testing (only about 10 stocks and 10 currency combinations) I found that cnbc had good results and was easy to regex.
But maybe my tests where too lazy? I don't really use them so I cannot tell if cnbc has as many stocks and currencies as bloomberg but I think for the meantime it's better then nothing ...

[Mordasius]

Maybe you can try cnbc.com as a new source.

I played around on a number of sites with APIs over the weekend but most like http://www.iextrading.com had only a limited selection of stocks / indexes or were for the US only. http://www.alphavantage.co has a good selection but you have to apply for a free API which I suspect a number of users wouldn't be bothered to do.

So I ended up scraping INO.com for the currency skins and CNBC.com for the stocks and shares (thanks for the suggestion). Neither is as comprehensive as Bloomberg, Yahoo or Google but neither seem to object to a little gentle scaping - yet.

They are up on Deviant Art:
https://www.deviantart.com/mordasius/art/Market-Prices-755702746
https://www.deviantart.com/mordasius/art/Currency-Markets-755399465