It is currently April 19th, 2024, 11:58 pm
Unable to access Bloomburg.com with webparser plugin
-
Sk1Bum
- Posts: 2
- Joined: July 5th, 2018, 6:59 pm
Re: Unable to access Bloomburg.com with webparser plugin
The "UserAgent=..." fix lasted for about 2 hours here, and then just quit completely again.
-
eclectic-tech
- Rainmeter Sage
- Posts: 5398
- Joined: April 12th, 2012, 9:40 pm
- Location: Cedar Point, Ohio, USA
Re: Unable to access Bloomburg.com with webparser plugin
Yes, it stopped working for me too...Sk1Bum wrote:The "UserAgent=..." fix lasted for about 2 hours here, and then just quit completely again.
Perhaps a solution can be found, but seeing this action of changing their website again, makes me think it will be a "short-lived" answer
Apparently Bloomberg doesn't want anyone accessing their data from anywhere other than looking at their site
:: My DA Gallery :: Rainmeter DA Gallery :: Rainmeter Workshops :: Rainmeter Documentation :: BBCode Guide ::
-
SilverAzide
- Rainmeter Sage
- Posts: 2604
- Joined: March 23rd, 2015, 5:26 pm
Re: Unable to access Bloomburg.com with webparser plugin
Just a thought... add a randomized bit of text to the user agent string AND don't hit the site at regular intervals like a bot would.eclectic-tech wrote:Yes, it stopped working for me too...
Perhaps a solution can be found, but seeing this action of changing their website again, makes me think it will be a "short-lived" answer
Apparently Bloomberg doesn't want anyone accessing their data from anywhere other than looking at their site
-
eclectic-tech
- Rainmeter Sage
- Posts: 5398
- Joined: April 12th, 2012, 9:40 pm
- Location: Cedar Point, Ohio, USA
Re: Unable to access Bloomburg.com with webparser plugin
Thanks for the thought, but the issue appears to be changes with Bloomberg's page.SilverAzide wrote:Just a thought... add a randomized bit of text to the user agent string AND don't hit the site at regular intervals like a bot would.
The UpdateRate of the webparser measure(s) is set to 1800 (every 30 minutes), so the site is not being hit excessively.
The page no longer contains the RegExp items in the same way it used to have them; it appears Bloomberg is using a different webpage format which causes the parser to fail to match the requested items... RegExp matching error (-1) in the log.
So until the skin can be corrected with a working RegExp, I don't think modifying the UserAgent text will work either. But thanks for the help.
I am sure Mordasius will not rest until this is resolved, one way or the other.
EDIT: Bloomberg is flagging these request as a "Violation of the terms of service". So I will stand by my first reply, that Bloomberg does not want their data accessed by a webparser.
Debug Info:
Bloomberg wrote:...
<body>
<div class="container">
<!– Customer messaging and logo ->
<img style="margin-bottom: 40px;" src="https://www.bloomberg.com/graphics/assets/img/BB-Logo-2line.svg"
width="310"/>
<h1 class="text-center" style="margin: 0 auto;">Terms of Service Violation</h1>
<p>Your usage has been flagged as a violation of our <a href="http://www.bloomberg.com/tos" target="_blank" rel="noopener noreferrer">terms of service</a>.
</p>
<p>
For inquiries related to this message please <a href="http://www.bloomberg.com/feedback">contact support</a>.
For sales
inquiries, please visit <a href="http://www.bloomberg.com/professional/request-demo">http://www.bloomberg.com/professional/request-demo</a>
</p>
...
:: My DA Gallery :: Rainmeter DA Gallery :: Rainmeter Workshops :: Rainmeter Documentation :: BBCode Guide ::
-
Brian
- Developer
- Posts: 2679
- Joined: November 24th, 2011, 1:42 am
- Location: Utah
Re: Unable to access Bloomburg.com with webparser plugin
I tried adding some basic headers to the measure and was able to get some results.
But I am not sure how long it will last...
-Brian
Code: Select all
Header=Accept: text/html
Header2=Accept-Encoding: gzip, deflate, br
Header3=Accept-Language: en-US
Header4=Connection: close
Header5=Dnt: 1
Header6=Host: www.bloomberg.com
Header7=Referer: https://wwww.google.com-Brian
-
Mordasius
- Posts: 1171
- Joined: January 22nd, 2011, 4:23 pm
- Location: GMT +8
Re: Unable to access Bloomburg.com with webparser plugin
Thanks Brian. Those headers get the skins working but, as you say, who knows for how long.
I had a look at the Bloomberg Terms of Service (https://www.bloomberg.com/notices/tos/) part of which states:
Except as expressly permitted by Bloomberg in writing you may not copy, reproduce, recompile, decompile, disassemble, reverse engineer, distribute, publish, display, perform, modify, upload to, create derivative works from, transmit, transfer, sell, license, upload, edit post, frame, link, or in any way exploit any part of the Service, except that you may download material from the Service and/or make one print copy for your own personal, noncommercial use, provided that you retain all copyright and other proprietary notices.
I don't think the skins violate any of the above but just to check I sent them an email to find out which part of the TOS is being violated. I guess the bottom line is that they collate and publish the data and if they really don't want us to access it using Rainmeter then I guess they have every right to block the webparser plugin.
I had a look at the Bloomberg Terms of Service (https://www.bloomberg.com/notices/tos/) part of which states:
Except as expressly permitted by Bloomberg in writing you may not copy, reproduce, recompile, decompile, disassemble, reverse engineer, distribute, publish, display, perform, modify, upload to, create derivative works from, transmit, transfer, sell, license, upload, edit post, frame, link, or in any way exploit any part of the Service, except that you may download material from the Service and/or make one print copy for your own personal, noncommercial use, provided that you retain all copyright and other proprietary notices.
I don't think the skins violate any of the above but just to check I sent them an email to find out which part of the TOS is being violated. I guess the bottom line is that they collate and publish the data and if they really don't want us to access it using Rainmeter then I guess they have every right to block the webparser plugin.
-
Brian
- Developer
- Posts: 2679
- Joined: November 24th, 2011, 1:42 am
- Location: Utah
Re: Unable to access Bloomburg.com with webparser plugin
I worry about the clause distribute, publish, display in the context of publishing a skin for others to download and displaying a quote on desktop.
However, the following suggests that as long as you aren't selling the information, it should be okay.
except that you may download material from the Service and/or make one print copy for your own personal, noncommercial use, provided that you retain all copyright and other proprietary notices
Ultimately they can block who and what they want. Normally websites do this with UserAgent, but can use Headers to block as well. Now that Rainmeter can send those as well, it will be harder to really block. Using some sort of scripting on the server would be the best way.
Just as an aside: There seems to be more and more companies trying to "lock down" access to their data and/or software. Whether it be a music player changing their API access, or websites enabling scripting to display content...it is becoming harder and harder to access and control data. Too many bad actors completely steal copy websites and/or data, causing the producer of that data to find creative was to "protect" that data. This directly effects software like Rainmeter, making it harder to legitimately process and display content to the user in a customizable way like a webbrowser does. Hopefully this trend will run its course.
-Brian
However, the following suggests that as long as you aren't selling the information, it should be okay.
except that you may download material from the Service and/or make one print copy for your own personal, noncommercial use, provided that you retain all copyright and other proprietary notices
Ultimately they can block who and what they want. Normally websites do this with UserAgent, but can use Headers to block as well. Now that Rainmeter can send those as well, it will be harder to really block. Using some sort of scripting on the server would be the best way.
Just as an aside: There seems to be more and more companies trying to "lock down" access to their data and/or software. Whether it be a music player changing their API access, or websites enabling scripting to display content...it is becoming harder and harder to access and control data. Too many bad actors completely steal copy websites and/or data, causing the producer of that data to find creative was to "protect" that data. This directly effects software like Rainmeter, making it harder to legitimately process and display content to the user in a customizable way like a webbrowser does. Hopefully this trend will run its course.
-Brian
-
jsmorley
- Developer
- Posts: 22629
- Joined: April 19th, 2009, 11:02 pm
- Location: Fort Hunt, Virginia, USA
Re: Unable to access Bloomburg.com with webparser plugin
I could make an argument that a company that produces and distributes data that they own, or that they expend effort to add value to, is entitled to make a reasonable profit off their efforts, their employees have to eat too... In the internet world, where everything is expected to be "free", advertising is often the only way to make money. I'm quite sure that Bloomburg sees efforts to access their information in a way that avoids at least being presented with the site advertising as "shoplifting", and they have a point. They have a product. They offer it to you with the sale price being that your are exposed to their advertisers.
I'm no huge fan of all the advertising, and I could also argue that it has gone way too far in a lot of ways, but I'm hard pressed to begrudge Bloomburg the right to try and restrict access to their site to "eyeballs". From a Rainmeter standpoint, it is regrettable, but not particularly unfair.
I say all this as I fine-tune the settings on my ad-blocker...
I'm no huge fan of all the advertising, and I could also argue that it has gone way too far in a lot of ways, but I'm hard pressed to begrudge Bloomburg the right to try and restrict access to their site to "eyeballs". From a Rainmeter standpoint, it is regrettable, but not particularly unfair.
I say all this as I fine-tune the settings on my ad-blocker...
-
Sk1Bum
- Posts: 2
- Joined: July 5th, 2018, 6:59 pm
Re: Unable to access Bloomburg.com with webparser plugin
Do these get added the same way as the UserAgent=... ?Brian wrote:I tried adding some basic headers to the measure and was able to get some results.But I am not sure how long it will last...Code: Select all
Header=Accept: text/html Header2=Accept-Encoding: gzip, deflate, br Header3=Accept-Language: en-US Header4=Connection: close Header5=Dnt: 1 Header6=Host: www.bloomberg.com Header7=Referer: https://wwww.google.com
-Brian
-
Brian
- Developer
- Posts: 2679
- Joined: November 24th, 2011, 1:42 am
- Location: Utah
Re: Unable to access Bloomburg.com with webparser plugin
The UserAgent option is used when setting up a wininet internet handle to be used when calling various wininet functions. The Header(s) option is sent directly when opening a URL. UserAgent is not added to the headers directly, but it is possible wininet is concatenating it to the headers when the actual request is made. It is not clear in the wininet documentation.Sk1Bum wrote:Do these get added the same way as the UserAgent=... ?
The UserAgent option provides a quick way to change the agent, and can also be set globally for all webparser measures on your system (this is done by editing the Rainmeter.data file). You can override any UserAgent option by defining a separate Header like this: Header=User-Agent: ...
-Brian