Skin Authors: openstreetmap usage

[Brian]

In reference: https://github.com/rainmeter/rainmeter/issues/367

It looks like too many skins are making geocoding requests to nominatim.openstreetmap.org.

These requests have become disruptive and they have asked us to limit these requests. While we cannot (and will not) limit web requests within Webparser, we can ask skin authors to provide a more efficient way of accessing these geocoding services.

Since every skin will be different, this is a good time to remind everyone on how and when Webparser updates: https://docs.rainmeter.net/tips/webparser-updaterate/

More helpful links:
https://docs.rainmeter.net/manual/skins/rainmeter-section/#Update
https://docs.rainmeter.net/manual/measures/webparser/#UpdateRate
https://docs.rainmeter.net/manual/measures/webparser/#UserAgent

While it may not help, you could also try NOT using ForceReload by making sure you have ForceReload=0 on the measure, or removing ForceReload from the Flags option.

Another option might be to "save" any data received to a variable and then disable the Webparser measure making the request - at least until the user wants a "new" location. This type of geocoding data is really only needed once per-location, so requesting it every time the skin runs or updates (or refreshes) isn't necessary.

Hopefully, we can get this sorted out so that our users (and theirs!) can continue to use this great resource!

Thanks!
-Brian

PS - If anyone comes across a skin that is requesting this geocoding data too often, please ask the skin author to update their skin(s)! Look for low Update/UpdateRate values, or repeated refreshes of Rainmeter or the skin.

[JelleDekkers]

Will do! Although I'm 100% sure it isn't caused by one of my skins as I have set the UserAgent to a browser.

[Yincognito]

Will do! Although I'm 100% sure it isn't caused by one of my skins as I have set the UserAgent to a browser.

This isn't so much about the user agent string, it's more about the potential high frequency of updating WebParser measures in some skins.

By the way, such a frequent polling of a site can be:
1. intentional:
a) to get data as soon as it changes (this not really a logical reason when it comes to Nominatim or even weather providers - after all, it's not like your location continuously drifts on the planet, or the weather is changing every minute, regardless of whether some folks could be obsessed by that)
b) during the building / testing phase of a skin when the access to the resource is more frequent, if one insists in using the online web resource for tests, despite the fact that using a locally saved version of the page via the file:// URI scheme allows the same thing with no drawbacks whatsoever
2. unintentional:
a) because of bad coding or missing key details, like having a lower skin Update rate at the level of a couple of milliseconds to facilitate nice animations in the skin, but forgetting to use either a larger UpdateDivider or a larger UpdateRate in WebParser measures to account for that and bring updating such measures at a reasonable rate (example: the animated variants of a skin called Small Clean Weather, which besides the fact that they're broken and don't work, update WebParsers excessively, see here for a brief explanation and here for an easy solution to it, which can be applied to any similar skin if needed)
b) simply due to the structure of the Rainmeter featured machines, e.g. if somebody has a network of 4000 machines, each having installed and loading a Rainmeter skin querying a web resource at Windows' startup, then it's easy to have that many nearly simultaneous requests to a site just because of the initial update of all measures when the skin is loaded

Obviously, in this case, since nearly all Nominatim requests are manual (or should be, given that a saved geocode can be reused without polling the resource afterwards), needing the user to first input his location to get the corresponding geocode to use going forward, only 1.a) and 1.b) normally apply - hence the general idea of this thread. Besides malicious intent which frankly is not even about Rainmeter since it can be done from any network capable software out there anyway, it's highly unlikely that any Rainmeter user would want to consciously harm his own chances of benefitting from such a nice free service like the one provided by Nominatim (or others, for that matter). So, at worst, such mistakes are caused by the fact that due to the hobbyist nature of Rainmeter, most users or skin developers are not actual professionals to be careful at every little detail that might make improperly using some skins a problem for the providers of various services.

[JelleDekkers]

This isn't so much about the user agent string, it's more about the potential high frequency of updating WebParser measures in some skins.

By the way, such a frequent polling of a site can be:
1. intentional:
a) to get data as soon as it changes (this not really a logical reason when it comes to Nominatim or even weather providers - after all, it's not like your location continuously drifts on the planet, or the weather is changing every minute, regardless of whether some folks could be obsessed by that)
b) during the building / testing phase of a skin when the access to the resource is more frequent, if one insists in using the online web resource for tests, despite the fact that using a locally saved version of the page via the file:// URI scheme allows the same thing with no drawbacks whatsoever
2. unintentional:
a) because of bad coding or missing key details, like having a lower skin Update rate at the level of a couple of milliseconds to facilitate nice animations in the skin, but forgetting to use either a larger UpdateDivider or a larger UpdateRate in WebParser measures to account for that and bring updating such measures at a reasonable rate (example: the animated variants of a skin called Small Clean Weather, which besides the fact that they're broken and don't work, update WebParsers excessively, see here for a brief explanation and here for an easy solution to it, which can be applied to any similar skin if needed)
b) simply due to the structure of the Rainmeter featured machines, e.g. if somebody has a network of 4000 machines, each having installed and loading a Rainmeter skin querying a web resource at Windows' startup, then it's easy to have that many nearly simultaneous requests to a site just because of the initial update of all measures when the skin is loaded

Obviously, in this case, since nearly all Nominatim requests are manual (or should be, given that a saved geocode can be reused without polling the resource afterwards), needing the user to first input his location to get the corresponding geocode to use going forward, only 1.a) and 1.b) normally apply - hence the general idea of this thread. Besides malicious intent which frankly is not even about Rainmeter since it can be done from any network capable software out there anyway, it's highly unlikely that any Rainmeter user would want to consciously harm his own chances of benefitting from such a nice free service like the one provided by Nominatim (or others, for that matter). So, at worst, such mistakes are caused by the fact that due to the hobbyist nature of Rainmeter, most users or skin developers are not actual professionals to be careful at every little detail that might make improperly using some skins a problem for the providers of various services.

I know what you mean, but I was referring specifically to the fact that Nominatim is seeing a lot of 'Rainmeter' user agents connecting to their service.

I will update my skins besides this point however, because they do fetch information 4 time an hour, which is exactly what the devs of Nominatim are complaining about.

[Yincognito]

I will update my skins besides this point however, because they do fetch information 4 time an hour, which is exactly what the devs of Nominatim are complaining about.

Sure thing. I think the problem was the number of additional users requesting the same thing at roughly the same time, and not necessarily retrieving data 3 or 4 times per hour from an online service, which seems perfectly reasonable. Anyway, like I said, since locations on Earth don't migrate and the geocode can be stored in some variables for future usage, in this case there's no point in doing more than 1 request when installing the skin (or when the user changes his place of residence).

[SilverAzide]

Looks like openstreetmap.org finally pulled the trigger and is blocking the Rainmeter WebParser.

I've gotten the first report from one of my Gadgets users, and I can confirm the issue. Using a custom UserAgent string corrects the problem, obviously.

Access blocked
You have been blocked because you have violated the usage policy of OSM's Nominatim geocoding service. Please be aware that OSM's resources are limited and shared between many users. The usage policy is there to ensure that the service remains usable for everybody.

Please review the terms and make sure that your software adheres to the terms. You should in particular verify that you have set a custom HTTP referrer or HTTP user agent that identifies your application, and that you are not overusing the service with massive bulk requests.

If you feel that this block is unjustified or remains after you have adopted your usage, you may contact the Nominatim system administrator at nominatim@openstreetmap.org to have this block lifted.

[Yincognito]

Thanks for reporting / confirming, SilverAzide. I don't use the service, but I guess this was bound to happen sooner or later. It's good that there are still ways to use it. This should serve as a lesson to use things moderately.

Personally, I saw the complaint on GitHub as contradictory from post to post, debatable in terms of gravity and slightly out of place trying to dictate how another software should enforce all kinds of limits for what is essentially a user issue and not a software one, but then, that's just me. I swear, at this rate, literally everything will be forbidden in what was once a free and open software development environment. Like I said on another occasion, we'll have tons of workarounds for another ton of workarounds by someone else and then wonder how bugs happen and everything became so complicated.

[SilverAzide]

One additional comment for skin authors who use the OSM API and who also wish to be a good netizen and follow OSM's terms of service:

It is a violation of the OSM API Usage Policy to spoof the UserAgent string.

See their API Usage Policy, particularly the section Technical Usage Requirements.

One reasonable suggestion posted on the Rainmeter GitHub thread is that skin authors create their own UserAgent string by including #CURRENTCONFIG# or #ROOTCONFIG# to identify the skin or skin suite; e.g., perhaps something like...

Code: Select all

UserAgent="#ROOTCONFIG#/n.n.n (powered by Rainmeter)"

---where "n.n.n" is the skin version number. This will yield a browser-like user agent string that meets OSM's TOS.

[Yincognito]

One reasonable suggestion posted on the Rainmeter GitHub thread is that skin authors create their own UserAgent string by including #CURRENTCONFIG# or #ROOTCONFIG# to identify the skin or skin suite...

I really hate to be the non positive guy in such matters, but ... careful with what you concede as reasonable when it comes to others' actions. It's only reasonable as long as your own skin suite is not banned for misuse by some user. Judging by the fact that they already blocked an application module and not the users abusing that module, you - or anyone else using their service in your skins - might end up in the same category with the "violating" Rainmeter WebParser, even after setting a custom HTTP referrer / user agent. Remember DeviantArt and how much they "cared" for requests to reconsider their actions and have such things lifted. Just saying - hopefully I'm wrong with this.

Also, there's also the possibility of folks setting their user agent to some other skin or suite (or letting the same user agent as the original skin they are modifying), so one could have all kinds of surprises in that regard (e.g. one skin being the cluprit, and the user agent of another being blocked).

[SilverAzide]

Meh. There's no good solution to this, and this thread isn't the place for a debate. Brian announced to the community that the Rainmeter devs had been approached by the OSM team to warn about misuse of the WebParser. My comments are to simply confirm that OSM has followed through on their threat to block WebParser, and to point out that responsible devs need to pay attention to OSM's terms of service. My suggestion for a UserAgent was only that, a suggestion, per their TOS. Further debate on whether this is effective or not should be in it's own thread, IMO, not this announcement thread.