⭐ Weather.com - Parsing the V3 JSON

[SilverAzide]

Hard to prove a negative, but I guess the test will be if we continue to get these weird reports here on the forums once folks have the new beta.

So I was looking at the weather on my cellphone (a Google phone) and I noticed the default Android weather app had a direct link to the weather.com page with a variation of the URL we are using. Not sure if everyone knows this already, maybe I'm last to the party as usual. It appears you can control the temperature units with a parameter. The phone app uses:

https://weather.com/weather/today/l/<lat>,<long>?temp=f
https://weather.com/weather/today/l/<lat>,<long>?temp=c

With a little experimenting, using the locale ID will control the language, and a "units" parameter is also accepted in lieu of "temp" (but not the hybrid option):

https://weather.com/<locale>/weather/today/l/<lat>,<long>?units=e
https://weather.com/<locale>/weather/today/l/<lat>,<long>?units=m

So using something wacky like...
https://weather.com/ru-RU/weather/today/l/48.8566969,2.3514616?temp=f
...will get you the weather in Paris France with Russian descriptions and Fahrenheit/English units.

So the trick of controlling the units with the locale isn't necessarily the only option.

[jsmorley]

So I was looking at the weather on my cellphone (a Google phone) and I noticed the default Android weather app had a direct link to the weather.com page with a variation of the URL we are using. Not sure if everyone knows this already, maybe I'm last to the party as usual. It appears you can control the temperature units with a parameter. The phone app uses:

https://weather.com/weather/today/l/<lat>,<long>?temp=f
https://weather.com/weather/today/l/<lat>,<long>?temp=c

With a little experimenting, using the locale ID will control the language, and a "units" parameter is also accepted in lieu of "temp" (but not the hybrid option):

https://weather.com/<locale>/weather/today/l/<lat>,<long>?units=e
https://weather.com/<locale>/weather/today/l/<lat>,<long>?units=m

So using something wacky like...
https://weather.com/ru-RU/weather/today/l/48.8566969,2.3514616?temp=f
...will get you the weather in Paris France with Russian descriptions and Fahrenheit/English units.

So the trick of controlling the units with the locale isn't necessarily the only option.

That's interesting indeed! I wonder if there are any other parameters supported.

Looks to me like units=m and temp=c both do the same thing. Set everything Metric. And yeah, looks like units=h is not (yet) supported for the UK. So I guess in a way just using the locale is just as easy, and a tad more flexible. The one thing this DOES offer is a way to get a language other than English, while using Imperial units. Not sure many are going to want that, but hey...

[Yincognito]

So I was looking at the weather on my cellphone (a Google phone) and I noticed the default Android weather app had a direct link to the weather.com page with a variation of the URL we are using. Not sure if everyone knows this already, maybe I'm last to the party as usual. It appears you can control the temperature units with a parameter. The phone app uses:

https://weather.com/weather/today/l/<lat>,<long>?temp=f
https://weather.com/weather/today/l/<lat>,<long>?temp=c

With a little experimenting, using the locale ID will control the language, and a "units" parameter is also accepted in lieu of "temp" (but not the hybrid option):

https://weather.com/<locale>/weather/today/l/<lat>,<long>?units=e
https://weather.com/<locale>/weather/today/l/<lat>,<long>?units=m

So using something wacky like...
https://weather.com/ru-RU/weather/today/l/48.8566969,2.3514616?temp=f
...will get you the weather in Paris France with Russian descriptions and Fahrenheit/English units.

So the trick of controlling the units with the locale isn't necessarily the only option.

Brilliant find! You were not the last to the party this time, and this is really useful. I was trying to set the units in the main page through a method like this for a while, but my mistake was probably the fact that I used the units query first, right after the last slash. My settings skin was already configured to accept setting both the language and the units, I just couldn't find a way to do this for the main page. Now the whole issue is solved thanks to you.

That being said, we must be careful that this doesn't introduce other subtle differences in the page source that could hamper our parsing, but so far it looks good, I already applied the suffix to the URL and it behaves as expected.

I wonder if there are any other parameters supported.

Yeah, just gave it a try with &format=json wondering if we could get lucky and get the pure JSON without other "workarounds", but as expected, it didn't work.

EDIT: Yep, discovered another parameter, actually some sort of parameter combination, e.g.:

Code: Select all

https://weather.com/weather/today/l/Los+Angeles+CA?canonicalCityId=84c64154109916077c8d3c2352410aaae5f6eeff682000e3a7470e38976128c2

or

Code: Select all

https://weather.com/weather/today/l/Mexico+City+Mexico+City+Mexico?canonicalCityId=be0befa047d3e1dee37c318f892246bcba0860bde3bbe17767ebcc35ee1d2d72

Unfortunately, simply something like:

Code: Select all

https://weather.com/weather/today/l/Los+Angeles+CA

in order to automatically find the location by name and displaying the data doesn't seem to work...or maybe another parameter must be added to the mix...

[dgmele]

I'd prioritise hourly forecasts, those seem more useful.

I agree and anxiously await a V3 JSON measures file.

[jsmorley]

Ummm... I work for Cisco, you are going to possibly have that out later today !! You want to come work for us I wish we had that good of customer service.

Well, it helps that we can code directly into Production, since a mistake doesn't take down the world's entire banking system.

You guys are our entire Q&A Department.

[jsmorley]

I have changed my "super parent" measure to look like this:

Code: Select all

[@EntireSiteSuperParent]
Measure=WebParser
URL=#URLSite#
UpdateRate=#UpdateRate#
Flags=Resync | NoCookies
UserAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0
DecodeCharacterReference=1
LogSubstringErrors=0
RegExp=(?siU)^(.*)$
StringIndex=1
FinishAction=[!EnableMeasureGroup Parents]
;Debug=2
;Debug2File=#@#EntireSite.txt

I think that the "cache" was never an issue with weather.com, just the "cookies", and I see no reason to have it work hard to re-download the entire site on each update, when often it doesn't need to. In addition, I don't think the UserAgent was ever really an issue either, and while I think it might be worth having one, I set it back to a Windows desktop browser string so what I get with debug=2 is going to be what I get in my browser.

[SilverAzide]

I have changed my "super parent" measure to look like this:
<snip>

Just wanted to double-check: the main WeatherComJSONMeasures.inc and WeatherComJSONMoon.inc files are encoded in UTF-8. Is this to keep the size down? Secondly, (separate topic) I am assuming that the ForceReload=1 and Flags=ForceReload are the same thing. The manual didn't explicitly say so last time I checked, but I assumed so.

[jsmorley]

Just wanted to double-check: the main WeatherComJSONMeasures.inc and WeatherComJSONMoon.inc files are encoded in UTF-8. Is this to keep the size down? Secondly, (separate topic) I am assuming that the ForceReload=1 and Flags=ForceReload are the same thing. The manual didn't explicitly say so last time I checked, but I assumed so.

1) My bad, those .inc files should really be encoded as UTF-16 LE, to be consistent with what is recommended for all .ini and .inc files in Rainmeter. Having them be ANSI / UTF-8 w/o BOM is ok in this case, since there is no Unicode or Extended-ASCII in them, but that was careless. I will correct that the next time I need to update them otherwise.

2) Yes, Flags=ForceReload should be used to replace the stand-alone ForecReload=1 going forward. It is still supported of course, for backward compatibility, but I would always recommend staying away from "deprecated" options.


If ForceReload is important to you, and you are going to be distributing your skins to folks who may or may NOT have the latest 4.4 beta of Rainmeter, it would do no harm to set it as:

Code: Select all

Flags=ForceReload | NoCookies
ForceReload=1

Those not on the latest beta won't get the "NoCookies" behavior of course, but would seamlessly get the ForceReload behavior even on Rainmeter 4.3.

Just as an aside, I'm a bit skeptical that ForecReload ever really makes sense. Why not take advantage of the speed of the "cache" when the resource hasn't changed since the last update. I was using it when we weren't quite sure what was causing the issues with weather.com, but I think it is clear now that the problem was "cookies", so I removed that.

[jsmorley]

Having WebParser ever react to "cookies" was in my view a bad idea from the start. I think that flag of NoCookies should probably have been the default behavior from the git-go.

There is just no useful purpose in having WebParser "send" a cookie set in Internet Explorer as a part of the HTTP/HTTPS header when connecting to a site, or having it "accept" and store a cookie sent by the site. It can only cause unexpected or even harmful behavior, since in theory you could log into your bank in Internet Explorer, have it set a cookie in the IE database, and then have a skin that goes to the URL for your bank and have it actually "connect" to your logged in account, until the cookie eventually expires. There is no good that can come from that.

Connecting to a site that sets "tracking" or other "advertising" cookies with WebParser and having your system remember them and send them back the next time you connect isn't anything to be happy with either.

For Backwards Compatibility reasons, we really can't default the desired behavior, but I do recommend the new NoCookies flag going forward.

[jsmorley]

If you want to see this "unexpected / bad" behavior in the wild, just do this:

Get into Internet Explorer, and log in to your Google account. Open GMail, and send yourself an email.

Close Internet Explorer

Have a skin that has a measure like:

Code: Select all

[MeasureMail]
Measure=WebParser
URL=https://mail.google.com/gmail/feed/atom/
RegExp="(?siU)(.*)$"
DecodeCharacterReference=1
Debug=2

Run that, and you will find the information for your logged in account in WebParserDump.txt.