Weather.com JSON not loading correctly

[mak_kawa]

Hi jsmorley

Thank you for clarification.
I am now eased my mind to hear that the WebParser does not use IE11 components.

I'd be delighted to completely re-write WebParser to fundamentally use a different approach, maybe some embedded core of the open-source "Chromium" or something like that.

Great!! I am looking forward without impatience...

[nek]

I feel really weird... This morning I too encountered this issue on scraping Weather.com. Checking the html source, I have found that the source syntax/contents is completely different from previous one. Why?! It is natural that the RegExp error has occurred.

Hi,
There are different html/json on the Weather.com. Desktop or Mobile version.
The contents(html/json) for desktop is API version.1 based, mobile is API version.3 based. Edited: 2020-06-04 / June 4th, 2020

Q1. which version of contents did you get?
A1. There is "desktop" or "mobile" in the html source.

Code: Select all

"deviceClass": "desktop"
"deviceClass": "mobile"

Code: Select all

"twc-device-class": "desktop"
"twc-device-class": "mobile"

Q2. How to get the mobile version of contents from weather.com?
A2. Just define the UserAgent when you get html page.

Code: Select all

[MeasureWebParser]
Measure=WebParser
;;mobile version.
UserAgent=Mozilla/5.0 (iPhone; CPU iPhone OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Mobile/15E148 Safari/604.1
...

My weather skin here is using mobile version of weather.com.
https://forum.rainmeter.net/viewtopic.php?f=27&t=34524

Edited: 2020-05-31 / May 31st, 2020
Q3. How to solve this problems? >> https://forum.rainmeter.net/viewtopic.php?f=118&t=35342#p177064
A3. I recommend you to append the UserAgent in the skin file like this.

Code: Select all

[MeasureWebParser]
Measure=WebParser
;;desktop version.
UserAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36
...

Added: 2020-06-27 / June 27th, 2020
If someone still have a problems, try this.
Changing hostname weather.com to weather.com. (dot com dot) for avoiding cookie related issues.

Code: Select all

[MeasureWebParser]
Measure=WebParser
URL=https://weather.com./en-US/weather/today/l/...
;;desktop version.
UserAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36
...

I guess the weather.com is testing/preparing for this.

Chrome browser
Intent to Deprecate and Freeze: The User-Agent string - Google group

WebParser measure UserAgent
https://docs.rainmeter.net/manual/measures/webparser/#UserAgent
Latest user agents for Web Browsers & Operating Systems
https://www.whatismybrowser.com/guides/the-latest-user-agent/

[Yincognito]

Hi,
There are different html/json on the Weather.com. Desktop or Mobile version.
The contents(html/json) for desktop is API version.1 based, mobile is API version.3 based.

Q1. which version of contents did you get?
A1. There is "desktop" or "mobile" in the html source.

Code: Select all

"deviceClass": "desktop"
"deviceClass": "mobile"

Code: Select all

"twc-device-class": "desktop"
"twc-device-class": "mobile"

Q2. How to get the mobile version of contents from weather.com?
A2. Just define the UserAgent when you get html page.

Code: Select all

[MeasureWebParser]
Measure=WebParser
;;mobile version.
UserAgent=Mozilla/5.0 (iPhone; CPU iPhone OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Mobile/15E148 Safari/604.1
...

My weather skin here is using mobile version of weather.com.
https://forum.rainmeter.net/viewtopic.php?f=27&t=34524

Q3. How to solve this problems?
A3. I recomend you to append the UserAgent in the skin file like this.

Code: Select all

[MeasureWebParser]
Measure=WebParser
;;desktop version.
UserAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36
...

I guess the weather.com is testing/preparing for this.



WebParser measure UserAgent
https://docs.rainmeter.net/manual/measures/webparser/#UserAgent
Latest user agents for Web Browsers & Operating Systems
https://www.whatismybrowser.com/guides/the-latest-user-agent/

Very useful information you have there, nek - well done.

However, while overall correct, there is a small correction to make: the desktop JSON content was (until the recent problem, obviously) Version 2 based, not Version 1. And yes, the mobile data has "always" (or at least since the start of this year) been Version 3, that's something I notified people here on the forum since the start of the year, when these problems first appeared, but it wasn't considered of interest at the time. Regarding versions, don't be fooled by the JSON API findings in the page source, because I just checked that out on my mobile and it's version 1 - I have no idea why, maybe the JSON API version is different from the actual JSON content version, who knows.

So, only for the "observation" (i.e. "current condition") part, in the interest of simplicity (the forecast days part is grouped differently between versions)...

This is how JSON content Version 1 looks like:

Code: Select all

{"metadata":{"language":"en-GB","transaction_id":"1590574256535:-1490119034","version":"1","latitude":47.96,"longitude":26.40,"units":"m","expire_time_gmt":1590580800,"status_code":200},"observation":{"key":"15020","class":"observation","expire_time_gmt":1590580800,"obs_id":"15020","obs_name":"Botosani","valid_time_gmt":1590573600,"day_ind":"D","temp":13,"wx_icon":11,"icon_extd":1101,"wx_phrase":"Light Rain Shower","pressure_tend":2,"pressure_desc":"Falling","dewPt":10,"heat_index":13,"rh":85,"pressure":1004.10,"vis":9,"wc":13,"wdir":330,"wdir_cardinal":"NNW","gust":29,"wspd":14,"max_temp":null,"min_temp":null,"precip_total":null,"precip_hrly":0.1,"snow_hrly":null,"uv_desc":"Moderate","feels_like":13,"uv_index":3,"qualifier":null,"qualifier_svrty":null,"blunt_phrase":null,"terse_phrase":null,"clds":50,"water_temp":null,"primary_wave_period":null,"primary_wave_height":null,"primary_swell_period":null,"primary_swell_height":null,"primary_swell_direction":null,"secondary_swell_period":null,"secondary_swell_height":null,"secondary_swell_direction":null}}

This is how JSON content Version 2 looks like:

Code: Select all

"vt1Observation":
{"altimeter":1023.71,"barometerTrend":"Falling","barometerCode":2,"barometerChange":-0.68,"dewPoint":11,"feelsLike":14,"gust":26,"humidity":76,"icon":12,"observationTime":"2020-05-27T13:11:14+0300","obsQualifierCode":null,"obsQualifierSeverity":null,"phrase":"Rain","precip24Hour":57.66,"snowDepth":0.0,"temperature":14,"temperatureMaxSince7am":14,"uvIndex":4,"uvDescription":"Moderate","visibility":3.22,"windSpeed":14,"windDirCompass":"N","windDirDegrees":350}

This is how JSON content Version 3 looks like:

Code: Select all

{"cloudCeiling":17800,"cloudCoverPhrase":"Cloudy","dayOfWeek":"Wednesday","dayOrNight":"D","expirationTimeUtc":1590574874,"iconCode":12,"iconCodeExtend":1200,"obsQualifierCode":null,"obsQualifierSeverity":null,"precip1Hour":1.02,"precip6Hour":12.7,"precip24Hour":57.66,"pressureAltimeter":1023.71,"pressureChange":-0.68,"pressureMeanSeaLevel":1023.9,"pressureTendencyCode":2,"pressureTendencyTrend":"Falling","relativeHumidity":76,"snow1Hour":0.0,"snow6Hour":0.0,"snow24Hour":0.0,"sunriseTimeLocal":"2020-05-27T05:22:33+0300","sunriseTimeUtc":1590546153,"sunsetTimeLocal":"2020-05-27T21:01:03+0300","sunsetTimeUtc":1590602463,"temperature":14,"temperatureChange24Hour":1,"temperatureDewPoint":11,"temperatureFeelsLike":14,"temperatureHeatIndex":14,"temperatureMax24Hour":16,"temperatureMaxSince7Am":14,"temperatureMin24Hour":11,"temperatureWindChill":14,"uvDescription":"Moderate","uvIndex":4,"validTimeLocal":"2020-05-27T13:11:14+0300","validTimeUtc":1590574274,"visibility":3.22,"windDirection":350,"windDirectionCardinal":"N","windGust":26,"windSpeed":14,"wxPhraseLong":"Rain","wxPhraseMedium":null,"wxPhraseShort":null}

A last thought: maybe weather.com is "unifying" the JSON content versions presented on desktops with those presented on mobile, kind of how Windows switched to the mobile (i.e. "Metro") design a few years back, among other things?

[mak_kawa]

Hi nek

Ah... it was the mobile version of HTML source... maybe now I understand why/how. Thank you for clarification of the issue.

[jsmorley]

In digging a bit more I actually think it might just be a "cookie" issue, and I think if you simply do:

1.jpg

it should correct the issue. This is ongoing however, and you might have to do this from time to time until whatever is going on with the site settles down.

Note that you should exit Rainmeter before doing this, then restart Rainmeter after.

[nek]

- removed -

[Yincognito]

In digging a bit more I actually think it might just be a "cookie" issue, and I think if you simply do:



1.jpg



it should correct the issue. This is ongoing however, and you might have to do this from time to time until whatever is going on with the site settles down.

Note that you should exit Rainmeter before doing this, then restart Rainmeter after.

Just speculating, but would a cookie blocking on weather.com (i.e. the Internet Options -> Privacy tab -> Sites) stop this annoyance happening again? Also, do you think not allowing cookies from whatever site would have a negative impact on a WebParser measure operations on that site?

[jsmorley]

Just speculating, but would a cookie blocking on weather.com (i.e. the Internet Options -> Privacy tab -> Sites) stop this annoyance happening again? Also, do you think not allowing cookies from whatever site would have a negative impact on a WebParser measure operations on that site?

I have tried that, and blocking cookies from / to weather.com seems to stop it from connecting entirely. It's why I strongly recommend exiting Rainmeter, then kill the cookies, then restart. If you kill the cookie while the skin is running, it really hates having that rug pulled out from under it.

What seems to work for me, although I can't say for sure what the key to this is, is to:

1) Use UserAgent with a string for a recent version of Chrome, Firefox, Edge.
2) Use ForceReload=1.

Code: Select all

[@EntireSiteSuperParent]
Measure=WebParser
URL=#URLSite#
UserAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0
ForceReload=1
DecodeCharacterReference=1
UpdateRate=#UpdateRate#
LogSubstringErrors=0
RegExp=(?siU)^(.*)$
StringIndex=1
FinishAction=[!EnableMeasureGroup Parents]
;Debug=2
;Debug2File=#@#EntireSite.txt

3) When it gets hinky, delete the cookies as above.

[Yincognito]

I have tried that, and blocking cookies from / to weather.com seems to stop it from connecting entirely. It's why I strongly recommend exiting Rainmeter, then kill the cookies, then restart. If you kill the cookie while the skin is running, it really hates having that rug pulled out from under it.

I see. Thanks for letting me know.

[Yincognito]

Q3. How to solve this problems?
A3. I recomend you to append the UserAgent in the skin file like this.

Code: Select all

[MeasureWebParser]
Measure=WebParser
;;desktop version.
UserAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36
...

I guess the weather.com is testing/preparing for this.

Just tried your suggested method, and unfortunately, in my case, it didn't work (same results in jsmorley's JSON parsing skin, by the way):

Desktop User Agent.jpg

Will try again the way jsmorley suggested, not necessarily to fix things (because I know they'll come back sooner or later) but to make another save of the page the "old way", if possible.