It is currently December 1st, 2020, 5:39 pm

Weather.com JSON not loading correctly

General topics related to Rainmeter.
User avatar
jsmorley
Developer
Posts: 21563
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com JSON not loading correctly

Post by jsmorley »

mak_kawa wrote: May 26th, 2020, 11:18 pm Hi everyone

I feel really weird... This morning I too encountered this issue on scraping Weather.com. Checking the html source, I have found that the source syntax/contents is completely different from previous one. Why?! It is natural that the RegExp error has occurred.

Following jsmorley and pul53dr1v3r's instruction, I doubtfully cleared cookies on *Internet Explorer 11* (!). I am not using IE11 since two or three years ago. And... surprisingly the issue is gone!.
I am not sure why clearing the cookie for IE11 can solve the issue on my weather skin... One thing I can guess is that "the WebParser measure uses IE11 components", am I right?

If so, I am a bit anxious that Rainmeter uses obsolete(?) components for the important and indispensable WebParser measure... Is there a possibility that IE11 and its components will be removed completely from Windows10 (or later version)??
It doesn't matter. Rainmeter does NOT use Internet Explorer. What it uses are some Windows API calls for accessing the internet, that are also used by Internet Explorer. Those are not ever going to go away, as they are used by all kinds of things in Windows that need to go out to the outside world.

That's why when you clear the cache and cookies you use a Control Panel applet called "Internet Options", and not something called "Internet Explorer Options".

Now mind you, I'd be delighted to completely re-write WebParser to fundamentally use a different approach, maybe some embedded core of the open-source "Chromium" or something like that. I also feel "weird", if not actually "dirty", even having to talk about Internet Explorer. I don't see that happening in the short-term however.
mak_kawa
Posts: 905
Joined: December 30th, 2015, 9:47 am

Re: ⭐ Weather.com JSON not loading correctly

Post by mak_kawa »

Hi jsmorley

Thank you for clarification.
I am now eased my mind to hear that the WebParser does not use IE11 components. :-)
I'd be delighted to completely re-write WebParser to fundamentally use a different approach, maybe some embedded core of the open-source "Chromium" or something like that.
Great!! I am looking forward without impatience...
nek
Posts: 30
Joined: November 3rd, 2019, 12:00 am

Re: ⭐ Weather.com JSON not loading correctly

Post by nek »

mak_kawa wrote: May 26th, 2020, 11:18 pm I feel really weird... This morning I too encountered this issue on scraping Weather.com. Checking the html source, I have found that the source syntax/contents is completely different from previous one. Why?! It is natural that the RegExp error has occurred.
Hi,
There are different html/json on the Weather.com. Desktop or Mobile version.
The contents(html/json) for desktop is API version.1 based, mobile is API version.3 based. Edited: 2020-06-04 / June 4th, 2020

Q1. which version of contents did you get?
A1. There is "desktop" or "mobile" in the html source.

Code: Select all

"deviceClass": "desktop"
"deviceClass": "mobile"

Code: Select all

"twc-device-class": "desktop"
"twc-device-class": "mobile"
Q2. How to get the mobile version of contents from weather.com?
A2. Just define the UserAgent when you get html page.

Code: Select all

[MeasureWebParser]
Measure=WebParser
;;mobile version.
UserAgent=Mozilla/5.0 (iPhone; CPU iPhone OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Mobile/15E148 Safari/604.1
...
My weather skin here is using mobile version of weather.com.
https://forum.rainmeter.net/viewtopic.php?f=27&t=34524

Edited: 2020-05-31 / May 31st, 2020
Q3. How to solve this problems? >> https://forum.rainmeter.net/viewtopic.php?f=118&t=35342#p177064
A3. I recommend you to append the UserAgent in the skin file like this.

Code: Select all

[MeasureWebParser]
Measure=WebParser
;;desktop version.
UserAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36
...
Added: 2020-06-27 / June 27th, 2020
If someone still have a problems, try this.
Changing hostname weather.com to weather.com. (dot com dot) for avoiding cookie related issues.

Code: Select all

[MeasureWebParser]
Measure=WebParser
URL=https://weather.com./en-US/weather/today/l/...
;;desktop version.
UserAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36
...

I guess the weather.com is testing/preparing for this.
Chrome browser
Intent to Deprecate and Freeze: The User-Agent string - Google group

WebParser measure UserAgent
https://docs.rainmeter.net/manual/measures/webparser/#UserAgent
Latest user agents for Web Browsers & Operating Systems
https://www.whatismybrowser.com/guides/the-latest-user-agent/
Last edited by nek on June 28th, 2020, 12:38 pm, edited 6 times in total.
User avatar
Yincognito
Posts: 2933
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com JSON not loading correctly

Post by Yincognito »

nek wrote: May 27th, 2020, 5:56 am Hi,
There are different html/json on the Weather.com. Desktop or Mobile version.
The contents(html/json) for desktop is API version.1 based, mobile is API version.3 based.

Q1. which version of contents did you get?
A1. There is "desktop" or "mobile" in the html source.

Code: Select all

"deviceClass": "desktop"
"deviceClass": "mobile"

Code: Select all

"twc-device-class": "desktop"
"twc-device-class": "mobile"
Q2. How to get the mobile version of contents from weather.com?
A2. Just define the UserAgent when you get html page.

Code: Select all

[MeasureWebParser]
Measure=WebParser
;;mobile version.
UserAgent=Mozilla/5.0 (iPhone; CPU iPhone OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Mobile/15E148 Safari/604.1
...
My weather skin here is using mobile version of weather.com.
https://forum.rainmeter.net/viewtopic.php?f=27&t=34524

Q3. How to solve this problems?
A3. I recomend you to append the UserAgent in the skin file like this.

Code: Select all

[MeasureWebParser]
Measure=WebParser
;;desktop version.
UserAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36
...
I guess the weather.com is testing/preparing for this.



WebParser measure UserAgent
https://docs.rainmeter.net/manual/measures/webparser/#UserAgent
Latest user agents for Web Browsers & Operating Systems
https://www.whatismybrowser.com/guides/the-latest-user-agent/
Very useful information you have there, nek - well done. :thumbup:

However, while overall correct, there is a small correction to make: the desktop JSON content was (until the recent problem, obviously) Version 2 based, not Version 1. And yes, the mobile data has "always" (or at least since the start of this year) been Version 3, that's something I notified people here on the forum since the start of the year, when these problems first appeared, but it wasn't considered of interest at the time. Regarding versions, don't be fooled by the JSON API findings in the page source, because I just checked that out on my mobile and it's version 1 - I have no idea why, maybe the JSON API version is different from the actual JSON content version, who knows.

So, only for the "observation" (i.e. "current condition") part, in the interest of simplicity (the forecast days part is grouped differently between versions)...

This is how JSON content Version 1 looks like:

Code: Select all

{"metadata":{"language":"en-GB","transaction_id":"1590574256535:-1490119034","version":"1","latitude":47.96,"longitude":26.40,"units":"m","expire_time_gmt":1590580800,"status_code":200},"observation":{"key":"15020","class":"observation","expire_time_gmt":1590580800,"obs_id":"15020","obs_name":"Botosani","valid_time_gmt":1590573600,"day_ind":"D","temp":13,"wx_icon":11,"icon_extd":1101,"wx_phrase":"Light Rain Shower","pressure_tend":2,"pressure_desc":"Falling","dewPt":10,"heat_index":13,"rh":85,"pressure":1004.10,"vis":9,"wc":13,"wdir":330,"wdir_cardinal":"NNW","gust":29,"wspd":14,"max_temp":null,"min_temp":null,"precip_total":null,"precip_hrly":0.1,"snow_hrly":null,"uv_desc":"Moderate","feels_like":13,"uv_index":3,"qualifier":null,"qualifier_svrty":null,"blunt_phrase":null,"terse_phrase":null,"clds":50,"water_temp":null,"primary_wave_period":null,"primary_wave_height":null,"primary_swell_period":null,"primary_swell_height":null,"primary_swell_direction":null,"secondary_swell_period":null,"secondary_swell_height":null,"secondary_swell_direction":null}}
This is how JSON content Version 2 looks like:

Code: Select all

"vt1Observation":
{"altimeter":1023.71,"barometerTrend":"Falling","barometerCode":2,"barometerChange":-0.68,"dewPoint":11,"feelsLike":14,"gust":26,"humidity":76,"icon":12,"observationTime":"2020-05-27T13:11:14+0300","obsQualifierCode":null,"obsQualifierSeverity":null,"phrase":"Rain","precip24Hour":57.66,"snowDepth":0.0,"temperature":14,"temperatureMaxSince7am":14,"uvIndex":4,"uvDescription":"Moderate","visibility":3.22,"windSpeed":14,"windDirCompass":"N","windDirDegrees":350}
This is how JSON content Version 3 looks like:

Code: Select all

{"cloudCeiling":17800,"cloudCoverPhrase":"Cloudy","dayOfWeek":"Wednesday","dayOrNight":"D","expirationTimeUtc":1590574874,"iconCode":12,"iconCodeExtend":1200,"obsQualifierCode":null,"obsQualifierSeverity":null,"precip1Hour":1.02,"precip6Hour":12.7,"precip24Hour":57.66,"pressureAltimeter":1023.71,"pressureChange":-0.68,"pressureMeanSeaLevel":1023.9,"pressureTendencyCode":2,"pressureTendencyTrend":"Falling","relativeHumidity":76,"snow1Hour":0.0,"snow6Hour":0.0,"snow24Hour":0.0,"sunriseTimeLocal":"2020-05-27T05:22:33+0300","sunriseTimeUtc":1590546153,"sunsetTimeLocal":"2020-05-27T21:01:03+0300","sunsetTimeUtc":1590602463,"temperature":14,"temperatureChange24Hour":1,"temperatureDewPoint":11,"temperatureFeelsLike":14,"temperatureHeatIndex":14,"temperatureMax24Hour":16,"temperatureMaxSince7Am":14,"temperatureMin24Hour":11,"temperatureWindChill":14,"uvDescription":"Moderate","uvIndex":4,"validTimeLocal":"2020-05-27T13:11:14+0300","validTimeUtc":1590574274,"visibility":3.22,"windDirection":350,"windDirectionCardinal":"N","windGust":26,"windSpeed":14,"wxPhraseLong":"Rain","wxPhraseMedium":null,"wxPhraseShort":null}
A last thought: maybe weather.com is "unifying" the JSON content versions presented on desktops with those presented on mobile, kind of how Windows switched to the mobile (i.e. "Metro") design a few years back, among other things?
mak_kawa
Posts: 905
Joined: December 30th, 2015, 9:47 am

Re: ⭐ Weather.com JSON not loading correctly

Post by mak_kawa »

Hi nek

Ah... it was the mobile version of HTML source... maybe now I understand why/how. Thank you for clarification of the issue. :-)
User avatar
jsmorley
Developer
Posts: 21563
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com JSON not loading correctly

Post by jsmorley »

In digging a bit more I actually think it might just be a "cookie" issue, and I think if you simply do:


1.jpg


it should correct the issue. This is ongoing however, and you might have to do this from time to time until whatever is going on with the site settles down.

Note that you should exit Rainmeter before doing this, then restart Rainmeter after.
You do not have the required permissions to view the files attached to this post.
nek
Posts: 30
Joined: November 3rd, 2019, 12:00 am

Re: ⭐ Weather.com JSON not loading correctly

Post by nek »

- removed -
Last edited by nek on June 10th, 2020, 4:40 pm, edited 1 time in total.
User avatar
Yincognito
Posts: 2933
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com JSON not loading correctly

Post by Yincognito »

jsmorley wrote: May 27th, 2020, 11:38 am In digging a bit more I actually think it might just be a "cookie" issue, and I think if you simply do:



1.jpg



it should correct the issue. This is ongoing however, and you might have to do this from time to time until whatever is going on with the site settles down.

Note that you should exit Rainmeter before doing this, then restart Rainmeter after.
Just speculating, but would a cookie blocking on weather.com (i.e. the Internet Options -> Privacy tab -> Sites) stop this annoyance happening again? Also, do you think not allowing cookies from whatever site would have a negative impact on a WebParser measure operations on that site?
User avatar
jsmorley
Developer
Posts: 21563
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com JSON not loading correctly

Post by jsmorley »

Yincognito wrote: May 27th, 2020, 11:49 am Just speculating, but would a cookie blocking on weather.com (i.e. the Internet Options -> Privacy tab -> Sites) stop this annoyance happening again? Also, do you think not allowing cookies from whatever site would have a negative impact on a WebParser measure operations on that site?
I have tried that, and blocking cookies from / to weather.com seems to stop it from connecting entirely. It's why I strongly recommend exiting Rainmeter, then kill the cookies, then restart. If you kill the cookie while the skin is running, it really hates having that rug pulled out from under it.

What seems to work for me, although I can't say for sure what the key to this is, is to:

1) Use UserAgent with a string for a recent version of Chrome, Firefox, Edge.
2) Use ForceReload=1.

Code: Select all

[@EntireSiteSuperParent]
Measure=WebParser
URL=#URLSite#
UserAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0
ForceReload=1
DecodeCharacterReference=1
UpdateRate=#UpdateRate#
LogSubstringErrors=0
RegExp=(?siU)^(.*)$
StringIndex=1
FinishAction=[!EnableMeasureGroup Parents]
;Debug=2
;Debug2File=#@#EntireSite.txt

3) When it gets hinky, delete the cookies as above.
User avatar
Yincognito
Posts: 2933
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com JSON not loading correctly

Post by Yincognito »

jsmorley wrote: May 27th, 2020, 11:57 am I have tried that, and blocking cookies from / to weather.com seems to stop it from connecting entirely. It's why I strongly recommend exiting Rainmeter, then kill the cookies, then restart. If you kill the cookie while the skin is running, it really hates having that rug pulled out from under it.
I see. Thanks for letting me know. ;-)