It is currently September 22nd, 2021, 5:14 pm

Weather.com JSON not loading correctly

General topics related to Rainmeter.
User avatar
Yincognito
Rainmeter Sage
Posts: 4052
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com JSON not loading correctly

Post by Yincognito »

dvo wrote: May 28th, 2020, 5:20 pmthat was me :P
Yes, you were. You are the culprit! :twisted:
User avatar
SilverAzide
Rainmeter Sage
Posts: 1564
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ Weather.com JSON not loading correctly

Post by SilverAzide »

jsmorley wrote: May 28th, 2020, 5:06 pm Yeah, I like the new design, but it really doesn't have anything to do with the JSON.
Has anyone who is having the JSON parsing issues tried "downgrading" the UserAgent string to IE7? I'm not having the problem, so I'm unable to test if the JSON changes.

IE7
UserAgent="Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"

IE11
UserAgent="Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko"
Gadgets Wiki GitHub More Gadgets...
User avatar
SilverAzide
Rainmeter Sage
Posts: 1564
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ Weather.com JSON not loading correctly

Post by SilverAzide »

Setting the user agent to one for Android definitely causes the "new" site to load, with different HTML and JSON. I'm wondering if dumbing down the browser/WebParser will cause weather.com to return the "old" site.
Gadgets Wiki GitHub More Gadgets...
User avatar
Yincognito
Rainmeter Sage
Posts: 4052
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com JSON not loading correctly

Post by Yincognito »

SilverAzide wrote: May 28th, 2020, 5:24 pm Has anyone who is having the JSON parsing issues tried "downgrading" the UserAgent string to IE7? I'm not having the problem, so I'm unable to test if the JSON changes.

IE7
UserAgent="Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"

IE11
UserAgent="Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko"
I'm not able to test it either, since I recently "fixed" the issue as per jsmorley's advice (and now I'm working locally anyway, until the regexes are finished). One other thing, regarding what you said about the measurement units, because I forgot to say it when replying to you earlier: it's not a big problem, since the "new format" only has 3 measurement unit systems: e=imperial, m=metric and h=hybrid (i.e. UK), and those strings (aka e,m,h) are all over the JSON and they can be extracted.
SilverAzide wrote: May 28th, 2020, 5:39 pm Setting the user agent to one for Android definitely causes the "new" site to load, with different HTML and JSON. I'm wondering if dumbing down the browser/WebParser will cause weather.com to return the "old" site.
They might just unify the platforms to match the mobile one, as suspected earlier, because I don't think they suddenly can't detect whether the system is a desktop or a mobile... Bottom line, now it's the "Win 8 period" on weather.com (i.e. little bugs, things being inconsistent, etc). I wonder how long will it take them to get to the "stable" "Win10 period" ... :D
User avatar
SilverAzide
Rainmeter Sage
Posts: 1564
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ Weather.com JSON not loading correctly

Post by SilverAzide »

Yincognito wrote: May 28th, 2020, 5:45 pmOne other thing, regarding what you said about the measurement units, because I forgot to say it when replying to you earlier: it's not a big problem, since the "new format" only has 3 measurement unit systems: e=imperial, m=metric and h=hybrid (i.e. UK), and those strings (aka e,m,h) are all over the JSON and they can be extracted.
Oh, I was not referring to that part, I was referring to the actual units of measure ("F", "C", "in", "mm", "mph", "kph", etc.). But you're right, it's not too big of a deal. More of an annoyance... :)
Gadgets Wiki GitHub More Gadgets...
User avatar
Yincognito
Rainmeter Sage
Posts: 4052
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com JSON not loading correctly

Post by Yincognito »

SilverAzide wrote: May 28th, 2020, 7:57 pm Oh, I was not referring to that part, I was referring to the actual units of measure ("F", "C", "in", "mm", "mph", "kph", etc.). But you're right, it's not too big of a deal. More of an annoyance... :)
Indeed.
nek
Posts: 44
Joined: November 3rd, 2019, 12:00 am

Re: ⭐ Weather.com JSON not loading correctly

Post by nek »

FYI: HTTP request (User-Agent, without Cookies) and response from weather.com

1. Environment

Windows 10 Pro 1909 64-bit (build 18363) - Japanese (1041)
curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
Rainmeter 4.4.0.3338 beta (64-bit), portable installed.

2. Target URL

https://weather.com/en-US/weather/today/l/42.31,-71.11

3. Results (2020-05-29 / May 29th, 2020)

(1) No User-Agent specified.

Code: Select all

curl.exe -s -I -A "" --url "https://weather.com/en-US/weather/today/l/42.31,-71.11"
## Request Header ##
HEAD /en-US/weather/today/l/42.31,-71.11 HTTP/1.1
Host: weather.com
Accept: */*
## Response Header ##
HTTP/1.1 200 OK
Set-Cookie: ci=TWC-Connection-Speed=4G&TWC-Locale-Group=US&TWC-Device-Class=desktop
TWC-Device-Class: desktop
(2) Rainmeter WebParser Measure default.

Code: Select all

curl.exe -s -I -A "Rainmeter WebParser plugin" --url "https://weather.com/en-US/weather/today/l/42.31,-71.11"
## Request Header ##
HEAD /en-US/weather/today/l/42.31,-71.11 HTTP/1.1
Host: weather.com
User-Agent: Rainmeter WebParser plugin
Accept: */*
## Response Header ##
HTTP/1.1 200 OK
Set-Cookie: ci=TWC-Connection-Speed=4G&TWC-Locale-Group=US&TWC-Device-Class=desktop
TWC-Device-Class: desktop
(3) Web browser for Desktop

Code: Select all

curl.exe -s -I -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0" --url "https://weather.com/en-US/weather/today/l/42.31,-71.11"
## Request Header ##
HEAD /en-US/weather/today/l/42.31,-71.11 HTTP/1.1
Host: weather.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0
Accept: */*
## Response Header ##
HTTP/1.1 200 OK
Set-Cookie: ci=TWC-Connection-Speed=4G&TWC-Locale-Group=US&TWC-Device-Class=desktop
TWC-Device-Class: desktop
(4) Web browser for Mobile

Code: Select all

curl.exe -s -I -A "Mozilla/5.0 (iPhone; CPU iPhone OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Mobile/15E148 Safari/604.1" --url "https://weather.com/en-US/weather/today/l/42.31,-71.11"
## Request Header ##
HEAD /en-US/weather/today/l/42.31,-71.11 HTTP/1.1
Host: weather.com
User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Mobile/15E148 Safari/604.1
Accept: */*
## Response Header##
HTTP/1.1 200 OK
Set-Cookie: ci=TWC-Connection-Speed=4G&TWC-Locale-Group=US&TWC-Device-Class=mobile
TWC-Device-Class: mobile
(5) Other Results.
### TWC-Device-Class=desktop ###
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:73.0) Gecko/20100101 Firefox/73.0
Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko
Mozilla/5.0 (Windows NT 10.0; Trident/7.0; rv:11.0) like Gecko
Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36 Edg/83.0.478.37
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)
### TWC-Device-Class=mobile ###
Mozilla/5.0 (Linux; Android 10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.83 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 10; Pixel 4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Mobile Safari/537.36
Mozilla/5.0 (iPhone; CPU iPhone OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Mobile/15E148 Safari/604.1
### TWC-Device-Class=bot ### Add: 2020-06-05 / June 5h, 2020
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

WebParser measure UserAgent
https://docs.rainmeter.net/manual/measures/webparser/#UserAgent
Latest user agents for Web Browsers & Operating Systems
https://www.whatismybrowser.com/guides/the-latest-user-agent/
HTTP headers
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers
Last edited by nek on June 5th, 2020, 5:53 am, edited 3 times in total.
User avatar
Yincognito
Rainmeter Sage
Posts: 4052
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com JSON not loading correctly

Post by Yincognito »

nek wrote: May 29th, 2020, 1:15 pmFYI: HTTP request (User-Agent, without Cookies) and response from weather.com
Nicely done. :thumbup:

The "ci" cookie is a session cookie (mine is something like TWC-Locale-Group=GLS+&X-Origin-Hint=Prod-IBM-LS&TWC-GeoIP-Country=RO&TWC-Privacy=gdpr, as my region is EU). Apart from that, I saved another 2 cookies (probably persistent ones) on my hard drive with the weather.com thing still working "normally". I might compared those with their version when the weather.com parsing will become broken again, to see if there's any difference.

That being said, it appears that visiting any page in IE has the potential to produce the issue, as mentioned here, so in the end it might not have anything to do with the cookies from weather.com at all... :confused:

Thing is, we can't avoid the IE stuff mixing into this, since WebParser relies on it. :Whistle
User avatar
pul53dr1v3r
Posts: 437
Joined: July 30th, 2014, 10:30 am

Re: ⭐ Weather.com JSON not loading correctly

Post by pul53dr1v3r »

just to mention that the issue can be reproduced wnever you delete IE data. For instance, even if a weather skin works and you delete the data, it may work or not. No rules.

And this is the most accurate way how to fix it based on my experience with the issue so far:
https://forum.rainmeter.net/viewtopic.php?f=27&p=177437#p177435
User avatar
Yincognito
Rainmeter Sage
Posts: 4052
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com JSON not loading correctly

Post by Yincognito »

nek wrote: May 29th, 2020, 1:15 pmFYI: HTTP request (User-Agent, without Cookies) and response from weather.com
It appears that one can view the session cookies and other bowser data from weather.com right on their site, by choosing Data Rights at the bottom of their main page, then opening Request A Portable Copy Of Data, followed by clicking on the View Browser Data button.

The ci session cookie shown by the site is slightly different from what I saved a few days ago after applying the fix, but the weather skin still gets the "old" format (i.e. it's still working).