Yes, you were. You are the culprit!
It is currently April 27th, 2024, 7:03 am
Weather.com JSON not loading correctly
-
- Rainmeter Sage
- Posts: 7175
- Joined: February 27th, 2015, 2:38 pm
- Location: Terra Yincognita
-
- Rainmeter Sage
- Posts: 2611
- Joined: March 23rd, 2015, 5:26 pm
Re: ⭐ Weather.com JSON not loading correctly
Has anyone who is having the JSON parsing issues tried "downgrading" the UserAgent string to IE7? I'm not having the problem, so I'm unable to test if the JSON changes.
IE7
UserAgent="Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"
IE11
UserAgent="Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko"
-
- Rainmeter Sage
- Posts: 2611
- Joined: March 23rd, 2015, 5:26 pm
Re: ⭐ Weather.com JSON not loading correctly
Setting the user agent to one for Android definitely causes the "new" site to load, with different HTML and JSON. I'm wondering if dumbing down the browser/WebParser will cause weather.com to return the "old" site.
-
- Rainmeter Sage
- Posts: 7175
- Joined: February 27th, 2015, 2:38 pm
- Location: Terra Yincognita
Re: ⭐ Weather.com JSON not loading correctly
I'm not able to test it either, since I recently "fixed" the issue as per jsmorley's advice (and now I'm working locally anyway, until the regexes are finished). One other thing, regarding what you said about the measurement units, because I forgot to say it when replying to you earlier: it's not a big problem, since the "new format" only has 3 measurement unit systems: e=imperial, m=metric and h=hybrid (i.e. UK), and those strings (aka e,m,h) are all over the JSON and they can be extracted.SilverAzide wrote: ↑May 28th, 2020, 5:24 pm Has anyone who is having the JSON parsing issues tried "downgrading" the UserAgent string to IE7? I'm not having the problem, so I'm unable to test if the JSON changes.
IE7
UserAgent="Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"
IE11
UserAgent="Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko"
They might just unify the platforms to match the mobile one, as suspected earlier, because I don't think they suddenly can't detect whether the system is a desktop or a mobile... Bottom line, now it's the "Win 8 period" on weather.com (i.e. little bugs, things being inconsistent, etc). I wonder how long will it take them to get to the "stable" "Win10 period" ...SilverAzide wrote: ↑May 28th, 2020, 5:39 pm Setting the user agent to one for Android definitely causes the "new" site to load, with different HTML and JSON. I'm wondering if dumbing down the browser/WebParser will cause weather.com to return the "old" site.
-
- Rainmeter Sage
- Posts: 2611
- Joined: March 23rd, 2015, 5:26 pm
Re: ⭐ Weather.com JSON not loading correctly
Oh, I was not referring to that part, I was referring to the actual units of measure ("F", "C", "in", "mm", "mph", "kph", etc.). But you're right, it's not too big of a deal. More of an annoyance...Yincognito wrote: ↑May 28th, 2020, 5:45 pmOne other thing, regarding what you said about the measurement units, because I forgot to say it when replying to you earlier: it's not a big problem, since the "new format" only has 3 measurement unit systems: e=imperial, m=metric and h=hybrid (i.e. UK), and those strings (aka e,m,h) are all over the JSON and they can be extracted.
-
- Rainmeter Sage
- Posts: 7175
- Joined: February 27th, 2015, 2:38 pm
- Location: Terra Yincognita
Re: ⭐ Weather.com JSON not loading correctly
Indeed.SilverAzide wrote: ↑May 28th, 2020, 7:57 pm Oh, I was not referring to that part, I was referring to the actual units of measure ("F", "C", "in", "mm", "mph", "kph", etc.). But you're right, it's not too big of a deal. More of an annoyance...
-
- Posts: 105
- Joined: November 3rd, 2019, 12:00 am
Re: ⭐ Weather.com JSON not loading correctly
FYI: HTTP request (User-Agent, without Cookies) and response from weather.com
1. Environment
Windows 10 Pro 1909 64-bit (build 18363) - Japanese (1041)
curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
Rainmeter 4.4.0.3338 beta (64-bit), portable installed.
2. Target URL
https://weather.com/en-US/weather/today/l/42.31,-71.11
3. Results (2020-05-29 / May 29th, 2020)
(1) No User-Agent specified.
WebParser measure UserAgent
https://docs.rainmeter.net/manual/measures/webparser/#UserAgent
Latest user agents for Web Browsers & Operating Systems
https://www.whatismybrowser.com/guides/the-latest-user-agent/
HTTP headers
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers
1. Environment
Windows 10 Pro 1909 64-bit (build 18363) - Japanese (1041)
curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
Rainmeter 4.4.0.3338 beta (64-bit), portable installed.
2. Target URL
https://weather.com/en-US/weather/today/l/42.31,-71.11
3. Results (2020-05-29 / May 29th, 2020)
(1) No User-Agent specified.
Code: Select all
curl.exe -s -I -A "" --url "https://weather.com/en-US/weather/today/l/42.31,-71.11"
(2) Rainmeter WebParser Measure default.## Request Header ##
HEAD /en-US/weather/today/l/42.31,-71.11 HTTP/1.1
Host: weather.com
Accept: */*
## Response Header ##
HTTP/1.1 200 OK
Set-Cookie: ci=TWC-Connection-Speed=4G&TWC-Locale-Group=US&TWC-Device-Class=desktop
TWC-Device-Class: desktop
Code: Select all
curl.exe -s -I -A "Rainmeter WebParser plugin" --url "https://weather.com/en-US/weather/today/l/42.31,-71.11"
(3) Web browser for Desktop## Request Header ##
HEAD /en-US/weather/today/l/42.31,-71.11 HTTP/1.1
Host: weather.com
User-Agent: Rainmeter WebParser plugin
Accept: */*
## Response Header ##
HTTP/1.1 200 OK
Set-Cookie: ci=TWC-Connection-Speed=4G&TWC-Locale-Group=US&TWC-Device-Class=desktop
TWC-Device-Class: desktop
Code: Select all
curl.exe -s -I -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0" --url "https://weather.com/en-US/weather/today/l/42.31,-71.11"
(4) Web browser for Mobile## Request Header ##
HEAD /en-US/weather/today/l/42.31,-71.11 HTTP/1.1
Host: weather.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0
Accept: */*
## Response Header ##
HTTP/1.1 200 OK
Set-Cookie: ci=TWC-Connection-Speed=4G&TWC-Locale-Group=US&TWC-Device-Class=desktop
TWC-Device-Class: desktop
Code: Select all
curl.exe -s -I -A "Mozilla/5.0 (iPhone; CPU iPhone OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Mobile/15E148 Safari/604.1" --url "https://weather.com/en-US/weather/today/l/42.31,-71.11"
(5) Other Results.## Request Header ##
HEAD /en-US/weather/today/l/42.31,-71.11 HTTP/1.1
Host: weather.com
User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Mobile/15E148 Safari/604.1
Accept: */*
## Response Header##
HTTP/1.1 200 OK
Set-Cookie: ci=TWC-Connection-Speed=4G&TWC-Locale-Group=US&TWC-Device-Class=mobile
TWC-Device-Class: mobile
### TWC-Device-Class=desktop ###
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:73.0) Gecko/20100101 Firefox/73.0
Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko
Mozilla/5.0 (Windows NT 10.0; Trident/7.0; rv:11.0) like Gecko
Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36 Edg/83.0.478.37
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)
### TWC-Device-Class=mobile ###
Mozilla/5.0 (Linux; Android 10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.83 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 10; Pixel 4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Mobile Safari/537.36
Mozilla/5.0 (iPhone; CPU iPhone OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Mobile/15E148 Safari/604.1
### TWC-Device-Class=bot ### Add: 2020-06-05 / June 5h, 2020
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
WebParser measure UserAgent
https://docs.rainmeter.net/manual/measures/webparser/#UserAgent
Latest user agents for Web Browsers & Operating Systems
https://www.whatismybrowser.com/guides/the-latest-user-agent/
HTTP headers
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers
Last edited by nek on June 5th, 2020, 5:53 am, edited 3 times in total.
-
- Rainmeter Sage
- Posts: 7175
- Joined: February 27th, 2015, 2:38 pm
- Location: Terra Yincognita
Re: ⭐ Weather.com JSON not loading correctly
Nicely done.
The "ci" cookie is a session cookie (mine is something like TWC-Locale-Group=GLS+&X-Origin-Hint=Prod-IBM-LS&TWC-GeoIP-Country=RO&TWC-Privacy=gdpr, as my region is EU). Apart from that, I saved another 2 cookies (probably persistent ones) on my hard drive with the weather.com thing still working "normally". I might compared those with their version when the weather.com parsing will become broken again, to see if there's any difference.
That being said, it appears that visiting any page in IE has the potential to produce the issue, as mentioned here, so in the end it might not have anything to do with the cookies from weather.com at all...
Thing is, we can't avoid the IE stuff mixing into this, since WebParser relies on it.
-
- Posts: 442
- Joined: July 30th, 2014, 10:30 am
Re: ⭐ Weather.com JSON not loading correctly
just to mention that the issue can be reproduced wnever you delete IE data. For instance, even if a weather skin works and you delete the data, it may work or not. No rules.
And this is the most accurate way how to fix it based on my experience with the issue so far:
https://forum.rainmeter.net/viewtopic.php?f=27&p=177437#p177435
And this is the most accurate way how to fix it based on my experience with the issue so far:
https://forum.rainmeter.net/viewtopic.php?f=27&p=177437#p177435
-
- Rainmeter Sage
- Posts: 7175
- Joined: February 27th, 2015, 2:38 pm
- Location: Terra Yincognita
Re: ⭐ Weather.com JSON not loading correctly
It appears that one can view the session cookies and other bowser data from weather.com right on their site, by choosing Data Rights at the bottom of their main page, then opening Request A Portable Copy Of Data, followed by clicking on the View Browser Data button.
The ci session cookie shown by the site is slightly different from what I saved a few days ago after applying the fix, but the weather skin still gets the "old" format (i.e. it's still working).