Re: ⭐ Weather.com JSON not loading correctly
Posted: May 28th, 2020, 5:24 pm
Has anyone who is having the JSON parsing issues tried "downgrading" the UserAgent string to IE7? I'm not having the problem, so I'm unable to test if the JSON changes.
I'm not able to test it either, since I recently "fixed" the issue as per jsmorley's advice (and now I'm working locally anyway, until the regexes are finished). One other thing, regarding what you said about the measurement units, because I forgot to say it when replying to you earlier: it's not a big problem, since the "new format" only has 3 measurement unit systems: e=imperial, m=metric and h=hybrid (i.e. UK), and those strings (aka e,m,h) are all over the JSON and they can be extracted.SilverAzide wrote: ↑May 28th, 2020, 5:24 pm Has anyone who is having the JSON parsing issues tried "downgrading" the UserAgent string to IE7? I'm not having the problem, so I'm unable to test if the JSON changes.
IE7
UserAgent="Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"
IE11
UserAgent="Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko"
They might just unify the platforms to match the mobile one, as suspected earlier, because I don't think they suddenly can't detect whether the system is a desktop or a mobile... Bottom line, now it's the "Win 8 period" on weather.com (i.e. little bugs, things being inconsistent, etc). I wonder how long will it take them to get to the "stable" "Win10 period" ...SilverAzide wrote: ↑May 28th, 2020, 5:39 pm Setting the user agent to one for Android definitely causes the "new" site to load, with different HTML and JSON. I'm wondering if dumbing down the browser/WebParser will cause weather.com to return the "old" site.
Oh, I was not referring to that part, I was referring to the actual units of measure ("F", "C", "in", "mm", "mph", "kph", etc.). But you're right, it's not too big of a deal. More of an annoyance...Yincognito wrote: ↑May 28th, 2020, 5:45 pmOne other thing, regarding what you said about the measurement units, because I forgot to say it when replying to you earlier: it's not a big problem, since the "new format" only has 3 measurement unit systems: e=imperial, m=metric and h=hybrid (i.e. UK), and those strings (aka e,m,h) are all over the JSON and they can be extracted.
Indeed.SilverAzide wrote: ↑May 28th, 2020, 7:57 pm Oh, I was not referring to that part, I was referring to the actual units of measure ("F", "C", "in", "mm", "mph", "kph", etc.). But you're right, it's not too big of a deal. More of an annoyance...
Code: Select all
curl.exe -s -I -A "" --url "https://weather.com/en-US/weather/today/l/42.31,-71.11"
(2) Rainmeter WebParser Measure default.## Request Header ##
HEAD /en-US/weather/today/l/42.31,-71.11 HTTP/1.1
Host: weather.com
Accept: */*
## Response Header ##
HTTP/1.1 200 OK
Set-Cookie: ci=TWC-Connection-Speed=4G&TWC-Locale-Group=US&TWC-Device-Class=desktop
TWC-Device-Class: desktop
Code: Select all
curl.exe -s -I -A "Rainmeter WebParser plugin" --url "https://weather.com/en-US/weather/today/l/42.31,-71.11"
(3) Web browser for Desktop## Request Header ##
HEAD /en-US/weather/today/l/42.31,-71.11 HTTP/1.1
Host: weather.com
User-Agent: Rainmeter WebParser plugin
Accept: */*
## Response Header ##
HTTP/1.1 200 OK
Set-Cookie: ci=TWC-Connection-Speed=4G&TWC-Locale-Group=US&TWC-Device-Class=desktop
TWC-Device-Class: desktop
Code: Select all
curl.exe -s -I -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0" --url "https://weather.com/en-US/weather/today/l/42.31,-71.11"
(4) Web browser for Mobile## Request Header ##
HEAD /en-US/weather/today/l/42.31,-71.11 HTTP/1.1
Host: weather.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0
Accept: */*
## Response Header ##
HTTP/1.1 200 OK
Set-Cookie: ci=TWC-Connection-Speed=4G&TWC-Locale-Group=US&TWC-Device-Class=desktop
TWC-Device-Class: desktop
Code: Select all
curl.exe -s -I -A "Mozilla/5.0 (iPhone; CPU iPhone OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Mobile/15E148 Safari/604.1" --url "https://weather.com/en-US/weather/today/l/42.31,-71.11"
(5) Other Results.## Request Header ##
HEAD /en-US/weather/today/l/42.31,-71.11 HTTP/1.1
Host: weather.com
User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Mobile/15E148 Safari/604.1
Accept: */*
## Response Header##
HTTP/1.1 200 OK
Set-Cookie: ci=TWC-Connection-Speed=4G&TWC-Locale-Group=US&TWC-Device-Class=mobile
TWC-Device-Class: mobile
### TWC-Device-Class=desktop ###
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:73.0) Gecko/20100101 Firefox/73.0
Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko
Mozilla/5.0 (Windows NT 10.0; Trident/7.0; rv:11.0) like Gecko
Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36 Edg/83.0.478.37
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)
### TWC-Device-Class=mobile ###
Mozilla/5.0 (Linux; Android 10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.83 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 10; Pixel 4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Mobile Safari/537.36
Mozilla/5.0 (iPhone; CPU iPhone OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Mobile/15E148 Safari/604.1
### TWC-Device-Class=bot ### Add: 2020-06-05 / June 5h, 2020
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Nicely done.
It appears that one can view the session cookies and other bowser data from weather.com right on their site, by choosing Data Rights at the bottom of their main page, then opening Request A Portable Copy Of Data, followed by clicking on the View Browser Data button.