⭐ Weather.com - Parsing the V3 JSON

[pul53dr1v3r]

I guess it makes sense that the data to fill up a little 6" screen would been less than my 32" monitor.

believe it or not, but i've never openet the site on my phone to see the visual difference, let alone the code. But hope there are all the data as on the desktop site.

when i don't use the Forecast it's 823 KB, and with it and details that show up on mouse hover, the size is 1,9.
As so i had to use 2 SuperParent. One for https://weather.com/#Lang#/weather/today/l/
and another for Link2=https://weather.com/#Lang#/weather/tenday/l/.

Probably it could be more optimized, but for now i don't see a way.

Code: Select all

;Super Parents
[@EntireSiteSuperParent]
Measure=WebParser
URL=#URL#
UpdateRate=#UpRate#
Flags=Resync | NoCookies
UserAgent=Mozilla/5.0 (Linux; Android 10; Pixel 4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Mobile Safari/537.36
DecodeCharacterReference=1
LogSubstringErrors=0
RegExp=(?siU)^(.*)$
StringIndex=1
FinishAction=[!EnableMeasureGroup Parents]

[@EntireSiteSuperParentForecast]
Measure=WebParser
URL=#URL2#
UpdateRate=#UpRate#
Flags=Resync | NoCookies
UserAgent=Mozilla/5.0 (Linux; Android 10; Pixel 4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Mobile Safari/537.36
DecodeCharacterReference=1
LogSubstringErrors=0
RegExp=(?siU)^(.*)$
StringIndex=1
FinishAction=[!EnableMeasureGroup ParentsForecast]
Disabled=(#showForecast12_1#=0)

;Parents
[MeasureNow]
Measure=WebParser
Url=[@EntireSiteSuperParent]
RegExp=(?siU)"getSunV3CurrentObservationsUrlConfig":.*{"cloudCeiling":.*,"cloudCoverPhrase":".*","dayOfWeek":".*","dayOrNight":".*","expirationTimeUtc":.*,"iconCode":(.*),"iconCodeExtend":.*,"obsQualifierCode":.*,"obsQualifierSeverity":.*,"precip1Hour":.*,"precip6Hour":.*,"precip24Hour":.*,"pressureAltimeter":(.*),"pressureChange":.*,"pressureMeanSeaLevel":.*"pressureTendencyCode":(.*),"pressureTendencyTrend":".*","relativeHumidity":(.*),"snow1Hour":.*,"snow6Hour":.*,"snow24Hour":.*,"sunriseTimeLocal":".*","sunriseTimeUtc":.*"sunsetTimeLocal":".*","sunsetTimeUtc":.*"temperature":(.*),"temperatureChange24Hour":.*,"temperatureDewPoint":(.*),"temperatureFeelsLike":(.*),"temperatureHeatIndex":.*"temperatureMax24Hour":.*"temperatureMaxSince7Am":.*"temperatureMin24Hour":.*"temperatureWindChill":.*"uvDescription":".*","uvIndex":(.*),"validTimeLocal":"(.*)","validTimeUtc":.*"visibility":(.*),"windDirection":(.*)"windDirectionCardinal":"(.*)","windGust":.*,"windSpeed":(.*),"wxPhraseLong":"(.*)","wxPhraseMedium":.*,"wxPhraseShort":.*}
LogSubstringErrors=0
FinishAction=[!EnableMeasureGroup "Weather"][!UpdateMeasureGroup Weather][!UpdateMeterGroup Weather][!UpdateMeterGroup Background][!Redraw]
Disabled=1
Group=Parents

[MeasureLocation]
Measure=WebParser
Url=[@EntireSiteSuperParent]
RegExp=(?siU)(?(?=.*{"locale1":.*,"locale2":".*","locale3":.*,"locale4":.*}).*"neighborhood":.*,"adminDistrict":(.*),"adminDistrictCode":(.*),"postalCode":(.*),"postalKey":.*,"country":(.*),"countryCode":(.*),"ianaTimeZone":.*,"displayName":(.*),.*)
LogSubstringErrors=0
FinishAction=[!EnableMeasureGroup Location][!UpdateMeasureGroup Location][!UpdateMeterGroup Location][!UpdateMeterGroup Background][!Redraw]
Disabled=1
Group=Parents

[MeasureForecast]
Measure=WebParser
Url=[@EntireSiteSuperParentForecast]
RegExp=(?siU)"getSunV3DailyForecastUrlConfig":.*{"dayOfWeek":.*"(.*)","(.*)","(.*)","(.*)","(.*)","(.*)","(.*)","(.*)",.*"temperatureMax":(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),.*"temperatureMin":(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),.*"validTimeLocal":.*"(.*)","(.*)","(.*)","(.*)","(.*)","(.*)","(.*)","(.*)".*"iconCode":(.*),(.*),(.*),.*,(.*),.*,(.*),.*,(.*),.*,(.*),.*,(.*),.*,(.*),.*
LogSubstringErrors=0
FinishAction=[!EnableMeasureGroup Forecast][!UpdateMeasureGroup Forecast][!UpdateMeterGroup Forecast][!UpdateMeterGroup Background][!Redraw]
Disabled=1
Group=ParentsForecast

[MeasureDetailed]
Measure=WebParser
Url=[@EntireSiteSuperParentForecast]
RegExp=(?siU)"getSunV3DailyForecastUrlConfig":.*{.*"precipChance":(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),.*"windDirectionCardinal":(.*),"(.*)","(.*)","(.*)","(.*)","(.*)","(.*)","(.*)"(.*)","(.*)","(.*)","(.*)","(.*)","(.*)","(.*)","(.*)",.*"windSpeed":(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),.*"wxPhraseLong":(.*),"(.*)","(.*)","(.*)","(.*)","(.*)","(.*)","(.*)","(.*)","(.*)","(.*)","(.*)","(.*)","(.*)","(.*)","(.*)"
LogSubstringErrors=0
FinishAction=[!EnableMeasureGroup Detailed][!UpdateMeasureGroup Detailed][!UpdateMeterGroup Detailed][!UpdateMeterGroup Background][!Redraw]
Disabled=1
Group=ParentsForecast

;Child measures
...

[Yincognito]

Well, I would use StringIndex / StringIndex2 to create "child" measures that are based on parsing a StringIndex of a "parent" measure. StringIndex2 creates a "child" stringindex. I'm not doing that in this case. The "parent" measures are still just fully parents, each with a RegExp that creates their own top-level StringIndex numbers, just based on parsing the "superparent" instead of parsing the "site". The "child" measures then reference the "parent" measures. That way I only need to hit the site once.

Indeed. I think it's more or less of the same, two ways / approaches achieving largely the same result. In my case, I had to use dynamic variables due to the nature of my skin, so at one point in the chain of WebParsers, I chose to go with a String measure instead of another WebParser measure, because if I'm not mistaken, the manual says one has to command update the WebParser parent in order to use a slightly different (and thus, dynamic) RegExp, and I had no intention of polling the site again...

[Yincognito]

As so i had to use 2 SuperParent. One for https://weather.com/#Lang#/weather/today/l/
and another for Link2=https://weather.com/#Lang#/weather/tenday/l/.

Apart from the code, which we'll take a look at in a moment (jsmorley as well, probably), why would you need the 10 day link?! The today URL contains the forecast for 15 days (not just 10) in its JSON (notice the current date in the top right corner and the last forecast day displayed in the center):

Weather 15 days.jpg

P.S. JsMorley's skin is getting the first instance of the forecast though, which is for 7 days. I believe some slight modifications in the corresponding measure(s) would get the 15 days one, if you really needed it.

[pul53dr1v3r]

Apart from the code, which we'll take a look at in a moment (jsmorley as well, probably), why would you need the 10 day link?! The today URL contains the forecast for 15 days (not just 10) in its JSON (notice the current date in the top right corner and the last forecast day displayed in the center):
Weather 15 days.jpg

Indeed, it's all on the today URL. i'm obviously blind sometimes. JSON is the same, i just changed the URL, and now the one is for the whole config.
Now with the forecast it uses about 837KB. Thanks for opening my eyes.

[Yincognito]

Indeed, it's all on the today URL. i'm obviously blind sometimes. JSON is the same, i just changed the URL, one for the whole config.
Now with the forecast it uses about 837KB. Thanks for opening my eyes.

Yeah, no worries. You also opened our eyes with the cookie thingy, so now we're even...

[pul53dr1v3r]

Yeah, no worries. You also opened our eyes with the cookie thingy, so now we're even...

i'm glad we get rid of the solution, most likely for good thanks to hardworking Morley's hands (probably Brian's too) and the NoCookies option.
But apart from it, the site seems settled down, at least to me.

[Yincognito]

i'm glad we get rid of the solution, most likely for good thanks to hardworking Morley's hands (probably Brian's too) and the NoCookies option.
But apart from it, the site seems settled down, at least to me.

Yeah, well, I don't like to make the same mistake twice (i.e. trusting that it's all settled), so I'm not relying on the new WebParser features (yet), but instead made my regexes to be "version independent". Meaning I don't care if I get V2 or V3, cookie or no cookie, or if the order of the fields or their names slightly change, I'm gonna get the weather either way - no user agent, no flag, no force reload, nothing. I'm done believing this is stable, so I'm adaptive / flexible as well.

[jsmorley]

Yeah, well, I don't like to make the same mistake twice (i.e. trusting that it's all settled), so I'm not relying on the new WebParser features (yet), but instead made my regexes to be "version independent". Meaning I don't care if I get V2 or V3, cookie or no cookie, or if the order of the fields or their names slightly change, I'm gonna get the weather either way - no user agent, no flag, no force reload, nothing. I'm done believing this is stable, so I'm adaptive / flexible as well.

I think you can in fact overdose on Viagra. Just sayin...

[Yincognito]

I think you can in fact overdose on Viagra. Just sayin...

Better product quality leads to increased consumption, so you may be right on that one...
I'm not sure yet, but ... can you overdose on ... erm ... cookies?!

[pbutler6]

Thanks for all your arduous work. I was almost finished with a new skin to replace the one from last year when Weather.com suddenly stopped working. After all your speedy effort I fixed things in a couple of weeks instead of the 4 months from last time.

I don't know if you plan to update the JSON measures, but if you do, I suggest that you include sunrise and sunset data for future days. In the evening I am no longer interested in when the sun rose this morning, what I want to know is when is the sun coming up tomorrow. I don't know what data is available on Weather.com, but right now sunrise and sunset are only available in your parsing of the Current section. There are no sunrise and sunset variables for the Forecast section, so I can't retrieve tomorrow's sunrise.