It is currently March 30th, 2020, 12:48 pm

⭐ weather.com - Parsing the JSON

Our most popular Tips and Tricks from the Rainmeter Team and others
User avatar
jsmorley
Developer
Posts: 20425
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

⭐ weather.com - Parsing the JSON

Post by jsmorley »

With the passing of the wxdata weather feed, several of us thought we might be able to simply parse the information directly from the https://weather.com site.

There are two approaches you can take with this. The first, parsing or "screen scraping" the HTML the site returns, is described in this post:

weather.com - Parsing the HTML

However, there are some risks with parsing HTML directly, as any change to the appearance of the site, which can happen with some regularity, can, and almost certainly will, break any regular expression using the HTML.




Included in the site when you go to it, in the "source" that is returned, there is also embedded JSON data. This is a method, similar to XML, to structure "data" in a way that is isolated from the "display". This should, in theory, change much less often, and thus be a lot more reliable over time.

Here is my weather skin, converted to use only the JSON data from the site:
Note that this only goes out to the site one time, and gets all information in one pass.


WeatherComJSON_Feb 20, 2020.rmskin

1.png


Much as the other example did, this uses a series of @Include .inc files to eliminate having to figure out regular expression or even create any measures for your skin. It does all the hard work for you!

Then you can just concentrate on the Meters in your skin. All the measures you will need are already there! Just create the meters, and use the MeasureName= options you can easily get by looking at About/Skins to see what the measure names are that are populated. I tried to make the measure names as descriptive as possible, and if you look in the include files, I have done as much commenting as I can.

If you want to "overload" an included measure, for instance to use Substitute or put an IfCondition or IfMatch on the measure, just create a copy of the measure as a Measure=String or Measure=Calc in the skin, and dynamically use the value of the original measure as a [SectionVariable].

I strongly recommend that you DON'T alter WeatherComJSONMeasures.inc.
That just reduces its usefulness as something you can just plug into another new skin, and makes is very difficult indeed for you if I update it.


WeatherComJSONVariables.inc
This is where you set your Location Code and Language. Read it for help.
Always @Include this file.

WeatherComJSONMeasures.inc
This parses all the JSON code and creates the measures you will need.
Always @Include this file.


WeatherComJSONMoon.inc
This will create measures for the Moon Phase and MoonSet / MoonRise times for 6 days.
It is optional to @Include this file.


WeatherComJSONPollen.inc
This will create measures for the Tree/Grass/Weed Pollen levels (0-5) for 4 days.
It is optional to @Include this file.


WeatherComJSONLanguage.inc
This will allow you to translate the "labels" I use in my skin to any other language you want.
This is specific to this skin, and you may not need it for yours.

GeneralVariables.inc
This is where I set all the general font and color variables.
This is specific to this skin, and you may not need it for yours.



Once you @Include WeatherComJSONVariables.inc AND WeatherComJSONMeasures.inc in your skin, the following measures will be created. You can use About / Skins to see them AND the values they currently have, but here is a list for reference:
[@UnitsType] : This will be "m" or "e"
[@UnitsTemperature] : This will be "C" or "F"
[@UnitsSpeed] : This will be "km/h" or "mph"
[@UnitsDistance] : This will be "km" or "mi"
[@UnitsAccumulation] : This will be "cm" or "in"
[@UnitsPrecipitation] : This will be "mm" or "in"
[@UnitsPressure] This will be "mb" or "in"

[@LocationLatitude] : i.e. 38.805
[@LocationLongitude] : i.e -77.047
[@LocationAdminDistrict]
[@LocationAdminDistrictCode]
These AdminDistrict values will vary in meaning depending on where you live.
In the US, this will be the "state". Other places it may be "region".
[@LocationCountry] : i.e. United States of America
[@LocationCountryCode] : i.e. US
[@LocationName] : Location Display Name - Generally the "city"
[@LocationLanguageCode] : i.e. en-US
[@LocationTimeZoneAbbreviation] : i.e. EST

[@CurrentPressure]
[@CurrentPressureChangeTrend]
[@CurrentPressureCode]
[@CurrentPressureChangeAmount]
[@CurrentDewPoint]
[@CurrentFeelsLike]
[@CurrentWindGust]
[@CurrentHumidity]
[@CurrentIcon]
[@CurrentObservationTimeStamp]
[@CurrentObservationDate]
[@CurrentObservationTime]
[@CurrentConditions]
[@CurrentPrecipitationLast24Hours] : Use @UnitsPrecipitation
[@CurrentSnowDepth] : Use @UnitsAccumulation
[@CurrentTemperature]
[@CurrentTemperatureMaxSince7AM]
[@CurrentUVIndex] : A range of 1-10
[@CurrentUVDescription]
[@CurrentVisibilityDistance]
[@CurrentWindSpeed]
[@CurrentWindDirectionCompass] : i.e. NNW
[@CurrentWindDirectionDegrees] : i.e. 220
[@CurrentSunriseTimeStamp]
[@CurrentSunriseTime]
[@CurrentSunSetTimeStamp]
[@CurrentSunsetTime]

[@ForecastTodayDayLong] : Full day of the week i.e. Saturday
[@ForecastTodayDayShort] : Always "Today", or an empty string. See Note below.
[@ForecastTodayDayPrecipitationPercent] : Percentage likely
[@ForecastTodayDayPrecipitationAmount] : Use @UnitsPrecipitation
[@ForecastTodayDayPrecipitationType] : "rain", "snow" or the generic "precip"
[@ForecastTodayDayTemperature]
[@ForecastTodayDayUVIndex] : A range of 1-10
[@ForecastTodayDayUVIndexDescription]
[@ForecastTodayDayIcon]
[@ForecastTodayDayConditions]
[@ForecastTodayDayNarrative]
[@ForecastTodayDayCloudPercent]
[@ForecastTodayDayWindDirectionCompass]
[@ForecastTodayDayWindDirectionDegrees]
[@ForecastTodayDayWindSpeed]
[@ForecastTodayDayHumidity]
[@ForecastTodayDaySnowRange] : Use @UnitsAccumulation
Note: At some point in the afternoon, all "TodayDay" measures except
[@ForecastTodayDayLong] will return an empty string. If [@ForecastTodayDayShort]
is an empty string, then it is "Tonight". There can't be a "forecast" for a part of day
that has already gone by. You must react to that, and perhaps use "TodayNight"
measures in their place.

[@ForecastTodayNightShort] : Always "Tonight"
[@ForecastTodayNightPrecipitationPercent] : Percentage likely
[@ForecastTodayNightPrecipitationAmount] : Use @UnitsPrecipitation
[@ForecastTodayNightPrecipitationType] : "rain", "snow" or the generic "precip"
[@ForecastTodayNightTemperature]
[@ForecastTodayNightUVIndex] : A range of 1-10
[@ForecastTodayNightUVIndexDescription]
[@ForecastTodayNightIcon]
[@ForecastTodayNightConditions]
[@ForecastTodayNightNarrative]
[@ForecastTodayNightCloudPercent]
[@ForecastTodayNightWindDirectionCompass]
[@ForecastTodayNightWindDirectionDegrees]
[@ForecastTodayNightWindSpeed]
[@ForecastTodayNightHumidity]
[@ForecastTodayNightSnowRange] : Use @UnitsAccumulation

[@ForecastTodayMonthLong] : i.e. February
[@ForecastTodayMonthShort] : i.e. Feb
[@ForecastTodayDayOfMonth] : i.e. 15

[@ForecastDay2DayLong] : Full day of the week i.e. Sunday
[@ForecastDay2DayShort] : Short day of the week i.e. Sun
[@ForecastDay2DayPrecipitationPercent] : Percentage likely
[@ForecastDay2DayPrecipitationAmount] : Use @UnitsPrecipitation
[@ForecastDay2DayPrecipitationType] : "rain", "snow" or the generic "precip"
[@ForecastDay2DayTemperature]
[@ForecastDay2DayUVIndex] : A range of 1-10
[@ForecastDay2DayUVIndexDescription]
[@ForecastDay2DayIcon]
[@ForecastDay2DayConditions]
[@ForecastDay2DayNarrative]
[@ForecastDay2DayCloudPercent]
[@ForecastDay2DayWindDirectionCompass]
[@ForecastDay2DayWindDirectionDegrees]
[@ForecastDay2DayWindSpeed]
[@ForecastDay2DayHumidity]
[@ForecastDay2DaySnowRange] : Use @UnitsAccumulation

[@ForecastDay2NightShort] : Short day of the week i.e. Sun
[@ForecastDay2NightPrecipitationPercent] : Percentage likely
[@ForecastDay2NightPrecipitationAmount] : Use @UnitsPrecipitation
[@ForecastDay2NightPrecipitationType] : "rain", "snow" or the generic "precip"
[@ForecastDay2NightTemperature]
[@ForecastDay2NightUVIndex] : A range of 1-10
[@ForecastDay2NightUVIndexDescription]
[@ForecastDay2NightIcon]
[@ForecastDay2NightConditions]
[@ForecastDay2NightNarrative]
[@ForecastDay2NightCloudPercent]
[@ForecastDay2NightWindDirectionCompass]
[@ForecastDay2NightWindDirectionDegrees]
[@ForecastDay2NightWindSpeed]
[@ForecastDay2NightHumidity]
[@ForecastDay2NightSnowRange] : Use @UnitsAccumulation

[@ForecastDay2MonthLong] : i.e. February
[@ForecastDay2MonthShort] : i.e. Feb
[@ForecastDay2DayOfMonth] : i.e. 16

...

This will follow the same pattern for Day3 through Day6
You do not have the required permissions to view the files attached to this post.
User avatar
jsmorley
Developer
Posts: 20425
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Parsing the JSON

Post by jsmorley »

This is how I "react" to the fact that after some point in the afternoon all the "forecast" values for "today day" will be empty:

Code: Select all

[MeasureToggleAtNight]
Measure=String
Group=Weather
String=[@ForecastTodayDayShort]
DynamicVariables=1
IfMatch=^$
IfMatchAction=[!SetOption MeterTodayIcon MeasureName "@ForecastTodayNightIcon"][!SetOption MeterTodayName MeasureName "@ForecastTodayNightShort"][!SetOption MeterTodayHighLow Measurename "@CurrentTemperatureMaxSince7AM"][!SetOption MeterTodayPrecipitation MeasureName "@ForecastTodayNightPrecipitationPercent"]
IfNotMatchAction=[!SetOption MeterTodayIcon MeasureName "@ForecastTodayDayIcon"][!SetOption MeterTodayName MeasureName "@ForecastTodayDayShort"][!SetOption MeterTodayHighLow Measurename "@ForecastTodayDayTemperature"][!SetOption MeterTodayPrecipitation MeasureName "@ForecastTodayDayPrecipitationPercent"]
User avatar
jsmorley
Developer
Posts: 20425
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Parsing the JSON

Post by jsmorley »

I'm going to let this percolate on my system for a few days, and if I don't find any issues with it, I will convert all my skins to use this JSON approach. There is no downside to using it instead of the HTML, and some possibility that it will have far greater longevity.
User avatar
eclectic-tech
Rainmeter Sage
Posts: 3835
Joined: April 12th, 2012, 9:40 pm
Location: Cedar Point, Ohio, USA

Re: ⭐ weather.com - Parsing the JSON

Post by eclectic-tech »

Gaff: You've done a man's job, sir. I guess you're through, huh?
Deckard: Finished.

Good work! :thumbup:
User avatar
jsmorley
Developer
Posts: 20425
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Parsing the JSON

Post by jsmorley »

eclectic-tech wrote:
February 8th, 2020, 4:16 am
Gaff: You've done a man's job, sir. I guess you're through, huh?
Deckard: Finished.

Good work! :thumbup:
Thanks. There are several advantages and disadvantages to using the JSON.

Advantages:
- More reliable over time. This is a big one...
- Has several additional bits of information that the HTML doesn't.
- All the data, location/current/forecast/hourly are all in one place, one call to the site.
I have not bothered to parse the "hourly", as honestly, I just don't care...It's there though.
- Is FAR easier to parse, the regular expression is just falling off a log easy. You never need any "lookahead".
If the site doesn't have a particular bit of information it returns a literal string of null, rather than leaving it out.
There is really never "missing" data. I just RegExpSubstitute=1 / Substitute="^null$":"" and I have an empty string with no missing StringIndex.


Disadvantages:
- While changing the LanguageCode will make the site return the numeric values in the appropriate units of measure, and string values in the appropriate language, just as before, there isn't an easy way to get the "labels" like "Temperature/Wind/Pressure/Sunrise/Feels Like" and such that you will likely want in your skin. I worked around that by providing a file to allow you to "translate" those labels as desired. There are some translations of these labels in the JSON, but I wasn't happy with them. They are not nearly as "pithy" as the labels on the site HTML are, and there are just a billion of them to parse if you want to be exhaustive.
- Dates and Times are returned in an ISO standard format like 2020-02-07T23:10:51-0500. You really need one or more Time measures to pretty those up and return 12hr-24hr time as desired.
- High and Low temperatures are not distinctly returned as such. There is just a temperature for "day" and that can be treated as the "high" and a temperature for "night" and that can be treated as the "low".

Overall, the advantages greatly outweigh the disadvantages, for which there are ways around.
User avatar
jsmorley
Developer
Posts: 20425
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Parsing the JSON

Post by jsmorley »

I'll add the "moon" times and phases at some point soon.
User avatar
SilverAzide
Posts: 716
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ weather.com - Parsing the JSON

Post by SilverAzide »

Thank you! :great:
DeviantArt Gadgets More...
User avatar
jsmorley
Developer
Posts: 20425
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Parsing the JSON

Post by jsmorley »

SilverAzide wrote:
February 8th, 2020, 4:43 pm
Thank you! :great:
You are welcome. Unless I find some "gotcha" with this, I think we can be done with parsing the HTML, which has all kinds of risks.

Parsing any "website" that isn't a pure, static, RSS/ATOM/XML/JSON "feed" always has some risk, but it looks to me like they use the embedded JSON data to feed the "display" HTML, and them changing the HTML, which is almost certain, may not impact the underlying data. That's my hope anyway.
User avatar
jsmorley
Developer
Posts: 20425
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Parsing the JSON

Post by jsmorley »

Added a measure for [@CurrentPressureCode], which to be honest I'm not sure what it represents, but perhaps a numeric code that indicates how quickly the barometer is "changing". Need to do some testing of this to be sure. I wanted to include it for completeness.

New .rmskin in the first post of this thread.
SCR
Posts: 44
Joined: April 15th, 2015, 11:13 pm

Re: ⭐ weather.com - Parsing the JSON

Post by SCR »

Late last night I was congratulating myself on completing the conversion of my weather skin. Feeling proud of myself that it worked, I went to sleep happy that the weather was once again on my screen and the job was done.

This morning as I settled in with my cup of coffee to read the latest and greatest from Rainmeter I was greeted with your post of your latest miracle.

Now I have to get back to another conversion project. However I don't mind a bit as I am now and forever addicted to Rainmeter. I have never had so much fun with any other software.

Besides, now that I'm in my late 70's Rainmeter keeps the gray matter in my head churning so you can add a health benefit to all the things Rainmeter can do. I don't understand a lot of it but it sure is fun to try and figure out.

Once again I offer my sincerest thanks for your hard work. Maybe not hard for you but near impossible for me. I appreciate all you do.