Market Prices

[Yincognito]

That sounds like a perfect-fit solution! I really tried: reading, understanding, finding examples (I need concrete examples for understanding...).

I could provide the ready made regex pattern, but I promise it's actually really simple and you should be able to do it yourself - if it was complicated I would have posted it already, you know...

You already posted the <quickQuote> structure that your [mIndex1_Price] is using as the "input string" via its StringIndex=1 option:

Code: Select all

<quickQuote>
<altName>NVIDIA Corp</altName>
<altSymbol>NVDA.O</altSymbol>
<assetSubType>Common Stock</assetSubType>
<assetType>STOCK</assetType>
<cacheServed>false</cacheServed>
<cachedTime>Mon May 27 03:51:02 EDT 2024</cachedTime>
<change>-0.00</change>
<change_pct>-0.00</change_pct>
<cnbcId>0</cnbcId>
<code>0</code>
<comments>Composite</comments>
<countryCode>US</countryCode>
<curmktstatus>POST_MKT</curmktstatus>
<currencyCode>USD</currencyCode>
<exchange>NASDAQ</exchange>
<extendedMktQuote>
<afthrs_last_time>2024-05-24T01:00:00.000-0400</afthrs_last_time>
<change>4.31</change>
<change_pct>0.4048</change_pct>
<exchange>NASDAQ</exchange>
<fullchange>4.3099</fullchange>
<fullchange_pct>0.4048</fullchange_pct>
<last>1069.00</last>
<last_time>2024-05-24</last_time>
<last_time_msec>1716796262674</last_time_msec>
<source>Last NYSE Arca, VOL From CTA</source>
<timeZone>EDT</timeZone>
<type>POST_MKT</type>
<volume>1529669</volume>
</extendedMktQuote>
<feedSymbol>NVDA.O</feedSymbol>
<fullVolume>42949367</fullVolume>
<fundamentalData>
<beta>1.6785</beta>
<DEBTEQTYQ>19.759066</DEBTEQTYQ>
<dividend>0.40</dividend>
<dividendyield>0.0004</dividendyield>
<eps>17.0868</eps>
<feps>24.448</feps>
<fpe>43.549164</fpe>
<fpsales>22.263304</fpsales>
<fsales>117635760128</fsales>
<GROSMGNTTM>75.43435</GROSMGNTTM>
<MPreviousClose>1037.98999</MPreviousClose>
<mktcap>2618960642048</mktcap>
<mktcapView>2618960.5M</mktcapView>
<NETPROFTTM>53.3971</NETPROFTTM>
<PDYTDPCHG>109.60179</PDYTDPCHG>
<pcttendayvol>1.0593</pcttendayvol>
<pe>62.31067</pe>
<psales>32.829754</psales>
<ROETTM>115.6553</ROETTM>
<revenuettm>79773999104</revenuettm>
<revenuettmView>79774.0M</revenuettmView>
<sharesout>2459834112</sharesout>
<sharesoutView>2459.83M</sharesoutView>
<TTMEBITD>49853.0</TTMEBITD>
<tendayavgvol>40543152.00</tendayavgvol>
<yragoprice>305.38</yragoprice>
<yragopricechange>759.30994</yragopricechange>
<yragopricechangepct>248.64429</yragopricechangepct>
<yrhidate>2024-05-24</yrhidate>
<yrhiprice>1064.75</yrhiprice>
<yrlodate>2023-05-24</yrlodate>
<yrloprice>298.06</yrloprice>
</fundamentalData>
<high>1064.75</high>
<issue_id>218647</issue_id>
<issuer_id>27095</issuer_id>
<last>1064.69</last>
<last_time>2024-05-24</last_time>
<last_time_msec>1716796262674</last_time_msec>
<low>1030.00</low>
<mainmktstatus>CLOSE</mainmktstatus>
<name>NVIDIA Corp</name>
<onAirName>NVIDIA</onAirName>
<open>1044.49</open>
<prev_prev_closing>1037.99</prev_prev_closing>
<previous_day_closing>1037.99</previous_day_closing>
<provider>CNBC QUOTE CACHE</provider>
<providerSymbol>NVDA</providerSymbol>
<quoteDesc/>
<realTime>true</realTime>
<reg_last_time>2024-05-24T16:00:00.000-0400</reg_last_time>
<reg_market_close>16:00:00</reg_market_close>
<reg_market_open>09:30:00</reg_market_open>
<responseTime>Mon May 27 03:51:02 EDT 2024</responseTime>
<shortName>NVDA</shortName>
<source>Last NYSE Arca, VOL From CTA</source>
<streamable>1</streamable>
<symbol>NVDA</symbol>
<symbolType>symbol</symbolType>
<timeZone>EDT</timeZone>
<todays_closing>1037.99</todays_closing>
<trading_day_type>HOLIDAY</trading_day_type>
<volume>39728626</volume>
</quickQuote>

You already know how to get the last <last> field value, via the the value of the RegExp option used for the StringIndex2=1 option:

Code: Select all

(?siU)^.*?<last>(.*)<\/last>

You already know the (?|...|...) form of a branch reset pattern and the fact that the last <last> pattern should be the last in the branch reset:

Code: Select all

(?siU)^(?|...|.*?<last>(.*)<\/last>)

All you need to do is come up with the pattern that captures the <last> value that's enclosed between <extendedMktQuote> tags and fill the ... part above.

Oh, and you already have the testing sites where you can copy paste and see if what you write works. See where I'm going with this and why it's so easy that you can also do it in a heartbeat?

P.S. I moved the (?siU)^ out of the branch reset since this is just the collection of space ignoring (?s), case insensitive (?i) and ungreedy (?U) flags, followed by the ^ symbol aka the start of the string - and these are the same for both branches. Let me know if you don't understand something from the above.

[emp00]

All you need to do is come up with the pattern that captures the <last> value that's enclosed between <extendedMktQuote> tags and fill the ... part above.

Thanks for trying to guide me ... Really appreciated. I have pasted the xml snip for one stock having extendedMktQuote values, complete xml enclosed between <quickQuote> ... </quickQuote>, to https://regex101.com/r/dU5lQ3/2 (I already did that earlier today).

I "understand" the principal functionality of the branch reset groups but only based on examples which do not exactly fit to my problem. See my "best shot" below, my thinking is that I need .*? twice to ensure these parts can be arbitrary characters? Unfortunately, this RexEx still only gives me the bottom <last> price, and not the one in the <extendedMktQuote> ... </extendedMktQuote> section.

I understand, the branch groups between "|" are prioritized in their order, so obviously there still a bug in my expression. For me this is frustrating, I always try to copy as good as I can from examples, but even with such a "simple task" this fails and all I can do is "play" around with the expression counting on my luck. Just one tiny mistake, the expression fails and I'm left in the dark. That's how it feels. Ok, there's tons of documentation - but it's so dense and cryptic = above my head.

Code: Select all

(?siU)^(?|.*?<extendedMktQuote>.*?<last>(.*)<\/last><\/extendedMktQuote>|.*?<last>(.*)<\/last>)

[Yincognito]

Thanks for trying to guide me ... Really appreciated. I have pasted the xml snip for one stock having extendedMktQuote values, complete xml enclosed between <quickQuote> ... </quickQuote>, to https://regex101.com/r/dU5lQ3/2 (I already did that earlier today).

I "understand" the principal functionality of the branch reset groups but only based on examples which do not exactly fit to my problem. See my "best shot" below, my thinking is that I need .*? twice to ensure these parts can be arbitrary characters? Unfortunately, this RexEx still only gives me the bottom <last> price, and not the one in the <extendedMktQuote> ... </extendedMktQuote> section.

I understand, the branch groups between "|" are prioritized in their order, so obviously there still a bug in my expression. For me this is frustrating, I always try to copy as good as I can from examples, but even with such a "simple task" this fails and all I can do is "play" around with the expression counting on my luck. Just one tiny mistake, the expression fails and I'm left in the dark. That's how it feels. Ok, there's tons of documentation - but it's so dense and cryptic = above my head.

Code: Select all

(?siU)^(?|.*?<extendedMktQuote>.*?<last>(.*)<\/last><\/extendedMktQuote>|.*?<last>(.*)<\/last>)

You're very close - I used (no need to abuse the .*? ungreedy toggling in this case):

Code: Select all

(?siU)^(?|.*<extendedMktQuote>.*<last>(.*)<\/last>.*<\/extendedMktQuote>|.*?<last>(.*)<\/last>)

for the test string and it returned the correct stuff:

Last.jpg

Didn't test it in the skin, but then, I don't see why it shouldn't work there as well.

P.S. By the way, the docs are nothing dense and cryptic if you click on the RegExReference button on the left sidebar in the regexr.com page (obviously, after choosing the PCRE flavor from the button at top right) - it's only a short sentence description and a single example for most of the patterns. It's one of the reasons I prefer that site to all the others, it's very clean and straightforward.

[emp00]

You're very close - I used (no need to abuse the .*? ungreedy toggling in this case):

Code: Select all

(?siU)^(?|.*<extendedMktQuote>.*<last>(.*)<\/last>.*<\/extendedMktQuote>|.*?<last>(.*)<\/last>)

Thank you!!! I agree that https://regexr.com has somewhat better explaining capabilities (thanks for the hint switching to PCRE, how should I have known) but on the other hand I like https://regex101.com because it highlights the match in green color, which shows me directly what the expression result will be. Somehow regexr.com only showed "1 match" and in the popup "no match found in 3613 characters (?). Yes, I used exactly the same xml text and search expression...

What I still don't understand: Why does .*? work fine for the second "branch reset group" and why not in the first?

Using .* indeed makes the whole thing work... The additional ? means "Lazy. Makes the preceding quantifier greedy, causing it to match as many characters as possible." --> as far as I understood, "greedy mode" is default - so by removing, shouldn't it do the same? Seems like black magic to me - but I'm happy, it works!

However, I needed to do "one more thing": Found out, that CNBC delivers the extended market structure even if the regular market is already up again (tagged then as "previous extended market"). Of course, when this is the case I don't want to grab the extended market price... Found out, that <curmktstatus> is either REG_MKT or POST_MKT indicating current market status. By "simply" adding the additional criterion .*<curmktstatus>POST_MKT into the 1st branch reset group, I think I got this covered!

Code: Select all

RegExp=(?siU)^(?|.*<curmktstatus>POST_MKT.*<extendedMktQuote>.*<change>(.*)<\/change>.*<\/extendedMktQuote>|.*?<change>(.*)<\/change>)

I have used exactly the same expression for the <change> and <change_pct> RegEx measures, thereby hoping to also capture the price/price-change-data belonging together, all with prio1 on extended market (if market status = POST_MKT) and if not, using Mordasius expression capturing the regular market data. All thanks to your great hint using the "branch reset group" expression: This does not require any additional calc or whatever-measures in the skin! Error handling is included in this tricky expression. Nice^2! RegEx is really super-powerful...

So far, so good - will observe in the coming days, checking whether the new expression is robust under "all" conditions and will notify you guys if this is confirmed (maybe Mordasius will then include this in his official release? Hint: I also plan to using these new expressions in the My Portfolio skin...).

[Yincognito]

What I still don't understand: Why does .*? work fine for the second "branch reset group" and why not in the first?

Using .* indeed makes the whole thing work... The additional ? means "Lazy. Makes the preceding quantifier greedy, causing it to match as many characters as possible." --> as far as I understood, "greedy mode" is default - so by removing, shouldn't it do the same? Seems like black magic to me - but I'm happy, it works!

Correct. On a second look though, I think the problem was another. Your first branch was:

Code: Select all

.*?<extendedMktQuote>.*?<last>(.*)<\/last><\/extendedMktQuote>

Notice how there isn't any .* between the <\/last> and the <\/extendedMktQuote> patterns. As far as I remember and the earlier posts show, there was other stuff there in the input string, so this branch would have never matched since <\/extendedMktQuote> didn't directly follow <\/last> anyway. You can probably test and see which of the two possibilities was the actual culprit on regex101.com, and suddenly the "black magic" will become "duh, it's so obvious now". Remember, apart from the unorthodox syntax, regex is as logical as math, there is no hidden science behind it, especially if you correctly identify what happens when parsing the string character by character - just like a human would do.

P.S. I don't pay much attention to the pop-ups on regexr.com since they depend on where exactly you hover, but there is indeed only 1 match. A match is not the same as a capture, by the way, it's the whole pattern. The site also shows the result directly, as long as you specify whatever \1, \2 and such you want displayed in the box towards the bottom if the page (i.e. under "Tools"). For me, the layout on regex101.com is a bit too cluttered in key areas.

[emp00]

Notice how there isn't any .* between the <\/last> and the <\/extendedMktQuote> patterns.

Confirmed, you are right! My bad, this is exactly why the *.? pattern most probably did not work.

I just harmonized my expression, now using .*? for all string sections with arbitrary characters, just like in Mordasius original code. I still don't really understand if .* vs. .*? makes any difference at all, but since "the proven original" used this, it's probably not a wrong choice. I just checked with a bunch of saved xml's from CNBC and found everything working with the below harmonized regex, so far. Will keep this skin running in the coming days to gather full confidence.

Code: Select all

RegExp=(?siU)^(?|.*?<curmktstatus>POST_MKT.*?<extendedMktQuote>.*?<last>(.*)<\/last>.*?<\/extendedMktQuote>|.*?<last>(.*)<\/last>)

Of course, regex is the opposite of "black magic" (was just kidding) - I know it's perfect "string-math" and I am liking it more and more. It offers so many possibilities- The hard stuff is that the impatient user has to learn all of this - there's no engine creating a regex by describing the problem. Yikes, ChatGPT ?! I should have given that a try, even though you'd probably hate it. For me, in such a case, this would have been probably a good approach! Never tested regex with ChatGPT yet, but I'm 90% sure, it would be capable to assist very well. Depending on the user's prompting skills, of course. But I'll keep ChatGPT in mind for my "next" regex problem, so that I don't need to bother you... Nevertheless, THANKS for your time + patience and especially the excellent idea!!

[Yincognito]

I just harmonized my expression, now using .*? for all string sections with arbitrary characters, just like in Mordasius original code. I still don't really understand if .* vs. .*? makes any difference at all, but since "the proven original" used this, it's probably not a wrong choice.

Well, "the proven original" was simply intended to get the last <last> value from the <quickQuote> string, that's all. There was no such need in the alternative branch since you probably have a single <last> between the <extendedMktQuote> tags anyway, hence why it's not absolutely required to use it. It's not technically wrong to do so (at least for now, can't guarantee for the future), it's just about the pattern suiting the needs. Generally speaking, in most cases, you'd want an ungreedy (?siU).* behavior to guarantee that you get the first match coming after .*, and the .*? inversion was an exception dictated by necessity, since there were one or two <last> instances in the <quickQuote> and the last of those instances contained the desired value in the case linked above.

[emp00]

Generally speaking, in most cases, you'd want an ungreedy (?siU).* behavior to guarantee that you get the first match coming after .*, and the .*? inversion was an exception dictated by necessity, since there were one or two <last> instances in the <quickQuote> and the last of those instances contained the desired value in the case linked above.

Aah?! Now I get it?! ==> .*? is an "inversion" - matching the "last occurrence" in case more than 1 "<last>" tags are found while .* matches the first occurrence? Damn - I really tried to find such an explanation but e.g. regexr.com gives this explanation:

. Dot. Matches any character except line breaks.
* Quantifier. Match 0 or more of the preceding token.
? Lazy. Makes the preceding quantifier lazy, causing it to match as few characters as possible.

Maybe it's because I "ain't no native speaker" --> lazy means "inversion"? Maybe this is becoming too much off-topic... Sorry!!!

[Yincognito]

Damn - I really tried to find such an explanation but e.g. regexr.com gives this explanation:

. Dot. Matches any character except line breaks.
* Quantifier. Match 0 or more of the preceding token.
? Lazy. Makes the preceding quantifier lazy, causing it to match as few characters as possible.

The same regexr.com also says on the last line:

This behaviour is reversed by the ungreedy (U) flag/modifier.

which is the case here, see the beginning of the pattern.

In other words, the (?U) ungreedy flag at the start (which already makes the pattern lazy and match as few characters as possible) makes the ? mean the opposite to the "standard" behavior (so .*? will be greedy and match as many characters as possible in this case). This is supported by the entry for Flags > U, which says:

Makes quantifiers ungreedy (lazy) by default. Quantifiers followed by ? will become greedy.

Technically, the .*? locally inverted the (standard greedy behavior) overall inversion by (?U) at the start, if it makes sense.

Default greedy overall (i.e. for the whole pattern). Inverted to lazy overall by (?U). Then inverted back to greedy locally by the ? (just for the .* part).

[emp00]

Technically, the .*? locally inverted the (standard greedy behavior) overall inversion by (?U) at the start, if it makes sense.

It sure makes sense, but - honestly - I still don't get it completely. It's like looking around two corners and with mirrors facing each other, WTF?! I'm a practical thinking person, too much theory for my small and aging head... Nevertheless, I never give up and this offers levers to adjust the expressions to make them do what I need.

Turns out, today I noticed another complexity in the CNBC data: For <curmktstatus> they actually have 3 different variants: REG_MKT, PRE_MKT and POST_MKT! If one of the latter two is "set" we need to grab the <extendedMktQuote> price-data. I actually googled a regex solution called "negative lookahead" --> now using (?!REG_MKT) to find both alternatives. However, as it turned out due to the positioning of the <extendedMktQuote> block BETWEEN the <last>, <change> and <change_per> tags, I needed to modifiy the greedy/lazy switch only for <change> and <change_per> ... By the way, initially I started with (REG_MKT|PRE_MKT) as alternative words expression, but due to the following .* this expression always failed because it was used as "match" and maybe because it is already part of "branch reset group"? WTF^2.

Long-story-short - the below code seems to cover all variants that I'm aware of, as of today. We need 3 (actually 2) different expressions for the 3 relevant tags - this will always give prio1 to <extendedMktQuote> if REG_MKT is not active and it delivers the correct corresponding regular price data [f REG_MKT is running!

Code: Select all

; Price e.g. [mIndex1_Price]
RegExp=(?siU)^(?|.*<curmktstatus>(?!REG_MKT).*<extendedMktQuote>.*<last>(.*)<\/last>.*?|.*?<last>(.*)<\/last>)
; Price change e.g. [mIndex1_UpDown]
RegExp=(?siU)^(?|.*<curmktstatus>(?!REG_MKT).*<extendedMktQuote>.*<change>(.*)<\/change>.*?|.*<change>(.*)<\/change>)
; Price change% e.g. [mIndex1_ChangePer]
RegExp=(?siU)^(?|.*<curmktstatus>(?!REG_MKT).*<extendedMktQuote>.*<change_pct>(.*)<\/change_pct>.*?|.*<change_pct>(.*)<\/change_pct>)

Note the subtle .*? vs. .* differences, due to the "BETWEEN positioning" issue. I also played around removing the global ungreedy U because I seem to "dislike" this mode. Just not my way of thinking but by adjusting the ? reversions, I somehow achieved my desired effect anyway. I'm 100% sure this is not ideal!

In the end, I really invested several hours of testing with sweat, tears & black coffee. Thanks to regex101.com (I like it better, just because it directly displays the relevant match) I permutated the above 3 expressions on the 3 xml variants for several stocks and a few others that I saved... Hard, hard work. Yeah, I wanted to use ChatGPT but since I always thought "I'm so close, I can do it" -> never asked "the machine": all human-made. Learned a lot, but still don't understand more than 5% of regex basics. But who cares if "it works" - still I'm very open for any improvements and corrections of potential mistakes or redundancies in my hacky mess of regex code.

I will also confirm if this turns out to be robust in the coming days. Need to get confidence, hoping not to find another CNCB-tripwire... Hasta-la-vista, over-and-out. Regex is cool, kindof - WTF^3. Amen.