Thursday, 2023-06-01

*** lxo <lxo!~lxo@gateway/tor-sasl/lxo> has quit IRC00:50
*** lxo <lxo!~lxo@gateway/tor-sasl/lxo> has joined #libre-soc00:51
*** yambo <yambo!~yambo@069-145-110-003.biz.spectrum.com> has quit IRC01:19
*** yambo <yambo!~yambo@069-145-110-003.biz.spectrum.com> has joined #libre-soc01:30
*** lxo <lxo!~lxo@gateway/tor-sasl/lxo> has quit IRC01:47
*** lxo <lxo!~lxo@gateway/tor-sasl/lxo> has joined #libre-soc01:47
*** lxo <lxo!~lxo@gateway/tor-sasl/lxo> has quit IRC07:22
*** lxo <lxo!~lxo@gateway/tor-sasl/lxo> has joined #libre-soc07:23
*** ghostmansd <ghostmansd!~ghostmans@broadband-109-173-83-100.ip.moscow.rt.ru> has joined #libre-soc07:29
*** ghostmansd <ghostmansd!~ghostmans@broadband-109-173-83-100.ip.moscow.rt.ru> has quit IRC07:37
*** ghostmansd <ghostmansd!~ghostmans@broadband-109-173-83-100.ip.moscow.rt.ru> has joined #libre-soc08:37
*** ghostmansd <ghostmansd!~ghostmans@broadband-109-173-83-100.ip.moscow.rt.ru> has quit IRC08:53
*** ghostmansd <ghostmansd!~ghostmans@broadband-109-173-83-100.ip.moscow.rt.ru> has joined #libre-soc09:19
*** ghostmansd <ghostmansd!~ghostmans@broadband-109-173-83-100.ip.moscow.rt.ru> has quit IRC09:51
*** ghostmansd <ghostmansd!~ghostmans@broadband-109-173-83-100.ip.moscow.rt.ru> has joined #libre-soc10:12
*** ghostmansd <ghostmansd!~ghostmans@broadband-109-173-83-100.ip.moscow.rt.ru> has quit IRC11:01
*** ghostmansd <ghostmansd!~ghostmans@broadband-109-173-83-100.ip.moscow.rt.ru> has joined #libre-soc11:12
*** octavius <octavius!~octavius@92.40.168.36.threembb.co.uk> has joined #libre-soc11:17
*** ghostmansd[m] <ghostmansd[m]!~ghostmans@broadband-109-173-83-100.ip.moscow.rt.ru> has quit IRC11:19
*** ghostmansd <ghostmansd!~ghostmans@broadband-109-173-83-100.ip.moscow.rt.ru> has quit IRC11:19
*** ghostmansd[m] <ghostmansd[m]!~ghostmans@176.59.174.94> has joined #libre-soc11:20
lkclghostmansd[m], arse i just messed up the branch-checkouts.12:14
lkcli might be able to fix it12:15
lkclok sorted12:16
lkcland also rebased the new branch, extra322_ldst_idx12:18
lkclhttps://git.libre-soc.org/?p=openpower-isa.git;a=shortlog;h=refs/heads/extra322_ldst_idx12:18
*** octavius_ <octavius_!~octavius@92.40.169.186.threembb.co.uk> has joined #libre-soc14:03
*** octavius <octavius!~octavius@92.40.168.36.threembb.co.uk> has quit IRC14:06
*** octavius_ <octavius_!~octavius@92.40.169.186.threembb.co.uk> has quit IRC14:22
*** octavius_ <octavius_!~octavius@92.40.169.139.threembb.co.uk> has joined #libre-soc14:23
lkclghostmansd[m], when you've a moment (ignore me for now i'm just typing)15:00
lkcla couple of thoughts occurred to me15:00
lkcl1. for visitor-walking, the priority at the moment is disassembly (decoding).  redoing PowerDecoder for example15:02
lkcl2. i have an idea - don't try *separate* visitor-walking, actually have a function-pattern which if we do it really carefully can be part of the base-class of the entire insnsdb set!15:03
lkcland if we call the "start-header" function "__entry__" and the "end-header" function "__exit__", that *is* the very definition of a context-manager!15:04
lkcland so, if every class gains a pattern-of-functions (start-children-end) *and says what its children are* then it's a total no-brainer15:07
lkclfirst page google search "contextmanager visitor pattern" shows this https://pyomo.readthedocs.io/en/stable/_modules/pyomo/core/expr/visitor.html15:07
lkclclass StreamBasedExpressionVisitor(object)15:08
lkclahhhhh15:08
lkclthat's it15:09
lkclprogrammerjake, ah ha!  notice the child_idx in "data = acceptChildResult(self, node, data, child_result, child_idx)" ?15:10
lkclthat's what you were talking about - i didn't quite understand it at the time15:10
lkcli think we can literally use StreamBasedExpressionVisitor *as it is*15:11
lkcland "conform" with it.15:11
*** ghostmansd[m] <ghostmansd[m]!~ghostmans@176.59.174.94> has quit IRC15:42
*** ghostmansd[m] <ghostmansd[m]!~ghostmans@broadband-109-173-83-100.ip.moscow.rt.ru> has joined #libre-soc15:44
*** ghostmansd <ghostmansd!~ghostmans@broadband-109-173-83-100.ip.moscow.rt.ru> has joined #libre-soc15:45
*** ghostmansd[hexch <ghostmansd[hexch!~ghostmans@broadband-109-173-83-100.ip.moscow.rt.ru> has joined #libre-soc15:52
*** ghostmansd <ghostmansd!~ghostmans@broadband-109-173-83-100.ip.moscow.rt.ru> has quit IRC15:54
*** ghostmansd[hexch <ghostmansd[hexch!~ghostmans@broadband-109-173-83-100.ip.moscow.rt.ru> has quit IRC16:11
*** ghostmansd[hexch <ghostmansd[hexch!~ghostmans@broadband-109-173-83-100.ip.moscow.rt.ru> has joined #libre-soc16:15
*** ghostmansd[hexch <ghostmansd[hexch!~ghostmans@broadband-109-173-83-100.ip.moscow.rt.ru> has quit IRC16:25
*** ghostmansd[hexch <ghostmansd[hexch!~ghostmans@broadband-109-173-83-100.ip.moscow.rt.ru> has joined #libre-soc16:31
*** ghostmansd[hexch <ghostmansd[hexch!~ghostmans@broadband-109-173-83-100.ip.moscow.rt.ru> has quit IRC16:36
*** ghostmansd[m] <ghostmansd[m]!~ghostmans@broadband-109-173-83-100.ip.moscow.rt.ru> has quit IRC16:52
*** ghostmansd[m] <ghostmansd[m]!~ghostmans@176.59.42.252> has joined #libre-soc16:53
*** ghostmansd[m] <ghostmansd[m]!~ghostmans@176.59.42.252> has quit IRC17:02
*** ghostmansd[m] <ghostmansd[m]!~ghostmans@46.39.229.42> has joined #libre-soc17:03
*** ghostmansd[m] <ghostmansd[m]!~ghostmans@46.39.229.42> has quit IRC17:08
*** ghostmansd[m] <ghostmansd[m]!~ghostmans@176.59.42.252> has joined #libre-soc17:10
*** ghostmansd[m] <ghostmansd[m]!~ghostmans@176.59.42.252> has quit IRC17:13
*** ghostmansd[m] <ghostmansd[m]!~ghostmans@46.39.229.55> has joined #libre-soc17:13
lkclurrr the visitor-walker i added in nmutil is coming back to me18:03
*** ghostmansd[m] <ghostmansd[m]!~ghostmans@46.39.229.55> has quit IRC18:21
*** ghostmansd[m] <ghostmansd[m]!~ghostmans@176.59.55.126> has joined #libre-soc18:22
*** ghostmansd[m] <ghostmansd[m]!~ghostmans@176.59.55.126> has quit IRC18:37
*** ghostmansd[m] <ghostmansd[m]!~ghostmans@broadband-109-173-83-100.ip.moscow.rt.ru> has joined #libre-soc18:37
lkclprogrammerjake, apologies my laptop is getting so overloaded (swap-thrashing) that i had to disable utf-8 and fcvt unit tests.18:55
lkclat 56 GB resident RAM when my laptop is 64 GB it put the loadavg up to 100 and made it unresponsive.18:56
lkcli can't have that situation occur18:56
lkclcan you sort that out by drastically reducing the number of tests?18:56
lkclaside from anything it will kill the CI machine by putting it permanently into thrashing18:57
*** ghostmansd <ghostmansd!~ghostmans@broadband-109-173-83-100.ip.moscow.rt.ru> has joined #libre-soc19:35
markos_does it need to run everything in parallel?20:13
lkclonly for "speed"20:33
lkclit takes about 12 minutes @ 8 processes20:33
ghostmansdA huge comment on visitors: https://bugs.libre-soc.org/show_bug.cgi?id=1094#c520:34
lkclon my 4.8 ghz laptop20:34
lkclghostmansd, awesome20:34
ghostmansdTake your time, this is huge indeed.20:34
ghostmansdIf you don't have time to dive into it as whole, please answer in chunks, that's perfectly normal. I tried to address several parts so that this can be convenient for all of you guys.20:35
lkclwill do quick replies i have to take my daughter out shopping20:35
ghostmansdThese comments are not call for action, but my thoughts. Let's discuss.20:36
lkclgive me 20 mins?20:36
ghostmansdNo rush, this is a serious topic.20:36
ghostmansdTake as many minutes you need. :-)20:36
ghostmansdEven hours... but not years. :-)20:36
lkclyep everything you said makes sense20:37
lkclbbl20:37
lkclbtw do look at nmutil visitor20:39
lkclhttps://git.libre-soc.org/?p=nmutil.git;a=blob;f=src/nmutil/nmoperator.py;hb=HEAD20:40
lkclthe important function is "flatten"20:41
lkclit's very specific/explicit, the type-calling "if isinstance(ai, Record)"20:41
lkclbad hack20:42
programmerjakelkcl, we still need those tests, but i was thinking we could skip utf8 and fcvt tests completely unless `ENABLE_SLOW_TESTS` env var is set21:33
programmerjakealso, if you want to not run out of ram, just set SILENCELOG='!*,default'21:35
programmerjakesince the memory is being used by the *literally dozens of gigabytes* of log messages21:36
programmerjakeimho we need to remove/comment-out log calls in selectableint and other places that are called a bunch but already known to work21:37
programmerjakethe tests themselves likely use much less than 1GB ram per parallel testing process21:40
programmerjakeas an example, my build server is currently running ci on 8 cores and is using 3.47GB ram on the whole system21:45
programmerjakebecause ci has SILENCELOG='!*,default' set21:46
programmerjakeok, it's at 97% complete running pytest and is now up to 3.64GB21:49
programmerjakeanother idea: have SILENCELOG disable `default` log messages by default, you need to pass SILENCELOG='!*' to show them all21:52
programmerjakeor SILENCELOG=021:58
*** octavius_ <octavius_!~octavius@92.40.169.139.threembb.co.uk> has quit IRC22:03
lkclprogrammerjake, no, i need to be able to run them23:12
lkclthat's inconvenient as i run them frequently and examine them23:14
lkclbit-for-bit and look for particular keywords to trace the behaviour of ISACaller.23:14
lkcl> because ci has SILENCELOG='!*,default' set23:14
lkclgood, that's perfect *for ci*23:14
lkcleven if classified as "slow" i *have* to have them running, and running locally23:17
lkcland if "slow" and they cause *my* machine to run out of *64* GB of RAM that's not going to work, is it?23:18
lkclas the person who has to keep an eye on everything23:18
lkcl(and whose much smaller laptop only has 16 GB of RAM)23:18
lkclwe *cannot* make the assumption that everyone (including me) has massive-resourced-machines at their disposal23:19
programmerjakewell, then run all tests with SILENCELOG and if some fail, rerun *just those failing tests* without SILENCELOG so you can analyze their output23:27
programmerjakeyou can easily do that with pytest --lf which runs only the tests that failed last time23:30
programmerjakethe way i see it, this problem is ultimately caused by our having too many log calls in places where we no longer need them, such as selectableint's methods23:32
lkcli do it the other way round23:36
lkcli *need* those tests to be able to run, ok?23:36
lkcland i need them to run with full log files, *please listen*23:37
programmerjakeoh, i have an idea, maybe we can make a simple pytest plugin that stores logs on disk instead of in ram...23:37
lkclah that would work great23:37
programmerjakeor maybe just fix the pytest bug for that that's been open for a while iirc, lemme find it...23:39
lkclahhh https://github.com/pytest-dev/pytest/issues/2968#issuecomment-34820410023:51
lkcltearDown needs to explicitly close objects23:51
programmerjakei think i'll just write a plugin that captures output to a file for each test and gives the filename and first and last 5k lines back to pytest, if you want the rest you'll just have to look in the relevant file23:51
programmerjakethe bug is that pytest passes the whole captured output in memory to the main process even for passed tests where it'll just ignore the output, so large outputs cause the main process to run out of memory23:54
programmerjakeiirc23:54
lkclyoww23:55
programmerjake(not the bug you linked which is already fixed)23:55
lkclthat sounds like a good idea23:55
programmerjakethe store in files idea?23:56
programmerjakewith just giving back first/last few lines?23:57
lkclyes love it23:58
programmerjakeok, i'll work on that today23:59
programmerjakemaybe can get some project management budget and publish it on pip23:59

Generated by irclog2html.py 2.17.1 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!