Tuesday, 2021-12-07

mikolajwprogress so far: finished Robert Baruch's tutorial, watched Luke's video about Libre-SoC, completed the "First steps" tutorial, superficially examined the Nmigen simulator code01:22
mikolajwnow I'm going to try to write some small Nmigen simulations01:23
mikolajwand I ordered an Icebreaker FPGA board to be able to run my code01:25
mikolajw(I don't know if I can run Libre-SoC on it, probably not, but will still be useful)01:27
mikolajwmade a blinker in Nmigen. It works, I can see the waveforms with GTKWave03:59
mikolajwNow I'm going to see the Python code Nmigen generated to simulate it04:00
programmerjake:)04:02
mikolajwok, so it turns out all you need to do to get the code generated for the simulator is to set `NMIGEN_pysim_dump` environment variable, but it's only available in Nmigen development branch at the moment04:35
mikolajwNow I'm going to try to monkey-patch Nmigen to generate C code instead by reimplementing _pyrtl.py contents, as requested04:36
mikolajwand yeah, getting the C unsigned integer types to properly represent signals that aren't necessarily byte-aligned or less than 64 bits in size is going to be the hard part. I remember this was not fun when connecting Verilator-generated code with SDL204:44
mikolajwless than 65 bits in size*05:12
programmerjakeimho the hard part is the >64 bit signals and/or the signed signals...signed overflow is UB cuz C is annoying05:14
programmerjake<= 64-bit unsigned signals can just be done with unsigned long long (or uint64_t -- probably better) and masking off the top bits05:15
programmerjakeyou can use cxxrtl for a reference for implementing the required bigint types:05:20
programmerjakehttps://github.com/YosysHQ/yosys/blob/d186ea7a2d74f859972746e03996e9eddc6a5157/backends/cxxrtl/cxxrtl.h#L11005:20
programmerjakeassuming we're targeting gcc/clang, you can use the builtin add/sub/mul overflow instructions to implement bigint arithmetic more efficiently (cuz it will use the cpu's carry flag):05:33
programmerjakehttps://gcc.gnu.org/onlinedocs/gcc/Integer-Overflow-Builtins.html05:33
programmerjakefunctions, not instructions. oops...05:34
programmerjakethey also fix the signed overflow is UB issue05:34
mikolajwI wonder if using https://gcc.gnu.org/onlinedocs/gcc/_005f_005fint128.html for signals between 64 and 128 bits could improve performance05:48
mikolajwif we're into microoptimization05:49
mikolajwor even worse: https://clang.llvm.org/docs/LanguageExtensions.html#extended-integer-types05:54
mikolajwI don't know what are your portability requirements05:56
programmerjakeiirc we're targeting the linux kernel, so __int128 is out cuz it's only supported on 64-bit cpus...we'd likely want to run 32-bit powerpc at some point05:59
programmerjakeclang extended int types are out cuz gcc doesn't support them06:00
programmerjakeit'd be nice to target all gcc/clang supported targets, but that isn't a requirement06:01
programmerjakelinux kernel is an intended target, but it'd be nice to work outside the kernel too06:03
programmerjakethough double-check with lkcl cuz he may have something else in mind06:04
programmerjakeif you run into issues, i may be able to help cuz I've implemented bigint libraries before, as well as arbitrary-constant-bitwidth integers06:06
programmerjakethough that was in Rust and C++06:06
mikolajwok, sourcing the simulator engine from the current directory works, all I had to do is copy `pysim.py` and `_pyrtl.py` and pass `engine=` argument in `Simulator` constructor06:37
mikolajwNow analyzing `_FragmentCompiler` class06:45
mikolajwI'm going to just patch _pyrtl.py to output C instead of Python -- not going to rewrite it from scratch, as this will probably introduce less errors07:00
programmerjakesounds good!07:00
ghostmansd-pcHi folks. I'm trying to push the upstream changes from gdb into our local repository (we're 8 months behind). However, upon $(git push origin master) I get "FATAL: W any binutils-gdb dmitry3mdeb DENIED by fallthru".07:31
ghostmansd-pcI cloned the repo via $(git clone gitolite3@git.libre-soc.org:binutils-gdb.git), and $(git remote -v) shows exactly this (plus upstream branch I added to vanilla binutils-gdb).07:32
ghostmansd-pcBasically all I did is:07:33
ghostmansd-pcgit remote add upstream git://sourceware.org/git/binutils-gdb.git07:33
ghostmansd-pcgit fetch upstream07:33
ghostmansd-pcgit merge ustream/master07:33
ghostmansd-pcgit push origin master07:33
ghostmansd-pcThe last one breaks, with "FATAL: W any binutils-gdb dmitry3mdeb DENIED by fallthru".07:33
programmerjakeit means you don't have write permission to that git repo, lkcl will have to fix that once he wakes up07:41
ghostmansd-pcOK, thanks! I suspected that, but still had to check it. :-)07:59
octaviusmikolajw just be aware, as good as Robert Baruch's tutorial files are, they are a little out of date, so there may be things used in our nmigen files that need further clarification. I'm not an expert by a long shot (only started a few months ago), but if you have questions I'll try to answer them XD08:46
lkclmikolajw, good progress on the nmigen tutorials.  if you're used to verilog/vhdl/gate-level, always always use yosys "show top" on the .il file11:36
lkclread_ilang {insert filename}11:36
lkclshow <press tab twice to get full list of modules>11:36
lkclahh please don't monkey-patch nmigen.  please follow the instructions i gave here https://bugs.libre-soc.org/show_bug.cgi?id=665#c1211:37
lkclprogrammerjake, greater than 64-bit is specifically off the table for the first version11:37
lkclthere are no types greater than 64 bit in PowerDecoder2 which is the primary (exclusive) target of focus11:38
lkcladding more than 64-bit - which is unnecessary and will take much longer than the budget allocated for the task - is unnecessary11:39
lkclplease do not give other developers just starting out the confusing impression that the tasks they are being given are much larger than necessary11:39
lkclbecause mikolajw is just starting out i very specifically limited the scope that he does not get completely overwhelmed11:40
lkclif he follows your advice he will get completely overwhelmed with too much to do in one go11:41
lkcland it is scope-creep, doing far more than is actually needed11:41
lkclmikolajw: just so you are aware - programmerjake does not have the experience to make project management / planning decisions, and frequently will design things "far in advance of what is actually needed"11:42
lkcli have to be very strict with him about this because he keeps doing it. jeapordising our ability to complete within reasonable time-frames in the process.11:43
lkclprogrammerjake, sorry to have to mention this (again)11:43
lkclmikolajw, take a complete copy of _pyrtl.py as i described https://bugs.libre-soc.org/show_bug.cgi?id=665#c1211:43
lkclpick a location in the openpower-isa repo (a new subdirectory, suitably named)11:44
lkclcall the copy of _pyrtl.py .... i dunno... _crtl.py or something11:45
lkclprogrammerjake, no it most definitely is not a goal to target all gcc/clang variants, that is massive scope-creep11:46
lkclno, bigint libraries are not required here, that is massive scope-creep11:46
lkclit would turn a 1.5-to-2-week task easily into a 3-5 week task, particularly when unit tests get involved11:48
lkclghostmansd-pc, done, added you to binutils, binutils-gdb and gcc11:55
lkclthere's supposed to be a stack of tutorials here https://libre-soc.org/3d_gpu/tutorial/11:56
lkclbut linking through to lambdaconcept it's been removed11:57
lkclarchive.org still has it11:57
lkclhttps://web.archive.org/web/20210123052724/http://blog.lambdaconcept.com/doku.php?id=nmigen:nmigen_install11:58
lkclmikolajw, you'll see the extremely bad practice of wildcard imports has, unfortunately, infected the entirety of the HDL community11:59
lkclthis comes unfortunately from m-labs migen, over 15 years ago, when the scope of migen was absolutely tiny and they weren't ever expecting to have 100+ thousand line projects12:00
lkclwildcard imports are great for tiny projects, one file, maybe a thousand lines12:00
lkclfor a massive codebase involving multiple developers, multiple libraries, and hierarchical layouts, they are an absolutey f*****g nightmare12:00
lkcllitex is one of the worst offenders here (one of the best examples of how not to organise a large project): they have deep nested hierarchical class layouts, with Factory classes (data structures that generate classes)12:02
lkclwhere wildcard imports bring in classes *of the exact same name*, from different locations deep inside the file hierarchy... *across multiple code-repositories*12:02
lkclbottom line is that hardware engineers are not properly trained as software engineers, and as a community simply haven't had the time (or enough really truly painful experiences) yet to learn from these kinds of mistakes12:04
lkclnavigating litex is almost impossible because even the normal practice of "recursive grep" for a class that you see used, you find that for example there are 50 different implementations of class "Platform"12:05
lkclwhich one is used? you absolutely cannot tell and so have to examine all of them, looking at the full contents of the class, searching for function names and properties that look reasonably / vaguely like the ones being used12:06
lkclso please, please, for god's sake, don't follow that practice when writing nmigen. there are very few circumstances where wildcard imports are useful.12:07
programmerjake> programmerjake, no it most definitely is not a goal to target all gcc/clang variants, that is massive scope-creep16:09
lkclholy cow the i-cache / mmu stuff is complex.16:11
lkclam barely keeping up with how the microwatt code works16:12
lkclbut... i now have a (first) unit test which walks through the process of loading an instruction when virtual memory is enabled16:13
programmerjakeit may be scope-creep, however it is extremely easy to do (and in fact what any decent C programmer will likely do by default)...just don't use stuff not in the latest version of gcc/clang -- aka. use portable C (as well as optionally the gcc builtin overflowing functions -- they're still the best option for wrapping signed arithmetic)16:13
programmerjakelkcl: yay!16:14
lkclagain: (1) it's easy *for you* (2) it's extra work (3) this is mikolajw's very first contribution and it's (4) far more important that he not be overloaded than "extra features be added" and (5) reminder - again - of the incremental code-development policy16:15
lkclyeah, i had to hack things to communicate to the MMU16:15
lkclafter it returns "fetch failed", the MMU has to be triggered to do a RADIX page-walk16:15
lkclwith a special notification "iside" (instruction-side)16:16
lkclthat's not been used before (and isn't supported by LoadStore1 or by the MMU FSM)16:16
lkclyeah it's a big deal.16:17
lkclfrankly i'm astounded it worked16:17
programmerjakei'd argue that writing mostly portable C is just about as much work as for you to avoid importing winreg when writing python...soo easy you could do it in your sleep...basically don't go out of your way to depend on platform-specific stuff16:19
lkclyeah that's fairly normal practice.16:21
lkclbasically the real target is the linux kernel.16:23
lkclthe entire output has to be useable inside the linux kernel.16:23
lkcltherefore, even depending on <stdio.h> etc. is completely out of the question16:23
programmerjakesoo...why are you arguing that "fairly normal practice" is out of scope?16:23
lkcli'm not.16:24
lkcli thought you were advocating "support this version of a compiler AND support that version of a compiler AND support version X on platform Y times Z"16:24
lkclwhich would require autoconf or other such system and that's a definite "No" at this extremely early stage16:25
programmerjake"support all the compilers...": well, yes (more or less), but just cuz that's how portable C works, not cuz you'd actually add anything extra to do that16:26
programmerjakeno autoconf16:26
lkclwhilst it would be nice at some point to auto-generate python c-based modules (done that before, for python-webkit) and actually have simulations actually really running16:29
lkclthat's definitely on the "nice-to-have" list so not a priority16:29
lkclunless... it becomes so damn awkward to test the auto-generated code that it turns out to be easier to do that16:30
lkclin which case, the budget gets upped...16:30
programmerjakeoh, if we're doing that, we should target https://cffi.readthedocs.io/en/latest/16:31
lkclyyeah that's pretty dreadful. as is ctypes16:32
lkclctypes i used with pyjamas-desktop on w32 to access the entirety of the MSRPC interface in IEXPLORE.DLL16:33
programmerjakefrom what i heard cffi is by far the best way to portably connect C with Python16:33
lkclportable, yes, fragile yes.16:34
lkclit's based on calling functions by creating the function call stack... *by hand*16:34
programmerjakecuz it depends on the c compiler to resolve the ABI, so is portable and fast, unlike ctypes16:34
lkclmmm ok.16:35
programmerjakecffi is what i'm planning on using for the python rewrite of pia16:36
programmerjaketo be specific, i'm referring to the cffi mode of operation that just calls the c compiler to compile against python's headers, not the mode that uses libffi16:44
tplatennow having a look at test_loadstore1_ifetch.vcd18:30
lkcltplaten: i'm amazed it works18:30
lkclthe key signal to keep an eye on is mmu.iside (instruction-side)18:30
lkcland "itlb_load"18:31
lkclsome cesar-style gtkwave/css for this would be real handy18:31
tplatenI saw that yesterday when I compared parts of libre-soc with microwatt18:38
tplatenI am asking myself how would this be integrated into core and issuer, as issuer does the instruction fetch.18:39
tplatenI agree with the cesar-style gtkwave/css18:40
ghostmansd-pcfor anyone interested, here's the progress so far for gas: https://git.libre-soc.org/?p=binutils-gdb.git;a=shortlog;h=refs/heads/svp6418:57
ghostmansd-pcfor now, we're able to parse many bits, except for stuff marked as is_bc; we're not doing anything after parsing18:58
ghostmansd-pcThat's mostly a sketch for now, we parse stuff but drop it; I'm interested to hear you opinions, if I'm going into the wrong direction, please let me know now, before it's too late. :-)18:59
tplatenI recently saw the commit18:59
octaviusghostmansd-pc, I looked at your commitdiff. I roughly understand it (looks similar to parsing code I wrote for my project a while back), and at least from a syntax point of view, I didn't see any obvious mistakes/issues19:16
octaviusA lot of macro use, but that's almost expected in C, right? :)19:17
ghostmansd-pcoctavius, thank you for taking a look! basically what we do is simple: whenever we encounter something that starts with "sv.whatever", we try to gather all stuff separated by SVP64_SEP (currently '/'); we have tables of such stuff, and mostly it boils down to converting string into something integer-like.19:18
ghostmansd-pcWell, I actually don't like how it looks, but I tried to stick to binutils-gdb style. I personally don't like it, but they use it there. :-)19:19
octaviusSo is your code going to take the SVP64 instructions, and convert them to just the OpenPOWER ISA?19:19
ghostmansd-pcSo macros to e.g. define one entry in array of structures are direct reflection of other bits of code I saw there.19:19
octaviushehe fair enough19:20
ghostmansd-pcYes, for now this. Actually we should remap these, but this is for later.19:20
ghostmansd-pcSo we take instructions, grab everything we can decode, but don't bother after that. :-)19:21
ghostmansd-pcThis, of course, is somewhat an intermediate stage. Once we're able to decode, I'll squash the subsequent patches.19:21
octaviusYeah given that time limit that makes sense. Still very cool. Also nice to see a bit of C, even if I appreciate Python a lot more XD19:22
ghostmansd-pcI guess at this stage this is simply "OK, now we can parse some bits, but don't really consider these at all"19:22
ghostmansd-pc:-D19:22
ghostmansd-pcThat's actually an awful C. I'd have written many bits differently.19:22
ghostmansd-pcWelcome to GNU coding style, lol19:23
octaviusWell, better than some of what I've seen. At least it's not trying to cram too much19:23
octaviusIn general I try to avoid macros in my stuff, just so the compiler actually works for me19:23
ghostmansd-pcI tried gas on some stuff like `sv.extsw./sm=r3/dm=r3 5, 31`, and it's able to eat this, so we're now at least somewhere. :-)19:24
octaviusBut real projects probably can't escape from them19:24
octaviusNice!19:24
ghostmansd-pcI'm actually OK with macros as long as they're not abused, e.g. I'd use macros for intrusive linked lists, but not in place of inline functions.19:25
octaviusYeah, that makes sense19:25
octaviusVery small, verbose bits of code19:25
ghostmansd[m]Yep. Actually some C++ people I saw around go to great lengths to avoid macros replacing those with templates. Well, in my experience, some templates were _much_ worse than macros. Really, I cannot recall a C macro I could not understand, but with templates, I saw quite a lot.19:29
ghostmansd[m]So I tend to treat dont-use-macros advice with a grain of salt if it comes from C++ programmer, but totally accept in from anyone else :-D19:30
octaviusIt's a shame that c++ is such a rabbit hole. Even the hello world is esoteric (to put it lightly). So much software I use is written in it, but I don't really want to even touch it. So my points on macros are valid eh? ;)19:31
jnone of the main issues (as i see it) with C++ is, that the language is so big that everyone only uses a subset of it. of course, this isn't the same subset for everyone19:33
ghostmansd[m]Lol, about hello world in C++, I recall one of my colleagues who told he was surprised as fuck when he saw they tried to do bit shifting on strings...19:35
ghostmansd[m]It took me some time to realize he was mentioning std::cout << "hello, world" << std::endl or something like this.19:36
octaviusstd::cout << "Hello World!";19:36
octaviushttps://www.programiz.com/cpp-programming/examples/print-sentence19:36
octaviusSo this is a bitshift? I thought the string was sent to STDOUT19:36
jnyeah, this (ab/re)use of the leftshift operator was a bit of a weird choice19:36
octaviuslike a unix <<19:36
jnoctavius: that's the intention19:36
jnand the effect19:36
octaviusYeah19:37
jnbut << is still primarily the leftshift operator in C++19:37
ghostmansd[m]Maybe, lol. For him it was bitshift. :-)19:37
octaviusSomehow I can separate the two in my head (in a terminal vs writing C code). In c++ I would probably also think of bitshift only19:38
jneh19:38
jnyou'd get used to the dual meaning19:38
jnat some point19:38
octaviussure19:38
octaviusbut that requires *using* c++ :P19:39
ghostmansd[m]Not the case for this project it looks like :-P19:40
octaviusI guess at least untill there's more graphics development19:41
programmerjakewell...for a lot of the graphics stuff we're planning on using Rust rather than C++, though there will still be a bit of C++ for the work on gcc and llvm19:48
octaviusAh ok19:56
octaviusWhich version of Rust will we use? Aren't there fairly frequent release? Or is this not going to be a major issue?19:57
*** tplaten <tplaten!~isengaara@55d4844b.access.ecotel.net> has left #libre-soc20:00
programmerjakerust changing versions isn't a major issue because they test against *all* published crates and public projects on github...the compiler is very reliable (as long as you just use stable features)20:06
octaviusAh ok, good.20:08
octaviusDo you have any good recommendations for how to get started with rust? Just pick a project to implement and start using it? At some point I'd like to get round to using it20:09
programmerjakei'm assuming that whatever distros decide to package our graphics driver will end up using a rust compiler released around the same time as our driver, so us using newer rust features won't be an issue till then, at which point we can limit ourselves to the features of that rust release20:11
programmerjakei'd suggest trying: https://doc.rust-lang.org/book/20:12
programmerjakei learned rust in a pretty unusual way: i read all the library and language docs for months then actually used it waay later...so if you want to learn rust, following my approach may not be the best idea20:14
programmerjakeif you have questions feel free to ask here or on https://users.rust-lang.org/ or somewhere else...20:16
programmerjakethe rust community is quite friendly (one of their strongest points)20:17
octaviusHehehe, I don't think I read full documentation for anything that I've used before (perhaps wishbone is one of the few exceptions), so I'll probably do a bit of both20:24
programmerjake:)20:25
lkclghostmansd[m], try to keep in a branch for binutils-gdb, it will make review and pull requests easier when merging upstream (upstream will like branches)20:39
lkclit's looking pretty good20:40
lkcloctavius: the official Power ISA 3.0 and 3.1 does not have SVP64.20:42
lkclif i can reword the question (to clarify, to one for which the answer is "yes") - "is this going to take SVP64 assembly instructions and turn them into binary-formatted output"? the answer to that is "yes"20:43
lkclthat's what gnu-gas does: turns assembly mnemonics (of any supported language) into executable binary object format.20:44
lkcloctavius: c++ was fantastic when i was using it 20+ years ago.  the prevalent use of standard templates is now quite insane.20:45
octaviusYeah, I didn't word it too well. From my limited reading, SVP64 instructions essentially become for-loops, or similar instructions to make vector operations from scalars20:46
octaviusby iterating over a set of registers20:46
lkclghostmansd[m], ah, saw the bugreport message, it's a branch already. whewww20:47
lkclbasically.... yes.20:47
lkclfascinatingly, this is not a novel idea.20:47
lkclPeter Hsu, the designer of the MIPS R8000, came up with the exact same concept in 1994.20:47
lkcllike: to the letter.  prefix on a MIPS scalar-instruction as a suffix, the prefix covering src and dest regs, marking each individually as either scalar or vector, predication - everything.20:48
lkclthey were even going to do it as a multi-issue out-of-order execution engine20:49
octaviusIt seems to me that most ideas are reused/recycled. That's what happened with coding theory in electronics (when it became to run complex algos from the 50s/60s on embedded hardware). I'm sure there's plenty of good research we haven't rediscovered yet20:49
lkclthe *only reason* they didn't go ahead - back in 1994-1995 - was because MIPS could not work out how to do multi-issue out-of-order20:49
octaviusYes, always some issue preventing the overall idea from being used (due to efficiency I guess)20:50
ghostmansd[m]lkcl: I had an experience with FSF, I know the rules, lol :-D21:06
lkclghostmansd[m]: :)21:06
ghostmansd[m]Well, at least in general, I for sure forgot some details over the years, bit still the main course is known21:06
ghostmansd[m]Branch for the win21:06
lkclthey have to be very strict about it (and the copyright assignment)21:06
lkclmanaging enormous projects, you just... have to be21:07
ghostmansd[m]Yeah, I went through copyright assignment once, perhaps it's still there in mail, will check21:07
ghostmansd[m]I will check bc and ldst in the next days. Then, I think, we can go to remap.21:08
ghostmansd[m]What's in branch is really a sketch. The code will likely be updated based on my understanding.21:09
ghostmansd[m]For now this is what I was able to understand and collect, I hope that my understanding will improve, so will the code.21:09
ghostmansd[m]But don't really rely on that: regardless of how deeply I understand the original code, it will still be consumed by the coding style I adopt. :-)21:10
lkclghostmansd[m], ah cool, if you've done it already it'll still be on file, but worth checking with the copyright clerk21:48
lkclprogrammerjake, octavius jn cesar rsc toshywoshy meeting 10min21:49
toshywoshyIs Alain joining the ISA WG meeting?22:06
toshywoshyI won't be in the weekly meeting as the ISA TWG meeting is also now22:08
lkclahh ok22:09
lkclwill caall him22:09
lkcltoshywoshy, left a message, i think he's asleep :)22:22

Generated by irclog2html.py 2.17.1 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!