Thursday, 2021-08-05

lkclRockchip's latest offerings are pretty damn good. even the 32-bit RK3288 pissed all over budget-end Intel Celery processors11:46
lkclsome people are now using samsung smartphones for native builds11:46
justinrestivo[m]Hello! A bunch of us over at Summer of NIx ( are hoping to package the libre-soc tooling and repos with Nix. I'm hoping you could help us understand which repo(s) on correspond to each NLNet project. There's a bunch projects listed:,,,14:26
justinrestivo[m],, We'd like to package these. (As a bit of contex:  the elevator pitch for using Nix is reproducible builds and trivializing spinning up a dev environments) Thanks for any insight yall could provide. We're new to the project and still trying to grokk how things are organized.14:26
lkclcool, that's fantastic justinrestivo[m]16:36
lkclbasically you can start here16:37
lkcland it should be pretty obvious, directly from that, what is needed16:37
lkclbecause those are the reproducible-build install scripts we currently use16:38
lkcltrying to fit per NLnet project is not really a useful exercise.  there is too much overlap and too large a dependency chain.16:48
lkclnone of the NLnet projects are independent projects.16:48
justinrestivo[m]Thank you for the info!16:53
lkcljustinrestivo[m]: no problem.16:54
lkclbtw i've just sent Michiel the web-of-trust requirements document i created 4 years ago, for package-distribution-systems16:55
justinrestivo[m]awesome. I'll ask michiel about it17:04
lkclunfortunately at the time that i wrote it (just writing out the requirements took *3 weeks*) i didn't move on to the functional specification phase17:20
lkclthat's where practical details are, such as (and i gave this as an example to michiel), "the GPG web-of-trust must itself be distributed as a package that is itself signed as a package just like any other"17:21
lkclthe reason the document was written at all was because i was fighting some bike-shedding morons (quantity several dozen) trying to say that "SSL adds security to package distribution".17:23
lkclit actually does f***-all, but worse that that, SSL (TLS) instead makes servers high-priority hacking targets.17:23
lkclwithout a web-of-trust and without distributing that same web-of-trust, you *are* guaranteed to have a vulnerable distribution. this is just a fact.17:36
lkclalso if the number of participants in the web-of-trust (the number of package maintainers) is below a (hard-to-define) threshold, then it is possible to compromise security through blackmail, bribes or other influence.17:37
lkclubuntu for example is *NOT* trustable due to Canonical exerting financial control over the package maintainers, of whom there are too few.17:38
lkcldespite the fact that ubuntu uses exactly the same *technical* package distribution system as debian17:38
lkclbtw justinrestivo[m] a static reproduceable build is useful however much more useful would be us actually having the ability to update them.17:46
Las[m]What do you mean?17:47
Las[m]Updating what exactly?17:47
lkclLas[m], creating our own packages.17:47
lkcland distributing them ourselves so that we are not critically dependent on you, nixos developers and maintainers, for the provision of the exact and required revision for a given reproduceable build.17:48
Las[m]The optimal way we're supposed to do this SoN thing, is upstream a flake.nix file (and possibly some extra utility files) to the project's repository, which you can then use to build the project17:48
Las[m]So it would be in-sync with the repository, since it is in the repository17:48
lkclexample: we had the 180nm ASIC.17:48
Las[m]We say packaging but it is much more like an alternative to Make, or rather, a wrapper over it?17:48
Las[m]So you would never be dependent on us, or anyone17:49
Las[m]It would be in your full control17:49
lkclwe needed to create some very specific versions (tags) on all repros17:49
lkclwhat's "the project" in the above?17:49
Las[m]Are those written down anywhere?17:49
lkclthey're tags in the git repositories17:49
Las[m]Well, we have a bunch of issues handed to us in NLnet17:50
Las[m]E.g. one issue for the AMDVLK/RADV port, but I'm not sure that's relevant anymore17:50
* lkcl has to go out (walk) will be back later17:50
Las[m]I think it's based on what you got funding for17:50
Las[m]But it may not make sense to partition it that way, so it might be better to change that17:50
lkclit is indeed relevant: the mesa-3d driver17:50
Las[m]Thanks for the help, I'll ask some other questions later17:51
lkclbeing developed by Vivek;a=shortlog;h=refs/heads/libresoc_dev17:51
justinrestivo[m]<lkcl> "btw justinrestivo a static..." <- Nix would build all dependencies from source (or using a cached version). It would be able to set dependency git hashes. Updating dependencies/toolchain would be updating the hash manually or bumping the hash with nix. Like Las said, it's more of a wrapper over build tools like `make`.17:55
justinrestivo[m]As far as security goes, our packages would be able to be built entirely from source on a trusted build machine then distributed out to users. There's also trustix being worked on which allows for some degree of "trust" between binary caches of builds and the end user. That's still very much in the works though18:01
justinrestivo[m]as far as "the project" goes, we can essentially build each "repo" as a flake, and thereby include dev tools required by that repo, and also provide a way to build that package reproducibly18:06
justinrestivo[m] * As far as security goes, our packages would be able to be built entirely from source on a trusted build machine then distributed out to users. There's also trustix being worked on which allows for some degree of "trust" between binary caches of builds and the end user. That's still very much in the works though. Unsure how it relates to this "web of trust" you've mentioned.18:07
lkclit sounds like it's trustix that needs that document.18:22
lkcl"all work in debian is done by developers who can be identified"18:23
lkclif it is not truly the case that "s/debian/trustix" then trustix *cannot* be trusted - period.18:23
lkclif they don't know how debian works, then the trustix team urgently, urgently needs to get up-to-speed on it, very fast.18:24
lkclall debian developers participate in physical-presense GPG key-signing parties, showing government-issued ID or other proof of identity.18:25
lkclthose signatures are then placed into a "web of trust" aka a "keyring".18:25
lkclthose exact same signatures in that web-of-trust keyring are then themselves distributed *as* a package through the debian package distribution system18:25
lkclwhich itself has GPG signatures on:18:26
lkcl* the package18:26
lkcl* the archive18:26
lkcl* the release file which contains a SHA checksum of the SHA checksums of the packages within the release18:26
lkcla full inviolate chain of trust18:27
lkclANY ONE of those links not being satisfied and the ENTIRE chain is a catastrophic failure18:27
lkclre: "built entirely from source on a trusted build machine then distributed out to users" - and are the packages signed? if the build machine is compromised, and rogue packages substituted, can that be detected?18:33
lkclin debian's package management system, because the identities of the developers (and of the FTP Master) are known (because the keyring package includes their GPG signatures that were signed by other people's GPG signatures that were signed with...)18:34
lkcla rogue package is easily detectable18:34
lkclbecause at the immediate level, the archive release summary file (a GPG signature on all SHA checksums of all packages) will not match up.18:35
justinrestivo[m]As far as I know, Nix does something similar to debian currently.18:39
justinrestivo[m]I don't know a huge amount about trustix other than its existence. I know the devs understand how debian's trust model works. I think actually does a comparison between the two, but it's been a while since I looked at it.18:39
richardwilbur[m]<lkcl> "Rockchip's latest offerings..." <- Are there laptops available built on the Rockchip processors you speak of?18:47
lkclrichardwilbur[m], search "chromebook rockchip" or "chromebook rk3399" or other processor models18:53
lkcljustinrestivo[m], that should be fully documented and explained somewhere, so that a full audit and analysis can be carried out.  where is the build and distribution process for nixos explained, online?18:53
lkcl"similar to" is, unfortunately, much as i hate to say it plainly, not good enough.18:55
lkclif *any one* of the *18* separate and distinct requirements i list in the document i sent to Michiel is not respected, it breaks the *entire* chain of trust between developer and user.18:56
lkclyou can't have 17 of the 18 requirements fulfilled and miss one out, because (go figure) it's a chain.18:56
lkcltherefore, logically, "similar to" is simply not good enough... *unless*... the nixos package distribution has fulfilled the exact same requirements that debian also fulfils...18:57
lkcl... but just in a different (technical) fashion.18:57
lkclthis will tell me (and you) immediately if nixos fulfils the requirements necessary for full secure package distribution:18:58
lkclis there a nixos GPG archive keyring package?18:58
lkcl(distributed through nixos, installable with a nixos "install" command)18:59
justinrestivo[m]I believe the context is a bit different since nix is source based (like gentoo). You can, if you want, choose to not trust anyone besides the source nix uses. As for the implementation details behind distribution, I don't know off the top of my head. Let me ask around and get back to you. I have to hop out a bit for a meeting.19:00
justinrestivo[m]Gpg works on nixos (if I'm understanding correctly). I am able to sign emails/commits/ssh with my gpg keys. Easy to install and use.19:01
lkclsource or binary, is irrelevant: it's the *distribution* that requires a chain of trust19:09
lkclright the way back to the actual developer.19:09
lkclor, "person who made the declaration that the package contains what it says it contains, and contains nothing else"19:09
lkcldebian has source build download capability, just as it has package build capability.19:10
lkclthe source of a debian package contains (another) GPG signature, which can be checked prior to trying to build the debian package yourself19:11
lkcli am not asking "is GPG installed as a nixos package" - i would expect that19:11
lkcli am asking:19:12
lkcl1) do the nixos developers engage in GPG keyring signing parties, showing Government ID before doing so?19:12
Las[m]lkcl: Currently, Nixpkgs, the main repository, is not signed in any way.19:12
lkcl2) do the nixos developers upload the GPG signatures....19:12
Las[m]The NixOS Foundation trusts GitHub unfortunately19:12
lkclthen it's not secure.19:12
Las[m]This isn't a problem with Nix, and this can be "worked around" in various ways19:13
Las[m]Notably, you can fix Nixpkgs to a version you trust, i.e. a hash19:13
Las[m]But then it will be a pain to update it19:13
lkcland, as we saw a couple years back, an archlinux developer quit, terminated his account, and a rogue agent registered the same username and the same package19:13
lkclyes, indeed it will (rolling releases get out-of-date very rapidly, i have this problem with some debian pacakges)19:14
Las[m]Trustix doesn't solve this problem because it only verifies that the binaries built by the NixOS Foundation aren't compromised19:14
Las[m]No one checks whether the source code is19:14
lkclthat's where the keyring comes into play.19:14
lkclyou trust the developer because their identity is known (by other developers)19:14
Las[m]I searched about this issue before, and the usual "but it's TLS so it's secure" argument came up19:14
lkcland you then get "social effects"19:14
Las[m]This obviously isn't true and I don't think I need to explain that to you19:14
lkcloh jesus christ.19:15
lkclno, you don't.19:15
Las[m]Nix is regardless a great tool, but the current situation regarding that is a bit problematic...19:15
lkclok, this is why i asked, because there are very few people who understand this to an adequate extent19:16
Las[m]There might be something we can do to solve the trust problem, by e.g. avoiding use of Nixpkgs altogether19:16
lkcli literally had to field a dozen morons systematically suggesting - each one of them - that SSL and HTTPS would help with GNU package distribution19:16
Las[m]I hope they trust the CAs19:16
lkclwell the issue goes beyond the CAs19:17
lkcldistribution via HTTPS makes the *server itself* a high-priority hacking target.19:17
Las[m]Since you can't verify the author, only where it came from19:17
lkclall a hacker has to do is: compromise the server, replace the packages with rogue ones, and quotes everyone trusts the downloads quotes, right?19:18
lkclbecause it's "secure", right?19:18
lkclSSL protects the *channel* (the pipe), it doesn't prove the provenance of the *data*19:18
lkclthere is - unfortunately - a mode in SSL (TLS, i keep saying "SSL" out of habit), which is used by apple and the BBC19:19
lkclwhich is that the actual "pipe" (the network socket) applies a Private Certificate to the data.19:19
lkclif you have the right public key, you can "unlock" the pipe.19:20
lkclthe BBC used that to prevent and prohibit BBC iPlayer "illegal" downloads19:20
lkcland because that public key is not issued to Libre Browser Projects, you can't use anything other than *Microsoft* or *Apple* issued web browsers to view BBC iPlayer content19:21
* lxo doesn't like how the following conversation is going:
lxowould anyone with factual data care to step in?19:21
lkcllxo: thx19:22
lkcli missed that one19:22
lkcllxo: is it on phoronix, or is it on gnusocial?19:22
Las[m]lkcl: I'll forward your worries in any case, since I am of the exact same opinion. Hopefully, for the good of everyone, the situation will be improved.19:23
lxothe conversation is in the fediverse, but it started with a link to the phoronix post19:23
lkclLas[m]: well, the purpose of this conversation has been to provide Michiel with valuable information.  they fund *trustable* projects.19:23
lkclif they're funding something where the developers are fundamentally ignoring basic security and end-user trust, that's something he needs to be aware of19:24
Las[m]We're using GitHub lol19:24
lkclLas[m]: :)19:24
lxoI haven't looked at the phoronix post, I just saw the conversation between a friend of mine asking questions and someone I don't know casting a negative light on us19:25
Las[m]Well, it's still a step forward, so I don't want to be critical of the project.19:25
lkclwell, using github "because it can distribute stuff with git" is fine19:25
lkclLas[m]: true19:25
lkcllxo: i'll probably have to write something here, if you're able to drop it into the context that would be awesome19:25
lkclbasically, the key fundamental difference between RISC-V and OpenPOWER comes down to the fact that ISA Extensions may be proposed to the Power ISA *without a mandatory membership of the respective Foundation*19:27
lkclIBM *REQUIRED* this before it would allow the Power ISA to be transferred into the hands of the OpenPOWER Foundation.19:27
lkclif you only need to *use* RISC-V, there is no problem.19:27
lkclif you are happy with the governance of RISC-V, there is no problem19:28
lkclif you intend to modify RISC-V in a secretive, customised fashion, where as part of the custom secretive modifications to the RISC-V ISA you effectively HARD FORK the toolchain, linux kernel, glibc6, and all other packages, and (eyes-open) take on the responsibility indefinitely for the maintenance and distribution of the same, there is no problem19:29
lkclif however - as we are doing - you have the following combined goals:19:30
lkcl1) the full and public and transparent development of *PUBLIC* modifications to the ISA, intended to go into mass-volume products19:30
lkcl2) where said products are intended to be programmed by tens to hundreds of thousands of developers19:31
lkcl3) that all such development of all tools and all ISA Modifications, and all discussion of the same SHALL be ENTIRELY public19:31
lkcl**THEN** you have a real serious problem with the RISC-V Foundation and the RISC-V Foundation Charter because:19:32
lkcla) there are blanket commercial confidentiality clauses in the RISC-V Membership Agreement *AND*19:32
lkclb) no ISA Extensions are permitted to be submitted UNLESS you are a Member of the RISC-V Foundation *AND*19:33
lkclc) even the documentation describing **HOW** to submit a public ISA Extension are behind a secretive closed-doors "ITU-Style" paywall19:34
lkcli.e. you don't get access - even to the documentation on how to *begin* to submit ISA extensions - until you sign the agreement that takes away all rights to publicly discuss the ISA extension and the process of doing so.19:34
lkclnow, IBM's ISA WG also has some "secrecy" as part of the review of an ISA Extension.19:35
lkclthat review process does **NOT** prevent or prohibit anyone from actually *SUBMITTING* an ISA Extension19:36
lkclsorry, s/IBM's/OpenPOWER Foundation's19:36
lkclso with the Power ISA, you can still submit an extension, and they will also be setting up a publicly-accessible mailing list for the discussion prior to submission19:37
lkclbut what you *can't* do is participate in the voting of the *ratification* of that ISA extension.19:37
lkclthe RISC-V Foundation simply blanket prevent and prohibit even the submission of Extensions.19:38
lkclhere's where it all went to hell in a handbasket for the RISC-V Foundation:19:38
lkcli made somewhere around *FIFTEEN* public, reasonable, and in-good-faith requests to participate, requesting information on the ISA Extension Submission Process, requesting the unique Commercial circumstances of our project to be taken into consideration19:39
lkclnot ONE SINGLE response was received.19:39
lkclthese requests are all cc'd to the google groups RISC-V mailing lists, you can see clearly that i cc'd the Director of the RISC-V Foundation in each and every request19:40
lkclunder Trademark Law, this means that the RISC-V Foundation *SYSTEMATICALLY* ignored reasonable and in-good-faith requests for Fair, Reasonable and Non-Discriminatory participation in the evolution of the RISC-V ISA.19:41
justinrestivo[m]Received? as in they just ignored your email entirely?19:41
lkcland under Trademark Law, that is grounds for invalidation of the RISC-V Trademark.19:41
lkcljustinrestivo[m]: yes.19:41
lkclthe Founder of RISC-V was also a participant on the same mailing list.19:42
lkclit cannot in any way, shape or form be claimed by the RISC-V Foundation that they did not see those messages, sent systematically over an 18 month period19:42
lkclnow, the really interesting thing is that it turns out that independent technical reviews of the RISC-V ISA show that it's deeply flawed for High-Performance Supercomputing19:43
programmerjakeso RISC-V is a mess organizationally, though they do have some very good ideas worth considering emulating, such as TileLink19:44
lkclso the Libre-SOC Project seriously dodged a bullet on that one.  we should be grateful for the RISC-V Foundation's blatant arrogance.19:44
lkclTileLink came out of the same team, the only problem with it being that it was designed by a team that is used to OO HDL Programming19:44
justinrestivo[m]Do you have access to any papers or anything about those technical reviews?19:45
lkclpeople who have tried implementing it in non-Object-Orientated languages such as Verilog or VHDL have had a bitch of a job19:45
lkcljustinrestivo[m]: yes, 1 sec19:45
lxothanks, lkcl.  is there any of this that you'd like me to keep out of the response?19:45
justinrestivo[m]I'm quite curious as to how/why19:45
lkcllxo: if you post the link to the irc log, 1 sec19:46
lkcland yes, may be useful19:46
programmerjakewell, VHDL is OO iirc, since it's based on Ada19:46
programmerjakeVerilog is just a poor programming language19:46
lkclVHDL is based on ADA? cool! i didn't know that. ohhh i imagine that's where the in/out comes from.19:46
lkclthat one.19:47
programmerjakeiirc it's based on Ada, it's worth double-checking19:48
lkclthat contains the link to the paper by the Alibaba Group19:48
lkclprogrammerjake: it resonates, if you know what i mean. it makes sense19:48
lkclthe Alibaba Group had to add some custom instructions (Indexed LD/ST, LD-ST-with-shift-immediate)19:48
lkcland others19:48
justinrestivo[m]The paper seems to be behind some sort of wall. It's asking me for credentials.19:49
lkclthis was the only way they were able to get high-performance and drastically reduce instruction count in critical loops19:49
lkcl1 sec19:49
lkclit's an extremely comprehensive paper, the processor they built is extreme high-performance19:50
programmerjakefrom reading the TileLink spec, I'd expect to be able to implement it in under 3000 lines of nmigen19:51
lxolkcl, thanks19:51
lkclprogrammerjake: interesting: nmigen would be completely different, we could use OO programming19:51
justinrestivo[m]Thanks a lot. I'm keen to read this.19:51
lkcljustinrestivo[m]: p60 is where the "nitty-gritty" starts19:52
lkclthe POWER ISA has had register + register (INDEXED) Addressing since POWER1, back in 1994/5.19:52
lkclbut the kicker is: *none of these modifications were submitted to the RISC-V ISA WG*19:53
lkcland consequently, if their processor ever becomes mainstream, it will cause an absolute nightmare dog's dinner mess19:53
Las[m]Is instruction fusion that hard to do?19:54
lkclthe very first RISC-V CPU which uses the CUSTOM Opcodes *and* becomes ubuqiutous, where end-user programmers start demanding that gcc and linux kernel and llvm all have mainline support for it19:54
justinrestivo[m]I know next to nothing about the power isa. Is this the most recent/used version of it?
lkclthat's it: it's Game Over for RISC-V.  no other vendor will ever be able to use those same CUSTOM Opcodes, publicly.19:55
lkclLas[m]: YES!!! :)19:55
lkclit's taken us 3 and a half years to develop SVP64.19:55
lkclthe first version was on top of RISC-V, that was 18+ months of work19:55
lkclwe started SVP64 (on top of Power ISA) also 18+ months ago19:56
lkcljustinrestivo[m]: yes.  that's the version that's "Approved" for public use by the EULA19:56
lkclonce we have written drafts, we have to begin the submission process.  this i expect to take at least a year, to go through the review, so that the ISA WG participants actually fully understand what's being submitted, and why19:59
lkclthen, we have to develop Compliancy Tests (our existing unit tests actually cover that)19:59
Las[m]Does the implementation of SVP64 work currently?20:00
lkclall of which are to help make sure that the Extension actually does the job, but more than that, if it's implemented by anyone, the implementors can check it works *before* they drop USD 16 million on 7nm production masks20:00
lkclby "work" in this context, there's two implementations, and also a concept20:00
lkclthe "concept" of SVP64 "works"20:01
lkclthe Simulator "works".  you can write assembler, right now, and "execute" it.20:01
lkclat a rate of 1-2000 instructions per second because it's in python, but you _can_ execute it :)20:01
lkclthis one's a good one:20:02
lkclthat's a full triple-loop Inverse-DCT (Type III) *in-place* - no register copying - in 9 instructions.20:03
lkclcaveat 1) "element 0 /= 2.0" isn't included in that.  that's an extra 2-3 instructions20:03
lkclcaveat 2) you have to have the COS tables pre-computed: calculating those is about... mmm.... 12 more instructions20:04
lkclhere's some more examples, which are the inner loop of FFMPEG MP3 CODEC:20:05
lkclthis compiles down to below 100 instructions (equivalent, 32-bit);a=blob;f=media/audio/mp3/mp3_0_apply_window_float_basicsv.s;hb=HEAD20:05
programmerjakelkcl: generally the COS tables are computed at compile-time and embedded in the program as read-only data20:05
lkclwhereas the VSX version of the same function (apply_window_float) is 450+20:06
lkclprogrammerjake: yes, in ffmpeg general-purpose dct they can pre-compute them and cache them20:06
Las[m]That is really nice. Do you have experience from working with GPU architectures which you're applying here?20:06
lkclbut the general-purpose version actually uses FFT (in a really interesting way)20:07
lkclLas[m]: i don't, so i studied several and also listened to the advice of people who know what they're doing.20:07
lkclfor example, programmerjake knows what he's doing because (a) he's really interested in it and (b) he's the author of Kazan20:07
Las[m]BTW, what is up with the 3D drivers for Libre SoC?20:08
Las[m]What is the difference between the Mesa fork and Kazan?20:08
lkcli also spent several months talking to Jeff Bush (Nyuzi)20:08
lkclthe MESA driver is part of.. well... MESA.20:08
programmerjakeKazan is a mostly independent Vulkan driver written in Rust20:08
Las[m]Do they solve the same problem then?20:09
lkclwe initially wanted to take AMDVLK or one of the other 3D MESA drivers and "morph" it into userspace20:09
lkclyes - or - they both aim to provide a Vulkan-compliant API20:09
Las[m]Seems like duplicated effort to me honestly20:10
lkclboth entirely in (heavily-3D-custom-accelerated) userspace20:10
lkclthere's crossover20:10
Las[m]I'm glad Kazan exists though, I don't like the monorepo 3D driver idea20:10
lkclthey'll likely end up using the same LLVM IR back-end, although jacob's developing Kazan to be back-end independent20:10
Las[m]I don't think LLVM dependency is something you should be aiming for20:11
programmerjakeKazan was started due to Mesa's architecture issues for software drivers20:11
Las[m]LLVM isn't used for AMD anymore due to various issues20:11
programmerjakesoftware rasterizers*20:11
Las[m]Specifically, RADV20:12
programmerjakeI'm aiming for Kazan to work with LLVM and/or Cranelift20:12
* lkcl needs to get up and walk about20:12
Las[m]Hopefully Cranelift will eventually be competitive with LLVM20:12
Las[m]Is Libre SoC currently testable/usable on an FPGA?20:12
programmerjakeCranelift is intended to fill a different niche -- the JIT compiler niche20:13
programmerjakeLLVM is very much geared towards ahead-of-time compilation20:13
programmerjakeYes, Libre-SOC is currently usable on an FPGA, however it isn't very fast since the instruction scheduling isn't finished yet, so it uses a stop-gap slow FSM-based instruction scheduler20:15
Las[m]Does it run Linux then?20:15
programmerjakeone instruction every several cycles20:15
programmerjakenot yet iirc, we still need to implement some system instructions, though lkcl would know for sure20:16
lkclthe RADIX MMU needs completing, the L1 D/I-Caches integrating20:17
lkcland for that we need LD/ST Exceptions20:17
lkcli've been focussing on SVP64 Discrete Cosine Transform and in-place Matrix Multiply20:18
lkclyes, it builds and runs on the (entirely libre-licensed toolchain) VERSA_ECP5 and ULX3S20:18
Las[m]Makes sense, sounds much more interesting20:18
Las[m]Do you get continued funding from the EU?20:18
Las[m]Or is it sparse?20:18
lkclwhich is why we need yosys and nextpnr-ecp5 on the nixos list, btw :)20:18
lkclseven separate NLnet grants of EUR 50,00020:19
Las[m]I've already noted those down!20:19
lkclone for each project20:19
lkclstar :)20:19
Las[m]Hopefully you'll get much more :)20:19
lkcland we've also got one that's under review for NGI POINTER, for EUR 200,00020:19
lkcland i just put in one EUR 50,000 request for coriolis220:19
lkcl(for, not specifically for Libre-SOC)20:20
lkcland another one for a port of cavatools to Power ISA.20:20
programmerjakeall our funding from nlnet is on a task-based basis, so we get paid for completing tasks. NGI POINTER is a lump sum iirc20:20
lkclprogrammerjake: Michiel's been talking with Mirko, to arrange something, because of the way we're set up (not a commercial company)20:20
Las[m]When you get the MMU and such working I'll try making a Libre SoC desktop, on an FPGA if necessary20:21
lkclLas[m], cool! nice!20:21
programmerjakeidk if nlnet gets their money in a lump sum or a little at a time20:21
lkclactually, NixOS would be perfect for that20:21
Las[m]Why do you think that?20:21
lkclthey put in invoices to the EU, based on the Milestones RFPs they receive20:21
programmerjakeah, ok20:22
lkclwe need a full custom rebuild of the entire source code, completely disabling Power ISA SIMD (-mnovsx -mnoaltivec)20:22
lkcllong story.20:23
Las[m]Ah, yeah, something like Nix or Guix would be perfect for that yeah20:23
lkclIBM didn't exactly mess up, but in Power ISA EABI 2.0 they made VSX mandatory, because it significantly improves performance for POWER8 and POWER9 hardware20:23
Las[m]It's very simple to do20:23
Las[m]And you don't implement it I assume?20:24
lkclunnnnfortunately, there was *no other Power ISA vendor* around to tell them, "errr that's going to have consequences"20:24
lkclthe "Embedded" Scalar Fixed-Point Compliancy Subset is 150 instructions20:24
programmerjakethen later OpenPower made VMX/VSX optional in their ISA spec20:24
lkclthe "Embedded" Floating-Point Compliancy Subset is 21420:24
lkclthe "Linux" Compliancy subset - which even there has VSX as optional - is a staggering, mind-numbing 950.20:25
programmerjakeand then AIX compliancy is basically everything -- huge20:26
lkclwe would be effectively stabbing ourselves in the head with blunt instruments if we attempted to QUADRUPLE the number of instructions that we tried to implement20:26
lkclprogrammerjake: yeah, it's like almost 1,100 instructions20:26
programmerjake"stabbing" with a bowling ball...XD20:26
lkclthe Power ISA Decoder in Libre-SOC is over 4,000 gates, and that's not even including any FP instructions.20:27
Las[m]By the way, I have some more answers regarding trust. It seems that Debian, Arch Linux, most common distributions don't actually sign their commits, only the binaries. The binaries in the case of NixOS, delivered by, are already signed by the NixOS Foundation, but you can choose to build any package yourself and be sure you're getting the same result (except in the case of bugs).20:28
Las[m]The commits to Nixpkgs, the repository for Nix packages, i.e. the source code, is also signed, but it's signed by GitHub!20:28
programmerjakeone place where the OpenPower ISA is problematic is it has like 200 different formats for encoding instructions, about half of which come from VSX/VMX20:29
lkclthen, as you can see from the FFMPEG MP3 CODEC example: once we'd completely obliterated our chances of achieving any reasonable power-efficient hardware in any reasonable amount of time, we'd be "supporting and endorsing" a SIMD Programming Model that ran binaries that were FOUR times larger than what we can achieve with SVP6420:29
Las[m]So you'd have to trust GitHub for the source in any case, but it's not much worse than Debian, because you can't trust the Debian source AFAICT, they only sign their binaries I think.20:29
lkclLas[m]: that's where trust in the *developers* comes into play, and you have "developer reputation" because their identities are known to several thousand other developers20:30
lkclany debian developer who tried to introduce rogue software would never work in the industry again.20:30
programmerjakeone thing I've been using for our Rust code is
Las[m]Yes but you trust the packages they sign, right?20:30
Las[m]But how do they build their packages?20:30
programmerjakecrev is a GPG web-of-trust style code review system20:31
Las[m]Also: There has been an RFC for requiring signed commits, but it was rejected due to some issues with the RFC itself, and the conclusion was that a better RFC was needed, since it was an important issue.20:31
lkclLas[m]: yes, you trust them to audit the source code before building it.  given that they also require a "Copyright provenance" list on every file, this review does actually happen20:31
lkcltrustix will help solve that developer-trust issue by having SHA checksums of reproduceable builds.20:32
Las[m]Do you remember the Linux buggy patch scandal?20:32
lkclwhich, interestingly, debian is also investigating20:32
Las[m]Nobody noticed.20:32
lkclthere was a university that recently had their entire contributions pulled due to a stupid researcher.20:33
programmerjakesounds right20:33
Las[m]Yes, but that was only because they made their findings public20:33
Las[m]Had they not publicized it no one would have known20:33
lkclthese are... how to say.. "out of scope" for the purposes of *distribution* of packages20:33
Las[m]I would say the same problem applies, because building packages also requires some code.20:34
lkclor, it's an orthogonal problem20:34
lkclyes, and here, over time, debian developers have self-bootstrapped up to be able to trust the packages that build the packages20:34
lkclover 2 decades20:35
programmerjakedebian's been around awhile20:35
lkclsolving the "source code review" problem is one that i would consider to be orthogonal / an-additional-problem-needing-solving20:36
lkcldebian solves it by way of "developer maintainer reputation" (consequences for them if they try to compromise it)20:36
Las[m]In any case, the equivalent of binary packages in NixOS are already signed20:36
lkclok, let's focus on that.20:36
Las[m]You can specify the key for the binary packages too.20:36
lkclsigned... by whom?20:36
lkclare the individual packages signed?20:37
lkcland by whose key?20:37
Las[m]I don't know who actually has the private key, I assume it's either Eelco Dolstra or Graham Christensen20:37
programmerjakewell, if the source-code review can be cryptographically combined with the distribution cryptography, we'd likely have a more trustable system20:37
lkclok so it's one single person, signing the *entirety* of the binary packages distributed through nixos?20:37
Las[m]wdym individual packages?20:37
Las[m]Yeah, pretty much20:37
lkclis each nixos package signed by that one key?20:38
Las[m]You can choose to not use the binary packages and build from source though20:38
lkclor is there a "Release list" which is also signed?20:38
Las[m]Yes, if you use their binary cache, as you call it20:38
Las[m]There is not20:38
lkclok, then that's a violation of one of the requirements.... two violations20:38
Las[m]What do you mean by "Release list"?20:38
lkcldebian has over a thousand individual developers, all of whom have their own GPG key.  to compromise the entirety of debian packaging you'd have to compromise the GPG keys of 1,000 developers20:39
programmerjakethe list of what packages there are, their versions, crypto hashes, and where to find them20:39
lkclso there's a software build dependency list, in nixos?20:39
Las[m]How do they actually sign the packages? Do they build the same package?20:39
Las[m]Do they have reproducible builds?20:40
Las[m]There is inherently20:40
lkcl"package X version N depends on package Y, version M"20:40
Las[m]It's a part of Nix, it's just how it works20:40
Las[m]You can't have a package without its dependencies and the associated metadata20:40
lkcldebian is investigating reproducible builds, yes.  quite complex, given that timestamps (used) to be inserted into object files20:40
Las[m]So how do they sign the packages? Isn't it fundamentally flawed then?20:41
Las[m]If you're one of the 1000 developers, what do you actually sign?20:41
Las[m]I assume they don't build the same package from source, verify that the bits match?20:41
Las[m]Since they'd need reproducible bits for that20:41
lkcland if there is a package that has Library Z, and another totally different package also has Library Z, how many (binary) copies of Library Z get installed if the 2 packages are installed?20:41
programmerjakeand file paths, and occasionally compile-time generated random numbers and other annoying bits20:41
Las[m]They get two different copies if they depend on two different versions of it20:41
Las[m]If it's the same it's not duplicated obviously20:41
Las[m]reproducible bits -> reproducible builds20:42
lkclthey build the package by running "dpkg-buildpackage" and at the end of the build process, dpkg-buildpackage will go get their GPG private key and sign the package.20:42
lkcldead simple.20:42
Las[m]So each package only has one signature?20:42
lkclLas[m]: not so "obviously".  some distros (dockkkkaaa most notably, sigh) actually duplicate the entire set of libraries.20:43
programmerjakedocker isn't really a distro...20:43
lkclApple's DMG system duplicates entire suites of libraries (self-contained, they call it.  spectacular waste of disk space)20:43
Las[m]It kind of is I assume?20:43
lkclit's a grey area :)20:44
Las[m]It's a trivial thing to do in Nix because of how Nix is set up20:44
lkclit makes it *possible* to distribute "stuff"20:44
Las[m]In any case, I think you'll have better security with Nix rather than Debian now that I think about it20:44
Las[m]Especially if you only fetch Nixpkgs through Git20:44
lkclok, so, is there a "Master List" of package that are available from NixOS?20:44
lkcla list of "everything-that's-been-built-and-available-from-the-cache"?20:45
Las[m]Kind of20:45
Las[m]I'll need to explain how Nix works for it to make sense20:45
Las[m]That exists yeah20:45
lkclwell, that list basically also needs to be GPG-signed.20:45
lkclor, more accurately (and less work), the SHA checksum of the "list-of-everything-built-and-available" needs to *itself* be GPG signed20:46
Las[m]Nix is simply a way to describe isolated builds, i.e. they are run in a container. You specify the args, the executable, the environment variables, the expected output hash if applicable, etc., and Nix builds it.20:46
Las[m]When you fetch a package from the binary cache, what you're doing is evaluating the source code for building the packages, then when you try to build the package, Nix checks if it's already been built by a trusted cache.20:46
Las[m]If it's not been built by the cache (e.g. if you're trying to build a custom package not in Nixpkgs) it will fail and will be built locally20:47
lkclharry_hk_99 from m-labs used nix packages for distributing nmigen-related stuff, btw.  very cool.20:47
Las[m]It is really cool20:47
Las[m]After using NixOS it's not possible for me to use a standard Linux distro anymore, it's simply too annoying20:47
Las[m]Because it's not just building standard packages that is reproducible, the OS configuration is also its own package20:47
lkclnice.  my primary interest is in the security of the "trusted cache".20:48
Las[m]For LibreSoC, what it would allow is users doing something like `nix build git://` to build the software of that repository, without any other steps, at all (besides installing Nix)20:48
Las[m]Well I don't think it matters much since you really don't have to use it20:49
Las[m]If you're at all worried about it just disable it20:49
lkclack, got it20:49
Las[m]It's only needed if you're trying to build chromium20:49
lkcloh btw, although it will be a hell of a lot of work... deep breath... we need pip3 out of the equation.20:49
Las[m]What would you use instead?20:49
lkclbecause it fundamentally violates safe / secure package management20:50
lkclmanual installation - by nixos - of each dependency - as a nixos package.20:50
Las[m]What you can do with Nix is ensure that it always fetches what's specified by some hash20:50
* lkcl drums fingers20:50
lkclhave to think about that.  in theory it should be ok.20:51
Las[m]It's not just possible with Nix, it's actually required20:51
lkclyou'd need to hook into pip3 itself and prevent it from running install20:51
Las[m]Since otherwise you can't be sure it fetches the same things from the internet20:51
Las[m]Yep, that's what's done currently20:52
lkclthat's really good to hear20:52
Las[m]It's how Python packages work on Nix20:52
Las[m]Cargo, etc. all can not fetch when you build packages that use them20:52
* lkcl oven's beeping here btw, gotta run :)20:52
Las[m]There is a single exception for that, which is that if a build process is declared to have a fixed output, i.e. the hash of the output of the build is specified, then the build can access the internet, since that's all you really care about.20:53
Las[m]If some download from e.g. was compromised, the output hash would change20:53
Las[m]The build would fail then, since the output is wrong20:54
Las[m]This is how fetching things from the internet works fundamentally with Nix, so e.g. to fetch something to use in Nix, you can use a helper build that uses curl to fetch the URL, and then the output is matched against some fixed hash.20:55
Las[m]There are no exceptions to this.20:55
lkclLas[m], i missed your question above about instruction fusion.  answer: when you want to do 8-way multi-issue (8 instructions fetched, decoded and executed at once), hell yes22:02
lkclif you have a fixed 32-bit width, you can fetch 8x32 bytes, decode 8x 32 byte instructions, issue 8x 32-byte instructions22:03
lkclthe only thing you need to do is: the first instruction you respect the read-write register hazards "as normal"22:04
lkclthe second instruction you include the *first* instruction's read-write register hazards *as well* as those of the second22:04
lkclthe third you include all of the register hazards from instructions 1, 2 *and* 3 and so on22:04
Las[m]Ah, yeah22:04
lkclinstruction fusion you now have to have an additional phase in between decode and issue (that is 8-wide)22:05
Las[m]I assume instruction fusion would be much easier if the instructions were more like a "pipeline"22:05
Las[m]Didn't Itanium have something like this?22:05
lkclLas[m]: itanium from what i gather was VLIW?22:05
lkclso you have a buffer of 8 instructions with your 32-bit instructions in it22:05
Las[m]Yeah, and I seem to remember that you basically had a pipeline of instructions, but I might be wrong22:06
lkclin the non-fused model, each of those goes *directly* to a pipeline.22:06
lkclinstruction 1 goes to pipeline 122:06
lkclinstruction 2 goes to pipeline 222:06
lkclwith fusion, all bets are off.22:06
lkclif you spot a linear sequence of 2 instructions (which is itself expensive to spot, given that they're now potentially overlapping)22:07
lkclthe instructions to fuse could be in buffer slot 1 and 2, or 2+3, or 3+4, or 4+5 .... 7+822:07
lkcland that's assuming there *is* only one such macro-fused instruction in buffers 1-8....22:07
lkclyou now have to *REPLACE* that with a *SINGLE* fused operation...22:08
lkcl... now you only have *seven* actual instructions22:08
lkcland they have to get shuffled down by one, dropping one spot in the 8-way multi-issue execution22:08
lkclnow let's say you have 4 such sequences22:08
lkclby some annoying coincidence22:08
lkclinstruction 1+2 is fuseable22:09
lkclinstruction 2+3 is fuseable22:09
lkcl4+5 *and* 6+7 are all fuseable22:09
lkclnow you must shuffle 1+2 down to pipeline 122:09
lkcl3+4 down to pipeline 222:09
lkcl5+6 into pipeline 322:09
lkcl7+8 into pipeline 422:09
lkclstart to see how that quickly becomes Absolute Hell?22:10
Las[m]Yeah, that's a lot of extra complexity.22:10
lkclyou have a massive multiplexer of the instructions, right between decode and execute22:11
lkclit's several thousand gates.22:11
lkclif you wanted to keep all 8 of the pipelines completely full you actually need to read *sixteen* instructions per clock cycle!22:11
lkclotherwise the pipelines might not be 100% busy.22:11
lkclso, fundamentally, it was a serious mistake to assume that this would be acceptable.22:12
programmerjakeit's exactly like compressed instruction decoding22:12
lkclindeed, except the detection logic of what the possible combinations are is far more complex in gate terms22:12
Las[m]You mean Arm Thumb?22:13
programmerjakeor RISC-V C, or others22:13
lkclwhere the algorithm that you came up with, programmerjake, for compressed instruction decoding (and 32-64 bit prefix) is brain-dead-simple22:13
lkcl(to detect, that is)22:13
lkclthe complexity of *detection* of the opportunity for fusing means that it is likely to require yet another slot in the decode phase.22:14
programmerjakeinstruction fusion can pretty easily be added to the algorithm I designed22:14
lkclyes it could indeed.22:15
lkclor, it could be applied a 2nd time22:15
lkclwith this time "identified fusion opportunity detected" as the input, instead of "64-bit prefix detected"22:16
lkclthe issue with RISC-V is: they didn't give anyone a *choice* about using macro-op fusion: adrian_b points out it out, clearly, that if you don't have it, you get worse performance.22:18
lkcl(because more instructions)22:18
lkcllike, twice as many as other ISAs22:18
lkcland because that was a fundamental base assumption *during the design of the ISA*, there's now not enough spare opcode space, plus all of the compilers and software are a massive multi-million-dollar multi-man-year task, to retroactively go back and fix it.22:20
lkclprogrammerjake, btw, sigh, i just realised that the SVP64 Branch-Conditional encoding has to not depend on SVSTATE's "Vertical-First" flag22:21
lkclor, it _can_ depend on it, but it creates a hazard between instruction decode and the last update to SVP64.22:21
lkclSVSTATE sorry22:21
lkclmind you, the decoder already has that (due to the looping).22:21
lkcli'm in two minds about that one.22:22
lkclis anyone else here a fan of Rick and Morty? :)22:26

Generated by 2.17.1 by Marius Gedminas - find it at!