Friday, 2021-08-06

lkclLas[m], justinrestivo[m]: ^10:10
lkclthat tells us that debian does indeed use the (massive, real-world-identity-backed, verified) debian keyring / web-of-trust to verify the integrity of source code used for all and any packages.10:11
lkclthe reports there show how people are having problems due to bugs in gpg (and also reports that there is a huge amount of misinformation, some of which has quite serious security implications for the users running the random "blind-leading-the-blind" commands being "advised")10:13
lkclalso there is this:
lkclwhich is a way that packages may be marked for auto-building (and their dependencies), originally designed for porters to keep up with other architectures10:15
lkclaside from a few (really useful) areas, it's looking, unfortunately, like nixos is displaying NIH syndrome.10:16
lkclthe pip3 checksumming is brilliant, and extremely useful.10:16
lkclwhere are the modifications to pip3 available?10:17
lkclmy feeling is that we should extract those and actually use them in the reproducible build dev-env-setup scripts.10:17
lkclalso, it would be fantastic to have the checksum modifications submitted upstream, such that the manifest file can be installed by OSes or end-users10:18
Las[m]It's not really possible without using Nix10:19
lkclLas[m]: some of the programming that i've done over the past 35 years, i don't believe in "not possible" :)10:20
lkclin 2010 i combined 4 different projects and used 12 different programming languages in 2 weeks to create 20,000 python bindings to webkit DOM, making it a FULL peer of javascript.10:21
lkcl"not possible" is just an invitation to prove that it is :)10:21
Las[m]There's no reason not to use Nix though10:22
lkclis there a pointer to where the source code used for pip3 may be viewed?10:22
Las[m]You can just avoid using Nixpkgs10:22
lkcldoes it have a GPG keyring backed web-of-trust for source packages, of over a thousand developers?10:23
lkclfor the *source* packages?10:23
lkclnot the binary packages, the *source* packages?10:23
lkclah, is this it?
Las[m]It doesn't AFAIK, which is why you might choose to avoid using Nixpkgs10:25
Las[m]Yep that's one solution, and probably what we'd use for Libre SoC10:26
Las[m]FWIW, I've been thinking about it a bit, and even without cryptographically signing the commits, I can't think of a trivial way to tamper with the source code.10:33
Las[m]People would notice if the upstream hash suddenly changes incompatibly10:33
Las[m]The link to Nixpkgs:
Las[m]It's a huge monorepo as you can see10:35
Las[m]FWIW, I can't see how Debian signs their commits:
Las[m]None of these commits are signed10:37
lkclcommits are not signed, the entire package source is signed.12:13
lkclthe reason is because packages may be from .tgz archives or other types of repositories12:14
lkclin the case of git-buildpackage, a tar archive is automatically extracted, and, once available, a SHA checksum may be generated and the signature verified on that12:14
lkclalso included in the signature is the debian maintainer's debian/rules, debian/control and all other information required to build the package.12:15
lkclit's much more sophisticated than people think [who don't know how it works, and consequently end up duplicating it]12:16
lkcl"people would notice" - how?12:17
lkclif there's no automated "noticing" that's a black mark12:18
Las[m]You are right12:19
lkcli.e. if there's no tool which can provably verify the upstream hash, then it can be tampered with in a way that causes damage12:19
lkclyes people would "notice", but the damage would potentially already be done12:20
Las[m]But there is a high chance that you'll notice that the git commit you got is not what other people have12:20
lkclif however each commit was GPG signed, by (any one of hundreds of maintainers) then not only could automated tools "notice", you'd also have a way to track who was responsible12:20
lkcland issue a revocation of their GPG key12:21
Las[m]It's not like Debian where packages might be any arbitrary version12:22
Las[m]If we assume that the situation with signing commits will not change, what you'd do to ensure security is to pin your Nixpkgs dependency to a specific commit12:23
Las[m]Then you might, perhaps, once every 6 months, find another valid commit to change to12:23
Las[m]You could check that this is the commit that other people are using12:23
Las[m]It could then be that the source that everyone is using has been tampered with, but this is much harder12:24
Las[m]Since people are looking at the source code12:24
lkclapt-get install package/version-number12:39
lkcli presume also apt-get source package/version-number (although i never used it)12:39
lkcland apt-pinning then ensures that version is not replaced12:39
Las[m]Yes but that doesn't check checksums12:40
lkclyes it does.12:40
lkclthe checksums are what are GPG signed12:40
Las[m]Ah, really?12:40
Las[m]So a package/version-number is tied to a specific checksum?12:40
lkcl... from the Release File, yes.12:41
Las[m]Ah, huh12:41
lkclremember, debian distributes .tar.gz archives as well as git repositories12:41
lkcle.g. most GNU packages are distributed as tarballs, not git repositories12:41
Las[m]Is the checksum checked when downloading or is it only the signature that is checked?12:42
Las[m]Makes sense, that's nice12:42
lkclwhich is the GPG-signed
Las[m]And I assume this is also signed somewhere?12:43
Las[m]Makes sense12:43
Las[m]What I mean before though wasn't in this way12:48
Las[m]In your install scripts, you don't check the checksums of the libraries you install AFAICT12:48
Las[m]Or am I wrong?12:48
Las[m]I.e. the development environment isn't exactly reproducible12:49
Las[m]With Nix, you will always get exactly the same versions, unless you explicitly update the dependencies12:49
Las[m]Version being defined as checksum12:50
Las[m]The reason I bring this up is that this is also a security measure in itself, since even if there isn't a person to trust, you can check that the checksums match what everyone else is using trivially.12:52
Las[m]Although obviously signing is still beneficial, which is why I'm drafting an RFC to introduce that12:52
lkclpackage dependencies are also packaged.13:07
lkcltherefore, inherently, if it's a dependency, it's also signed.13:07
lkcland with the work for the past... (5?) years in debian on reproduceable builds, then, clearly, that's also possible13:08
lkcl(where do you think the work on reproducible builds came from, such that nixos could consider using it?)13:08
lkclanswer: it came from work done by redhat for RPM distros and by debian developers for apt/dpkg packages13:09
Las[m]It's not possible without something like Nix though13:09
lkclthat's incorrect.13:09
Las[m]Then how would you do it?13:09
lkcldebian (should, by now) have reproduceable build checking as part of dpkg.13:09
lkclah, found the status / progress page
lkcl"We have a new control file *.buildinfo that records the build environment"13:10
Las[m]How do you actually ensure the hashen of the dependencies you are using when developing/building Libre SoC though13:12
lkclso out of appx 32,000 packages, appx 27,000 are currently reproducible13:12
lkclwe don't.  we rely on debian/stable13:12
Las[m]And how do you ensure they correspond to the source code for building the packages13:12
lkcland on manual git tags.13:12
lkclwe don't.13:12
Las[m]So effectively you only rely on trust13:13
lkclbecause there's not enough people helping, therefore we have to trust the external dependencies13:13
Las[m]You trust the Debian developers not to be malicious13:13
lkclthat, and if there's an external...13:13
lkclof course.  by way of their reputation13:13
Las[m]Whereas with Nix, you can just see that it hasn't changed and is correct13:13
lkclas in: if they were ever caught acting maliciously their career is over13:13
lkclthat's incorrect13:13
Las[m]How so?13:14
lkclthe chain of trust goes back to the original source, as picked up by the debian developer13:14
lkcland the debian developer takes responsibility for reviewing the source code (as mentioned yesterday, due to the copyright file, debian developers *have* to do a full source code audit)13:14
lkcl85% of debian is reproduceable according to that graph, btw13:15
lkclsorry, 85% of amd64 is reproduceable13:15
lkcli have a meeting in 45m i have to prepare for13:15
lkclLas[m], when writing that RFC this should be useful13:19
lkclif the *EIGHTEEN* requirements are not met, the RFC should be considered, effectively, harmful.  as in: if all requirements are not met, implementation of the RFC will result in a false sense of security.13:20
Las[m]What FPGA do you recommend for testing Libre SoC?21:53
programmerjakeI didn't actually get around to buying one yet, but I think we're planning on getting some ECP5-based boards made that have a decent amount of ram and other stuff, lkcl would know for sure22:25
matthewcroughanWhat page do I read to understand how to simulate the libreSOC?22:39
lkclLas[m], _probably_ the easiest one right now is the ULX3S
lkclalthough i've just got a Digilent Arty 100, in *theory* it should be dead easy to build / upload.23:22
lkclwe need the nextpnr toolchain for it though23:22
Las[m]What won't work without the nextpnr toolchain?23:23
lkclmatthewcroughan: if you just want to run simulations of the HDL, then run the devscripts and then pick one of the unit tests23:23
lkclFPGA building23:23
lkclfor FPGA targets supported by nextpnr23:23
Las[m]So the board isn't usable without some work?23:23
lkclwhich board? (there are hundreds)23:23
lkclULX3S "just works"23:24
lkclhowever, the larger Digilent Arty-A7 100, i have one but haven't had time yet to get *its* version of nextpnr built.23:26
lkclbtw get the largest FPGA you can.23:27
Las[m]Does it being larger make any difference for Libre SoC?23:27
lkclPower ISA is *expensive* in terms of gates, due to it being a high performance ISA23:27
lkclohh yes23:27
lkclbecause it's a minimum 20,000 LUT4s, that's without an MMU and FP.23:27
Las[m]So it uses as many gates as possible?23:27
lkclPaul Mackerras added *partial* VSX and that required a staggering additional 30,000 LUT4s.23:28
lkclwhich is one of the reasons we're not doing VSX / SIMD.23:28
lkclsigh vivado is proprietary.  frickin nuisance
programmerjakelater we will want to use a FPGA to run something with multi-issue, out-of-order, speculation, multi-core, L1 and L2 caches, etc. that needs a big FPGA to fit it all23:29
Las[m]The ECP5 only has "85k LUTs", is that not enough?23:30
lkclah ha!
lkclLas[m], it's enough... for now.23:30
lkcli have a VERSA_ECP5, only 45k LUT4s, and 50% of it is used up by the *integer-only* first version23:31
Las[m]Thanks for the recommend then23:31
lkclcompare that to e.g. a "simple" RISC-V 32-bit core, that can be done in only around 3,000 LUT4s.23:31
lkcl64-bit Power ISA is *massive*23:31
* lkcl spews23:33
lkclthere's another toolchain around somewhere.  run by google so of the usual form "wget | bash". sigh23:33
programmerjakespilling your coffee? :P23:34
Las[m]Isn't that still better than doing it in Python?23:34
lkclLas[m], "serious" low-level programming isn't done in java.23:36
lkclah ha!
programmerjakethat's a point of disagreement...I think Rust would be better than Java or Python, but Java could be better than Python -- depends23:36
lkclrust due to lack of adoption becomes a serious barrier to contributing. we've already made that mistake.23:37
lkcljava requires insane 150mb+ runtime environments, even to run the tiniest program23:38
Las[m]Not anymore23:39
Las[m]It can be compiled AOT into small executables23:39
lkcli heard gcc has a java frontend. the language itself is so anathemic i won't touch it. my only experience with java (a red5 plugin) it was as bad as i expected it to be, haven't used it since23:40
lkclngggggh dockkkhaaa23:41
lkclthat's where i remember the "wget googleurl | tar -x | bash" comes from23:42
lkclfor symbiflow23:42
lkclngggh stress! :)23:44
programmerjakedo note that Rust is more popular than you might think, according to the 2021 stack overflow survey Rust is known by a little more than 7% or respondents -- about the same as Go and more than Perl and Assembly and Matlab23:45
programmerjakeRust is also growing quickly23:45
lkclprogrammerjake: none of these things make it appropriate. "quickly growing" means "unstable".23:48
lkclwe've learned this lesson already23:48
programmerjakeumm...quickly growing doesn't mean unstable, Rust hasn't had any noticable breaking changes in 5+ years23:50
programmerjakein fact, they're specifically intending to not ever have breaking changes23:51
lkclprogrammerjake: i'll be absolutely honest with you, here: the biggest alarm bell regarding rust is how much you promote it, and how you won't do any programming in anything other than rust.23:52
lkclor, as you pointed out on libre-soc-misc, you'll do "non-rust" programming "if absolutely forced to".23:53
programmerjakewell, that's because I honestly think it has critically important benefits that all the other mainstream languages missed23:53
lkclthis has been a huge concern for me for over 18 months, now.23:53
lkcli know that: it does not automatically mean that "all other languages are total s*** and not worth bothering with"23:54
lkclmore than that, the language *itself* is not "the ecosystem".23:54
programmerjakeI never said they weren't woth bothering with or that they were trash23:55
lkcljust because the *syntax* is "great for a given specific purpose" does not automatically and inherently mean that all other languages are...23:55
lkclyou did indeed.  1 sec...23:55
programmerjakeI just said I think Rust is better in almost all cases23:56
programmerjakebeing better than a thing doesn't mean that thing is bad23:56
lkclit has in practice meant that you've not contributed to the nmigen codebase in over a year, and before that it was somewhere around 6-8 months to the previous contribution23:58
lkclwhich has put extreme pressure on me as the sole exclusive full-time HDL and Simulator developer23:59
programmerjakethat's not caused by my liking Rust -- that's caused by my other problems...sorry23:59
lkcli know. i get it.  i've been there (intriguingly, around the same age as you are, now)23:59

Generated by 2.17.1 by Marius Gedminas - find it at!