Discussion of commit frequency - Tuesday 16th January 17:00 UTC

Discussion of commit frequency and other points mentioned in this email thread. After meeting discussion, points agreed upon will be documented in the wiki.

Points to be discussed

Dmitry's points of concern:

  • As a developer, I should have a liberty to interactively rebase, fixup, squash commits, commit when I can and want, and same for push. If there are restrictions on this from NLnet side, these must be explicitly published on their own guidelines. I'm sure they have different forms than the ones exposed.

  • My point is, as I work on it as an individual in my free time, I should have flexibility to a certain degree to develop it when and how I want. And I commit the code when it's at least somewhat "ready" and "stable" and "beautiful" according to my personal standards. I extensively use interactive rebase to make the history readable. When I find that some branch commit introduced an error, I go back there and edit it interactively, then force push. This makes the linear history work when the master is rebased, decreases the chance of the later bisect and simply avoids the trash like "oh look I fixed a typo" for the code which just got rebased into the master (unless required).

  • Obviously this works better in branches or in master if you have force push rights (and likely develop alone); but it has its own benefits. Obviously it isn't a silver bullet, but it makes the history way better, and that's the approach I always follow. And now, despite that nobody had problems with this before, now we have this discussion. This is not the first time we have some rules established for which we lack the explicit requirements. The lack of these is fine as long as there's a technical rationale. I doubt any real SW project has a "commit whatever you have" policy. Real SW projects follow "commit respectively and wisely" policies instead.

  • Indeed, these statements are vague and reflect my own personal experience and preferences. But, unless you have a real severe cause to completely bend someone else's behavior and habits into the way you like it, I'd prefer to avoid it. You know I don't have massive chunks of the code with me (I wouldn't even if I had time, what's the purpose?). But I try to avoid "trash" commits, I try to commit things which work, and I always worked this way. After all, it's an Open Source development, not some corporate idiocracy.

  • As for "we're afraid something might get lost", I'd recommend starting with the different places to think about. For example, git and CI. These seem to be much better targets; these are the real points of failure.

  • Yes this is subjective; I hope we all are subjects though. "Commit once you think it's good enough to show to others, it breaks nothing, there's nothing to be ashamed of, you find it appropriate to publish at this stage" -- these all look reasonable. The beauty is in the eyes of the beholder.

Dmitry's point on his personal time:

  • My point is, I really have not much time to work, and I indeed try hard to dedicate all the free time I have. Please don't make me feel that I waste this time taking the spokes out of the wheels. :-)

Jacob's response:

  • Assuming you're not working directly on the master branch, I think it should be sufficient to push a WIP commit with whatever changes have been made by the end of the day, that can be rebased, squashed, amended, etc. next time you work on it. (if you're working on master, please don't rebase/squash/amend any commits, use a separate branch if you want that, then push to master once the commits won't change anymore)

  • It would also be useful but not required to push your current code if you're posting major results or asking questions, so others can see your code -- after all it's really hard to spot bugs in code you can't see!

  • I don't think there are any NLNet restrictions other than that code needs to be public before submitting RFPs.

  • we should probably put our conclusions in the developer documentation, Andrey would you be willing to do that? icr if that docs bug is closed yet...there's probably a good task to put this under.

  • maybe just expand the existing wiki section somewhat?

(Andrey: I agree. A first rough draft will probably be this page. Only the most important points the team agrees on will be written into HDL_workflow, if not already present.)

Luke's responses:

The "regular commit push" approach imitates corporate backup systems:

if you were working for a Corporation (and they were any good)
your private machine would be backed up by the IT Staff,
onto redundant servers, every day, without fail, so that there
is abolutely no chance work could be lost.

it would not be your problem, because the Corporation IT Staff
set up your machine, give it to you with backup software
pre-installed, it s basically not your problem, and you would
not be "wasting your time" on backups.

the FOSS equivalent of that is that you push, again, without
fail, to a public repository, but this is of course taking
up *your* time to do it. this is unfortunate but necessary,
as we obviously cannot invade your privacy of *your* personal
local machine to forcibly push to Libre-SOC servers to ensure
backups and Transparency!

Luke ensures there are at least 3 backups of the LibreSOC project:

i have TWO remote machines doing daily backups of the server
and i want it increased to 3. i also do a 3rd intermittent
server backup (manual rsync) because being on a Mobile Internet
it costs me USD 3 per Gigabyte, i only do that if absolutely
critical.

so that is 3 backups and i really want it to be 4.

then also there is salsa and other servers and all developers also have
clones of the repos, but they are intermittent and
cannot be relied on, plus cannot back up the full server itself,
just the repos.

Luke's point whether Nlnet requires such restrictive measures (answer to Jacob's point):

yes you are correct, but the backgground/context is that it
is far stronger than that with very serious implications.

their mandate is "Works for the Public Good".
if it's not public, then they would be in flagrant
direct violation of Dutch Foundation Law to give money
out for private work, which no messing about
is jail time for the Directors.

Luke also asked Dmitry why is it an issue to follow the approach of committing and pushing frequently.

One of the reasons perhaps could be slow or low-bandwidth internet, but given the low overhead of git pulls/pushes, this isn't an issue in practice.

The point to perform regular git commits is documented in HDL workflow (documents LibreSOC standard procedures), in the "git" section.

Luke mentions that if an argument exists for not following the procedure, it needs to be mentioned:

if you can provide an explanation as to why you consider it
reasonable to not follow the Project Standard Procedures
as outlined in the HDL Workflow page, then we have a
justification to put in front of NLnet and the EU Auditors,
and everything is fine.

this is basic ISO 9001 QA Procedure, which I trained in back in
1993. the "red flag fail" is not when you don't document
something, it's when you say you are going to follow a certain
procedure / practice *and then do not follow it*.

it is documented in HDL Workflow *all* commits shall be pushed,
and you are requesting, Dmitry, not to follow that Procedure.
you therefore explicitly need to explain clearly (and publicly),
no fuss, and there is no criticism here expressed *or implied*:
why you are not going to follow that Documented Procedure, ok?

(Andrey: I didn't see HDL_workflow say "all commits shall be pushed". It makes sense given that Nlnet only pays for published work, but perhaps this needs to be written explicitly?)

Luke also made a point that there may be exceptions due to privacy/security concerns, but those must be discussed outside of LibreSOC public sytems:

if there is a privacy or security reason
that you cannot reveal publicly, then NLnet will be perfectly
understanding of that as well, but the discussion there
has to move *offline* and not take place on Libre-SOC public
servers. it is one of the *only* exceptions to the rule
"everything is public". NLnet do actually fund some very
senstive projects, where anonymity of the developers is
absolute top priority... *but* the work still has to be
public, to fulfil "Works for the Public Good" Mandate.

Also a point on Nlnet:

as Jacob points out, you absolutely cannot
get paid for private work, as this directly conflicts with
NLnet's "Works for the Public Good" mandate.

Luke on git history cleanup:

i worked out Dmitry, that you probably actually enjoy doing the
git history cleanup, it is a sense of satisfaction and pride :)

Point made by Andrey to Dmitry:

  • I understand your concern of being able to develop independently, committing changes when ready for comment/review (so long as the development history is clear; which is easy to do with separating big changes into many commits, before pushing).