1. LibreSOC Bug Process
  2. Why raise issues
  3. Adding sub-tasks to tasks under existing milestone
  4. Budget Estimation
  5. "I'm thinking of doing... procedure"

LibreSOC Bug Process

This page describes in detail how to raise new tasks (bugs) and how to approach development within the project in order to get appropriate amount of funding for the tasks completed.

Why raise issues

If you have discovered a problem in Libre-SOC (software, hardware, etc.), please raise a bug report! Bug reports allow tracking of issues, both to make the developers lives easier, as well as for tracking completed grant-funded work.

It is extremely important to link the new bug to previous ones. As Luke mentioned on this bug, "it is a mandatory project requirement that the graph from any bug contain all other bugs (one "Group")".

The primary reason for this is to ensure bugs don't get buried and lost, and will aid those tackling similar problems at a later time.

Also, for project management and financing purposes, it helps developers to keep an up-to-date list of their tasks.

How to raise issues

  1. Create a bug report.
  2. Add in any links from the mailing list or IRC logs to the bug report for back tracking (this is mandatory). Also fill in the URL field if there is a relevant wiki page.
  3. CC in relevant team members
  4. Make absolutely sure to fill in "blocks", "depends on" or "see also" so that the bug is not isolated (otherwise bugs are too hard to find if isolated from everything else)
  5. Ping on IRC to say a bug has been created
  6. Unless you know exactly which milestone to use, leave blank initially. This also applies when editing milestone, budget parent/child, toml fields. See section [[HDL_workflow#Task management guidelines]] further down.
  7. After setting the milestone, it is absolutely required to run budget-sync, as it will point out any discrepancies. The budget allocations will be used for accounting purposes and MUST be correct. Note you can only get paid for stuff done after the nlnet grant is approved (before the MOU is signed)

If a developer ever needs to check which bugs are assigned to them, go to the Libre-SOC bug tracker advanced search page, and in the "Search by People" section, check "Bug Assignee" and "contains"

Additional info

Linking bugs - 'use bug#NNN' format

  • When mentioning other bugs in bug description or comment, use the "bug #NNN" format, and not "#NNN". For example, writing bug #1000 in in the bugtracker comment section will create a link to bug #1000. Following this syntax ensures the Bugzilla system converts every bug reference into a hyperlink which makes things easier to track (as you see the interdependencies between various tasks/bugs/milestones etc.).

Do not attempt to re-use bugs

  • As LibreSOC uses the bugtracker system for task management and grant/budget allocation, it is critical not to change the purpose of a previously created bug. If a duplicate bug has been created, mark as DUPLICATE (see the procedure section further down on this page on additional types of bugs which would not be worked on).
  • All comments cannot be removed (with the exception of having no comments other than the initial description, and no link/references to other bugs).
  • The bug may also be referenced externally, and thus re-use is not permitted.

Adding sub-tasks to tasks under existing milestone

  • notify Michiel and the relevant NNNN-NN@nlnet.nl team of advance notice of intent to add new sub-tasks, cc'ing bob goudriaan
    • confirm with them that this is NOT a change in the AGREED MILESTONE BUDGETs, because it is just sub-task allocation.
    • confirm that they are happy to add the sub-tasks to the MoU (this needs approval of bob goudriaan)
  • re-generate the JSON file
  • make a DIFF against the PREVIOUS JSON file
  • create a MANUAL report/summary of "changes" that NLnet may easily action
    • "add the following task X to parent Y of amount W",
    • and if any: "change parent Z available amount to V as a WRAPUP") (this latter is because occasionally there are subtasks not totalling the full parent amount, usually because a summary report is needed. Michiel and I privately agreed to call this "wrapup")
  • obtain a confirmation that this has been actioned
  • double-check that the RFP database correctly matches the new bugzilla status.

PLEASE NOTE: YOU CANNOT ACTION THE ABOVE UNDER THE FOLLOWING CIRCUMSTANCES

  1. to make a change to the actual budgetary amounts of the Grant Milestones, without written authorization from Bob and Michiel. a DIFFERENT PROCEDURE is needed. this is because NLnet had to go through a 3rd party Verification Process with the European Union: changing the amounts without consent is therefore tantamount to fraud.

  2. if there has been an RFP already submitted under a given Milestone, it becomes NO LONGER POSSIBLE to change the JSON file in NLnet's system because it is too complex.

    there is one Grant in this complex situation: bug #690, the crypto grant. it is made much more complex because it pre-dates NLnet's current RFP system, where RFPs were submitted by EMAIL and there are manual records not fully integrated into the database.

also note: as the addition of sub-tasks requires a change to the MOU it should NOT be taken lightly, i.e. should not be arbitrarily done one by one, but only in "batches".

considerable care therefore has to be taken to ensure that NLnet are not overloaded, nor that the EU Auditor is given grounds to become "suspicious" because of a dozen or more alterations to the MOU. and write your nickname (i.e. andrey etc.).

Budget Estimation

Working out a time taken (and budget) for a sub-tasks requires guestimating. A small self-contained task should take approximately 1/2 a day up to 8 days (+/- 40%).

The total for a group of sub-tasks should be approximately 5-25 days. If a single tasks looks like it might take longer than 8 days, it is required to break it into smaller subtasks. Big tasks can quickly get out of hand, so if in doubt, splitting a task is the better option.

Assume 1 month is appx EUR 3000 (this is an average; the value may be higher depending on circumstances) and back-calculate.

These numbers come from Luke's comment #8 on bug #1126

Statistically speaking using the +/- 40% variance for each task, and adding up over 20+ tasks will give a time estimate that is accurate to +/- 10%.

(Any sources on this?)

However it is very important to have a clear idea of what is actually needed, and to not leave anything out.

For example, when determining the task of adding instructions:

  • For each instruction perform a thought experiment: "how many lines of HDL, how many unit tests?"
  • Then from - past experience - estimate the total number of days.
  • Assume 1 month is appx EUR 3000 and back-calculate.
  • Put that number for each instruction (or group) into comment 0, add them up, and make that the total for the task.

(Luke has used this method for the last 5 years based on 20 years of project management, and it is expected for the team to familiarise themselves with it)

Also, make sure not to forget including documentation in your estimate. This ensures a portion of grant money is allocated to actually documenting the work involved.

Without documentation, it is not only difficult to teach newcomers about the code in question, it makes it difficult to come back to the code 6-12 months later for maintenance and/or improvement (not a rare situation in LibreSOC).

Don't forget to ask fellow project for help, they might be able to help determine the scope of the work involved.

"I'm thinking of doing... procedure"

Preamble

Given the scale of this project and the critical reliance on certain parts of it (such as devscripts, ISA csv files, ISACaller, etc.) on the work done by the team, it is extremely important to raise any proposed changes and/or improvements, and to wait for feedback before implementing said changes.

Going forward, we all need to keep this in mind when working on critical parts of the codebase.

To make good use of available time and budget, the LibreSOC team should focus on:

  1. Completing tasks under grant budget
  2. Make small, incremental changes which keep the overall codebase functional.
  3. When coming up with fixes or improvements which are intrusive to the current workflow (which may slow the team down from completing tasks under grant budget), assign them to 'Future' milestone for grant applications going forward.

Why these three points?

  1. Work that cannot be related to grant sub-tasks (even if indirectly, by bringing us closer to eventual completion), should be put aside until future funding is secured/confirmed.
  2. Small changes make it easier and quicker to find mistakes. That's one of the reasons Luke has specified on HDL workflow to stick to small commits. (Andrey: I need to improve on this myself)
  3. Big changes are inherently risky. When LibreSOC was just a few people (Luke and Jacob), it was easier to keep track of each other's progress. 5 years down the line, the situation has changed. Keep in mind that changes to critical parts (whether big or small) will now affect at minimum Luke, Dmitry, Jacob, Sadoon, myself (perhaps also Cesar and Konstantinos, and so on). By going through the process of documenting a change in a new bug report, not only there's an opportunity to take a pause and think about repercussions, it also adds to the list of work for future grant applications (which will make it easier to draft focussed grants with realistic timescales and budget).

Procedure

  • If you discover a problem in code, raise a bug report, and use a corresponding 'importance' setting depending on how serious you perceive the issue to be. This will start a discussion.

No work is to be started yet.

  • Based on generated discussion, determine if the issue is a blocker to current tasks under budget. If it is a blocker , then the task 'importance' to be set to 'major' or 'critical (or 'blocker'). Andrey: need to clarify this If possible, a budget may be assigned after discussion and confirmation with Luke and Andrey (depends on remaining budget/tasks).

  • If the issue is not a blocker, but useful in future work, then the 'Future' milestone is to be assigned. The issue will be evaluated at a later stage. At this point, no further time should be spent on this issue (to prioritise outstanding tasks).

(Andrey): Sometimes determining whether to use WONTFIX or INVALID is difficult. Perhaps more examples would help?

  • If the issue is not a blocker, and the discussion shows that it is not an issue at all, it is to be set to either of the following:
    • RESOLVED DUPLICATE - If the issue raised already exists. Example, bug #962
    • RESOLVED WONTFIX - If the issue requires too much time or budget. Example, bug #921
    • RESOLVED INVALID - If the issue does not align with project goals or methodology. Example, bug #76

  • The final status will be confirmed after at least two other people (other than the reporter) look at the bug report. For cases considered to be WONTFIX or INVALID, 48h should be given before the bug report is closed. This ensures the team has enough time to see the discussion before the issue disappears.

  • Once the issue has been discussed and determined to be critical to current grant sub-task/s, and budget considered, then work can proceed in a separate branch. Only after fixes have been confirmed to keep the CI tests passing, can they be rebased (to keep commit history) into the master branch.

    • Note: branch names should not include personal info such as names. This is because others may require to use or work on the branch, and no single developer owns a branch.

This procedure adds a time delay between the issue discovery and start of work. This is important however because it allows for team members to read bug updates without being overwhelmed and have time to add input.