Contributor Minutes on 2020-06-14

Contributor Minutes on 2020-06-14

Attendees

• Zixing Liu
• Leo Shen
• Gumblex

Agenda

• OSPP (Continued)
• AOINB

Minutes

OSPP (Continued)

• Liu: Bai's not here today, so I want to ask is there any student sending their proposal email your way? Currently I know there is one interested in localization project and one interested in automated testing.
• Gumblex: Well yes, but they lost contact afterwards.
• Liu: The deadline for enrolment is June 20, so any pending contacts should hurry up.
• Leo: So I guess there is no other student enrolled in any other projects.
• Liu: We will be reviewing the proposals from June 21 to June 28.

AOINB

• Liu: I have compiled a list of questions for you based on the messaging history. I would like to go through those questions.

AOINB - Abstract Data Types

• Liu: Have you decided on the basic data structures?
  • Leo: Well... I am using Websockets... And it worked fine?
  • Gumblex: I think we don't need to use Websocket, normal http requests would fit the bill nicely.
  • Liu: Lion said he would like to use Zero MQ. But let's set the protocol implementation aside for now.
• Liu: I think it is important to first finalize or at least fix the abstract data types as this will greatly affect all the future designs. So have you decided detailed structures for the metadata and etc?
  • Leo: I am working on this alone... therefore... it's been designed carelessly...
  • Gumblex: I feel like I am in a competitive position to Leo.
  • Liu: That's not good. This project meant to be cooperative.
  • Liu: Okay, so Leo asked the question about the ADT before on where to store those things. I think it probably doesn't matter in Python.
• Gumblex: Which "basic data structures" are you referring to?
  • Liu: The data representation format, for example, how to represent packages on scheduler and so forth.
• Liu: I hope we can reach an agreement on this as this is a cooperative project.
  • Leo: My hope as well.
• Liu: So I can see there's the recursive dependency of Package class in the Package class. Is that intentional?
  • Gumblex: You obviously can't do that.
  • Liu: The alternative is to have a list of package names I guess?
  • Leo: Yeah, that's how it is going to be implemented.
  • Gumblex: If I were you, I would use the ones in abbs-meta. Also your current implementation is not considering sub-packages and package groups.
    • Leo: Alright. Honestly I didn't know these even existed.
  • Liu: Also, for sub-packages, you need to build them sequentially in the same folder.
    • Leo: ... I really didn't know that.
    • Liu: Write that down then. Any other issues you can identify?
• Gumblex & Leo: That's everything.

AOINB - Workflow Changes

• Liu: Yesterday we have discussed that we might put a Git server on the scheduler. It's a common choice to commit your changes first so that the CI will build using your changes afterwards. Therefore, our current workflow would change since you can't make test builds on the build agents anymore.
  • Leo: Yes. You can upload a patch to the Git server.
  • Gumblex: That's part of the original design too.
• Liu: So we going to put a copy of the package tree somewhere, should it be scheduler? And then, what if someone submitted a patch that conflicts with another person's patch?
  • Leo: The job associated with the offending patches will be cancelled, per previously discussed.
  • Gumblex: Well, we make another copy of the tree so there can't be any conflicts. You know, using CoW (copy-on-write) implementations or even simply cp -ar.
  • Leo: That's way too complicated for me.
  • Liu: If that's difficult, then we can first making it rejecting patches. We can always change this behavior later.
  • Liu: Time for another question. Taking the issues with the Chromium on ARM64 for instance, I need to access the environment to debug the problem since I may not own such devices. In which case I need to have direct access to the container. How can we do that?
  • Leo: You can always ssh into the builder I guess.
• Liu: How about the build data? What if I want to examine the build folder? If you want an example on how the others did this, take a look at Circle CI, where they keep the container for 60~120 minutes if you want.
  • Gumblex & Leo: (combined paraphase) We can keep those on a counting basis. For instance, keeping the latest N failures, where N varies according to the disk space available on the agent.

AOINB - Mutual Communication Between Scheduler and Builder

• Liu: Now we can talk about the protocols. Why I didn't talk about this earlier? Because we need a basis on what we have, what we know, and more importantly, what is known to the scheduler and builder. So that they can communicate with each other (with what they themselves don't know).
• Liu: I already seen the document provided by Leo. So the "hello" message is indicating "I am coming"?
  • Leo: Correct. I will add authentication data into that later. My current implementation relies very much on Websocket.
  • Gumblex: Which you definitely could split into separate simple HTTP requests.
  • Leo: Well some of them requires scheduler to builder communications.
  • Gumblex: You can always make builder actively sending requests to scheduler for updates.
  • Liu: Did you mean long-polling?
  • Leo: Hah. That's making me more want to use Websocket.
  • Gumblex: How do you resolve the issues with Websocket disconnections?
    • Leo: The library I have chosen can handle that transparently.
    • Liu: You can use heartbeat packets like many Javascript libraries.
  • Gumblex: My argument is that using simple HTTP request can ensure we don't need to keep track of the states of the connections or requests.
    • Leo: I am familar of the Websocket. So it seems to me that Websocket is easier.
  • Liu: Lion said we should use Zero MQ. It's light-weight and efficient.
    • Leo: I am not familar with this, so it's more complicated to me.
    • Gumblex: I only use Zero MQ for inter-process communication.
• Liu: So in conclusion, Websocket and plain HTTP request are our options. Both are very good candidates. Websocket's strength is having a long-connection for less latency, HTTP request for easier server and client code.
  • Gumblex: We don't need low-latency in our case so much. And we need to transfer huge files like payload bundles, which Websocket would struggle to handle.
  • Liu: You don't say. How do you handle network interruptions when Websocket is currently transferring that huge payload?
  • Gumblex: And builder can report their status or statistics back to scheduler when doing the long-polling.
  • Liu: Or you can send status and statistics in the heartbeat packet in Websocket.
  • Leo: That's... why we want to do this in the Websocket implementation?
  • Liu & Leo & Gumblex: (talking about issues with async mechanism in Python)
• Leo: (to Gumblex) Merge reorganize branch into master first?
  • Gumblex: Just directly merge it?
  • Leo: Yep. Hmm... I realized you can actually implement those with plain HTTP requests. But I need the interface documentation from you (Gumblex).
  • Gumblex: Will do. But I don't know how to write Org-mode format.
  • Leo: I will convert that.
• Leo: ... Speaking of that... Let's settle on what to use in unison? We are going in different directions right now.
• Liu: Indeed. How about we just use plain HTTP requests?
  • Leo: Well, I don't like to use that library actually. Hmm. Yeah. What library should I use for the HTTP server?
  • Gumblex: Use bottle.
• Liu: I have one question left, but it seems like it was answered previously so I will skip that.
• Liu: Any other questions?
• Gumblex: About authentication. I have already prepared the implemention using nacl, which is a super nice crypto library.
  • Leo: We can insert the authentication data somewhere.
  • Liu: Headers? Query parameters?
  • Gumblex: Doesn't matter, as long as it exists.
  • Liu: I suggest put it in the header like many others do.
  • Leo: Alright.

Actionables

• Leo: Convert all Org-mode format documents to Markdown format.
• Gumblex: Undetermined for now.