Virtual Labs Hosting Process

Introduction

Virtual Labs is a mission mode project initiated by the Ministry of Human Resources and Development (MHRD). The objective of this project is to provide laboratory learning experience to the students who do not have access to adequate laboratory infrastructure. Currently there are around 90 labs developed by various institutes in the consortium. A streamlined software development life cycle process followed for the development of these labs ensures high quality labs. This document defines the hosting process to be followed by the developers (open source community) of the Virtual Labs project.

Purpose

This document defines the experiment on-boarding and hosting process that should be followed by the developers of the lab while requesting the hosting of their lab on AWS. The roles and responsibilities of the various parties involved in the hosting process is also discussed in detailed in the document.

Motivation

A well defined experiment hosting and on-boarding process will help maintain a consistent user experience and enable experiment-authors to focus on the content. Consolidated information regarding all the deployments will also facilitate reporting.

Audience

The target audience for this document is the hosting team at CPE, IIITH and all the lab authors and owners who want to avail the hosting service provided by Central Platform Engineering Team (CPE), IIITH team.

Definitions and Pre-requisites

Virtual Labs hosting process will require certain pre-requisites to be met by the experiments that need to be hosted. The following paragraphs define terminologies used during the hosting process.

Hosting

Hosting is a service provided by the Central Platform Engineering (CPE) which includes the following:

1. Providing infrastructure including all dependencies required to run the Lab/Experiments as specified in the Onboarding request of the hosting unit.
2. Ensuring round the clock availability of the Lab/Experiments.
3. Ensuring that a minimum acceptable performance criteria are met by the Lab/Experiments.

Following are not part of the hosting service:

1. Testing: The deployable code is a black box for the CPE team. The CPE team is not responsible for testing the deployments. The developer/owner of the Lab/Experiment is responsible for confirming that the Lab/Experiments are deployed and functioning as intended. This confirmation is part of the hosting process.
2. Implementation: The CPE team is not responsible for any bug fixes or enhancements or implementation on the Lab/Experiment. The developers should ensure that their product is complete and functional before raising a hosting request.
3. Approvals: The CPE team does not provide any approvals regarding the completeness of the Lab/Experiment. The developers are expected to have sought the necessary approvals regarding completeness of the Lab/Experiment before raising a hosting request.

Hosting Unit

Hosting unit is the code base/repository that will hosted in its enirety during the hosting process. It will not be possible for a part of a hosting unit to be hosted separately. A hosting unit will correspond to one public repository.A hosting unit could be a stand alone experiment or could be a lab consisting of a set of related experiments.

Repository Owner

Each repository, corresponding to a hosting unit, will have one designated primary owner. This owner will be the single entity responsible for all the code that resides in the repository. The owner will also be the final approval authority for a hosting request. The owner should ensure that the master branch of the repository only contains the code that needs to be hosted. The latest code in the master branch should represent the hosted experiment.

The primary repository owner does not have to be a person. It may be any uniquely identifiable entity. From the perpective of Virtual Labs hosting team, this entity will the single point of contact for all clarifications. The hosting team will not be responsible for any confusions arising due to shared accessibility of the repo owner entity.

Requester

A repository owner or a developer who raises a hosting request.

Tags

The repository owner will be responsible for tagging each merge to the master branch. The owner will have to follow Semantic Versioning . In short, each version is a combination of three numbers (MAJOR.MINOR.PATCH) separated by dots. The changes to these numbers represent the following:

1. MAJOR: A change incompatible with previous versions.
2. MINOR: A backwards compatible new feature.
3. PATCH: A backwards compatible bug fix.

All Virtual Labs hosting request should specify a tag.

Build Process

Each hosting unit should have a well defined build process. The hosting team will treat the build process as a black box action. This action will be triggerred to generate the artifacts to deploy. The repository owner should ensure that the build process is well documented and functioning as desired before raising a hosting request.

Testing

The hosting process will result in a deployment with an accessible url. The responsibility of testing the deployments will lie with the repository owner. The hosting team will be responsible for providing the link to the deployment. The hosting team will not provide any testing environment. The repository owner will be responsible for ensuring that the hosting unit works as expected in their own specific testing environments.

Hosting Process

Virtual Labs hosting process consists of two steps -

1. Request the onboarding of a hosting unit
2. Request the hosting of hosting unit

On-Boarding process of hosting unit

Each hosting unit that needs to be hosted by the Virtual Labs hosting team, must follow an onboarding process. The repository owner of the hosting unit will need to raise an issue of type on-boarding request in the engineers-forum. The issue will need to filled with the following pieces of information :

On-Boarding Request -

1. A non-conflicting name for the hosting unit.
2. Repository URL for the hosting unit.
3. A designated repository owner with the following details: a. Repository owner github id b. Repository owner email address
4. The build command for the hosting unit along with the link to the build process documentation.
5. Hardware specification including: a. CPU b. Memory c. Network bandwidth
6. Software environment specification including: a. Host operating system with minimum compatible version b. Any additional software packages with minimum compatile versions c. Browser plugins required to view experiment d. DB

Repository owner will be responsible for keeping the above information current and intimating the hosting team of any updates/changes as and when needed.

On receipt of an onboarding request the hosting team will verify the data and acknowledge the same as part of the request.

The hosting team will be responsible for storing the this information at a central location and making it available for all stakeholders.

Hosting Process of hosting unit

A repository owner/developer (requester ) will have to raise an issue of type hosting request in the GitHub repository engineers-forum under Virtual-Labs organisation for hosting of a hosting unit. The issue will need to filled with the following pieces of information :

1. Repository URL
2. Branch to be deployed
3. Tag to be deployed
4. Contact email address and github handle to be used in cases of clarifications

A hosting request will be tied to this information. This issue will be a single source of truth for that hosting request. All communication related to the hosting will be recorded on the raised hosting request.

Lifecycle of a Hosting Request

The hosting request will go through the following lifecycle:

1. Requester will raise a hosting request issue by duly filling in the details in the hosting request.
2. The hosting team will attach the url of the corresponding onboarding request to the hosting request. If the onboarding request is not found ( by searching on the repo url), then the hosting team will add its findings and label the issue as Failed.
3. If requester is not the repository owner, the hosting team will seek approval from the repository owner. This will be through the above created GitHub hosting request issue.
4. The hosting team will honour the hosting request only on receiving the approval from repository owner else will label the hosting request issue as Not Approved
5. On a successful hosting, the hosting team will add the link of the newly hosted content to the hosting request issue. They will also label the issue as Hosted.
6. However, on failure ( because of missing or incorrect information or repository problems,), the hosting team will label the issue as Failed.
7. The requester will be responsible for fixing the failed hosting bug and will need to raise a new hosting request to get the unit hosted.
8. Once the hosting is successful, the requester will be responsible for the verification of the hosted link. If the hosting is not as expected, the requester should change the label from Hosted to Reopened. The requester will also need to specify if he/she would like to revert to the previous hosted image.
9. An issue with Reopened label will by default host the repository and branch/tag as specified in the hosting request initially. However, if the requester has specifically requested a revert, the previously hosted image will be restored and issue will be labelled as Reverted.
10. If the requester wants to revert to any earlier branch/tag, a new hosting request will need to be raised to get the unit hosted.
11. Requester will be responsible for closing a hosting request issue by changing the issue status from Open to Closed when the issue is labelled as Hosted or Reopened or Reverted or Not Approved.

Labels & Status of a Hosting Request

At any given time a hosting issue should be marked with only one of the following labels. To change the label of an issue, the current label of the issue should be unchecked and the new label should be checked.

Hosted : This label indicates that the hosting request has been successful and the hosted url has been shared in the hosting request issue. This label is used only by the hosting team.

Failed : This label indicates that the hosting request has not been successful. This label is used only by the hosting team.

Reopened : This label is used by the requester to indicate that the hosting url provided on a successful hosting of the hosting unit is not passing the validation.

Not Approved : This label indicates that the hosting request issue raised by the requester was not approved by the Repository Owner. This label is used only by the hosting team.

Reverted : This label indicates that the hosting request has been successfully revert to the previous hosted image. This label is used only by the hosting team.

Apart from the above mentioned labels a hosting request issue can either be in a closed or open status as provided by GitHub.

Open : This status indicates that a new hosting request issue has been raised and is waiting for the services of the hosting team. All issues are in this status when they are created on GitHub.

Closed : This status indicates that a hosting request has been serviced by the hosting team and is labelled as Hosted or Reopened or Reverted or Not Approved.

Hosting Process for Phase III Experiment/Labs

Keeping the deliverables/due date and the current hosting model in mind, the following process will need to be followed for getting a Lab hosted on AWS during the Phase III of the Virtual Labs project ( ending on March 31st 2020). This is as per many discussions with IITB and the meeting on [2020-03-03 Tue].

VLEAD and IITB will follow the process detailed below for the hosting of labs which have been developed with each experiment having its own repository. All the Phase 3 labs and experiments hosted on AWS will have a common UI and report analytics at both lab and experiment level.

Step 1

IIITH will create lab repo for the lab that is ready ( marked in green) in the 200224_Deliverable_Status_Phase3 sheet shared by IITB and also populate the lab urls in the sheet.

Step 2

IITB will raise an issue of type Phase 3 Onboarding Request and fill the lab name and lab url ( as populated in the excel sheet by IIITH) and attach the R0 of the lab.

Step 3

IIITH will tag the head of the master branch of all the experiments in that lab as mentioned in the 200224_Deliverable_Status_Phase3 as v1.0.0

Step 4

IIITH will populate the received Phase 3 Onboarding Request with experiment details ( name, url, tag, branch and build command) and Owner details and request for approval from IITB . It will be assumed by IIITH that no special hardware or software is required for the running of the experiments. IITB will have to specially state if this is not the case.

Step 5

IITB will validate the above data and approve the same on the received Phase 3 Onboarding Request .

Step 6

On approval from IITB , IIITH will host all the experiments and populate the lab landing page information (Introduction, Objective, List of experiments, target audience, course alignment ) from the R0 shared by IITB in the Phase 3 Onboarding Request.

Step 7

IIITH will host the lab and share the hosted url link as part of the onboarding request issue and seek approval from the github handle of the owner of the lab ( as recorded in the onboarding request)

Step 8

On approval from the Lab owner, IIITH will share the link with IIT Delhi.

Rehosting Process for Phase III Lab

VLEAD and the institute requesting the rehosting of Phase 3 lab while changing the content of the lab will follow the process detailed below-

Step 1

Institute requesting rehosting will follow the link and locate the Phase III Lab/Experiment(s) OnBoarding Request issue corresponding to the lab that they wish to rehost.

Step 2

They will comment on the issue with the content that they like to add to the lab pages following the format - Lab Section Name - New content

Step 3

VLEAD's hosting engineer will host the lab with the requested changes and share the link on the same issue to seek approval from institute requesting rehosting.

Step 4

Institute requesting rehosting will need to test the hosted link and on their approval the hosted link will be shared with IITD (to be added to vlab.co.in).

Rehosting Process for Phase III Experiments

VLEAD and the institute requesting the rehosting of Phase 3 lab after changing the content of experiment will follow the process detailed below-

Step 1

Institute requesting rehosting will make the necessary changes to the experiment content in the experiment repository on GitLab ( IITB server ) or Github (virtual-labs organization) as per the process followed during experiment development. After they make the changes to the experiment sources , they will need to tag it.

Step 2

Then will then follow the link and locate the Phase III Lab/Experiment(s) OnBoarding Request issue corresponding to the lab containing the experiment that they wish to rehost.

Step 3

They will comment on the same Phase III Lab/Experiment(s) OnBoarding Request with the following text "Request to rehost the experiment <exp repo link> with the tag <tag number>". They may include more than one experiment ( belonging to the same lab) rehosting request at a time.

Step 4

VLEAD's hosting engineer will host the experiments and share the link on the same issue to seek approval from them.

Step 5

Institute requesting rehosting will need to test the hosted link and comment on the same issue stating their approval.

Moving Phase 3 experiment repositories from IITB GitLab Server to virtual-labs Github organization

VLEAD and the institute requesting the movement of Phase 3 experiment repositories from IITB Gitlab Server to virtual-labs GitHub organization will follow the process detailed below-

Step 1

Institute requesting movement will raise an issue to request creation of experiment repositories for the lab.

Step 2

They will fill the information requested for on the issue and attach R0 ( proposal of the lab)

Step 3

VLEAD's hosting engineer will create experiment repositories for each of the experiment on the proposal on Github and share the links of the created repositories on the same issue.

Step 4

Institute requesting movement will populate the dev branches of the created repositories with the source code of the experiments ( same phase 3 structure as defined in the IIT B exp repos) and unit test the experiments locally.

Step 5

Institute requesting movement will merge the fully tested dev branch to testing branch. This will automatically deploy the experiment along with UI on GitHub pages for testing the complete experiment. The Institute requesting movement will receive an email about the success/failure of the deployment on the email id associated with the github handle provided in the issue.

Step 6

Institute requesting movement will then merge from a fully tested testing branch to the main branch and tag the commit.

Step 7

Institute requesting rehosting will then follow the link and locate the Phase III Lab/Experiment(s) OnBoarding Request issue corresponding to the lab that they wish to rehost. If such an issue does not exist they will create will raise an issue of type Phase 3 Onboarding Request. In the issue, for each experiment they will provide the Experiment Name|Experiment Source Repo link|Tag . They will also attach the R0 of the lab to the issue if not already attached.

Step 8

VLEAD's hosting engineer will host the experiments and the lab and share the link on the same issue to seek approval from them.

Step 9

Institute requesting momement will need to test the hosted link and comment on the same issue stating their approval.

Creation of Experiment Repositories

Steps to be followed by VLEAD's Hosting Engineer

Step 1

On receiving a request for creation of experiment repositories for the lab, VLEAD's hosting engineer will create the experiment(s) repository from the ph3-exp-template repository.

Step 2

VLEAD's hosting engineer will create two branches - dev and testing from the main branch in the created experiment repositories.

Step 3

VLEAD's hosting engineer will add the Github Handle of the experiment developer provided in the request issue to the newly created experiment repository with maintainer access. VLEAD's hosting engineer will also add developer's common email address to github notification center

Step 4

VLEAD's hosting engineer will comment on the repository creation request providing the links to the created repositories.

Central Hosting Data

The following information will be stored at a central place for a quick reference by the hosting team:

1. Hosting unit name
2. Repository URL
3. Currently hosted Branch/Tag
4. Previously hosted Branch/Tag
5. Date/Time of hosting

Please follow the link for the Central Hosting Data.

Flow Diagram of the Hosting Process

The flow diagram depicting lifecycle of a hosting request is below:

image

Roles and Responsibilities

S.No.	Task	Responsibility
1.	Raising an On-Boarding Request	Repository owner
2.	Tagging the hosting unit release in the	Requester
	master branch while raising hosting request
3.	Testing the application before	Requester
	making hosting request
4.	Raising the hosting request and	Requester
	providing all required information
5.	Seeking approval of the repository owner	Hosting team
	if the request is raised by someone other
	than the repository owner.
	Labelling the hosting request issue as
	Not Approved if the request is not
	approved by the repository owner
6.	Validating the information provided in the	Hosting team
	hosting and on-boarding request
7.	Running the hosting process	Hosting team
8.	Providing the hosted link to the	Hosting team
	requester and labelling the hosting
	request issue to Hosted
9.	Providing the causes/nature of failure	Hosting team
	and labelling the hosting request issue to
	Failed
10.	Validating the hosted code	Requester
11.	Labelling the hosting request issue as	Requester
	Reopened if validation is unsuccessful
12.	Reverting the hosting unit to previous hosted	Hosting team
	image if validation fails and
	labelling the issue as Reverted
13.	Changing the status to Closed if the	Requester
	request is labelled Hosted or Reverted
	or Not Approved or Failed

Repository Creation of a Service

Steps to be followed by VLEAD's Hosting Engineer while creating a service repository

Step 1 On receiving a request for creation of service repository, VLEAD's hosting engineer will create the service repository from the svc-template repository.

Step 2 VLEAD's hosting engineer will create two branches - dev and testing from the main branch in the created service repository.

Step 3 VLEAD's hosting engineer will add the Github Handle of the developer provided in the request issue to the newly created service repository with maintainer access.

Step 4 VLEAD's hosting engineer will comment on the repository creation request providing the links to the created repositories.

Hosting a service

Steps to be followed by VLEAD's Hosting Engineer while hosting a service

FAQ

Why should I use the Hosting Service?

To get more for nothing. Hosting Service provides you a high-availability, high-performance deployment environment. You do not have to spend time, money, people or other resources to ensure that your website is available to your target audience. You can also take advantage of other services, like Analytics, provided by the Central Platform Engineering team and monitor multiple statistics about the users and usage patterns of your Lab.

Do I get better website performance if I use Hosting Service?

Almost all the time. The CPE team works continuously to improve the overall and individual Lab/Experiment performance based on the Analytics data. You can compare the performance of the Labs/Experiments deployed through the hosting service to those deployed elsewhere. If you use the Analytics service integration, the CPE team can individually target your Lab/Experiment for better performance.

Does the CPE team help me improve my Lab performance?

Yes. The CPE team continuously works to improve the infrastructure in order to achieve best performance results for your Lab. The CPE team does not improve your code, however, it can provide you guidelines and actionable performance improvement advice.

Does Hosting Service add a UI to my webpages?

No. Hosting service does not alter your code in any way. As the developer, you are responsible for ensuring that your Lab/Experiment is complete in all respects before you request it to be hosted. Hosting service is strictly limited to its stated objectives.

How can I get Analytics for my experiments?

By working with the CPE team. Analytics service is another service provided by the CPE team. You can work with the CPE team to integrate Analytics service into your Labs. You can also get access to customized dashboards/reports in order to monitor multiple statistics about the users and usage patterns of your Lab.

Does the CPE team test my Lab for correctness as part of hosting?

No. It is not possible for the CPE team to test your Lab because only you understand the right definition of correctness for your Lab/Experiment. As part of hosting, the CPE team ensures that your Lab is up and accessible. You are responsible for ensuring the completeness and correctness of your code and deployment.

Will the CPE team tell me about broken links as part of hosting?

No. Your code is a complete blackbox for the CPE team. You are responsible for ensuring the completeness and correctness of your code and deployment.

Conclusion

This document highlights the hosting process of Virtual Labs from on-boarding to hosting. It aims at collaborating contributions of the developers across the open source platform - GitHub. This streamlined process would help in our strive for excellence in delivering high standard labs.

Glossary

• Repository : The distributed version control system - git is used here. Every time the term repository or repo is used, it refers to a git repository. A repository is an on-disk data structure which stores metadata for a set of files and/or directory structure. The whole set of information in the repository may be duplicated on every user's system or may be maintained on a single server.
• Virtual Labs : Virtual Labs is an initiative by MHRD under NMEICT. The objective of this project is to make engineering education engaging, enjoyable, immersive and online.
• Hosting team : VLEAD at IIITH which provides Virtual Labs hosting service