Umbrella Issue: Project GitHub integration flow

Problem

This issue should serve as an umbrella issue to discuss and implement the full GitHub integrations flow for connecting GitHub repositories to a Code Corps project.

Integrations flow

Project integrations page

• Each project has an integrations tab, only visible to the project owner
• The integrations tab has a GitHub section
• Show if the user is already connected to GitHub and:
  • If not, display a button to connect their GitHub account.
  • If so, display a button to add the project GitHub integration.

The user’s GitHub id is used to match the webhook to the User and the originating Project. Once the user has returned from GitHub, we can use the originating project’s id to redirect the user to the integrations tab on the project page. Unfortunately, GitHub does not currently have a better redirect flow to the requesting application.

Redirect to GitHub

• Creates a GitHubAppInstallation record on the API (see model details below)
• [ ] Redirects to GitHub
  • the user selects where to install (user or organization)
  • the user selects repositories
  • the user clicks install

Webhooks

• Receive and handle webhook from GitHub API of installation event with id
  • Find existing GitHubAppInstallation for the GitHub user
  • Requests repositories from GitHub’s https://api.github.com/installation/repositories endpoint using the GitHub app’s installation token. Use these repositories to create or update GitHubRepo records and ProjectGitHubRepo records.
• Listen for installation_repositories webhook which will notify you of added/removed repositories on an installation. Again, use this to create or update GitHubRepo records and ProjectGitHubRepo records.

Back to client process

• User is redirected back to Code Corps /integrations/github
• Query parameters should contain an installation_id. Retrieve the user’s only GitHubAppInstallation with a status from the API.

Instead, we opted to open the github page in a new window and instruct the user to come back and refresh. Simpler to deal with to begin with. Can improve on later.

Handling the statuses and other edge cases

There are various status types that we can get back from the API depending on where the user was in the installation process.

• initiated_on_code_corps - we haven’t received the webhook yet. Ideally we would ping the API for changes or wait for a channel to complete, but for now, we can have the user click refresh until the application fetches new model data. We should also warn them not to close their browser or navigate away.
• processing - we still need to fetch all the repos from the API to display on the project integrations page. As above, ideally we would ping the API for changes or wait for a channel to complete, but for now, we can have the user click refresh until the application fetches new model data.
• processed - Redirect them to the project integrations page.
• initiated_on_github - we received an installation webhook with a matching user, but an installation was never created on Code Corps. Edge case for now, but can be handled by asking them to identify a project where they are an owner in order to complete the process.
• unmatched_user - when we receive a webhook but there is no such Code Corps user with the given GitHub id. Edge case for now. This should only happen when the installation was initiated on GitHub. We can handle this by asking a user to connect with GitHub after the fact, and in the meantime matching the installation up to that user. Once the user has connected, we can move the status of the installation to initiated_on_github so they can pick a project.
• Missing installation_id in the query params. Edge case for now, but can be handled by providing an error message on the /integrations/github page specifically.

This has been made simpler instead and split up into separate fields, since we track multiple different statuses here

Redirecting to the project integrations page

• The GitHubAppInstallation record will have a Project relationship; we should redirect to that project’s “integrations” tab.
• On the integrations tab, we list the GitHubRepos for the project via the ProjectGitHubRepo relationships. None of them should be followed yet. The user can refresh
• There could be a case where the webhook has not been handled yet, so we may want to show a status for each, with an option to refresh
• Next to each repo is a button to connect the project with that repo. This creates a ProjectGitHubRepo on our API and we start handling webhooks. We call this “following a repo”. (see further notes on the need to create a separate model here).

Again, simplified a good amount. We list installations with the option to install more.

For each installation, if there is an OrganizationGithubAppInstallation, we display that installation as connected and list repos for that installation.

For each repo, if there is a ProjectGithubRepo, we display that repo as connected, with the option to remove (deletes ProjectGithubRepo). If the repo is not connected, there is an option to connect (creates ProjectGithubRepo).

API

Webhooks

See above for some specific webhook flows, and the discussion below of GitHubEvent.

Once a project is connected, we use webhooks to determine which projects will receive changes from issue and issue_comment events.

Models

GitHubAppInstallation

• user - the user who initiated the install. Matched to a user by the sender id in the installation webhook.
• originating_project - the project where the installation was initiated. Used to redirect back to the project’s integrations page to complete the installation process. There’s not necessarily a project for a given installation, as an installation really exists only on a per-GitHub account basis for a given GitHub app.
• state
  • initiated_on_code_corps - the user clicks from Code Corps’ UI to go to GitHub and install
  • initiated_on_github - we receive an installation webhook with a matching user, but an installation was never created on Code Corps
  • processing - when the installation webhook is received, was matched, and is now processing
  • processed - the integration process is completed from the perspective of there being an installation on the project.
  • unmatched_user - when we receive a webhook but there is no such Code Corps user with the given GitHub id

A user can only have one GitHubAppInstallation in the initiated state at any given time. This prevents a case where a user could unintentionally have two pending installations at the same time for different projects – we wouldn’t know which project to relate everything to by default.

GitHubEvent

• action - under the action key in the payload, e.g. created
• source - application, application oauth, user oauth
• status
  • errored
  • ignored
  • processing
  • processed
  • unhandled
  • unprocessed
• type - can be matched to either the "X-GitHub-Event" HTTP header or one of the JSON keys in the payload, e.g. installation or issue

Explicitly ignore some events with reasoning; we can just log the ignored actions.

GitHubRepo

• github_id
• github_account_id - the owner account’s GitHub id
• github_account_login - the owner account’s GitHub username
• github_account_avatar_url
• github_account_type - e.g. Organization or User

ProjectGitHubRepo

• github_repo
• project
• following - whether the repo is currently being followed by the project. When a user selects repos in the GitHub installation process manually, we will auto-follow all of the repositories.

Controllers

• GitHubAppInstallation has :create, :show, :index actions at a minimum (:index for filtering by the user’s installation, or we could have a special endpoint to return the latest relevant one as explained above).
• GitHubRepo with :index (for filtering) and :show.
• ProjectGitHubRepo :index, :show, :create, :delete

[begedin]

@LennonR I’m moving your tracking ideas into a separate issue.

@JoshSmith I believe this umbrella issue is in a good enough spot to be closed.

[joshsmith]

@begedin my big concern now is that we write some of the big picture “how it works” in documentation somewhere, since that’s spread across the client and server and not, I think, easily explained in summary form for a contributor to step into and explore.