Chapter 7. Bulk importing in Red Hat Developer Hub

Procedure

1. The Bulk Import plugins are installed but disabled by default. To enable the ./dynamic-plugins/dist/red-hat-developer-hub-backstage-plugin-bulk-import-backend-dynamic and ./dynamic-plugins/dist/red-hat-developer-hub-backstage-plugin-bulk-import plugins, edit your dynamic-plugins.yaml with the following content:

   plugins:
     - package: ./dynamic-plugins/dist/red-hat-developer-hub-backstage-plugin-bulk-import-backend-dynamic
       disabled: false
       pluginConfig:
         bulkImport:
           importAPI: open-pull-requests
   
     - package: ./dynamic-plugins/dist/red-hat-developer-hub-backstage-plugin-bulk-import
       disabled: false
       pluginConfig:
         dynamicPlugins:
           frontend:
             red-hat-developer-hub.backstage-plugin-bulk-import:
               translationResources:
                 - importName: bulkImportTranslations
                   module: Alpha
                   ref: bulkImportTranslationRef
               appIcons:
                 - name: bulkImportIcon
                   importName: BulkImportIcon
               dynamicRoutes:
                 - path: /bulk-import
                   importName: BulkImportPage
                   menuItem:
                     icon: bulkImportIcon
                     text: Bulk import
                     textKey: menuItem.bulkImport

   See Installing and viewing plugins in Red Hat Developer Hub.

2. Configure the required bulk.import RBAC permission for the users who are not administrators as shown in the following code:

   rbac-policy.csv fragment

   p, role:default/bulk-import, bulk.import, use, allow
   g, user:default/<your_user>, role:default/bulk-import

   Note that only Developer Hub administrators or users with the bulk.import permission can use the Bulk Import feature. See Permission policies in Red Hat Developer Hub.

Verification

1. The sidebar displays a Bulk Import option.
2. The Bulk Import page shows a list of added GitHub repositories and GitLab projects.

Procedure

1. Click Bulk Import in the Developer Hub left sidebar.
2. If your RHDH instance has multiple source control tools configured, select GitHub from the Source control tool list. The interface displays GitHub repositories that your authenticated user account can access, excluding repositories already present in the catalog.

3. Select the repositories to import, and click Add.

   Developer Hub creates a pull request in each selected repository to add the required catalog-info.yaml file using your GitHub credentials.

4. For each repository to import, click PR to review and merge the changes in GitHub.

Verification

1. Click Bulk Import in the Developer Hub left sidebar.
2. Verify that each imported GitHub repository in the Selected repositories list has the status Waiting for approval or Imported.

3. For each Waiting for approval repository, click the pull request link to review and merge the catalog-info.yaml file in the corresponding repository.

   The pull request is created using your GitHub user account, ensuring proper attribution in the repository history.

Procedure

1. In the Developer Hub left sidebar, click Bulk Import.

2. If your RHDH instance has multiple source control tools configured, select GitLab from the Source control tool list.

   The interface displays GitLab projects that your authenticated user account can access, excluding projects already present in the catalog.

3. Select the projects to import, and click Add.

   Developer Hub creates a merge request in each selected project to add the required catalog-info.yaml file using your GitLab credentials.

4. For each project to import, click MR to review and merge the changes in GitLab.

Verification

1. Click Bulk Import in the Developer Hub left sidebar.
2. Verify that each imported GitLab project in the Selected projects list has the status Waiting for approval or Imported.

3. For projects with the Waiting for approval status, click the merge request link to review and add the catalog-info.yaml file to the project repository.

   The merge request is created using your GitLab user account, ensuring proper attribution in the project history.

Procedure

1. Access your Developer Hub backend logs where audit log events are recorded.

2. Review the following Bulk Import audit log events to monitor repository operations:

   BulkImportUnknownEndpoint

   Tracks requests to unknown endpoints.

   BulkImportPing

   Tracks GET requests to the /ping endpoint, which allows us to make sure the bulk import backend is up and running.

   BulkImportFindAllOrganizations

   Tracks GET requests to the /organizations endpoint, which returns the list of organizations accessible from all configured GitHub Integrations.

   BulkImportFindRepositoriesByOrganization

   Tracks GET requests to the /organizations/:orgName/repositories endpoint, which returns the list of repositories for the specified organization (accessible from any of the configured GitHub Integrations).

   BulkImportFindAllRepositories

   Tracks GET requests to the /repositories endpoint, which returns the list of repositories accessible from all configured GitHub Integrations.

   BulkImportFindAllImports

   Tracks GET requests to the /imports endpoint, which returns the list of existing import jobs along with their statuses.

   BulkImportCreateImportJobs

   Tracks POST requests to the /imports endpoint, which allows to submit requests to bulk-import one or many repositories into the Developer Hub catalog, by eventually creating import pull requests in the target repositories.

   BulkImportFindImportStatusByRepo

   Tracks GET requests to the /import/by-repo endpoint, which fetches details about the import job for the specified repository.

   BulkImportDeleteImportByRepo

   Tracks DELETE requests to the /import/by-repo endpoint, which deletes any existing import job for the specified repository, by closing any open import pull request that could have been created.

   Example audit log output:

   {
     "actor": {
       "actorId": "user:default/myuser",
       "hostname": "localhost",
       "ip": "::1",
       "userAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
     },
     "eventName": "BulkImportFindAllOrganizations",
     "isAuditLog": true,
     "level": "info",
     "message": "'get /organizations' endpoint hit by user:default/myuser",
     "meta": {},
     "plugin": "bulk-import",
     "request": {
       "body": {},
       "method": "GET",
       "params": {},
       "query": {
         "pagePerIntegration": "1",
         "sizePerIntegration": "5"
       },
       "url": "/api/bulk-import/organizations?pagePerIntegration=1&sizePerIntegration=5"
     },
     "response": {
       "status": 200
     },
     "service": "backstage",
     "stage": "completion",
     "status": "succeeded",
     "timestamp": "2024-08-26 16:41:02"
   }

Procedure

1. Configure the Bulk Import plugin by editing your app-config.yaml file to enable Orchestrator mode.

   bulkImport:
     orchestratorWorkflow: your_workflow_id
     importAPI: 'orchestrator'

   where:

   orchestratorWorkflow

   The ID of the workflow to run for each repository.

   importAPI

   The execution mode for the workflow. Enter orchestrator to enable workflow execution.

2. Verify that the Orchestrator workflow receives the following input:

   {
     "inputData": {
       "owner": "redhat-developer",
       "repo": "rhdh-plugins",
       "baseBranch": "main",
       "targetBranch": "bulk-import-orchestrator"
     },
     "authTokens": [
       {
         "token": "<github_token>",
         "provider": "github"
       }
     ]
   }

   where:

   owner

   Specifies the repository owner (organization or user name).

   repo

   Specifies the repository name.

   baseBranch

   Specifies the default branch of the Git repository (for example, main).

   targetBranch

   Specifies the target branch for the import operation. By default, this is set to bulk-import-orchestrator.

   authTokens

   Specifies the authentication tokens for the Git provider:

3. For GitHub: { token: <github_token>, provider: github }
4. For GitLab: { token: <gitlab_token>, provider: gitlab }

5. Navigate to the Bulk Import page in the sidebar and complete the following steps:

   1. Select your Git provider (for example, GitHub or GitLab).
   2. Select the projects you want to import.
6. Click import to run the workflow.