Project

General

Profile

Feature #4024

[ce] Integrations/Webhooks

Added by Daniel D over 4 years ago. Updated about 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
15.06.2016
Due date:
% Done:

0%

Estimated time:
Sorting:
Commit Number:

Description

Currently there is a WIP to implement integrations in the rhodecode ui, in a way that is simple for an end user to configure.

There are a few tasks which must be done to make this work:

  1. Create a unified first class citizen event object system, these should be generated in relevant models eg. PullrequestModel.create to create and fire PullRequestCreateEvent(pullrequest) - they should also gather relevant environment data such as request context - each event has a string describing it, of the form 'repo.push' and a special wildcard '*' string will match any event

  2. These objects should expose a json dict which describe the entirety of the event - such as repo (plus repo data), user, relevant commits etc. - It would be a good idea to use the data structs returned by the rhodecode api, eg. {'event': 'pullrequest.create', 'repo': repo.get_repo_api_data()}

  3. This json dict is the basis of all integrations/webhooks - it must be as complete as possible to allow any integration/webhook to utilize the data without needing further apicalls/lookups (later we can allow post_processors "extensions" to be registered on the events - to let them add extra data).

  4. Now that we have a basis to describe an event - we can work on hooks - A table of event hooks of the form:

CREATE TABLE event_hooks (
      event_type VARCHAR,
      integration_id INT
)

and an integrations table of the form:

CREATE TABLE integrations (
      integration_id INT primary_key,

      # this is a string which is used as a
      # discriminator to load the correct integration
      # eg. 'slack' => SlackIntegration
      integration_type VARCHAR,

      # this is a json string which exposes settings for the integration
      # these will be exposed in the user interface as a form
      integration_settings_json VARCHAR

      # when this is null, will apply the integration on any repo
      # instead of having to create a new integration per repo
      # you can make all repo.push use same integration
      # it will also be null for global type events such as `user.create`
      repo_id INT,
)

which maps eg. events => integrations, where integration will be database objects that inherit from a base Integration class that define how to interact with the incoming json data - and do the analogous integration - for example with these values:

+------------+----------------+
| event_type | integration_id |
+------------+----------------+
| repo.push  |              1 |
| repo.pull  |              1 |
| repo.pull  |              2 |
| *          |              3 |
+------------+----------------+
+----------------+---------+------------------+---------------------------------+
| integration_id | repo_id | integration_type |    integration_settings_json    |
+----------------+---------+------------------+---------------------------------+
|              1 | 1       | slack            | '{"channel": "#develop"}'       |
|              2 | null    | slack            | '{"channel": "#general"}'       |
|              3 | null    | webhook          | '{"url": "http://logs-server"}' |
+----------------+---------+------------------+---------------------------------+

A push to repo(id=1) will fire two integrations:

  • integration(id=1) - sends a message to slack channel #develop
  • integration(id=3) - POSTS the repo push json to http://logs-server

A pull to repo(id=1) will fire three integrations

  • integration(id=1) - sends a message to slack channel #develop
  • integration(id=2) - sends a message to slack channel #general
  • integration(id=3) - POSTS the repo push json to http://logs-server

A pull to repo(id=2) will fire two integrations

  • integration(id=2) - sends a message to slack channel #general
  • integration(id=3) - POSTS the repo push json to http://logs-server

In fact any event will be posted to the http://log-server - and any pull event will be sent to slack #general.

History

#1 Updated by Daniel D over 4 years ago

The Integration base class should follow a pyramid class structure similar to auth plugins that makes it auto expose in the admin making new integrations easy to add

#2 Updated by Marcin Kuzminski [CTO] over 4 years ago

few notes:

  • could wildcard work like glob patterns, so one could use repo:*
  • would be cool if you can define an events for repo group as well. When we did per-repo settings we prepared some work for to make it also work for per-repo groups. However i'm not sure if this could be fast enough to determine an event for a member of groups (they could be nested indefinetly)

  • i'm not sure if we should use repo.get_repo_api_data() maybe more explicit function like repo.get_event_data() would be better even if at beginning calls get_api_data() this would allow us to have some flexibility on how stable the events data is.

I think we also should consider detection of same events, what if there's a global event, and you define same action on per-repo basis, or using other patterns. I think it's easy to check simply by hashing event type and it's params so same hashes can be skipped.

Any thoughts ?

#3 Updated by Daniel D over 4 years ago

Yep glob patters would work for the nested event type - the only issue I found there was the scheme would need some design eg:

repo
    push
       pre
       post
    pull
       pre
       post
    pullrequest
       create
       merge
       update

+1 on get_events_data

Hashing method to check same event + integration would require a stable settings json - something to keep in mind..maybe sort it before saving.

#4 Updated by Marcin Kuzminski [CTO] over 4 years ago

  • Private changed from Yes to No
  • Target version set to v4.2

#5 Updated by Marcin Kuzminski [CTO] about 4 years ago

  • Target version changed from v4.2 to v4.3

#6 Updated by Marcin Kuzminski [CTO] about 4 years ago

  • Status changed from New to In Progress

#7 Updated by Marcin Kuzminski [CTO] about 4 years ago

  • Assignee set to Daniel D

#8 Updated by Daniel D about 4 years ago

  • Status changed from In Progress to Feedback

Events/Integrations have been added to 4.3, we're now in testing phase.

#9 Updated by Marcin Kuzminski [CTO] about 4 years ago

  • Status changed from Feedback to Closed

Also available in: Atom PDF