Turn Sentry errors into bugs

Connect an app, step by step

1. In Sentry

   Create an internal integration

   Go to Settings → Developer Settings → Custom Integrations → Create New Integration, choose Internal Integration, and name it qarunbook.

2. In Sentry

   Give it permissions and the issue webhook

   Under Permissions, set:

   • Project: Read
   • Organization: Read — without it, Load projects in qarunbook says That token is not allowed to do this
   • Issue & Event: Read & Write — write is what lets qarunbook resolve an issue

   Under Webhooks, tick Issue. Leave Webhook URL empty for now, and choose Save Changes.

3. In Sentry

   Copy the token and the client secret

   In the integration's Tokens section, choose New Token and copy it — Sentry shows it once.

   Under Credentials, copy the Client Secret. It is shown in full only briefly after the integration is made; if it is hidden, choose Rotate client secret and copy the new one.

4. In qarunbook

   Open the app's Sentry settings

   Open the app, choose App configuration, then the Integrations tab, and choose the Sentry tile under Where errors are caught.

5. In qarunbook

   Give your Sentry, organisation and token

   Under Sentry, pick sentry.io — US, sentry.io — EU (de.sentry.io) or Self-hosted (then give its https Sentry address).

   Fill in Organization slug — the your-org in your-org.sentry.io — and paste the token into Auth token.

6. In qarunbook

   Load projects, pick one, add the secret

   Choose Load projects and pick the Project. Paste the client secret into Client secret (optional) — with it, every delivery's signature is checked.

7. In qarunbook

   Choose what becomes a bug, and connect

   Under What becomes a bug, set the Level, People affected, at least, Environments (like production), whether to raise Only issues Sentry first sees after connecting, and the platforms to mark when Sentry cannot tell.

   Under Keeping Sentry in step, tick what should happen each way (see Keeping Sentry in step). Then choose Connect.

   The card now reads Watching [project] in [org], with signatures checked when the client secret is set.

8. In qarunbook

   Copy the webhook URL

   Under Recommended on the card is the integration webhook URL. Choose Copy.

9. In Sentry

   Paste it into the integration

   Back in Settings → Developer Settings → Custom Integrations, open the qarunbook integration, paste the URL into Webhook URL, and choose Save Changes.

Check it works

1. In Your tool

   Make a new error in the project

   Throw an error in the app Sentry watches, or send one with Sentry.captureException(new Error("Test: qarunbook connection")). Give it a message Sentry has not seen before, so it is a new issue; a captureMessage is sent at info level, which the default Level leaves out.

2. In qarunbook

   See the bug arrive

   Within seconds the app has a bug raised by Sentry: its title first, then Where, Level, Platform, Environment, First seen, Users affected, Events and a link back. The bug shows a Sentry · SHOP-WEB-1A link, and the card reads Last heard from Sentry just now: created: raised QA-16.

3. In qarunbook

   Fix it and confirm the fix

   On the bug, choose Mark fixed, then Confirm fixed.

4. In Sentry

   See it resolved

   The issue in Sentry now shows Resolved via qarunbook.

5. In Your tool

   Optionally, send the same error again

   Sentry marks the issue regressed, and the bug in qarunbook reopens. Wait a couple of minutes after the resolve first: Sentry may not count an error that arrives within a minute or two of it as a regression.

If nothing arrives, Sentry keeps its own log of what it sent: Developer Settings → the integration → Dashboard → Request Log.

How it works

1. Sentry sees a new issue in the project you connect, and it passes your filters.
2. It becomes a bug on the app, raised by Sentry: its title first, then where it happened, the level, platform and environment, when it was first and last seen, how many people and events, the release, and a link back.
3. Someone fixes it and marks it fixed. A tester checks the fix on the live app and confirms it.
4. qarunbook resolves the issue in Sentry. If Sentry sees the error again, the bug reopens and the people on it are told.

It is the same bug as one raised by hand, so everything else follows: it is filed in Linear, Jira, GitHub or GitLab if the app is connected, it can be given people to fix it, Slack and Microsoft Teams hear of it, and it counts towards Free's monthly allowance of issues.

What becomes a bug

• Level — error and fatal by default. Warnings, info and debug messages can be let in too.
• Environments — say production to leave staging out. When a list is set, an issue whose environment cannot be told is left out too.
• People affected, at least — 0 raises from the first. More needs an alert, as below, to tell qarunbook when the number is reached.
• Only issues Sentry first sees after connecting — on by default: an issue Sentry already knew before you connected is not raised, even when an alert fires on it.

The bug's priority comes from the level: fatal and error are high, warning is medium, info and debug are low. Its platforms come from what the error ran on — a browser, a browser on a phone, iOS or Android — matched to the app's own platforms. When Sentry cannot tell, the bug affects the platforms you pick on the card, or every platform.

Keeping Sentry in step

• Resolve the issue in Sentry when the fix is confirmed on the live app — on by default. A fix is only done when someone has seen it hold.
• Resolve it as soon as the bug is marked fixed — off by default, for teams that would rather Sentry heard straight away.
• Mark the bug fixed when it is resolved in Sentry — on by default. Resolving the issue in Sentry marks the bug fixed here, ready for someone to confirm.
• When Sentry unresolves an issue it had resolved — a regression — the bug is reopened, whoever reopened it is named, and the people on it are told. It happens once per regression, however many times Sentry says so.

Or with an alert

Sentry tells an integration about an issue the moment it is created, when it has hit one person. To raise a bug only once an error has hit, say, fifty people, use an alert instead: set People affected, at least on the card, then in Sentry create an issue alert with a condition like the issue is seen by more than 50 users, and an action that sends it to qarunbook — either Send a notification via the internal integration (turn on Alert Rule Action in it), or the project's WebHooks integration pointed at the alert webhook URL under Or with an alert on the card.

The WebHooks action signs nothing, so its URL carries an unguessable token instead. If it gets out, choose Replace alert URL, then Make a new URL: the old one stops at once.

Good to know

• Sentry only brings bugs in and hears when they are fixed. It does not create checks or sections, and nothing already in Sentry is copied across when you connect.
• One internal integration has one webhook URL, so it serves one qarunbook app. To connect several apps, make an internal integration for each, or use the alert route above.
• At most 30 bugs an hour come in from Sentry for one app, so a release that breaks everything is one bad hour, not hundreds of bugs.
• The card says when Sentry last got through and what came of it — raised QA-41, or why it was held back — which is the quickest way to check the setup.
• A user auth token works in place of the integration's, with the scopes org:read, project:read, event:read and event:write. An internal integration's token belongs to the organisation rather than to a person, so it keeps working when people leave.
• The token and client secret are stored encrypted and never shown again — the card shows only their last four characters. To use others, choose Change.
• If the app goes back to testing, Sentry's errors are not raised until it is live again.
• The REST API returns the link on every issue as sentry (short_id and url), or null, and via is sentry.
• Self-hosted Sentry must be reachable over https from the internet. Private and local addresses are refused, so the token is never sent inside a network.
• Disconnect stops new bugs at once; delete the internal integration in Sentry too. Bugs already raised keep their links, and connecting again never raises the same Sentry issue twice.