Automating Security Triage with HackerOne and Deep Search
How Sourcegraph uses HackerOne webhooks and Deep Search to automatically validate, triage, and investigate vulnerability reports before engineers start their day.
You wake up ready to attack the day, you check your email and there it is: a critical bug report in a massive codebase you've never seen, only heard horror stories of. The report goes into technical details about components you don't understand with terms you heard back in school but haven't heard in years:
- How do you know the bug is actually valid?
- How do you know what protections may already be in place to prevent the bug?
- Where in the code is the bug and how do you fix it?
Finding the answer to all of these questions may take hours, days, or weeks depending on the codebase.
For a lot of programs, this happens multiple times a week, a direct side effect of the high-velocity vulnerability discovery now possible with tools like Claude Mythos. That's a lot of time and a lot of triage engineer burnout!
This happened to me a few too many times for my liking (once), so I decided to put my access to Deep Search to good use. We use HackerOne for our private bug bounty program which has the capability to send a webhook on any newly opened report. We took advantage of this by writing a small service that takes a signed webhook from HackerOne, verifies its signature, then passes along the bug report via the Deep Search API to investigate the report in the relevant codebase.
Once the investigation is complete, the service posts back a team-only comment stating if the bug was valid, a severity, links to the affected code, potential fixes, and a link to the Deep Search conversation. For example, in response to a bug report we got about not escaping filenames in src-cli (thanks @fauxthn!), the response looked like the following:
Overall vulnerability analysis and pointers to affected code
Continuation of vulnerability chain and vulnerability validity assessment
Severity and impact assessment and beginning of suggested remediations
More suggested remediations and a link to the Deep Search conversation
That's pretty cool! Now when I get to my computer in the morning and see the report, all of the investigative work is already done for me. And if I need to, I can follow the link to the conversation, fork it, and ask follow up questions. Neat!
Another cool thing it can do is point out tangentially related bugs to components that were mentioned in the report. For example, in a bug report related to GraphQL aliases, Deep Search pointed out a critical logic flaw in the code defining admin-level controls of these aliases.
This bug was not included in the original report at all and was independently discovered by Deep Search.
Critical bug independently discovered by Deep Search showing inverted configuration override logic
While this is pretty cool, it's not perfect. Due to the nature of the service, the bug report has to be passed along to Deep Search. This causes a few problems:
| Problem | Solution |
|---|---|
| Reporter Token Abuse: The reporter can influence control token consumption in Deep Search, creating an easy abuse channel. | Use Entitlements to limit the number of Deep Search questions allowed within a time period and utilize invite-only bounty programs to reduce overall volume. |
| Inaccurate Repository Mapping: Deep Search relies on report keywords to link vulnerable components to repositories, which is not guaranteed and can lead to incorrect matches. | Map each high-level in-scope service to one or more specific repositories within the Deep Search prompt to improve result quality. |
| Prompt Injection Payloads: The report can contain malicious prompt injection payloads. | Lean on entitlements and the model provider's built-in defenses to mitigate input risks. You can also add tags marking untrusted input in the prompt that's sent to Deep Search. |
Overall, this little service has dramatically improved the security team's life. It has cut down the hours we used to dedicate to triaging and manually validating vulnerabilities and it's improved our response times. This project is a powerful example of how AI can augment security teams, allowing them to focus on high-impact remediation rather than routine investigation. We anticipate that systems like this will become an essential part of modern vulnerability management.
Ready to stop manually triaging bugs and start solving them? Automate your own security workflows by integrating your bounty platform with Deep Search today. Schedule a demo.
A special thanks to Justin Dorfman and Andre Eleuterio for their contributions to this blog post.
Unblock your organization.
Ship faster.
With Sourcegraph, the code understanding platform for enterprise.
Schedule a demo