Automating Power-Ups to Aid Data Ingest, Enrichment, and Analysis
by savage | 2023-03-24
Synapse Power-Ups are Storm services and packages designed to facilitate activities such as data ingestion and enrichment to analysis and assigning additional tasking. Power-Ups are an integral part of the workflow for us analysts at The Vertex Project, and we often choose to automate many of them. This gives us several advantages:
Improved efficiency - automation runs in the background and ensures that tasks are completed in a timely manner;
Greater consistency in workflow - automated processes execute the same way each time, whereas relying on an analyst to manually do something can result in variations, especially if the analyst makes a mistake or forgets entirely;
Helps prevent analyst burn-out - leveraging automation in place of manual effort helps reduce the risk of analyst burn-out by alleviating the need for analysts to spend time on tedious tasks and allowing them to make better use of their time and effort.
In this piece, we’ll highlight different examples of ways analysts might automate Synapse Power-Ups to assist with their workflow. Our goal is for this to serve as a starting point for others looking for ways to automate Power-Ups in their environment.
Scheduling Power-Ups with Cron Jobs
We can create cron jobs that execute Power-Ups at specific times (either once or on an ongoing basis) to support data ingestion, enrichment, and analysis. The cron job below, for example, will use the Synapse-TOR Power-Up and run the tor.sync
command on the 37th minute of every hour to import an updated list of TOR exit nodes into the Power-up testing View:
Some other examples of cron jobs that analysts may use for research and analysis tasks include:
Ingesting Data:
Use the Synapse-MISP Power-Up to regularly import published MISP events into Synapse. The cron job can run the following Storm code:
misp.sync
Use the Synapse-Twitter Power-Up to regularly ingest tweets from a Twitter feed and scrape them for indicators of compromise:
twitter.feed.pull
Enriching Data:
Pipe all hash nodes with a #rep
tag and no accompanying file:bytes
node(s) to vxintel.download
to check for and download associated bytes through the Synapse-vxIntel Power-Up. The cron job can run the following Storm query:
hash:sha256#rep hash:sha1#rep hash:md5#rep -{-> file:bytes} | vxintel.download
Use the Synapse-Alienvault Power-Up to enrich all inet:fqdn
and inet:ipv4
nodes created within the last week and tagged with #rep
, #cno.threat
, or #cno.mal
to collect passive DNS A records:
inet:fqdn inet:ipv4 +.created@=(now, -7 days) +(#rep or #cno.mal or #cno.threat) | alienvault.otx.pdns
Analyzing Data:
Use the Synapse-YARA Power-Up to run all file:bytes
nodes tagged with #rep
, #cno.threat
, or #cno.mal
(but excluding those tagged #cno.mal.*
and any file:bytes
nodes with associated it:app:yara:match
nodes) against YARA rules tagged with #cno.detect
:
file:bytes +(#rep or #cno.threat or #cno.mal) -#cno.mal.* -{-> it:app:yara:match} | yara.match --rules ${it:app:yara:rule#cno.detect}
Executing Power-Ups with Event-Based Triggers
In some cases, creating an event-based trigger can be a more appropriate solution for some tasks rather than relying on a scheduled cron job. Triggers are better suited for on-demand tasks in particular, as they are designed to execute as soon as a predetermined condition is met. The trigger below, for example, is designed to enrich inet:ipv4
nodes tagged with a #rep
tag:
When an analyst working in the Power-up testing View adds a #rep
tag to an inet:ipv4
node, the trigger will pipe that node to the maxmind
command to pull in geolocation data through the Synapse-Maxmind Power-Up, and the alienvault.otx.pdns
command to pull in passive DNS data through the Synapse-Alienvault Power-Up.
Some other examples of triggers that analysts may use to support research and analysis tasks include:
Enriching Data:
When a hash:sha256
, hash:md5
, or hash:sha1
node is tagged with #rep
, use the Synapse-vxIntel Power-Up and Synapse-VirusTotal Power-Up to download the file and provide additional file information:
+(hash:sha256 or hash:sha1 or hash:md5) | vxintel.download | virustotal.file.report | virustotal.file.behavior
When an inet:fqdn
node is tagged with #cno.mal
or #cno.threat
, use the Synapse-Alienvault Power-Up to pull in passive DNS data, the Synapse-Nettools Power-Up to import WhoIs and DNS A records, and the Synapse-URLScan Power-Up to populate additional URLs:
+(#cno.mal or #cno.threat) | alienvault.otx.pdns | nettools.dns | urlscan.enrich
If an inet:email
node is tagged with #cno.threat
, use the Synapse-WhoXY Power-Up to perform a reverse search for related WhoIs records:
| whoxy.whois.reverse
Analyzing Data:
When a file:bytes
node is tagged with #cno.threat
or #cno.mal
and is not linked to an it:app:yara:match
node, use the Synapse-YARA Power-Up to run the file against YARA rules tagged with #cno.detect
:
+(#cno.mal or #cno.threat) -#cno.mal.* -{-> it:app:yara:match} | yara.match --rules ${it:app:yara:rule#cno.detect}
If a file:bytes
node matches a YARA rule, creating an it:app:yara:match
node, push the #cno.mal.*
tag from the it:app:yara:rule
node to it:app:yara:match
node and the matching file:bytes
node while also tagging both with #int.review.yara
to flag them for an analyst to review
+it:app:yara:match
{ -> it:app:yara:rule $mals = $node.globtags(cno.mal.*)}
-> file:bytes
for $m in $mals {[ +#cno.mal.$m +#int.review.mal.$m ]}
Creating Follow-on Tasking:
When the tag #int.review
is applied to a node, use the Synapse-Jira Power-Up to create a ticket tasking an analyst to review annotations on the node. An example of how this Storm code might look for a tagged inet:fqdn
node would be:
+#int.review -{-(refs)> proj:ticket}
switch $node.form() {
"inet:fqdn": {
$fqdn=$node.value()
//defang fqdn
$fqdn=$fqdn.replace(".", "[.]")
//create ticket description
$text=$lib.str.format("FQDN: {fqdn}", fqdn=$fqdn)
$desc=$lib.import(jira).text2ADF($text)
$fields=$lib.dict(labels=("int.review",),description=$desc)
jira.issue.create MDR support "Research request for tagged node review" --fields $fields
}
}
Automating Power-Ups to Improve Workflow Efficiency
We can improve our consistency and efficiency by automating Power-Ups to assist with data ingestion, enrichment, and analysis, among other tasks. Using cron jobs and triggers to automate tasks ensures that those actions are done consistently, freeing us from needing to remember when and how to do a task, and allowing us to use that time for other obligations. Event-based automation such as triggers, for example, can help us to operate more efficiently by ensuring that tasks are completed at machine speed so that data is enriched and tags are pushed to provide us with the data and context that we need to continue with our analysis. While the specifics of how we might decide to craft automation will depend on our operations and use cases, hopefully this blog can help serve as a starting point with ideas for how automating Synapse Power-Ups can facilitate analyst workflows.
For more information about Synapse, including tips, tricks, and use cases, join our Slack community, check out our videos on YouTube, and follow us on Twitter.