Post Archive
MISP Purge Events Tool - Released
In various cases it is useful to expire or purge older events, clean blocklist entries, or delete events from a specific organization. This post captures the original release flow and operational guidance.
What The Tool Does
The MISP Purge Event tool is written in Python and supports multiple cleaning operations for large datasets.
The tool can delete events based on first-seen and last-seen values, and it supports deleting by organization UUID or cleaning in Blocklisted Events.
To protect specific organizations, you can define exclusions. Feeds with fixed events are excluded by default.
Warning
- Always perform a backup before deletion tasks, and at minimum run a dry-run first to verify the events that would be deleted.
- Never run this process on an MISP instance you do not own.
Notice
If a large set of events must be purged, it is highly recommended to disable and flush the correlation table first. Otherwise, MISP databases can become unstable due to waits for correlation cleanup.
Deletion can take time because heavy cleanup can include:
- attributes
- event tags
- attribute tags
- objects
- proposals
- event reports
- discussion threads
- correlations
Correlations are typically the largest cleanup area, depending on relations to other events.
MISP Disable Correlations Database
This can be achieved by logging in to MISP as a site admin and following this path:
- Administration
- Server Settings and Administration
- MISP Settings
- Change MISP.completely_disable_correlation to True
Validate that the correlations table is clean by checking Diagnostics under SQL database status.
When the large task is completed, remember to enable correlations again.
Benchmarks
For a MISP instance with the below data volume, correlations were removed before running the tool:
Events
9,460
Attributes
1,424,251
Execution Time
12m11.239s
MISP Purge Event Config
The config file is divided into three elements:
- MISP connection information
- Chunk size (how many events to delete at a time)
- Exclusions (organization UUIDs excluded from deletion)
