Console
    Contact Us Start free

    Collect MISP IOC logs

    Supported in:

    This document explains how to ingest MISP (Malware Information Sharing Platform) IOC logs to Google Security Operations using Bindplane. The parser processes the data in both CSV and JSON formats. It extracts IOC attributes like IP addresses, domains, hashes, and URLs, mapping them to a unified data model (UDM) along with threat details like severity, confidence, and descriptions. The parser handles both single and multiple IOC entries within the input data, normalizing them into a consistent UDM output.

    Before you begin

    Make sure you have the following prerequisites:

    • A Google SecOps instance.
    • A Linux host with systemd .
    • If running behind a proxy, ensure firewall ports are open per the Bindplane agent requirements.
    • Privileged access to your MISP server.

    Get Google SecOps ingestion authentication file

    1. Sign in to the Google SecOps console.
    2. Go to SIEM Settings > Collection Agents.
    3. Download the Ingestion Authentication File.
      • Save the file securely on the system where Bindplane will be installed.

    Get Google SecOps customer ID

    1. Sign in to the Google SecOps console.
    2. Go to SIEM Settings* > Profile.
    3. Copy and save the Customer IDfrom the Organization detailssection.

    Get MISP API credentials

    1. Sign in to your MISP web interface as an Administrator.
    2. Go to Administration > List Auth Keys.
    3. Click Add authentication key.
    4. Provide the following configuration details:
      • User: Select the user account associated with the key.
      • Optional: Allowed IPs: Specify allowed IP addresses for the key.
      • Expiration: Leave empty for no expiration or set as needed.
    5. Click Submit.
    6. Copy and save the API key in a secure location.
    7. Click I have noted down my key.

    Configure MISP data export

    1. Install PyMISP on your MISP server:

       pip3  
install  
pymisp

    2. Create the export directory:

       sudo  
mkdir  
-p  
/opt/misp/scripts
sudo  
mkdir  
-p  
/opt/misp/ioc_export

    3. Create the credentials file /opt/misp/scripts/keys.py :

        misp_url 
 = 
 'https://<MISP_SERVER_URL>' 
 misp_key 
 = 
 '<MISP_API_KEY>' 
 misp_verifycert 
 = 
 True 
 misp_client_cert 
 = 
 ''
      • Replace <MISP_SERVER_URL> with your MISP server URL.
      • Replace <MISP_API_KEY> with the API key from the prerequisites.

    4. Create the export script /opt/misp/scripts/misp_export.py :

        #!/usr/bin/env python3 
 # -*- coding: utf-8 -*- 
 import 
  
 argparse 
 from 
  
 pymisp 
  
 import 
 ExpandedPyMISP 
 from 
  
 keys 
  
 import 
 misp_url 
 , 
 misp_key 
 , 
 misp_verifycert 
 if 
 __name__ 
 == 
 '__main__' 
 : 
 parser 
 = 
 argparse 
 . 
 ArgumentParser 
 ( 
 description 
 = 
 'Export MISP IOCs to CSV format.' 
 ) 
 parser 
 . 
 add_argument 
 ( 
 "--controller" 
 , 
 default 
 = 
 'attributes' 
 , 
 help 
 = 
 "Controller to use for search (events, objects, attributes)" 
 ) 
 parser 
 . 
 add_argument 
 ( 
 "--event_id" 
 , 
 help 
 = 
 "Event ID to fetch. Without it, fetches recent data." 
 ) 
 parser 
 . 
 add_argument 
 ( 
 "--attributes" 
 , 
 nargs 
 = 
 '*' 
 , 
 help 
 = 
 "Requested attributes for CSV export" 
 ) 
 parser 
 . 
 add_argument 
 ( 
 "--misp_types" 
 , 
 nargs 
 = 
 '+' 
 , 
 help 
 = 
 "MISP types to fetch (ip-src, hostname, domain, etc.)" 
 ) 
 parser 
 . 
 add_argument 
 ( 
 "--context" 
 , 
 action 
 = 
 'store_true' 
 , 
 help 
 = 
 "Add event level context (tags, metadata)" 
 ) 
 parser 
 . 
 add_argument 
 ( 
 "--outfile" 
 , 
 required 
 = 
 True 
 , 
 help 
 = 
 "Output file to write the CSV data" 
 ) 
 parser 
 . 
 add_argument 
 ( 
 "--last" 
 , 
 required 
 = 
 True 
 , 
 help 
 = 
 "Time period: days (d), hours (h), minutes (m) - e.g., 1d, 12h, 30m" 
 ) 
 args 
 = 
 parser 
 . 
 parse_args 
 () 
 api 
 = 
 ExpandedPyMISP 
 ( 
 misp_url 
 , 
 misp_key 
 , 
 misp_verifycert 
 , 
 debug 
 = 
 False 
 ) 
 response 
 = 
 api 
 . 
 search 
 ( 
 controller 
 = 
 args 
 . 
 controller 
 , 
 return_format 
 = 
 'csv' 
 , 
 type_attribute 
 = 
 args 
 . 
 misp_types 
 , 
 publish_timestamp 
 = 
 args 
 . 
 last 
 , 
 include_context 
 = 
 args 
 . 
 context 
 , 
 requested_attributes 
 = 
 args 
 . 
 attributes 
 or 
 None 
 ) 
 with 
 open 
 ( 
 args 
 . 
 outfile 
 , 
 'w' 
 ) 
 as 
 response_file 
 : 
 response_file 
 . 
 write 
 ( 
 response 
 )
      1. Make the script executable:
       sudo  
chmod  
+x  
/opt/misp/scripts/misp_export.py

      Schedule MISP data exports

      1. Create scheduled exports using crontab:
       sudo  
crontab  
-e

    5. Add the following cron entries:

        # Export different IOC types daily with context 
 0 
  
 0 
  
*  
*  
*  
python3  
/opt/misp/scripts/misp_export.py  
--outfile  
/opt/misp/ioc_export/domains.csv  
--misp_types  
domain  
--last  
1d  
--context  
--attributes  
uuid  
event_id  
category  
 type 
  
value  
comment  
to_ids  
date  
attribute_tag  
event_info 0 
  
 1 
  
*  
*  
*  
python3  
/opt/misp/scripts/misp_export.py  
--outfile  
/opt/misp/ioc_export/ip-src.csv  
--misp_types  
ip-src  
--last  
1d  
--context  
--attributes  
uuid  
event_id  
category  
 type 
  
value  
comment  
to_ids  
date  
attribute_tag  
event_info 0 
  
 2 
  
*  
*  
*  
python3  
/opt/misp/scripts/misp_export.py  
--outfile  
/opt/misp/ioc_export/ip-dst.csv  
--misp_types  
ip-dst  
--last  
1d  
--context  
--attributes  
uuid  
event_id  
category  
 type 
  
value  
comment  
to_ids  
date  
attribute_tag  
event_info 0 
  
 3 
  
*  
*  
*  
python3  
/opt/misp/scripts/misp_export.py  
--outfile  
/opt/misp/ioc_export/urls.csv  
--misp_types  
url  
--last  
1d  
--context  
--attributes  
uuid  
event_id  
category  
 type 
  
value  
comment  
to_ids  
date  
attribute_tag  
event_info 0 
  
 4 
  
*  
*  
*  
python3  
/opt/misp/scripts/misp_export.py  
--outfile  
/opt/misp/ioc_export/sha256.csv  
--misp_types  
sha256  
--last  
1d  
--context  
--attributes  
uuid  
event_id  
category  
 type 
  
value  
comment  
to_ids  
date  
attribute_tag  
event_info 0 
  
 5 
  
*  
*  
*  
python3  
/opt/misp/scripts/misp_export.py  
--outfile  
/opt/misp/ioc_export/filenames.csv  
--misp_types  
filename  
--last  
1d  
--context  
--attributes  
uuid  
event_id  
category  
 type 
  
value  
comment  
to_ids  
date  
attribute_tag  
event_info 0 
  
 6 
  
*  
*  
*  
python3  
/opt/misp/scripts/misp_export.py  
--outfile  
/opt/misp/ioc_export/registries.csv  
--misp_types  
regkey  
--last  
1d  
--context  
--attributes  
uuid  
event_id  
category  
 type 
  
value  
comment  
to_ids  
date  
attribute_tag  
event_info 0 
  
 7 
  
*  
*  
*  
python3  
/opt/misp/scripts/misp_export.py  
--outfile  
/opt/misp/ioc_export/mutexes.csv  
--misp_types  
mutex  
--last  
1d  
--context  
--attributes  
uuid  
event_id  
category  
 type 
  
value  
comment  
to_ids  
date  
attribute_tag  
event_info

    6. Optionally, schedule a pull of feeds from MISP:

        23 
  
 0 
  
*  
*  
*  
curl  
--insecure  
--header  
 "Authorization: <MISP_API_KEY>" 
  
--header  
 "Accept: application/json" 
  
--header  
 "Content-Type: application/json" 
  
https://<MISP_SERVER_URL>/feeds/fetchFromAllFeeds

    Install the Bindplane agent

    Install the Bindplane agent on your Linux operating system according to the following instructions.

    Linux installation

    1. Open a terminal with root or sudo privileges.

    2. Run the following command:

       sudo  
sh  
-c  
 " 
 $( 
curl  
-fsSlL  
https://github.com/observiq/bindplane-otel-collector/releases/latest/download/install_unix.sh ) 
 " 
  
install_unix.sh

    Additional installation resources

    Configure the Bindplane agent to ingest MISP logs and send to Google SecOps

    1. Access the configuration file:

      • Locate the config.yaml file. Typically, it's in the /etc/bindplane-agent/ directory on Linux.
      • Open the file using a text editor (for example, nano , vi ).

    2. Edit the config.yaml file as follows:

        receivers 
 : 
  
 filelog 
 : 
  
 file_path 
 : 
  
 /opt/misp/ioc_export/*.log 
 exporters 
 : 
  
 chronicle/chronicle_w_labels 
 : 
  
 compression 
 : 
  
 gzip 
  
 # Adjust the path to the credentials file you downloaded in Step 1 
  
 creds_file_path 
 : 
  
 '/path/to/ingestion-authentication-file.json' 
  
 # Replace with your actual customer ID from Step 2 
  
 customer_id 
 : 
  
< customer_id 
>  
 endpoint 
 : 
  
 malachiteingestion-pa.googleapis.com 
  
 # Add optional ingestion labels for better organization 
  
 ingestion_labels 
 : 
  
 log_type 
 : 
  
 'MISP_IOC' 
  
 raw_log_field 
 : 
  
 body 
 service 
 : 
  
 pipelines 
 : 
  
 logs/source0__chronicle_w_labels-0 
 : 
  
 receivers 
 : 
  
 - 
  
 filelog 
  
 exporters 
 : 
  
 - 
  
 chronicle/chronicle_w_labels
      • Replace <CUSTOMER_ID> with your actual Customer ID from the prerequisites.
      • Update /path/to/ingestion-authentication-file.json to the path where the authentication file was saved.

    Restart Bindplane agent to apply the changes

    1. To restart the Bindplane agent in Linux, run the following command:

       sudo  
systemctl  
restart  
observiq-otel-collector

    UDM mapping table

    Log field UDM mapping Logic
    Attribute.category
    		 entity.metadata.threat.category_details Direct mapping from the category field in the Attribute object.
    Attribute.comment
    		 entity.metadata.threat.summary Direct mapping from the comment field in the Attribute object.
    Attribute.deleted
    		 entity.metadata.threat.detection_fields.value Direct mapping from the deleted field in the Attribute object. The key is set to Attribute deleted .
    Attribute.event_id
    		 entity.metadata.threat.detection_fields.value Direct mapping from the event_id field in the Attribute object. The key is set to Attribute event_id .
    Attribute.first_seen
    		 entity.metadata.threat.detection_fields.value Direct mapping from the first_seen field in the Attribute object. The key is set to Attribute first_seen .
    Attribute.id
    		 entity.metadata.threat.detection_fields.value Direct mapping from the id field in the Attribute object. The key is set to Attribute id or Attribute id $$ depending on the format of the raw log.
    Attribute.timestamp
    		 entity.metadata.threat.detection_fields.value Direct mapping from the timestamp field in the Attribute object. The key is set to Attribute timestamp .
    Attribute.to_ids
    		 entity.metadata.threat.detection_fields.value Direct mapping from the to_ids field in the Attribute object. The key is set to Attribute to_ids .
    Attribute.type
    		 entity.metadata.threat.category_details Direct mapping from the type field in the Attribute object.
    Attribute.type
    		 log_type Used to determine the type of IOC and map it to the appropriate UDM fields.
    Attribute.uuid
    		 entity.metadata.product_entity_id Direct mapping from the uuid field in the Attribute object.
    Attribute.value
    		 entity.entity.file.full_path Mapped if the Attribute.type is filename .
    Attribute.value
    		 entity.entity.file.md5 Mapped if the Attribute.type is md5 .
    Attribute.value
    		 entity.entity.file.sha1 Mapped if the Attribute.type is sha1 .
    Attribute.value
    		 entity.entity.file.sha256 Mapped if the Attribute.type is sha256 .
    Attribute.value
    		 entity.entity.hostname Mapped if the Attribute.type is domain .
    Attribute.value
    		 entity.entity.ip Mapped if the Attribute.type is ip-dst , ip-dst|port , or ip-src . The value is extracted using a grok pattern.
    Attribute.value
    		 entity.entity.resource.name Mapped if the Attribute.type is mutex .
    Attribute.value
    		 entity.entity.registry.registry_key Mapped if the Attribute.type is regkey .
    Attribute.value
    		 entity.entity.url Mapped if the Attribute.type is uri or URL .
    column1
    		 entity.metadata.product_entity_id Direct mapping from the first column in the CSV data.
    column14
    		 event_info Used to append additional information to the threat_sr.description field.
    column16
    		 event_source_org Direct mapping from the 16th column in the CSV data.
    column18
    		 threat_level Direct mapping from the 18th column in the CSV data.
    column21
    		 description Direct mapping from the 21st column in the CSV data.
    column3
    		 misp_category Direct mapping from the third column in the CSV data.
    column4
    		 type Direct mapping from the fourth column in the CSV data.
    column5
    		 value Direct mapping from the fifth column in the CSV data.
    column6
    		 comment Direct mapping from the sixth column in the CSV data.
    column8
    		 ts1 Direct mapping from the eighth column in the CSV data.
    description
    		 ioc.description The value is generated by combining the description field with the event_info field, separated by - additional info: .
    description
    		 entity.metadata.threat.description Direct mapping from the description field.
    event_creator_email
    		 entity.entity.labels.value Direct mapping from the event_creator_email field. The key is set to event_creator_email .
    event_source_org
    		 ioc.feed_name Direct mapping from the event_source_org field.
    event_source_org
    		 entity.metadata.threat.threat_feed_name Direct mapping from the event_source_org field.
    Feed.publish
    		 entity.metadata.threat.detection_fields.value Direct mapping from the publish field in the Feed object. The key is set to Feed publish .
    first_seen
    		 ioc.active_timerange.start Direct mapping from the first_seen field. The value is parsed as a date.
    first_seen
    		 entity.metadata.interval.start_time Direct mapping from the first_seen field. The value is parsed as a date.
    info
    		 entity.metadata.description Direct mapping from the info field.
    last_seen
    		 ioc.active_timerange.end Direct mapping from the last_seen field. The value is parsed as a date.
    log.category
    		 ioc.categorization Direct mapping from the category field in the log object.
    log.category
    		 entity.metadata.threat.category_details Direct mapping from the category field in the log object.
    log.comment
    		 entity.entity.file.full_path Mapped if the log.type is filename and the comment field is not Artifacts dropped .
    log.comment
    		 entity.metadata.threat.detection_fields.value Direct mapping from the comment field in the log object. The key is set to Attribute comment .
    log.comment
    		 entity.metadata.threat.summary Direct mapping from the comment field in the log object.
    log.deleted
    		 entity.metadata.threat.detection_fields.value Direct mapping from the deleted field in the log object. The key is set to Attribute deleted .
    log.event_id
    		 entity.metadata.threat.detection_fields.value Direct mapping from the event_id field in the log object. The key is set to Attribute event_id .
    log.first_seen
    		 entity.metadata.threat.detection_fields.value Direct mapping from the first_seen field in the log object. The key is set to Attribute first_seen .
    log.id
    		 entity.metadata.threat.detection_fields.value Direct mapping from the id field in the log object. The key is set to Attribute id .
    log.timestamp
    		 entity.metadata.threat.detection_fields.value Direct mapping from the timestamp field in the log object. The key is set to Attribute timestamp .
    log.to_ids
    		 entity.metadata.threat.detection_fields.value Direct mapping from the to_ids field in the log object. The key is set to Attribute to_ids .
    log.type
    		 ioc.categorization Direct mapping from the type field in the log object.
    log.type
    		 log_type Used to determine the type of IOC and map it to the appropriate UDM fields.
    log.uuid
    		 entity.metadata.product_entity_id Direct mapping from the uuid field in the log object.
    log.value
    		 entity.entity.file.full_path Mapped if the log.type is filename .
    log.value
    		 entity.entity.file.md5 Mapped if the log.type is md5 .
    log.value
    		 entity.entity.file.sha1 Mapped if the log.type is sha1 .
    log.value
    		 entity.entity.file.sha256 Mapped if the log.type is sha256 .
    log.value
    		 entity.entity.hostname Mapped if the log.type is domain .
    log.value
    		 entity.entity.ip Mapped if the log.type is ip-dst , ip-dst|port , or ip-src . The value is extracted using a grok pattern.
    log.value
    		 entity.entity.resource.name Mapped if the log.type is mutex .
    log.value
    		 entity.entity.registry.registry_key Mapped if the log.type is regkey .
    log.value
    		 entity.entity.url Mapped if the log.type is uri or url .
    log.value
    		 ioc.domain_and_ports.domain Mapped if the log.type is domain .
    log.value
    		 entity.entity.user.email_addresses Mapped if the log.type is threat-actor .
    misp_category
    		 entity.metadata.threat.category_details Direct mapping from the misp_category field.
    Org.name
    		 entity.metadata.threat.detection_fields.value Direct mapping from the name field in the Org object. The key is set to Org name .
    published
    		 entity.metadata.threat.detection_fields.value Direct mapping from the published field. The key is set to published .
    Tag.colour
    		 entity.metadata.threat.detection_fields.value Direct mapping from the colour field in the Tag object. The key is set to tag colour .
    Tag.exportable
    		 entity.metadata.threat.detection_fields.value Direct mapping from the exportable field in the Tag object. The key is set to tag exportable .
    Tag.hide_tag
    		 entity.metadata.threat.detection_fields.value Direct mapping from the hide_tag field in the Tag object. The key is set to tag hide_tag .
    Tag.id
    		 entity.metadata.threat.detection_fields.value Direct mapping from the id field in the Tag object. The key is set to tag id .
    Tag.is_custom_galaxy
    		 entity.metadata.threat.detection_fields.value Direct mapping from the is_custom_galaxy field in the Tag object. The key is set to tag is_custom_galaxy .
    Tag.is_galaxy
    		 entity.metadata.threat.detection_fields.value Direct mapping from the is_galaxy field in the Tag object. The key is set to tag is_galaxy .
    Tag.isinherited
    		 entity.metadata.threat.detection_fields.value Direct mapping from the isinherited field in the Tag object. The key is set to tag isinherited .
    Tag.name
    		 entity.metadata.threat.detection_fields.value Direct mapping from the name field in the Tag object. The key is set to tag name .
    Tag.numerical_value
    		 entity.metadata.threat.detection_fields.value Direct mapping from the numerical_value field in the Tag object. The key is set to tag numerical_value .
    Tag.user_id
    		 entity.metadata.threat.detection_fields.value Direct mapping from the user_id field in the Tag object. The key is set to tag user_id .
    threat_level
    		 ioc.raw_severity Direct mapping from the threat_level field.
    threat_level
    		 entity.metadata.threat.severity_details Direct mapping from the threat_level field.
    threat_level_id
    		 entity.entity.labels.value Direct mapping from the threat_level_id field. The key is set to threat_level_id .
    ts1
    		 ioc.active_timerange.start Direct mapping from the ts1 field. The value is parsed as a date.
    ts1
    		 entity.metadata.interval.start_time Direct mapping from the ts1 field. The value is parsed as a date.
    		entity.entity.file.full_path Mapped if the type is filename .
    		entity.entity.file.md5 Mapped if the type is md5 .
    		entity.entity.file.sha1 Mapped if the type is sha1 .
    		entity.entity.file.sha256 Mapped if the type is sha256 .
    		entity.entity.hostname Mapped if the type is domain .
    		entity.entity.ip Mapped if the type is ip-dst , ip-dst|port , or ip-src . The value is extracted using a grok pattern.
    		entity.entity.port Mapped if the port field is not empty. The value is converted to an integer.
    		entity.entity.resource.name Mapped if the type is mutex .
    		entity.entity.resource.resource_subtype Mapped if the type is regkey . The value is set to regkey .
    		entity.entity.resource.resource_type Mapped if the type is mutex or regkey . The value is set to MUTEX or STORAGE_OBJECT respectively.
    		entity.entity.registry.registry_key Mapped if the type is regkey .
    		entity.entity.url Mapped if the type is uri or url .
    		entity.metadata.collected_timestamp The value is set to the timestamp of the raw log entry.
    		entity.metadata.description The value is set to the type field if the raw log is in CSV format. Otherwise, it's set to the info field.
    		entity.metadata.entity_type The value is determined based on the type or log_type field. It can be DOMAIN_NAME , FILE , IP_ADDRESS , MUTEX , RESOURCE , or URL .
    		entity.metadata.interval.end_time The value is set to a default value of 253402300799 seconds.
    		entity.metadata.interval.start_time The value is set to the first_seen field if it's not empty. Otherwise, it's set to a default value of 1 second or the timestamp of the raw log entry.
    		entity.metadata.product_name The value is set to MISP .
    		entity.metadata.threat.confidence The value is set to UNKNOWN_CONFIDENCE if the confidence field is empty or f . Otherwise, it's set to HIGH_CONFIDENCE , MEDIUM_CONFIDENCE , or LOW_CONFIDENCE based on the value of the confidence field.
    		entity.metadata.threat.confidence_details Direct mapping from the confidence field.
    		entity.metadata.threat.detection_fields The value is a list of key-value pairs extracted from various fields in the raw log.
    		entity.metadata.vendor_name The value is set to MISP .
    		ioc.active_timerange.end The value is set to the last_seen field if it's not empty.
    		ioc.active_timerange.start The value is set to the ts1 or first_seen field if they are not empty. Otherwise, it's set to a default value of 1 second.
    		ioc.categorization The value is set to misp_category IOCs if the raw log is in CSV format. Otherwise, it's set to the category field in the Attribute or log object.
    		ioc.confidence_score Direct mapping from the confidence field.
    		ioc.description The value is generated by combining the description field with the event_info field, separated by - additional info: .
    		ioc.domain_and_ports.domain Mapped if the type or log_type is domain .
    		ioc.feed_name The value is set to MISP if the event_source_org field is empty. Otherwise, it's set to the event_source_org field.
    		ioc.ip_and_ports.ip_address Mapped if the ip field is not empty. The value is converted to an IP address.
    		ioc.ip_and_ports.ports Mapped if the port field is not empty. The value is converted to an unsigned integer.
    		ioc.raw_severity Direct mapping from the threat_level field.
    		timestamp The value is set to the timestamp of the raw log entry.

    Need more help? Get answers from Community members and Google SecOps professionals.

    Design a Mobile Site
    View Site in Mobile | Classic
    Share by: