Collect MISP IOC logs

This guide outlines the steps to ingest MISP (Malware Information Sharing Platform) IOC (Indicators of Compromise) logs to Google Security Operations using Bindplane. The parser extracts IOCs from MISP data formatted as JSON or CSV. It parses the input, maps fields to the Unified Data Model (UDM), handles various IOC types (such as IP, domain, and file hashes), and enriches the data with threat intelligence context like confidence and severity. The parser also performs specific logic for different data formats and handles cases with missing or unsupported fields.

Before you begin

Make sure you have the following prerequisites:

• Google SecOps instance
• Linux host with systemd
• If running behind a proxy, firewall ports are open
• Privileged access to your MISP server

Get MISP API Key

1. Sign in to your MISPweb UI as an Administrator.
2. Go to Administration > List Auth Keys.
3. Click Add authentication key.
4. Provide the following Key configuration:
   • User: Select the user account associated with the key.
   • Allowed IPs: You can optionally specify allowed IP addresses for the key.
   • Copy and save the key in a secure location.
   • Click I have noted down my key.

Configure MISP logs export

1. Sign in to your MISPinstance using SSH.

2. Install PyMISPusing the following command:

    pip3  
   install  
   pymisp  
    
   

3. Modify the get_csv.py export script using the following:

     #!/usr/bin/env python3 
    # -*- coding: utf-8 -*- 
    import 
     
    argparse 
    from 
     
    pymisp 
     
    import 
    ExpandedPyMISP 
    from 
     
    keys 
     
    import 
    misp_url 
    , 
    misp_key 
    , 
    misp_verifycert 
    if 
    __name__ 
    == 
    '__main__' 
    : 
    parser 
    = 
    argparse 
    . 
    ArgumentParser 
    ( 
    description 
    = 
    'Get MISP data in a CSV format.' 
    ) 
    parser 
    . 
    add_argument 
    ( 
    "--controller" 
    , 
    default 
    = 
    'attributes' 
    , 
    help 
    = 
    "Attribute to use for the search (events, objects, attributes)" 
    ) 
    parser 
    . 
    add_argument 
    ( 
    "-e" 
    , 
    "--event_id" 
    , 
    help 
    = 
    "Event ID to fetch. Without it, it will fetch the whole database." 
    ) 
    parser 
    . 
    add_argument 
    ( 
    "-a" 
    , 
    "--attribute" 
    , 
    nargs 
    = 
    '+' 
    , 
    help 
    = 
    "Attribute column names" 
    ) 
    parser 
    . 
    add_argument 
    ( 
    "-o" 
    , 
    "--object_attribute" 
    , 
    nargs 
    = 
    '+' 
    , 
    help 
    = 
    "Object attribute column names" 
    ) 
    parser 
    . 
    add_argument 
    ( 
    "-t" 
    , 
    "--misp_types" 
    , 
    nargs 
    = 
    '+' 
    , 
    help 
    = 
    "MISP types to fetch (ip-src, hostname, ...)" 
    ) 
    parser 
    . 
    add_argument 
    ( 
    "-c" 
    , 
    "--context" 
    , 
    action 
    = 
    'store_true' 
    , 
    help 
    = 
    "Add event level context (tags...)" 
    ) 
    parser 
    . 
    add_argument 
    ( 
    "-f" 
    , 
    "--outfile" 
    , 
    help 
    = 
    "Output file to write the CSV." 
    ) 
    parser 
    . 
    add_argument 
    ( 
    "-l" 
    , 
    "--last" 
    , 
    required 
    = 
    True 
    , 
    help 
    = 
    "can be defined in days, hours, minutes (for example 5d or 12h or 30m)." 
    ) 
    args 
    = 
    parser 
    . 
    parse_args 
    () 
    pymisp 
    = 
    ExpandedPyMISP 
    ( 
    misp_url 
    , 
    misp_key 
    , 
    misp_verifycert 
    , 
    debug 
    = 
    True 
    ) 
    attr 
    = 
    [] 
    if 
    args 
    . 
    attribute 
    : 
    attr 
    += 
    args 
    . 
    attribute 
    if 
    args 
    . 
    object_attribute 
    : 
    attr 
    += 
    args 
    . 
    object_attribute 
    if 
    not 
    attr 
    : 
    attr 
    = 
    None 
    print 
    ( 
    args 
    . 
    context 
    ) 
    response 
    = 
    pymisp 
    . 
    search 
    ( 
    return_format 
    = 
    'csv' 
    , 
    controller 
    = 
    args 
    . 
    controller 
    , 
    eventid 
    = 
    args 
    . 
    event_id 
    , 
    requested_attributes 
    = 
    attr 
    , 
    publish_timestamp 
    = 
    args 
    . 
    last 
    , 
    type_attribute 
    = 
    args 
    . 
    misp_types 
    , 
    include_context 
    = 
    args 
    . 
    context 
    ) 
    if 
    args 
    . 
    outfile 
    : 
    with 
    open 
    ( 
    args 
    . 
    outfile 
    , 
    'w' 
    ) 
    as 
    f 
    : 
    f 
    . 
    write 
    ( 
    response 
    ) 
    else 
    : 
    print 
    ( 
    response 
    ) 
    
   

4. Edit the keys.py file to include your MISP API credentials and URL, as follows:

     misp_url 
    = 
    'https://<MISP_URL>' 
    misp_key 
    = 
    '<MISP_API_KEY>' 
    misp_verifycert 
    = 
    False 
    misp_client_cert 
    = 
    '' 
    
   

   • Replace <MISP_URL> with your MISP IP or hostname.
   • Replace <MISP_API_KEY with the actual API key generated previously.

5. Open crontab using the crontab -e command and enter the following:

     0 
     
    20 
     
   *  
   *  
   *  
   python3  
   /opt/misp/<YOUR_EXPORT_SCRIPT_PATH>  
   -f  
   /opt/misp/ioc_export/url.log  
   -t  
    "url" 
     
   -l  
   1d  
   -c 0 
     
    20 
     
   *  
   *  
   *  
   python3  
   /opt/misp/<YOUR_EXPORT_SCRIPT_PATH>  
   -f  
   /opt/misp/ioc_export/ip-dst.log  
   -t  
    "ip-dst" 
     
   -l  
   1d  
   -c 0 
     
    20 
     
   *  
   *  
   *  
   python3  
   /opt/misp/<YOUR_EXPORT_SCRIPT_PATH>  
   -f  
   /opt/misp/ioc_export/ip-src.log  
   -t  
    "ip-src" 
     
   -l  
   1d  
   -c 0 
     
    20 
     
   *  
   *  
   *  
   python3  
   /opt/misp/<YOUR_EXPORT_SCRIPT_PATH>  
   -f  
   /opt/misp/ioc_export/domain.log  
   -t  
    "domain" 
     
   -l  
   1d  
   -c 0 
     
    20 
     
   *  
   *  
   *  
   python3  
   /opt/misp/<YOUR_EXPORT_SCRIPT_PATH>  
   -f  
   /opt/misp/ioc_export/sha256.log  
   -t  
    "sha256" 
     
   -l  
   1d  
   -c 0 
     
    20 
     
   *  
   *  
   *  
   python3  
   /opt/misp/<YOUR_EXPORT_SCRIPT_PATH>  
   -f  
   /opt/misp/ioc_export/file.log  
   -t  
    "filename" 
     
   -l  
   1d  
   -c 0 
     
    20 
     
   *  
   *  
   *  
   python3  
   /opt/misp/<YOUR_EXPORT_SCRIPT_PATH>  
   -f  
   /opt/misp/ioc_export/registry.log  
   -t  
    "registry" 
     
   -l  
   1d  
   -c 0 
     
    20 
     
   *  
   *  
   *  
   python3  
   /opt/misp/<YOUR_EXPORT_SCRIPT_PATH>  
   -f  
   /opt/misp/ioc_export/mutex.log  
   -t  
    "mutex" 
     
   -l  
   1d  
   -c 0 
     
    20 
     
   *  
   *  
   *  
   python3  
   /opt/misp/<YOUR_EXPORT_SCRIPT_PATH>  
   -f  
   /opt/misp/ioc_export/threat-actor.log  
   -t  
    "threat-actor" 
     
   -l  
   1d  
   -c 
   

   • Update <YOUR_EXPORT_SCRIPT_PATH> according to the actual export script location.

Get Google SecOps ingestion authentication file

1. Sign in to the Google SecOps console.
2. Go to SIEM Settings > Collection Agents.
3. Download the Ingestion Authentication File. Save the file securely on the system where Bindplane will be installed.

Get Google SecOps customer ID

1. Sign in to the Google SecOps console.
2. Go to SIEM Settings > Profile.
3. Copy and save the Customer IDfrom the Organization Detailssection.

Install the Bindplane agent on MISP Server

Linux installation

1. Open a terminal with root or sudo privileges.

2. Run the following command:

    sudo  
   sh  
   -c  
    " 
    $( 
   curl  
   -fsSlL  
   https://github.com/observiq/bindplane-agent/releases/latest/download/install_unix.sh ) 
    " 
     
   install_unix.sh 
   

Additional installation resources

For additional installation options, consult the installation guide .

Configure the Bindplane agent to ingest MISP log files and send to Google SecOps

1. Access the configuration file:
   • Locate the config.yaml file. Typically, it's in the /etc/bindplane-agent/ directory on Linux.
   • Open the file using a text editor (for example, nano , vi , or Notepad).

2. Edit the config.yaml file as follows:

     receivers 
    : 
     
    filelog 
    : 
     
    file_path 
    : 
     
    /opt/misp/ioc_export/*.log 
     
    log_type 
    : 
     
    'file' 
    exporters 
    : 
     
    chronicle/chronicle_w_labels 
    : 
     
    compression 
    : 
     
    gzip 
     
    # Adjust the path to the credentials file you downloaded in Step 1 
     
    creds_file_path 
    : 
     
    '/path/to/ingestion-authentication-file.json' 
     
    # Replace with your actual customer ID from Step 2 
     
    customer_id 
    : 
     
   < customer_id 
   >  
    endpoint 
    : 
     
    malachiteingestion-pa.googleapis.com 
     
    # Add optional ingestion labels for better organization 
     
    ingestion_labels 
    : 
     
    log_type 
    : 
     
    'MISP_IOC' 
     
    raw_log_field 
    : 
     
    body 
    service 
    : 
     
    pipelines 
    : 
     
    logs/source0__chronicle_w_labels-0 
    : 
     
    receivers 
    : 
     
    - 
     
    filelog 
     
    exporters 
    : 
     
    - 
     
    chronicle/chronicle_w_labels 
    
   

   • Replace the port and IP address as required in your infrastructure.
   • Replace <customer_id> with the actual Customer ID.
   • Update /path/to/ingestion-authentication-file.json to the path where the authentication file was saved in the Get Google SecOps ingestion authentication file section.

Restart the Bindplane agent to apply the changes

To restart the Bindplane agent in Linux, run the following command:

   
sudo  
systemctl  
restart  
bindplane-agent 

UDM mapping table

Need more help? Get answers from Community members and Google SecOps professionals.