User Guide
Also available as:
PDF

Using CLI to Query pcap Data

You can search or filter the PCAP data with one of the following command line tools:

  • Fixed filter option

  • Query filter option (Metron Stellar)

Filtering can be done both by the packet header as well as via a binary regular expression which can be run on the packet payload itself. This filter can be specified via:

  • The -pf or --packet_filter options for the fixed query filter

  • The BYTEARRAY_MATCHER(pattern, data) Stellar function. The first argument is the regex pattern and the second argument is the data. The packet data will be exposed via the packet variable in Stellar.

Both the fixed filter option tool and query filter option tool are executed by ${metron_home}/bin/pcap_query.sh [fixed|query]

[Note]Note

Because the output from a query can be very large, we recommend appending -rpf $Number_of_records to the end of the filter command. This argument creates multiple files, populating them with the specified number of records and titling them with timestamps.

You can filter or query for the following fields in the PCAP data:

  • ip_scr_addr

  • ip_dst_addr

  • ip_src_port

  • ip_dst_port

  • protocol

  • timestamp

Fixed filter options:

 -bop,--base_output_path <arg>   Query result output path. Default is
                                 '/tmp'.
 -bp,--base_path <arg>           Base PCAP data path. Default is
                                 '/apps/metron/pcap'.
 -da,--ip_dst_addr <arg>         Destination IP address.
 -df,--date_format <arg>         Date format to use for parsing start_time
                                 and end_time. Default is to use time in
                                 millis since the epoch.
 -dp,--ip_dst_port <arg>         Destination port.
 -pf, --packet_filter <arg>      Packet filter regex
 -et,--end_time <arg>            Packet end time range. Default is current
                                 system time.
 -nr,--num_reducers <arg>        The number of reducers to use.  Default
                                 is 10.
 -h,--help                       Display help.
 -ir,--include_reverse           Indicates if filter should check swapped
                                 src/dest addresses and IPs.
 -p,--protocol <arg>             IP Protocol.
 -rpf                            Maximum number of records per file. 
 -sa,--ip_src_addr <arg>         Source IP address.
 -sp,--ip_src_port <arg>         Source port.
 -st,--start_time <arg>          (required) Packet start time range.

Fixed filter examples:

$METRON_HOME/bin/pcap_query.sh fixed \
                                      -st "20160617" \
                                      -df "yyyyMMdd" \
                                      -sa 192.168.138.158 \
                                      -da 123.456.789.012 \
                                      -sp 49197 \
                                      -dp 80 \
                                      -p 6
                                      -rpf 500

To search for every packet that has an ip_dst_port of 8080 and contains the text "persist", run:

$METRON_HOME/bin/pcap_query.sh fixed \
          --ip_dst_port 8080 \
          --packet_filter \
          "\`persist\`" \
          -st "20170425" \
          -df "yyyyMMdd"

Query filter options:

 -bop,--base_output_path <arg>   Query result output path. Default is
                                 '/tmp'.
 -bp,--base_path <arg>           Base PCAP data path. Default is
                                 '/apps/metron/pcap'.
 -df,--date_format <arg>         Date format to use for parsing start_time
                                 and end_time. Default is to use time in
                                 millis since the epoch.
 -et,--end_time <arg>            Packet end time range. Default is current
                                 system time.
 -nr,--num_reducers <arg>        The number of reducers to use.  Default
                                 is 10.
 -h,--help                       Display help.
 -q,--query <arg>                Query string to use as a filter.
 -rpf                            Maximum number of records per file. 
 -st,--start_time <arg>          (required) Packet start time range.

The Query filter's --query argument specifies the Stellar expression to execute on each packet. To interact with the packet, a few variables are exposed:

  • packet : The packet data (a byte[])

  • ip_src_addr : The source address for the packet (a String)

  • ip_src_port : The source port for the packet (an Integer)

  • ip_dst_addr : The destination address for the packet (a String)

  • ip_dst_port : The destination port for the packet (an Integer)

  • BYTEARRAY_MATCHER : The first argument is the regex pattern and the second argument is the data. The packet data will be exposed via thepacket variable in Stellar.

Query filter examples:

$METRON_HOME/bin/pcap_query.sh query \
                                      -st "20160617" \
                                      -df "yyyyMMdd" \
                                      --query "ip_src_addr == '192.168.138.158' and ip_src_port == '49197' \
                                               and ip_dst_addr == '123.456.789.012' and ip_dst_port == '80' \
                                               and protocol == '6'"
                                       -rpf 500

To search for every packet that has an ip_dst_port of 8080 and contains the text "persist", run:

$METRON_HOME/bin/pcap_query.sh query \
          --query "ip_dst_port == 8080 && 
            BYTEARRAY_MATCHER('\`persist\`', packet)" \
            -st "20170425" \
            -df "yyyyMMdd"

You can also do proper binary regexes that look for packets containing the text "persist" and the 2 byte sequence 0x1F909 (in hex):

$METRON_HOME/bin/pcap_query.sh query \ 
          --query "BYTEARRAY_MATCHER('1F90', packet) && 
          BYTEARRAY_MATCHER('\`persist\`', packet)" \ 
          -st "20170425" \
          -df "yyyyMMdd"

Other examples:

$METRON_HOME/bin/pcap_query.sh query \
                                      -st "1466136000000" \
                                      --query "IN_SUBNET(ip_src_addr, '192.168.0.0/24') and ip_src_port == '49197' \
                                               and ip_dst_addr == '123.456.789.012' and ip_dst_port == '80' \
                                               and protocol == '6'"
                                      -rpf 500
# subnet function checks IP is in specified subnet
      --query "IN_SUBNET(ip_src_addr, '192.168.0.0/24') \
               and ip_src_port == '49197' \
               and ip_dst_addr == '123.456.789.012' \
               and ip_dst_port == '80' \
               and protocol == '6'"
# range queries on ports
      --query "ip_src_port <= 50000 and ip_dst_port >= 30000"
# range queries with conditionals and parens
      --query "(ip_src_port < 50000 and ip_src_port > 40000) \
            or (ip_src_port < 20000 and ip_src_port > 10000)"
# in/not in list of values
      --query "ip_src_port < 10000 and ip_dst_port in ['54056', '54057', '8080']"