rwpmaplookup(1)

NAME
rwpmaplookup - Map keys to prefix map entries

SYNOPSIS
  rwpmaplookup { --map-file=MAP_FILE | --address-types[=MAP_FILE]
                 | --country-codes[=MAP_FILE] }
        [--fields=FIELDS] [--ipset-files] [--no-errors]
        [--ip-format=FORMAT] [--integer-ips] [--zero-pad-ips]
        [--no-titles] [--no-columns] [--column-separator=CHAR]
        [--no-final-delimiter] [{--delimited | --delimited=CHAR}]
        [{--output-path=PATH | --pager=PAGER_PROG}]
        [--no-files ARG [ARGS...] | --xargs[=FILE] | FILE [FILES...]]

  rwpmaplookup --help

  rwpmaplookup --version

DESCRIPTION
rwpmaplookup finds keys in a binary prefix map file and prints the key and its value in a textual, bar (|) delimited format.

By default, rwpmaplookup expects its arguments to be the names of text files containing keys---one key per line. When the --ipset-files switch is given, rwpmaplookup takes IPset files as arguments and uses the IPs as the keys. The --no-files switch causes rwpmaplookup to treat each command line argument itself as a key to find in the prefix map.

When --no-files is not specified, rwpmaplookup reads the keys from the files named on the command line or from the standard input when no file names are specified and neither --xargs nor --no-files is present. To read the standard input in addition to the named files, use "-" or "stdin" as a file name. When the --xargs switch is provided, rwpmaplookup will read the names of the files to process from the named text file, or from the standard input if no file name argument is provided to the switch. The input to --xargs must contain one file name per line.

You must tell rwpmaplookup the prefix map to use for look-ups using one of three switches:

•   To use an arbitrary prefix map, use the --map-file switch.

•   If you want to map IP addresses to country codes (see ccfilter(3)), use the --country-codes switch. To use the default country code prefix map, do not provide an argument to the switch. To use a specific country code mapping file, specify the file as the argument.

•   If you want to map IP addresses to address types (see addrtype(3)), use the --address-types switch. To use the default address types prefix map, do not provide an argument to the switch. To use a specific address types mapping file, specify the file as the argument.

If the --map-file switch specifies a prefix map containing protocol/port pairs, each input file should contain one protocol/port pair per line in the form PROTOCOL/PORT, where PROTOCOL is a number between 0 and 255 inclusive, and PORT is a number between 0 and 65535 inclusive. When the --ipset-files switch is specified, it is an error if the --map-file switch specifies a prefix map containing protocol/port pairs.

When querying any other type of prefix map and the --ipset-files switch is not present, each textual input file should contain one IP address per line, where the IP is a single IP address (not a CIDR block) in canonical form or the integer representation of an IPv4 address.

The --fields switch allows you to specify which columns appear in the output. The default columns are the key and the value, where the key is the IP address or protocol/port pair, and the value is the textual label for that key.

If the prefix map contains IPv6 addresses, any IPv4 address in the input is mapped into the ::ffff:0:0/96 netblock when searching.

If the prefix map contains IPv4 addresses only, any IPv6 address in the ::ffff:0:0/96 netblock is converted to IPv4 when searching. Any other IPv6 address is ignored, and it is not printed in the output unless the "input" field is requested.

Prefix map files are created by the rwpmapbuild(1) and rwgeoip2ccmap(1) utilities. IPset files are created most often by rwset(1) and rwsetbuild(1).

OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

One of --map-file, --address-types, or --country-codes is required.

--map-file=PMAP_FILE
    Find the IP addresses or protocol/port pairs in the prefix map file PMAP_FILE.

--address-types
    Find the IP addresses in the address types (see addrtype(3)) mapping file specified by the SILK_ADDRESS_TYPES environment variable, or in the default address types mapping file if that environment variable is not set.

--address-types=ADDRTYPE_FILE
    Find the IP addresses in the address types mapping file specified by ADDRTYPE_FILE.

--country-codes
    Find the IP addresses in the country code (see ccfilter(3)) mapping file specified by the SILK_COUNTRY_CODES environment variable, or in the default country code mapping file if that environment variable is not set.

--country-codes=COUNTRY_CODE_FILE
    Find the IP addresses in the country code mapping file specified by COUNTRY_CODE_FILE.

--fields=FIELDS
    Specify the columns to include in the output. The columns will be displayed in the order the fields are specified. FIELDS is a comma separated list of field-names. Field-names are case-insensitive. When this switch is not provided, the default fields are "key,value". The list of available fields are:

    key The key used to search the prefix map.

    value
        The label returned from the prefix map for the key.

    block
        The block in the prefix map that contains the key. For a prefix map file that contains IPv4 addresses, the result will be a CIDR block such as 10.18.26.32/27.

    start-block
        The value at the start of the block in the prefix map that contains the key.

    end-block
        The value at the end of the block in the prefix map that contains the key.

    input
        The text read from the input file that rwpmaplookup attempted to parse. Note that blank lines, lines containing only whitespace and comments, and lines longer than 2048 characters will not be printed. In addition, any comments appearing after the text are stripped. When --ipset-files is specified, this field contains the IP address in its canonical form.

--no-files
    Causes rwpmaplookup to treat the command line arguments as the text to be parsed. This allows one to look up a handful of values without having to create a temporary file. Use of the --no-files switch disables paging of the output. This switch may not be combined with --ipset-files.

--no-errors
    Disables printing of errors when the input cannot be parsed as an IP address or a protocol/port pair. This switch is ignored when --ipset-files is specified.

--ipset-files
    Causes rwpmaplookup to treat the command line arguments as the names of IPset files to read and use as keys into the prefix map. It is an error to use this switch when --map-file specifies a protocol/port prefix map. When --ipset-files is active, the "input" column of --fields contains the IP in its canonical form, regardless of the --ip-format switch. This switch may not be combined with --no-files.

--ip-format=FORMAT
    When printing the key of an prefix map containing IP addresses, specify how IP addresses are printed. When this switch is not specified, the SILK_IP_FORMAT environment variable is checked for a format. If it is empty or contains an invalid format, IPs are printed in the canonical format. The FORMAT is one of:

    canonical
        Print IP addresses in their canonical form: dotted quad for IPv4 (127.0.0.1) and hexadectet for IPv6 ("2001:db8::1"). Note that IPv6 addresses in ::ffff:0:0/96 and some IPv6 addresses in ::/96 will be printed as a mixture of IPv6 and IPv4.

    zero-padded
        Print IP addresses in their canonical form, but add zeros to the output so it fully fills the width of column. The addresses 127.0.0.1 and "2001:db8::1" are printed as 127.000.000.001 and "2001:0db8:0000:0000:0000:0000:0000:0001", respectively.

    decimal
        Print IP addresses as integers in decimal format. The addresses 127.0.0.1 and "2001:db8::1" are printed as 2130706433 and 42540766411282592856903984951653826561, respectively.

    hexadecimal
        Print IP addresses as integers in hexadecimal format. The addresses 127.0.0.1 and "2001:db8::1" are printed as "7f000001" and "20010db8000000000000000000000001", respectively.

    force-ipv6
        Print all IP addresses in the canonical form for IPv6 without using any IPv4 notation. Any IPv4 address is mapped into the ::ffff:0:0/96 netblock. The addresses 127.0.0.1 and "2001:db8::1" are printed as "::ffff:7f00:1" and "2001:db8::1", respectively.

--integer-ips
    Print IP addresses as integers. This switch is equivalent to --ip-format=decimal, it is deprecated as of SiLK 3.7.0, and it will be removed in the SiLK 4.0 release.

--zero-pad-ips
    Print IP addresses as fully-expanded, zero-padded values in their canonical form. This switch is equivalent to --ip-format=zero-padded, it is deprecated as of SiLK 3.7.0, and it will be removed in the SiLK 4.0 release.

--no-titles
    Turn off column titles. By default, titles are printed.

--no-columns
    Disable fixed-width columnar output.

--column-separator=C
    Use specified character between columns and after the final column. When this switch is not specified, the default of '|' is used.

--no-final-delimiter
    Do not print the column separator after the final column. Normally a delimiter is printed.

--delimited
--delimited=C
    Run as if --no-columns --no-final-delimiter --column-sep=C had been specified. That is, disable fixed-width columnar output; if character C is provided, it is used as the delimiter between columns instead of the default '|'.

--output-path=PATH
    Determines where the output of rwpmaplookup is written. If this option is not given, output is written to the standard output.

--pager=PAGER_PROG
    When the --no-files switch has not been specified and output is to a terminal, invoke the program PAGER_PROG to view the output one screen full at a time. This switch overrides the SILK_PAGER environment variable, which in turn overrides the PAGER variable. If the value of the pager is determined to be the empty string, no paging will be performed and all output will be printed to the terminal.

--xargs
--xargs=FILENAME
    Causes rwpmaplookup to read file names from FILENAME or from the standard input if FILENAME is not provided. The input should have one file name per line. rwpmaplookup will open each file in turn and read the IPset, textual IP addresses, or textual protocol/port pairs from it, as if the files had been listed on the command line.

--help
    Print the available options and exit.

--version
    Print the version number and information about how SiLK was configured, then exit the application.

EXAMPLES
In the following examples, the dollar sign ("$") represents the shell prompt. The text after the dollar sign represents the command line. Lines have been wrapped for improved readability, and the back slash ("\") is used to indicate a wrapped line.

  Country code examples
Print the country code for a list of addresses read from the standard input.

 $ cat my-addrs.txt
 128.2.0.0
 128.2.0.1
 $ cat my-addrs.txt | rwpmaplookup --country-codes
             key|               value|
       128.2.0.0|                  us|
       128.2.0.1|                  us|

Use --no-files to list the address on the command line.

 $ rwpmaplookup --country-codes  128.2.0.0 128.2.0.1
             key|               value|
       128.2.0.0|                  us|
       128.2.0.1|                  us|

Use --ipset-files to read the addresses from an IPset file.

 $ rwsetbuild my-addrs.txt my-addrs.set
 $ rwpmaplookup --country-codes --ipset-files my-addrs.set
             key|               value|
       128.2.0.0|                  us|
       128.2.0.1|                  us|

Use the --fields switch to control which columns are printed.

 $ rwpmaplookup --country-codes --fields=value my-addrs.txt
                value|
                   us|
                   us|

Add the --delimited and --no-titles switches so the output only contains the value column. Print the country code for a single address using the default country code prefix map.

 $ rwpmaplookup --country-codes --fields=value --delimited \
        --no-titles --no-files 128.2.0.0
 us

Alternatively

 $ echo 128.2.0.0   \
   | rwpmaplookup --country-codes --fields=value --delim --no-title
 us

To use a different country code mapping file, provide that file as the argument to the --country-codes switch.

 $ rwpmaplookup --country-code=old-address-map.pmap --no-files 128.2.0.0
           key|value|
     128.2.0.0|   us|

  CIDR block input
Note that rwpmaplookup does not parse text that contains CIDR blocks.

 $ echo '128.2.0.0/31'      \
   | rwpmaplookup --country-codes
             key|value|
 rwpmaplookup: Invalid IP '128.2.0.1/31' at -:1: Extra text follows value

For this case, use the IPset tool rwsetbuild(1) to parse the CIDR block list and create a binary IPset stream, and pipe the IPset to rwpmaplookup.

 $ echo '128.2.0.0/31'      \
   | rwsetbuild             \
   | rwpmaplookup --country-code --ipset-files
             key|value|
       128.2.0.0|   --|
       128.2.0.1|   --|

For versions of rwpmaplookup that do not have the --ipset-files switch, you can have rwsetcat(1) read the binary IPset stream and print the IP addresses as text, and pipe that into rwpmaplookup. Be sure to include the "--cidr-blocks=0" switch to rwsetcat which forces individual IP addresses to be printed.

 $ echo '128.2.0.0/31'              \
   | rwsetbuild                     \
   | rwsetcat --cidr-blocks=0       \
   | rwpmaplookup --country-code
             key|value|
       128.2.0.0|   --|
       128.2.0.1|   --|

  General prefix map usage
Consider a user-defined prefix map, assigned-slash-8s.pmap, that maps each /8 in the IPv4 address space to its assignment.

 $ rwpmapcat assigned-slash-8s.pmap | head -4
            ipBlock|                                         label|
          0.0.0.0/8|                   IANA - Local Identification|
          1.0.0.0/8|                                         APNIC|
          2.0.0.0/8|                                      RIPE NCC|

Use the --map-file switch to map from IPs to labels using this prefix map.

 $ cat my-addrs.txt
 17.17.17.17
 9.9.9.9
 $ cat my-addrs.txt | rwpmaplookup --map-file=assigned-slash-8s.pmap
             key|               value|
     17.17.17.17| Apple Computer Inc.|
         9.9.9.9|                 IBM|

Use --ip-format=decimal to print the output as integers.

 $ cat my-addrs.txt         \
   | rwpmaplookup --ip-format=decimal --map-file=assigned-slash-8s.pmap
        key|               value|
  286331153| Apple Computer Inc.|
  151587081|                 IBM|

Add the "input" field to see the input as well.

 $ cat my-addrs.txt         \
   | rwpmaplookup --ip-format=decimal --fields=key,value,input \
        --map-file=assigned-slash-8s.pmap
        key|               value|               input|
  286331153| Apple Computer Inc.|         17.17.17.17|
  151587081|                 IBM|             9.9.9.9|

Combine the "input" field with the --no-errors switch to see a row for each key.

 $ rwpmaplookup --fields=key,value,input --no-errors --no-files \
        --map-file=assigned-slash-8s.pmap 9.9.9.9 17.1717.17
             key|               value|               input|
         9.9.9.9| Apple Computer Inc.|             9.9.9.9|
                |                    |          17.1717.17|

The input can contain integer values.

 $ echo 151587081           \
   | rwpmaplookup --fields=key,value,input --delimited=, \
        --map-file=assigned-slash-8s.pmap
 key,value,input
 9.9.9.9,IBM,151587081

  Block output
Specifying "block" in the --fields switch causes rwpmaplookup to print the CIDR block that contains the address key.

 $ cat my-addrs.txt
 9.8.7.6
 9.10.11.12
 17.16.15.14
 17.18.19.20
 $ rwpmaplookup --map-file=assigned-slash-8s.pmap \
        --fields=key,value,block my-addrs.txt
             key|               value|             block|
         9.8.7.6|                 IBM|         9.0.0.0/8|
      9.10.11.12|                 IBM|         9.0.0.0/8|
     17.16.15.14| Apple Computer Inc.|        17.0.0.0/8|
     17.18.19.20| Apple Computer Inc.|        17.0.0.0/8|

To break the CIDR block into its starting and ending value, specify the "start-block" and "end-block" fields.

 $ rwpmaplookup --map-file=assigned-slash-8s.pmap               \
        --fields=key,value,start-block,end-block my-addrs.txt
             key|               value|    start-block|      end-block|
         9.8.7.6|                 IBM|        9.0.0.0|  9.255.255.255|
      9.10.11.12|                 IBM|        9.0.0.0|  9.255.255.255|
     17.16.15.14| Apple Computer Inc.|       17.0.0.0| 17.255.255.255|
     17.18.19.20| Apple Computer Inc.|       17.0.0.0| 17.255.255.255|

To get a unique list of blocks for the input keys, do not output the "key" field and pipe the output of rwpmaplookup to the uniq(1) command. (This works as long as the input data is sorted).

 $ cat my-addrs.txt                                 \
   | rwpmaplookup --map-file=assigned-slash-8s.pmap \
        --fields=block,value                        \
   | uniq
              block|               value|
          9.0.0.0/8|                 IBM|
         17.0.0.0/8| Apple Computer Inc.|

The values printed in the "block" column corresponds to the CIDR block that were used when the prefix map file was created.

 $ rwpmaplookup --map=assigned-slash-8s.pmap --fields=block,value   \
        --no-files 128.2.0.1 129.0.0.1
              block|               value|
        128.0.0.0/8|Administered by ARIN|
        129.0.0.0/8|Administered by ARIN|

In the output from rwpmapcat(1), those two blocks are combined into a larger range.

 $ rwpmapcat --map=assigned-slash-8s.pmap | grep 128
        128.0.0.0/6|Administered by ARIN|

  Working with IPsets
Assume you have a binary IPset file, my-ips.set, that has the contents shown here, and you want to find the list of unique assignments from the assigned-slash-8s.pmap file.

 $ rwsetcat --cidr-blocks=1 my-ips.set
 9.9.9.0/24
 13.13.13.0/24
 15.15.15.0/24
 16.16.16.0/24
 17.17.17.0/24
 18.18.18.0/24

Since the blocks in the assigned-slash-8s.pmap file are /8, use the rwsettool(1) command to mask the IPs in the IPset to the unique /8 that contains each of the IPs.

 $ rwsettool --mask=8 my-ips.set    \
   | rwpmaplookup --map-file=assigned-slash-8s.pmap
            key|                        value|
        9.0.0.0|                          IBM|
       13.0.0.0|            Xerox Corporation|
       15.0.0.0|      Hewlett-Packard Company|
       16.0.0.0|Digital Equipment Corporation|
       17.0.0.0|          Apple Computer Inc.|
       18.0.0.0|                          MIT|

  Protocol/port prefix maps
Assume the service.pmap prefix map file maps protocol/port pairs to the name of the service running on the named port.

 $ rwpmapcat service.pmap
 startPair|  endPair|    label|
       0/0|  0/65535|  unknown|
       1/0|  1/65535|     ICMP|
       2/0|  5/65535|  unknown|
       6/0|     6/21|      TCP|
      6/22|     6/22|  TCP/SSH|
 ...
      17/0|    17/52|      UDP|
     17/53|    17/53|  UDP/DNS|
 ...

To query this prefix map, the input must contain two numbers separated by a slash.

 $ rwpmaplookup --map-file=service.pmap --no-files 6/80
       key|    value|
      6/80| TCP/HTTP|

Specifying "block", "start-block", and "end-block" in the --fields switch also works for Protocol/port prefix map files. The "block" column contains the same information as the "start-block" and "end-block" columns separated by a single space.

 $ rwpmaplookup --map-file=service.pmap --no-files  \
        --fields=key,value,start,end,block          \
        6/80 6/6000 17/0 17/53 128/128
       key|     value|start-blo|end-block|              block|
      6/80|  TCP/HTTP|     6/80|     6/80|          6/80 6/80|
    6/6000|       TCP|   6/4096|   6/6143|      6/4096 6/6143|
      17/0|       UDP|     17/0|    17/31|         17/0 17/31|
     17/53|   UDP/DNS|    17/53|    17/53|        17/53 17/53|
   200/200|Unassigned|    192/0|223/65535|    192/0 223/65535|

Using the pmapfilter(3) plug-in to rwcut(1), you can print the label for the source port and destination port in the SiLK Flow file data.rw.

 $ rwcut --pmap-file=service.pmap --num-rec=5       \
        --fields=proto,sport,src-service,dport,dst-service data.rw
 pro|sPort|src-service|dPort|dst-service|
  17|29617|        UDP|   53|    UDP/DNS|
  17|   53|    UDP/DNS|29617|        UDP|
   6|29618|        TCP|   22|    TCP/SSH|
   6|   22|    TCP/SSH|29618|        TCP|
   1|    0|       ICMP|  771|       ICMP|

The pmapfilter plug-in does not provide a way to print the values based on the application field. You can get that information by having rwcut print the protocol and application separated by a slash, and pipe the result into rwpmaplookup.

 $ rwcut --fields=proto,application --num-rec=5     \
        --delimited=/ --no-title                    \
   | rwpmaplookup --map-file=service.pmap
       key|    value|
     17/53|  UDP/DNS|
     17/53|  UDP/DNS|
      6/22|  TCP/SSH|
      6/22|  TCP/SSH|
       1/0|     ICMP|

ENVIRONMENT

SILK_IP_FORMAT
    This environment variable is used as the value for --ip-format when that switch is not provided. Since SiLK 3.11.0.

SILK_PAGER
    When set to a non-empty string, rwpmaplookup automatically invokes this program to display its output a screen at a time unless the --no-files switch is given. If this variable is set to an empty string, rwpmaplookup does not automatically page its output.

PAGER
    When set and SILK_PAGER is not set, rwpmaplookup automatically invokes this program to display its output a screen at a time.

SILK_COUNTRY_CODES
    This environment variable allows the user to specify the country code mapping file to use when the --country-codes switch is specified without an argument. The variable's value may be a complete path or a file relative to SILK_PATH. See the "FILES" section for standard locations of this file.

SILK_ADDRESS_TYPES
    This environment variable allows the user to specify the address type mapping file to use when the --address-types switch is specified without an argument. The variable's value may be a complete path or a file relative to the SILK_PATH. See the "FILES" section for standard locations of this file.

SILK_CLOBBER
    The SiLK tools normally refuse to overwrite existing files. Setting SILK_CLOBBER to a non-empty value removes this restriction.

SILK_PATH
    This environment variable gives the root of the install tree. When searching for configuration files, rwpmaplookup may use this environment variable. See the "FILES" section for details.

FILES

${SILK_COUNTRY_CODES}
${SILK_PATH}/share/silk/country_codes.pmap
${SILK_PATH}/share/country_codes.pmap
/usr/share/silk/country_codes.pmap
/usr/share/country_codes.pmap
    Possible locations for the country codes mapping file when the --country-codes switch is specified without an argument.

${SILK_ADDRESS_TYPES}
${SILK_PATH}/share/silk/address_types.pmap
${SILK_PATH}/share/address_types.pmap
/usr/share/silk/address_types.pmap
/usr/share/address_types.pmap
    Possible locations for the address types mapping file when the --address-types switch is specified without an argument.

NOTES
rwpmaplookup was added in SiLK 3.0.

rwpmaplookup duplicates the functionality of rwip2cc(1). rwip2cc is deprecated, and it will be removed in the SiLK 4.0 release. Examples of using rwpmaplookup in place of rwip2cc are provided in the latter's manual page.

SEE ALSO
rwpmapbuild(1), rwpmapcat(1), ccfilter(3), addrtype(3), pmapfilter(3), rwgeoip2ccmap(1), rwcut(1), rwset(1), rwsetbuild(1), rwsetcat(1), rwsettool(1), silk(7)