rwpmapbuild(1)

NAME
rwpmapbuild - Create a binary prefix map from a text file

SYNOPSIS
  rwpmapbuild [--input-file=FILENAME] [--output-file=FILENAME]
        [--mode={ipv4|ipv6|proto-port}] [--dry-run] [--ignore-errors]
        [--note-add=TEXT] [--note-file-add=FILENAME]
        [--invocation-strip]

  rwpmapbuild --help

  rwpmapbuild --version

DESCRIPTION
Prefix maps provide a way to map field values (specifically either IP addresses or protocol-port pairs) to string labels based on a user-defined map file. rwpmapbuild reads textual input to create a binary prefix map file. The syntax of this input is described in the "INPUT FILE FORMAT" section below.

As described in pmapfilter(3), you can partition, count, sort and display SiLK flow records based on the string labels defined in the prefix map. To view the contents of a prefix map file, use rwpmapcat(1). To query the contents of a prefix map, use rwpmaplookup(1).

The textual input is read from the specified input file, or from the standard input when the --input-file switch is not provided. The binary output is written to the named output file, or to the standard output when the --output-file switch is not provided and the standard output is not connected to a terminal.

OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

--input-file=FILENAME
    Read the textual input from FILENAME. You may use "stdin" or "-" to represent the standard input. When this switch is not provided, the input is read from the standard input unless the standard input is a terminal. rwpmapbuild will read textual input from the terminal if the standard input is explicitly specified as the input. The input file format is described below.

--output-file=FILENAME
    Write the binary prefix map to FILENAME. You may use "stdout" or "-" to represent the standard output. When this switch is not provided, the prefix map is written to the standard output unless the standard output is connected to a terminal.

--mode={ipv4|ipv6|proto-port}
    Specify the type of the input, as if a mode statement appeared in the input stream. The value specified by this switch must not conflict with an explicit mode statement appearing in the input.

--dry-run
    Do not write the output file. Simply check the syntax of the input file.

--ignore-errors
    Write the output file regardless of any errors encountered while parsing the input file.

--note-add=TEXT
    Add the specified TEXT to the header of the output file as an annotation. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool.

--note-file-add=FILENAME
    Open FILENAME and add the contents of that file to the header of the output file as an annotation. This switch may be repeated to add multiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation.

--invocation-strip
    Do not record the command used to create the prefix map in the output. When this switch is not given, the invocation is written to the file's header, and the invocation may be viewed with rwfileinfo(1). Since SiLK 3.12.0.

--help
    Print the available options and exit.

--version
    Print the version number and information about how SiLK was configured, then exit the application.

INPUT FILE FORMAT
The input file format consists of any number of input lines of the forms described below. Note that there is not a form that accepts a single IP address and a label; therefore, to provide a label for a single IP address you must append "/32" to a single IPv4 address (or "/128" to a single IPv6 address).

Blank lines in the input file are ignored, as are comments. Comments begin with the first "#" character on a line and extend to the end of the line.

rwpmapbuild maps ranges to string labels. These string labels may be created either explicitly via the label statement or implicitly by specifying text after a range, but a single input file must use only one method to create labels. When the label statement is used, all labels must be pre-declared in the label statement prior to their use in the default statement or an range statements.

In the following, the label-value represents either a numerical label identifier that was created with the label statement or label-text.

NOTE: Unlike many SiLK input files, there is no explicit delimiter between the range and the string label. The range and string label are separated by whitespace. The first non-whitespace character after the range begins the label.

label-text is a textual string that begins at the first non-whitespace character and extends to the final non-whitespace character on that line that does not appear in a comment. The label-text may include embedded whitespace and non-alphanumeric characters. While a comma (",") is legal in the label-text, using a comma prevents the label from being used by the --pmap-src and --pmap-dest switches in rwfilter(1).

The following statements are supported:

map-name simple-string
    Creates a name for the data in this prefix map file. The simple-string cannot contain whitespace, a comma, or a colon. When the prefix map file is used by rwfilter(1), the simple-string is used to generate the filtering switch names. When the prefix map file is used by rwcut(1), rwgroup(1), rwsort(1), rwstats(1), or rwuniq(1), the simple-string is used to generate the field names. See pmapfilter(3) for details.

label num label-text
    Associate the numeric identifier num with the given label text label-text. It is an error if num or label-text appear in any other label statement. The maximum allowed value for num is 2147483647, but note that rwpmapbuild creates an empty label for all the unassigned numeric identifiers that are less than the maximum identifier used in the input file. The label statement must appear before the default statement and before range definitions. When a label statement appears in the input, rwpmapbuild will complain if you attempt to use a label-value that was not previously defined in a label statement.

default label-value
    Make the given label identifier or label text the default value for any ranges not explicitly mentioned in this input file. The default statement must appear before any ranges are specified. If the default statement does not appear in the input, the label "UNKNOWN" is automatically defined and used as the default.

mode { ipv4 | ipv6 | proto-port | ip }
    Specify how to process the file. The mode statement must appear before any ranges are specified. The mode can also be set using the --mode command line switch. When both the mode statement and the --mode switch are given, their values must match. When neither the mode statement nor the --mode switch is provided, rwpmapbuild processes the input in IPv4 address mode. The ip mode is deprecated; it is an alias for ipv4.

  Address Mode
When rwpmapbuild is in IPv4 address mode, any IPv6 address in the input file will raise an error.

cidr-block label-value
    Associate the given label identifier or label text with this CIDR block. The CIDR block is composed of an IP address in canonical notation (e.g, dotted-decimal for IPv4), a slash "/", and the number of significant bits.

low-ip high-ip label-value
    Associate the given label identifier or label text with this IP range, where low-ip and high-ip are in canonical notation.

low-int high-int label-value
    Treat low-int and high-int as 32-bit values, convert the values to IPv4 addresses, and associate the given label identifier or label text with the IPv4 range.

  Protocol/Port Mode

proto/port proto/port label-value
    Associate the given label identifier or label text with all protocols and port numbers between these two values inclusive. Note that while port is not meaningful for all protocols (specifically, it is meaningful for TCP and UDP and may contain type/code information for ICMP), this file allows port numbers to be given for any protocol.

proto proto label-value
    Associate the given label identifier or label text with all protocols between these two values.

NOTES
The IP Address input file can contain nested CIDR blocks. They should be ordered with the more general blocks first, and the more specific blocks last. That is, use:

  10.0.0.0/8     My-network
  10.1.0.0/16    Special-Subnet-1
  10.1.2.0/24    Special-Subnet-2

Likewise, the protocol/port data can be nested:

  6 6            TCP
  6/0  6/1024    TCP/Generic reserved
  6/22 6/22      TCP/SSH
  6/25 6/25      TCP/SMTP
  6/80 6/80      TCP/HTTP

EXAMPLE
In the following examples, the dollar sign ("$") represents the shell prompt. The text after the dollar sign represents the command line. Lines have been wrapped for improved readability, and the back slash ("\") is used to indicate a wrapped line.

Reading and writing to a file:

 $ echo "10.1.2.3/32 my favorite host" > fav.txt
 $ rwpmapbuild -i fav.txt -o fav.pmap

Reading and writing to stdin and stdout:

 $ echo "10.9.8.128/27 suspicious subnet" \
   | rwpmapbuild --input-file=stdin --output-file=stdout > suspicious.pmap

  Complex IP File
  #    Numerical mappings of labels

  label 0             non-routable
  label 1             internal
  label 2             external

  #    Default to "external" for all un-defined ranges.

  default             external

  #    Force IP-mode

  mode                ip

  #    Create a name
  #
  #        This will add --pmap-src-network and --pmap-dst-network
  #        switches to rwfilter, and src-network and dst-network
  #        fields to rwcut, rwgroup, rwsort, rwstats, and rwuniq

  map-name            network

  ## Reserved and non-routable blocks ###########################

  #    Addresses in this block refer to source hosts on "this"
  #    network.  Address 0.0.0.0/32 may be used as a source
  #    address for this host on this network; other addresses
  #    within 0.0.0.0/8 may be used to refer to specified hosts
  #    on this network [RFC1700, page 4].

  0.0.0.0/8           non-routable

  #    This block is set aside for use in private networks.  Its
  #    intended use is documented in [RFC1918].  Addresses within
  #    this block should not appear on the public Internet.

  10.0.0.0/8          non-routable

  #    This block is assigned for use as the Internet host
  #    loopback address.  A datagram sent by a higher level
  #    protocol to an address anywhere within this block should
  #    loop back inside the host.  This is ordinarily
  #    implemented using only 127.0.0.1/32 for loopback, but no
  #    addresses within this block should ever appear on any
  #    network anywhere [RFC1700, page 5].

  127.0.0.0/8         non-routable

  #    This is the "link local" block.  It is allocated for
  #    communication between hosts on a single link.  Hosts
  #    obtain these addresses by auto-configuration, such as when
  #    a DHCP server may not be found.

  169.254.0.0/16      non-routable

  #    This block is set aside for use in private networks.  Its
  #    intended use is documented in [RFC1918].  Addresses within
  #    this block should not appear on the public Internet.

  172.16.0.0/12       non-routable

  #    This block is assigned as "TEST-NET" for use in
  #    documentation and example code.  It is often used in
  #    conjunction with domain names example.com or example.net
  #    in vendor and protocol documentation.  Addresses within
  #    this block should not appear on the public Internet.

  192.0.2.0/24        non-routable

  #    This block is set aside for use in private networks.
  #    Its intended use is documented in [RFC1918].  Addresses
  #    within this block should not appear on the public Internet.

  192.168.0.0/16      non-routable

  #    240.0.0.0/4 - This block, formerly known as the Class E
  #    address space, is reserved.  The "limited broadcast"
  #    destination address 255.255.255.255 should never be
  #    forwarded outside the (sub-)net of the source.  The
  #    remainder of this space is reserved for future use.
  #    [RFC1700, page 4]

  255.255.255.255/32  non-routable

  # -- Below this line, would add any mappings appropriate to
  # -- the local network.

  Complex Protocol/Port File
  #    Default to a hyphen ("-") for all un-defined ranges.

  default             -

  #    Force Protocol/Port-mode
  #
  #        This MUST be present, since IP mode is the default.

  mode                proto-port

  #    Protocol Overview

   1  1               ICMP
   6  6               TCP
  17 17               UDP
  50 50               ESP
  58 58               ICMPv6

  #    TCP -- Specific Ports

  6/0 6/1024          TCP/Generic Reserved
  6/21 6/21           TCP/ftp
  6/22 6/22           TCP/ssh
  6/25 6/25           TCP/smtp
  6/53 6/53           TCP/dns
  6/80 6/80           TCP/http
  6/6000 6/6063       TCP/X11

  #    UDP -- Specific Ports

  17/53 17/53         UDP/dns

  #    ICMP -- Specific Type/Code
  #
  #    To convert a type/code to a "port" value as stored in SiLK:
  #        (type << 8) | code     OR    (type * 256) + code
  #    so 3/3 (Destination Unreachable/Port Unreachable) becomes:

  1/771 1/771         ICMP/Destination Unreachable/Port Unreachable

ENVIRONMENT

SILK_CLOBBER
    The SiLK tools normally refuse to overwrite existing files. Setting SILK_CLOBBER to a non-empty value removes this restriction.

SEE ALSO
pmapfilter(3), rwfilter(1), rwfileinfo(1), rwpmapcat(1), rwpmaplookup(1), rwcut(1), rwgroup(1), rwsort(1), rwstats(1), rwuniq(1), silk(7)