RWTUC(1)

NAME
rwtuc - Text Utility Converter - rwcut output to SiLK flows

SYNOPSIS
  rwtuc [--fields=FIELDS] [--column-separator=CHAR]
        [--output-path=FILEPATH] [--bad-input-lines=FILEPATH]
        [--verbose] [--stop-on-error] [--note-add=TEXT]
        [--note-file-add=FILE] [--compression-method=COMP_METHOD]
        [--site-config-file=FILENAME] [--saddress=IPADDR]
        [--daddress=IPADDR] [--sport=NUM] [--dport=NUM]
        [--protocol=NUM] [--packets=NUM] [--bytes=NUM]
        [--flags-all=TCPFLAGS] [--stime=TIME] [--duration=NUM]
        [--etime=TIME] [--sensor=SID] [--input-index=NUM]
        [--output-index=NUM] [--next-hop-ip=IPADDR]
        [--flags-initial=TCPFLAGS] [--flags-session=TCPFLAGS]
        [--attributes=ATTR] [--application=NUM] [--class=NAME]
        [--type=NAME] [--stime+msec=TIME] [--etime+msec=TIME]
        [--duration+msec=NUM] [FILES]

  rwtuc --help

  rwtuc --version

DESCRIPTION
rwtuc reads text files that have a format similar to that produced by rwcut(1) and attempts to create a SiLK Flow record for each line of input.

The field which make up a single record should be separated by the pipe character ('|'); use the --column-separator switch to change this delimiter. Note that space will not work as delimiter since several fields (e.g., time, tcp-flags) contain embedded spaces.

The fields to be read from each line can be specified with the --fields switch; if the switch is not provided, rwtuc treats the first line as a title and attempts to determine the fields from the title strings.

Fields can be forced to have a fixed value, which causes rwtuc to override the value read from the input file (if any).

The textual input is read from the files named on the command line; if no files are specified, rwtuc attempts to read the text from the standard input if it is not connected to a terminal.

When the --output-path switch is not provided, output is sent to the standard output when it is not connected to a terminal.

By default, lines that cannot be parsed are silently ignored (unless rwtuc is attempting to determine the fields from the title line). When the --verbose switch is specified, problems parsing an input line will be reported to the standard error, and rwtuc will continue to process the input. The --stop-on-error switch is similar to the --verbose switch, except processing stops after the first error. Input lines that cause parse errors can be copied to another output stream with the --bad-input-lines switch. Each bad line will have the source file name and line number prepended to it, separated from each other and the source line by colons (':').

Due to the way records are stored, some field combinations cannot be supported. The end time is computed by adding the duration to the start time; when start time and duration are specified, any end-time value is ignored. The initial TCP flags and session TCP flags fields must both be present or absent. Using the attributes or application fields requires using the initial TCP flags field. The class and type field must both be present for the fields to have any affect on the SiLK flow record.

OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

--fields=FIELDS
    FIELDS contains the list of fields (columns) to parse. FIELDS is a comma separated list of field-names, field-integers, and ranges of field-integers; a range is specified by separating the start and end of the range with a hyphen (-). Field-names are case insensitive.

    The field names and their descriptions are:

    sIP,1
        source IP address in the canonical form: dotted-quad for IPv4 or hex-encoded for IPv6 (when SiLK has been compiled with IPv6 support). Integers from 0 to 4294967295 will be treated as an IPv4 address.

    dIP,2
        destination IP address in the same format as "sIP,1"

    sPort,3
        source port as an integer from 0 to 65535 inclusive

    dPort,4
        destination port as an integer from 0 to 65535 inclusive

    protocol,5
        IP protocol as an integer from 0 to 255 inclusive

    packets,pkts,6
        packet count as an integer from 1 to 4294967295 inclusive

    bytes,7
        byte count as an integer from 1 to 4294967295 inclusive

    flags,8
        bitwise OR of TCP flags over all packets; the string may contain "F", "S", "R", "P", "A", "U", "E", "C" in upper- or lowercase

    sTime,9
        starting time of the flow, in the form "YYYY/MM/DD[:hh[:mm[:ss[.sss]]]]". A "T" may be used to separate the day and hour fields. A floating point value between 536870912 and 4294967295 is also allowed and will be treated as seconds since the UNIX epoch.

    dur,10
        duration of flow as a floating point value from 0.0 to 4294967.295

    eTime,11
        end time of flow in the same form as "sTime,9". This field is ignored when both "sTime" and "dur" are also present.

    sensor,12
        router sensor name or ID as given in silk.conf

    class
        class of router at collection point as given in silk.conf

    type
        type of router at collection point as given in silk.conf

    Many of our packed files do not store the following fields and their values will always be 0, but they are listed here for completeness:

    in,13
        router SNMP input interface or vlanId if packing tools were configured to capture it (see sensor.conf(5)); an integer from 1 to 65535

    out,14
        router SNMP output interface or portVlanId; an integer from 1 to 65535

    nhIP,15
        router next hop IP address in the same format as "sIP,1"

    SiLK can store flows generated by enhanced collection software that provides more information than NetFlow v5. These flows may support some or all of these additional fields; for flows without this additional information, the field's value is always 0.

    initialFlags,26
        TCP flags on first packet in the flow; same form as the "flags,8" field

    sessionFlags,27
        bitwise OR of TCP flags over all packets except the first in the flow; same form as the "flags,8" field

    attribute,28
        flow attributes set by the flow generator:

        "F" flow generator saw additional packets in this flow following a packet with a FIN flag (excluding ACK packets)

        "T" flow generator prematurely created a record for a long-running connection due to a timeout. (When the flow generator yaf(1) is run with the --silk switch, it will prematurely create a flow and mark it with "T" if the byte count of the flow cannot be stored in a 32-bit value.)

        "C" flow generator created this flow as a continuation of long-running connection, where the previous flow for this connection met a timeout (or a byte threshold in the case of yaf).

        Consider a long-running ssh session that exceeds the flow generator's active timeout. (This is the active timeout since the flow generator creates a flow for a connection that still has activity). The flow generator will create multiple flow records for this ssh session, each spanning some portion of the total session. The first flow record will be marked with a "T" indicating that it hit the timeout. The second through next-to-last records will be marked with "TC" indicating that this flow both timed out and is a continuation of a flow that timed out. The final flow will be marked with a "C", indicating that it was created as a continuation of an active flow.

    application,29
        guess as to the content the flow, as an integer from 0 to 65535. Some software that generates flow records from packet data, such as yaf, will inspect the contents of the packets that make up a flow and use traffic signatures to label the content of the flow. SiLK calls this label the application; yaf refers to it as the appLabel. The application is the port number that is traditionally used for that type of traffic (see the /etc/services file on most UNIX systems). For example, traffic that the flow generator recognizes as FTP will have a value of 21, even if that traffic is being routed through the standard HTTP/web port (80).

    Fields may not be specified more than once.

--column-separator=CHAR
    Expect the character CHAR to used as the delimiter between columns instead of the default '|'.

--output-path=FILEPATH
    Write the SiLK Flow records to FILEPATH. The strings "stdout" and "stderr" may be used for the standard output and standard error, respectively, as long as they are not connected to a terminal.

--bad-input-lines=FILEPATH
    Copy any lines which could not be parsed to FILEPATH. The strings "stdout" and "stderr" may be used for the standard output and standard error, respectively. Each bad line will be prepended by the source input file, a colon, the line number, and a colon. On exit, rwtuc will remove FILEPATH if all input lines were successfully parsed.

--verbose
    If an input line fails to parse, print a message describing the error to the standard error, and continue to process the input.

--stop-on-error
    If an input line fails to parse, print a message describing the error to the standard error and exit. The output file will contain any records successfully created prior to reading the bad input line.

--note-add=TEXT
    Add the specified TEXT to the header of the output file as an annotation. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool.

--note-file-add=FILENAME
    Open FILENAME and add the contents of that file to the header of the output file as an annotation. This switch may be repeated to add multiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation.

--compression-method=COMP_METHOD
    Set the compression method of the output to COMP_METHOD. Some SiLK tools can use an external library to compress their binary output. The list of available compression methods and the default method are set when SiLK is compiled (the --help and --version switches print the available and default compression methods) and depend on which supported libraries are found. SiLK can support:

    none
        Do not compress the output using an external library

    zlib
        Use the zlib(3) library for compressing the output

    lzo1x
        Use the lzo1x algorithm from the LZO real time compression library for compression

    best
        Use whichever available method gives the "best" compression in general, though not necessarily the "best" for this particular output.

--site-config-file=FILENAME
    Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, the location specified by the SILK_CONFIG_FILE environment variable is used if that variable is not empty. The value of SILK_CONFIG_FILE should include the name of the file. Otherwise, the application looks for a file named silk.conf in the following directories: the directory specified in the SILK_DATA_ROOTDIR environment variable; the data root directory that is compiled into SiLK (use the --version switch to view this value); the directories $SILK_PATH/share/silk/ and $SILK_PATH/share/; and the share/silk/ and share/ directories parallel to the application's directory.

--help
    Print the available options and exit.

--version
    Print the version number and information about how SiLK was configured, then exit the application.

The following switches can be used to set fields to fixed values:

--saddress=IPADDR
    Set the source address field to IPADDR for all records. IPADDR can be in canonical notation or an unsigned integer.

--daddress=IPADDR
    Set the destination address field to IPADDR for all records. IPADDR can be in canonical notation or an unsigned integer.

--sport=NUM
    Set the source port field to NUM for all records; a value between 0 and 65535.

--dport=NUM
    Set the destination port field to NUM for all records; a value between 0 and 65535.

--protocol=NUM
    Set the protocol field to NUM for all records; a value between 0 and 255.

--packets=NUM
    Set the packets field to NUM for all records; the value must be non-zero.

--bytes=NUM
    Set the bytes field to NUM for all records; the value must be non-zero.

--flags-all=TCPFLAGS
    Set the TCP flags field to TCPFLAGS for all records.

--stime=TIME
    Set the start time field to TIME for all records.

--duration=NUM
    Set the duration field to NUM for all records.

--etime=TIME
    Set the end time field to TIME for all records.

--sensor=SID
    Set the sensor field to SID for all records. This can either be a sensor name or sensor ID.

--input-index=NUM
    Set the SNMP input index field to NUM for all records; a value between 0 and 65535.

--output-index=NUM
    Set the SNMP output index field to NUM for all records; a value between 0 and 65535.

--next-hop-ip=IPADDR
    Set the next-hop-ip field to IPADDR for all records. IPADDR can be in canonical notation or an unsigned integer.

--flags-initial=TCPFLAGS
    Set the initial TCP flags field to TCPFLAGS for all records.

--flags-session=TCPFLAGS
    Set the session TCP flags field to TCPFLAGS for all records.

--attributes=ATTR
    Set the attributes field to ATTR for all records.

--application=NUM
    Set the application field to NUM for all records; a value between 0 and 65535.

--class=NAME
    Set the class field to NAME for all records. Should be a valid class name; use rwfilter --help to see the list of valid classes for your site.

--type=NAME
    Set the type field to NAME for all records. Should be a valid type name; use rwfilter --help to see the list of valid types for your site.

--stime+msec=TIME
    Set the start time field (including milliseconds) to TIME for all records.

--etime+msec=TIME
    Set the end time field (including milliseconds) to TIME for all records.

--duration+msec=NUM
    Set the duration field (including milliseconds) to NUM for all records.

EXAMPLES
 $ rwcut data.rwf > cut.txt
 $ md5 < cut.txt
 7e3d693cd2cba2510803935274e1debd
 $ rwtuc < cut.txt | rwcut | md5
 7e3d693cd2cba2510803935274e1debd

ENVIRONMENT

SILK_CONFIG_FILE
    This environment variable is used as the value for the --site-config-file when that switch is not provided.

SILK_DATA_ROOTDIR
    When the --site-config-file switch is not provided and the SILK_CONFIG_FILE environment variable is not set, rwtuc looks for the site configuration file in $SILK_DATA_ROOTDIR/silk.conf.

SILK_PATH
    This environment variable gives the root of the install tree. As part of its search for the SiLK site configuration file, rwtuc checks for a file named silk.conf in the directories $SILK_PATH/share/silk and $SILK_PATH/share.

SEE ALSO
rwfilter(1), rwfileinfo(1), rwcut(1), sensor.conf(5), yaf(1), zlib(3)