Software Components and Usage
Executables are located in the /opt/trafMon/bin and /opt/trafMon/testing directories:
| bin/tmon_collector |
| bin/tmon_probe |
| bin/trafMon_loader.py |
| bin/trafMon_updateIpInfo.py |
| bin/trafMon_FormatNetFlow.py |
| bin/trafMon_volumeReportGen.py |
| bin/trafMon_detailReportGen.py |
| bin/trafMon_probeResurrect.sh.sample |
| bin/trafMon_serverResurrect.sh.sample |
| testing/trafgen |
| testing/ftpgen.py |
| testing/trafgen.py |
Some MySQL stored procedures, provided in the trafMon_template database, are also for direct use:
| trafMon_template.Update_ftp_data_in_ipcttable() |
| trafMon_template.Aggr_activityvolumetable_first_level() |
| trafMon_template.Aggr_activityvolumetable_netflow_first_level() |
| trafMon_template.Drop_working_tables() |
| trafMon_template.Partition_drop() |
The most important commands are explained here below.
1. The Probes
The probe executable has the following usage:
tmon_probe [-l] [-c configXML] [-n NEWconfigXML] probeName
USAGE:
-l (local) means using ./tmon.xml, ./tmon-new.xml and,
if it exists, ./tmon_probe.diag
-c means use the given XML, and nothing else
-n monitors the given NEW XML for scheduled config update
(based on its `startAt' attribute
If -l is NOT given, /etc/trafMon/xml/tmon.xml is used
If -l and -n are NOT given, /etc/trafMon/xml/tmon-new.xml is looked at
If -l is NOT given, or ./tmon_probe.diag doesn't exist,
/etc/trafMon/diag/<probeName>.diag is used
2. The Central Processing System
2.1 The Collector
The collector executable has the following usage:
tmon_collector [-l] [-c configXML] [-n NEWconfigXML] collectorName
USAGE:
-l (local) means using ./tmon.xml, ./tmon-new.xml and, if it exists, ./tmon_collector.diag
-c means use the given XML, and nothing else
-n monitors the given NEW XML for scheduled config update
(based on its `startAt' attribute
If -l is NOT given, /etc/trafMon/xml/tmon.xml is used
If -l and -n are NOT given, /etc/trafMon/xml/tmon-new.xml is looked at
If -l is NOT given, or ./tmon_collector.diag doesn't exist,
/etc/trafMon/diag/<collectorname>.diag is used
2.2. The scripts
This section describes the different scripts used by the trafMon software for a variety of tasks. The scripts are located in /opt/trafMon/bin/ unless mentioned otherwise explicitly.
trafMon_loader.py [options]
Options:
-h, --help show this help message and exit
-p PATH, --path=PATH Path to the directory containing the data file to load
into the DB (default: /var/trafMon/collector/
-l, --localConfig If -l is specified, the db.cred file in the current
directory is used. Otherwise, the default
/etc/tmon/cred/db.cred file is loaded
-L LOGFILEDIRECTORY, --logFileDirectory=LOGFILEDIRECTORY
Path to log directory (default: /var/trafMon/log/)
This script loads the probe and collector data, and optionally the SiLK NetFlow records, into the MySQL database.
Bulk load in temporary tables, then update persistent lists and persistent counters aggregates at 1 minute, 1 hour and 1 day.
trafMon_updateIpInfo.py [options]
Options:
-h, --help show this help message and exit
-p INFOFILE, --path=INFOFILE
File pathname to the .ini file with information about
the known IP adresses/segments. (Default:
/etc/trafMon/ipInfo.ini)
-l, --localConfig If -l is specified, the db.cred file in the current
directory is used. Otherwise, the default
/etc/tmon/cred/db.cred DB configuration file is loaded
-g GEOIPPATH, --geoipPath=GEOIPPATH
Path to the directory with GeoLite2 or Maxmind GeoIP2
databases (Default: /var/trafMon/GeoIP/)
-a, --all Activate a full update of the ipInfoTable rather than
a partial one (partial means that we do not try to
resolve addresses which have already been successfully
resolved previously).
-L LOGDIR, --logFileDirectory=LOGDIR
Path to log directory (default: /var/trafMon/log/)
This script fills the MySQL “ipInfotable” table relating an IP with a location, an activity and/or a country, a city and an ASN, and performs a DNS reverse lookup.
It applies to unsolved IP addresses (typically those freshly discovered; or, with –all, it conducts a complete re-scan of all registered IP addresses.
MAXMIND® free GeoLite2™ IP geolocation database files
The MaxMind Company sells GeOIP registries. It is also providing free access to GeoLite2 semi-accurate version of their commercial product. But, since the legal enforcement of privacy protection of personal data (in particular in California and the EU), users must first register at their site before getting the database files: https://dev.maxmind.com/geoip/geoip2/geolite2/. Please carefully analyse the licensing conditions. The databases used to obtain information about an IP should be downloaded or copied in /var/trafMon/GeoIP/.
Previously, the following commands could be used while being in the said directory:
– wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
– wget http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz
– gunzip *.gz
These commands downloaded the two database files used by the script and decompress them.
trafMon_FormatNetFlow.py [options]
Options:
-h, --help show this help message and exit
-H nbOfHours, --hour=nbOfHours
Number of hours of data to process (starting from the
past hour). This option has precedence to the
start/end date options. E.g.: '-H 2' at 14:36:27 means
data for [12:00:00, 14:00:00]
-S startTime, --dateStart=startTime
Time at which the collection of data will begin
(format: YYYY/MM/DD:HH). The data for the given hour
are included in the results.
-E endTime, --dateEnd=endTime
Time at which the collection of data will end (format:
YYYY/MM/DD:HH). The data for the given hour are
included in the results.
-N SRCPATH, --netflowdata=SRCPATH
Path to the root of the SiLK collected NetFlow data
(default: /var/silk/data/)
-D DSTPATH, --collectordata=DSTPATH
Path to the runtime observations logs produced by the
trafMon Collector, where to place the NetFlow
extracted log (default: /var/trafMon/collector/)
-C SILKCONFIG, --silkconfig=SILKCONFIG
Path to SiLK config file (default: /var/silk/twoway-
silk.conf)
-s SILKBIN, --silkbin=SILKBIN
Path where to find SiLK rwfilter and rwcut utilities
(default: /usr/local/bin/)
-L LOGDIR, --logFileDirectory=LOGDIR
Path to log directory (default: /var/log/trafMon/)
This script retrieves, every hour, the NetFlow records received and stored in the SiLK tree of raw log binary log files. The resulting text file is added to the trafMon collector output, subject to next run of trafMon_loader.py.
trafMon_volumeReportGen.py [options]
Options:
-h, --help show this help message and exit
-d DBname, --db=DBname
Database to be used. Default to 'trafMon'.
-r report, --report=report
Type of synthesis report to be generated. Possible
choices are: [manager, operator, conversation].
Default to 'manager'.
-D destination, --destination=destination
Destination directory. Default to
'/var/trafMon/reports/2020/11/13', where the
'YYYY/MM/DD' part is the generation time of the report
(today).
-t top, --top=top Top-N to be used. Possible choices are: [5, 10, 15,
20, 25]. Default to top-5.
-T threshold, --threshold=threshold
Threshold bandwidth in b/s to be used. Possible
choices are: [0, 1000, 10000, 50000, 100000, 500000].
Default to 1000.
-A activityName, --activity=activityName
Activity to be used. Default to 'any'. Use quotes if
the activity name include a space.
-L locationName, --location=locationName
Location to be used. Default to 'any'. Use quotes if
the location name include a space.
-H hostName, --host=hostName
Host to be used. Default to 'any'. Use quotes if the
host name include a space.
-s startDate, --startDate=startDate
Start date to be used (format: 'YYYY-MM-DD'). Default
to first day of previous month.
-e endDate, --endDate=endDate
End date to be used (format: 'YYYY-MM-DD'). Default to
last day of previous month.
-l LOGDIR, --logFileDirectory=LOGDIR
Path to log directory (default: /var/log/trafMon/)
-R TEMPLATESFOLDER, --reportTemplatesDirectory=TEMPLATESFOLDER
Path to trafMon report templates directory. Default:
/opt/trafMon/trafMon_reports/
-g GENREPORT, --genReport_sh=GENREPORT
Full pathname to the Birt runtime 'genReport.sh'
utility. Default to /opt/trafMon/bin/genReport.sh,
which is typically a symbolic link to the BIRT RunTime
installation/ReportEngine/genReport.sh
This script is used to automatically produce a sysnthesis PDF report.
trafMon_detailReportGen.py [options]
Options:
-h, --help show this help message and exit
-l, --localConfig if -l is specified, db.cred file is fetched from the
current directory. Default: /etc/trafMon/cred/
-f FILENAME, --filename=FILENAME
Give a pathname or filename containing IP address
patterns in concerned Flow Instances. This file
basename is also the root of the tree of generated
reports. When relative, the file is fetched from
current directory when -l is specified, otherwise from
default /etc/trafMon/report/
-D destination, --destination=destination
Destination directory. Default to
'/var/trafMon/reports/'.
-s STARTDATE, --startDate=STARTDATE
Give a start date in format: 'YYYY-MM-DD'
-e ENDDATE, --endDate=ENDDATE
Optionally give an end date in format: 'YYYY-MM-DD'
-t TIMESPAN, --timespan=TIMESPAN
Without endDate: choose between 'weekly' or 'monthly'
report. With endDate: give any identifier for this
type of reports.
-L LOGDIR, --logFileDirectory=LOGDIR
Path to log directory. Default: /var/log/trafMon/
-T TEMPLATESFOLDER, --reportTemplatesDirectory=TEMPLATESFOLDER
Path to trafMon report templates directory. Default:
/opt/trafMon/report/
-g GENREPORT, --genReport_sh=GENREPORT
Full pathname to the Birt runtime 'genReport.sh'
utility. Default to /opt/trafMon/bin/genReport.sh,
which is typically a symbolic link to the BIRT RunTime
installation/ReportEngine/genReport.sh
This script is used to automatically produce the set of protocol details PDF reports for IP hosts whose address match the configured patterns.
trafMon_probeResurrect.sh.sample “<tmon_probe start cmd>” <logfile_pathname>
trafMon_serverResurrect.sh.sample <log_pathname> [(“” | “<tmon_coll start cmd>”)
[(“” | “<msqld start cmd>”) [“<tomcat start cmd>”]]]
These sample scripts can be adapted and scheduled every minute by the crontab or the appropriate Linux accounts, on the appropriate computers, to automatically launch (or re-launch after potential crash) the key daemon processed of an operational trafMon installation.
3. MySQL Stored Procedures
Update_ftp_data_in_ipcttable( _dbName, _rangeStart, _rangeEnd)
Assigns the volumes of actual FTP data connections to their original data flows, encompassing corresponding FTP control session.
Aggr_activityvolumetable_first_level( _dbName, _rangeStart, _rangeEnd)
For the given range of days (up to yesterday), or for yesterday when both rangeStart and rangeEnd are NULL, update the prepared tables, at 1 hour and at 1 day granularity, for the “quick” generation of synthesis reports on trafMon probe data: with every flow having full description of its peer IP systems, full volumes in each direction, and the corresponding FTP and TCP key performance values.
Aggr_activityvolumetable_netflow_first_level( _dbName, _rangeStart, _rangeEnd)
For the given range of days (up to yesterday), or for yesterday when both rangeStart and rangeEnd are NULL, update the prepared tables, at 1 hour and at 1 day granularity, for the “quick” generation of synthesis reports on SiLK gathered NetFlow data: with every flow having full description of its peer IP systems and full volumes in each direction.
Drop_working_tables( )
Drops, from all existing trafMon database instances, the remaining working tables, produced upon the generation of BIRT report instances, and that cannot be declared as temporary tables (automatically drop at end of database connection).Those working table names start with an ‘_’.
Partition_drop( SCHEMANAME, TABLENAME, NB_DAYS)
Cleanup the given trafMon database from obsolete fine-gain observations from the given table name (stored in separate physical partitions of a same logical table), while preserving the last NB_DAYS days of data.
4. Traffic Generators
trafgen [-l] [-c XML] tgenName
USAGE:
-l (local) uses ./.xml and if it exists, ./trafgen.diag
-c means use the given XML, and nothing else
If -l is NOT given, /etc/tmon/xml/<tgenName>.xml is used
If -l is NOT given, or ./trafgen.diag doesn't exist,
/etc/tmon/diag/<tgenName>.diag is used
Conducts fully controlled repetitive scenarios, defined in an XML configuration file, of sending specially crafted UDP and/or TCP packets.
ftpgen.py host user password number-of-simultaneous-connections-to-server filename(optional default=totransfer.tar)
Conducts an FTP transfer session.
trafgen.py
Continuously repeats scenario (selected by editing the script) mixing ICMP Echo (ping), SNMP, NTP, DNS and FTP traffic (this last via ftpgen.py).
These utilities are used to generate traffic during the trafMon setup and configuration phase, or to qualify data links under “worst case” traffic patterns.
On the receiving side, following system daemons are “receiving” the traffic:
-
- “vsftpd” for the ftp traffic
- “ntpd“ for ntp traffic
- “snmpd“ for snmp
- “named“ for dns traffic
- “httpd“ for http traffic
5. The cron tables
A typical trafMon collector cron table, for unprivileged trafmon Linux account, includes:
-
- Execution every 10 minutes (or so) of the trafMon_loader.py, followed by trafMon_updateIpInfo.py
- Once per week, or less, trafMon_updateIpInfo.py –all
- The NetFlow data are extracted from the SiLK database once every hour (for the previous hour, when we can be pretty sure that all the connections are closed): trafMon_FormatNetFlow.py
- During a quite period at night, call the Aggr_activityvolumetable_first_level(“trafMon”, NULL, NULL), just after invocation of Update_ftp_data_in_ipcttable(“trafMon”, NULL, NULL) MySQL stored procedure.
- At another quiet time at night, optionally call the Aggr_activityvolumetable_netflow_first_level(“trafMon”, NULL, NULL) MySQL stored procedure.
- The Drop_procedure() is called regularly to clean the database of temporary tables (typically, tables created when generating reports).
- The schedule for trafMon_volumeReportGen.py and/or trafMon_detailReportGen.py scripts can be added to generate reports automatically.
- After careful monitoring of the growing rate of fine-grain trafMon raw observations tables, specific invocations of the Partition_drop() MySQL stored procedure can be automatically scheduled.
- Every minute, the trafMon_serverResurrect.sh /var/log/trafMon/resurrect.log “/opt/trafMon/bin/tmon_collector TMonServer” will permit to (re-)start a missing trafMon collector process.
A typical cron table for the privileged root Linux account, would include:
-
- Every minute, the trafMon_serverResurrect.sh (for mysqld and tomcat services only) or trafMon_probeResurrect.sh (unless invoked by unprivileged trafmon Linux account, thanks to setcap tuning).
- Every hour, the logrotate for the trafMon specific logrotate configuration
6. Optional CERT® SiLK
The SiLK Netflow receiver and database has three main configuration files, located in /var/silk and /usr/local/etc/:
-
- rwflowpack.conf: defines how the software itself should behave: where to log events, where to store data, etc.
- twoway-silk.conf and sensor.conf: defines which version of NetFlow the sensor(s) will receive, whether it will be over TCP or UDP and how to categorize the IPs.
These configuration files will be set up as needed or can be used as is.
If any other information is needed, please refer to the SiLK documentation.
7. MySQL
The specific setup of a running MySQL server for its use by the trafMon tool consists of 3 steps:
-
- Defining two database users (e.g. tmon_db and tmon_birt: db_users_sample.sql); the first with full privileges over the trafMon databases, including bulk loading from files, and the second with limited right to query (read-only) and to execute pre-defined stored procedures. The user names and their encoded passwords must be configured in several (protected) files (/etc/trafMon/cred/db.cred, /var/www/html/trafMon/php/include.php, /var/lib/tomcat/webapps/birt/trafMon_reports/Library/trafMonDb.rptlibrary);
- Create and configure the file /etc/trafMon/ipInfo.ini with Activity/Location of all known IP address ranges;
- Executing the SQL script that creates and populates the (nearly empty) database with template schemas for temporary and persistent trafMon data tables and with the definition of all trafMon stored procedures: trafMon_templateDB.sql
Maybe the table trafMon_template.protocol_table needs to be tuned with service names and precedence assigned to TCP or UDP port numbers.
At first run of the trafMon_loader.py, the runtime database will be automatically created and initialised.
For all details, please refer to the trafMon Configuration and Administration Guide: trafMon v1.0.0 Configuration and Administration Guide, Oct 2020