fredag 9. oktober 2015

SQL self joins explained

In my quest to transform a database of SNMP traps into actual information, I need to master joins. Or specifically self joins. What follows is an attempt at explaining this.
I'll be using two examples. The first goes straight to the solution. The next example is more long-winded.

The TL;DR version

Consider the following table:

+------+--------+---------+------+
| id   | c1     | c2      | c3   |
+------+--------+---------+------+
|    1 | name   | letters | adam |
|    1 | weight | kg      | 80   |
|    1 | height | cm      | 180  |
|    2 | name   | letters | brad |
|    2 | weight | kg      | 70   |
|    2 | height | cm      | 170  |
+------+--------+---------+------+
One may find it natural to think that the data were entered in groups by id. But records may actually be grouped by any of the columns. From a database perspective, the table design may not appear optimal. But for illustrating self joins it works well.

How do I get from that table......to this:

+------+--------+---------+------+
| id   | name   | weight  |height|
+------+--------+---------+------+
|    1 | adam   | 80      | 180  |
|    2 | brad   | 70      | 170  |
+------+--------+---------+------+


The solution is shown below. To give you a head start understanding it:
Any column you specify in your query must be unambigous.
AS means we assign an alias to something.
The column headers in the output do not originate in the first table above, but is assigned in the query.


SELECT 
nameTable.id AS id,
nameTable.c3 AS name,
weightTable.c3 AS weight,
heightTable.c3 AS height 
FROM example AS nameTable  
JOIN example AS weightTable ON nameTable.id = weightTable.id AND weightTable.c1 = 'weight'  
JOIN example AS heightTable ON nameTable.id = heightTable.id AND heightTable.c1 = 'height' 
WHERE nameTable.c1 = 'name';


Got that? Great. You have mastered self joins and may now return to the/your original problem. Or read on.


Expanded example

The next example makes use of two tables ('beings' and 'stuff').

SELECT * FROM beings; 
+------+------------+---------+
| id   | recordtype | content |
+------+------------+---------+
|    1 | firstname  | Arnold  |
|    1 | height     | 180     |
|    1 | weight     | 80      |
|    1 | age        | 50      |
|    1 | species    | human   |
|    2 | firstname  | Camilla |
|    2 | width      | 48      |
|    2 | species    | dog     |
|    3 | firstname  | Arnold  |
|    3 | length     | 30      |
|    3 | food       | tuna    |
|    3 | species    | cat     |
|    4 | firstname  | Edmund  |
|    4 | length     | 32      |
|    4 | food       | chicken |
|    4 | species    | cat     |
+------+------------+---------+


Notice how the table beings contain a variable number of records per id. And to spice it up even further, two different ids share the same recordtype and content.

SELECT * FROM stuff; 
+------+-----------+
| id   | birthyear |
+------+-----------+
|    1 |      2001 |
|    2 |      2002 |
|    3 |      2003 |
|    4 |      2004 |
+------+-----------+

Nothing exciting about this table.



First, we try to join data from the two tables. We will join data where the column 'id' matches between the tables.

SELECT 
birthyear,
content
FROM beings 
JOIN stuff ON beings.id = stuff.id;
+-----------+---------+
| birthyear | content |
+-----------+---------+
|      2001 | Arnold  |
|      2001 | 180     |
|      2001 | 80      |
|      2001 | 50      |
|      2001 | human   |
|      2002 | Camilla |
|      2002 | 48      |
|      2002 | dog     |
|      2003 | Arnold  |
|      2003 | 30      |
|      2003 | tuna    |
|      2003 | cat     |
|      2004 | Edmund  |
|      2004 | 32      |
|      2004 | chicken |
|      2004 | cat     |
+-----------+---------+

So... we got what we asked for. Right? Notice how we have to prefix 'id' with the tablename and a period on the last line of the query.

SELECT 
birthyear,
content
FROM beings 
JOIN stuff ON beings.id = stuff.id
WHERE recordtype = 'firstname';
+-----------+---------+
| birthyear | content |
+-----------+---------+
|      2001 | Arnold  |
|      2002 | Camilla |
|      2003 | Arnold  |
|      2004 | Edmund  |
+-----------+---------+

Now we're moving in the right direction. Can we get the id in the output as well?

SELECT 
id,
birthyear,
content
FROM beings 
JOIN stuff ON beings.id = stuff.id
WHERE recordtype = 'firstname';
ERROR 1052 (23000): Column 'id' in field list is ambiguous


Not like that, anyway. Remember being unambiguous when you ask for something. As a general rule in life. And in particular when querying a database. Prefix the id with tablename and a period. It does not matter which table...

SELECT 
stuff.id,
birthyear,
content
FROM beings 
JOIN stuff ON beings.id = stuff.id
WHERE recordtype = 'firstname';
+------+-----------+---------+
| id   | birthyear | content |
+------+-----------+---------+
|    1 |      2001 | Arnold  |
|    2 |      2002 | Camilla |
|    3 |      2003 | Arnold  |
|    4 |      2004 | Edmund  |
+------+-----------+---------+


Much better!
From here on we'll skip the 'stuff' table and see if we can join table 'beings' with itself! To do that, we employ the 'AS'...verb? subject? thingy?. Whatever. We use 'AS'. Look:
SELECT 
idalias.id,
idalias.recordtype,
idalias.content,
namealias.recordtype,
namealias.content
FROM beings AS idalias
JOIN beings AS namealias ON namealias.id = idalias.id AND namealias.recordtype = 'firstname';
+------+------------+---------+------------+---------+
| id   | recordtype | content | recordtype | content |
+------+------------+---------+------------+---------+
|    1 | firstname  | Arnold  | firstname  | Arnold  |
|    1 | height     | 180     | firstname  | Arnold  |
|    1 | weight     | 80      | firstname  | Arnold  |
|    1 | age        | 50      | firstname  | Arnold  |
|    1 | species    | human   | firstname  | Arnold  |
|    2 | firstname  | Camilla | firstname  | Camilla |
|    2 | width      | 48      | firstname  | Camilla |
|    2 | species    | dog     | firstname  | Camilla |
|    3 | firstname  | Arnold  | firstname  | Arnold  |
|    3 | length     | 30      | firstname  | Arnold  |
|    3 | food       | tuna    | firstname  | Arnold  |
|    3 | species    | cat     | firstname  | Arnold  |
|    4 | firstname  | Edmund  | firstname  | Edmund  |
|    4 | length     | 32      | firstname  | Edmund  |
|    4 | food       | chicken | firstname  | Edmund  |
|    4 | species    | cat     | firstname  | Edmund  |
+------+------------+---------+------------+---------+


(I believe the term may be 'clause'.)
What happened here, is that we named table 'beings' as 'idalias' and extracted the columns 'id', 'recordtype' and 'content', and then we joined each of the new records with the same records from the 'beings' table (now renamed to 'namealias'), but only where the 'id' matches in both tables, and only where the 'recordtype' of namealias contains 'firstname'. Why on earth would I do that?!
Look what happens when I add one more condition for the join:

SELECT 
idalias.id,
idalias.recordtype,
idalias.content,
namealias.recordtype,
namealias.content
FROM beings AS idalias
JOIN beings AS namealias ON namealias.id = idalias.id AND namealias.recordtype = 'firstname' AND idalias.recordtype = 'species';
+------+------------+---------+------------+---------+
| id   | recordtype | content | recordtype | content |
+------+------------+---------+------------+---------+
|    1 | species    | human   | firstname  | Arnold  |
|    2 | species    | dog     | firstname  | Camilla |
|    3 | species    | cat     | firstname  | Arnold  |
|    4 | species    | cat     | firstname  | Edmund  |
+------+------------+---------+------------+---------+


If I know I don't care for all species, but only our feline friends, I can limit my selection even furrierfurther:

SELECT 
idalias.id,
idalias.recordtype,
idalias.content,
namealias.recordtype,
namealias.content
FROM beings AS idalias
JOIN beings AS namealias ON namealias.id = idalias.id AND namealias.recordtype = 'firstname' AND idalias.content = 'cat';
+------+------------+---------+------------+---------+
| id   | recordtype | content | recordtype | content |
+------+------------+---------+------------+---------+
|    3 | species    | cat     | firstname  | Arnold  |
|    4 | species    | cat     | firstname  | Edmund  |
+------+------------+---------+------------+---------+


We're getting there

SELECT 
idalias.id,
idalias.recordtype,
idalias.content,
namealias.content,
foodalias.content
FROM beings AS idalias
JOIN beings AS namealias ON namealias.id = idalias.id AND namealias.recordtype = 'firstname' AND idalias.content = 'cat'
JOIN beings AS foodalias ON foodalias.id = idalias.id AND foodalias.recordtype = 'food'
+------+------------+---------+---------+---------+
| id   | recordtype | content | content | content |
+------+------------+---------+---------+---------+
|    3 | species    | cat     | Arnold  | tuna    |
|    4 | species    | cat     | Edmund  | chicken |
+------+------------+---------+---------+---------+


Let us trim the number of columns. And give those columns more descriptive names:

SELECT 
idalias.id AS id,
namealias.content AS firstname,
foodalias.content AS food,
speciesalias.content AS species
FROM beings AS idalias
JOIN beings AS namealias ON namealias.id = idalias.id AND namealias.recordtype = 'firstname' AND idalias.content = 'cat'
JOIN beings AS foodalias ON foodalias.id = idalias.id AND foodalias.recordtype = 'food'
JOIN beings AS speciesalias ON speciesalias.id = idalias.id AND speciesalias.recordtype = 'species';
+------+-----------+---------+---------+
| id   | firstname | food    | species |
+------+-----------+---------+---------+
|    3 | Arnold    | tuna    | cat     |
|    4 | Edmund    | chicken | cat     |
+------+-----------+---------+---------+


Much better! Now, if we wanted to figure out the dietary needs of one furry friend in particular, we could add a WHERE clause. Like this:

SELECT 
idalias.id AS id,
namealias.content AS firstname,
foodalias.content AS food,
speciesalias.content AS species
FROM beings AS idalias
JOIN beings AS namealias ON namealias.id = idalias.id AND namealias.recordtype = 'firstname' AND idalias.content = 'cat'
JOIN beings AS foodalias ON foodalias.id = idalias.id AND foodalias.recordtype = 'food'
JOIN beings AS speciesalias ON speciesalias.id = idalias.id AND speciesalias.recordtype = 'species'
WHERE namealias.content = 'Arnold';
+------+-----------+------+---------+
| id   | firstname | food | species |
+------+-----------+------+---------+
|    3 | Arnold    | tuna | cat     |
+------+-----------+------+---------+


I'll let this rest for now. Hope it was of use to you!

lørdag 3. oktober 2015

Logging SNMP traps to MySQL/MariaDB.


I recently got a request formulated approximately like this:
"How can our servicedesk count the number of devices a user has on our wireless network." 

The reason for the request is that we enforce a maximum of two devices per user. But whenever a user tries with a third, there is no useful feedback to the end user that the number of devices has been exceeded. Hence they will call servicedesk to request help getting their device on the network. Because they are not aware of the limit, because they have forgotten about one or another device in their backpack, or because someone is abusing the account in question. Or any other reason. Ideally, we would give users nice feedback. Practically, we don't.

Our provider of wireless network gear got tools to manage the wireless network, which may or may not assist with counting devices per user. But they are not extremely user friendly. And we'd rather not let more users have access to these tools than strictly required. Can I make a simple tool that assists superusers and the servicedesk in simple debugging of wireless access clients? 

Turns out I can.

The wireless controllers already emit a huge stream of SNMP traps to another network device which makes use of them to match usernames with IPaddresses. For... reasons. I can make use of the same messages. The following is an account of how I get the messages into a database, and how I can query the database to extract the information I want.


Before we start, you should be aware of a few important things with regards to SNMP:
  • SNMP version 1 and 2c are not encrypted protocols. Easily eavesdropped upon, easily spoofed. 
  • The traffic is UDP-based. An adversary can flood spoofed packets from anywhere. See your friendly firewall admin for advice.
  • It is common to have separate SNMP communities for read and write access to an SNMP agent. Having no write community enabled until you really need it, it is a good idea. 
  • The SNMP community is not a password. It is more like a username, for which there is no password.
  • An SNMP agent is a process which can be queried (read) and possibly configured (write), as well as emit traps ('alerts') directed at an SNMP trap receiver. The agent typically runs on a piece of network equipment, like a router, switch, firewall or even a server. The SNMP agent is called snmpd in most Linux distributions.
  • An SNMP trap receiver accepts traps with specific communities, transmitted from SNMP agents. The trap receiver typically runs on a server. The SNMP trap receiver is called snmptrapd in most Linux distributions.
  • For proper security, you must employ SNMP version 3. This is not covered in this post.
This is greatly simplified. A link to suggested reading materials is provided at the end.

I got my friendly sysadmin to spin up a virtual Ubuntu machine for me. Then, I logged in and did:

user@ubuntu:~$ sudo apt-get install mariadb-server mariadb-client snmp snmpd snmp-mibs-downloader


You will be prompted for an admin user/password etc. for the database. Note that MySQL/MariaDB both use an internal user database, unrelated to the system userdatabase. Writing the admin user and password down and keeping it in a safe place is a good idea.

MariaDB is a drop-in replacement for MySQL. I choose MariaDB over MySQL, as MariaDB had support for an option not present in the version of MySQL available to me at the moment. More about this in later posts. The names of the binaries and the startup-scripts are the same for MySQL and MariaDB. Drop-in replacement, remember.

The command above adds mysql and snmpd to your default services, and starts them both. We don't need snmpd at the moment, so we will stop and disable it.

user@ubuntu:~$ sudo service snmpd stop
user@ubuntu:~$ sudo update-rc.d snmpd disable



Now, we need to create the database the traps will be stored in, as well as the database user we will use when logging traps.

user@ubuntu:~$ mysql -u root -p


When prompted, provide the password you gave during install of MariaDB. You will be greeted with this prompt:

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 47
Server version: 5.5.44-MariaDB-1ubuntu0.14.04.1 (Ubuntu)

Copyright (c) 2000, 2015, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.


MariaDB [(none)]> 



Create the database we will log traps to:

MariaDB [(none)]> create database net_snmp;
Query OK, 1 row affected (0.03 sec)


Keep the database name 'net_snmp' for the purpose of logging traps.
If you end up with a '     ->' prompt, you have forgotten the terminating semicolon in the command above. Always terminate an sql command with a ';'. You can add that terminating semicolon at the '     ->' prompt. After every successful command you give the database engine, you will get a line stating Query OK, xxx row affected (0.03 sec). This is normal, and I will not repeat this line below.



Create the database user we will use to log traps with, and give that user all rights to tables in that database:


MariaDB [(none)]> create user 'netsnmp'@'localhost' identified by 'sekritpass';
MariaDB [(none)]> grant all on net_snmp.* to 'netsnmp'@'localhost' identified by 'sekritpass';

 
Create the schema we will enter data into. 
A database schema is the actual structure we write data to, including the tables. Consider it the 'shape' and 'function' of the database. The file describing the database schema is, as far as I can tell, not installed with snmpd. But it is present in the net-snmp source code distribution. 
 I have reproduced it below. 

Copy and paste this at the MariaDB prompt:

USE net_snmp;
DROP TABLE IF EXISTS notifications;
CREATE TABLE IF NOT EXISTS `notifications` (
  `trap_id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `date_time` datetime NOT NULL,
  `host` varchar(255) NOT NULL,
  `auth` varchar(255) NOT NULL,
  `type` ENUM('get','getnext','response','set','trap','getbulk','inform','trap2','report') NOT NULL,
  `version` ENUM('v1','v2c', 'unsupported(v2u)','v3') NOT NULL,
  `request_id` int(11) unsigned NOT NULL,
  `snmpTrapOID` varchar(1024) NOT NULL,
  `transport` varchar(255) NOT NULL,
  `security_model` ENUM('snmpV1','snmpV2c','USM') NOT NULL,
  `v3msgid` int(11) unsigned,
  `v3security_level` ENUM('noAuthNoPriv','authNoPriv','authPriv'),
  `v3context_name` varchar(32),
  `v3context_engine` varchar(64),
  `v3security_name` varchar(32),
  `v3security_engine` varchar(64),
  PRIMARY KEY  (`trap_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

DROP TABLE IF EXISTS varbinds;
CREATE TABLE IF NOT EXISTS `varbinds` (
  `trap_id` int(11) unsigned NOT NULL default '0',
  `oid` varchar(1024) NOT NULL,
  `type` ENUM('boolean','integer','bit','octet','null','oid','ipaddress','counter','unsigned','timeticks','opaque','unused1','counter64','unused2') NOT NULL,
  `value` blob NOT NULL,
  KEY `trap_id` (`trap_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;


 
This states that we will use the net_snmp database, and creates the two tables our trap receiver expects to have access to. With the database schema in place, we are ready to configure the trap receiver. 


Drop out of the MariaDB shell by typing exit at the prompt. Then create the following three files and make sure they have the same content as shown below:

root@ubuntu:~# cat /etc/snmp/snmptrapd.conf 
authCommunity log mytrapcommunity
sqlMaxQueue 1
sqlSaveInterval 9


root@ubuntu:~# cat /etc/rc.local 
/usr/sbin/snmptrapd -c /etc/snmp/snmptrapd.conf
exit 0


root@ubuntu:~# cat /etc/mysql/conf.d/snmptrapd.cnf 
[snmptrapd]
user=netsnmp
password=sekritpass
host=localhost

 
The first file is the config file for the process receiving the snmp traps from your network device. The first line tells snmptrapd to log traps with the SNMP community mytrapcommunity.  The next two lines instructs the trap receiver to log to mysql, have at most 1 traps in the buffer before commiting to the database,  and commit to the database at least every 9 seconds. For production, you may want to increase the buffer to, say 20. Or a 100.

The second file is a dirty way of starting the SNMP trap receiver on boot. Any reader who wants to contribute a proper initfile for snmptrapd is welcome to do so.

The third file specifies the database user, password and hostname the trap receiver will use when logging to the database. You should recognise the user and password we created earlier.


And finally, we start snmptrapd:

user@ubuntu:~$ sudo sh /etc/rc.local



We should now be ready to receive SNMP traps and have them entered into the database. To send a test SNMP trap, execute the following command:

user@ubuntu:~$ sudo snmptrap -v 2c -c mytrapcommunity 127.0.0.1 "" 1.2.3.4.0


Log in to the database console again:

user@ubuntu:~$ mysql -u root -p 
MariaDB [(none)]> use net_snmp;
[.....]

Database changed
MariaDB [net_snmp]>
select * from varbinds;
+---------+------------------------+-----------+----------------------------------+
| trap_id | oid                    | type      | value                            |
+---------+------------------------+-----------+----------------------------------+
|       1 | .1.3.6.1.2.1.1.3.0     | timeticks | Timeticks: (8422516) 23:23:45.16 |
|       1 | .1.3.6.1.6.3.1.1.4.1.0 | oid       | OID: .1.2.3.4.0                  |
+---------+------------------------+-----------+----------------------------------+

2 rows in set (0.00 sec) 

Congratulations! You have logged the first trap to the database. You will also find stuff in the table 'notifications'. You may now point your network devices at your SNMP trap receiver and store wast amounts of data in your database.

But as you may or may not know, having a huge amount of data does not automatically translate into information. I intend to write a part 2, where I give an example of how to make use of the trap database.

For further reading about SQL syntax, check out w3schools' SQL tutorial.
For further reading about SNMP, you could do worse than starting at the Net-SNMP wiki