GULP Grand Unified Logging Program Do you REALLY know who your users are SPC April 12, 2018 Joel Rosenblatt Director, Computer & Network security Columbia University, CISO

Columbia Network Environment Large research university Decentralized management structure Over 250,000 network nodes Over 100,000 MAC addresses active on average Decentralized computer support No sniffing traffic or scanning machines allowed “Free Love” IP address assignments No university wide, corporate like, firewalls 130,000 email accounts Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Initial problems to solve We wanted to offer pain free use of our network to visiting people We needed to reduce the overhead of registering machines Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Free Love The solution is …. Copyright (c) 2018 The Trustees of Columbia University in the City of New York

What is “Free Love” From http://www.educause.edu/ir/library/pdf/erm0266.pdf “Free Love” and Secured Services, by Vace Kundakci “Free love” allowed all computers, whether public or private, wired or wireless, in residence halls, at the libraries, in faculty and staff offices, or anywhere else on campus to connect directly to the network, and thereby to the world, without further ado. Copyright (c) 2018 The Trustees of Columbia University in the City of New York

NEW Problems to solve How do you answer the question… Who is using a certain IP address? Who is using a certain MAC address? When was a certain IP address being used by a certain user? Copyright (c) 2018 The Trustees of Columbia University in the City of New York

GULP The NEW solution is … Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Grand Unified Logging Program - GULP Problem – How do you know who is using an IP/MAC address without registration? GULP processes the logs from 14+ different services that require authentication GULP includes information from LENEL It processes information from DHCP and the ARP cache to associate MAC address with IP address GULP correlates all information A user can be tracked by IP, MAC, or UNI – even if the IP is not on the Columbia network The data is kept for 28 days and then purged Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Basic GULP workflow Pull all logs that associate an authenticated user, process, timestamp and IP address Dump information into a database Pull information from the network that associates IP address, MAC address and time (DHCP and ARP cache) Add network information into appropriate records in the database Includes ID Card transactions – “Door Swipes” Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Some technical stuff … We are currently pulling logs from servers We use cron to run scripts to pull different logs at different times depending on service We use a Perl parser designed for each log to extract the relevant data A script runs overnight to correlate the user>IP>MAC mapping Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Network security vs Public Safety What machine used that IP address at 3:00pm Was the machine with MAC address XX connected to the network yesterday How many MACs used that jack Who used that IP address at 3:00pm Did the person named John Doe log in to the network yesterday How many people used that IP address – and when Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Nifty Web interface Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Sample GULP for UNI Joel Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Gulp for IP 68.197.91.126 Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Sample GULP for audrey0 Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Question No one has seen this student for 10 days, can you tell me anything? Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Lost person procedure Look up ID of missing person using GULP Analyze login records for location and times Work with Public Safety to establish if this information matches up with missing person report Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Question A (faculty, staff, student) received this anonymous email from Yahoo – can you tell me who sent it? Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Procedure to track down some anonymous email senders Get IP address of email sender from headers (this does not work easily with Gmail) Pop into GULP See what comes up We have found that, quite often, the offender will fire off the nasty email, then login to our systems to check on their own email, once they authenticate, GULP has them Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Question We got a call from LE that someone is applying for Credit Cards using the identities of employees, can you help? Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Procedure to help Law Enforcement find Bad Guys Get some data from LE – in this case, we got the IP address that the applications were being submitted from Pop into GULP and see what you get P.S. The person is currently in jail Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Question (Department that runs their own network – I know you have them ) We can’t find this machine anywhere. All I know is the IP address, can you help? Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Procedure to find lost computers Take the IP address and pop into GULP The user or users of that computer will be displayed – then it is a simple matter of calling them and asking where they are Copyright (c) 2018 The Trustees of Columbia University in the City of New York

GULP data mining Use GULP data to discover compromised passwords Use GULP data to satisfy Audit requirements Use GULP data to expose MAC spoofers Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Compromised Password Discovery Create a daily process that looks at the last few days of GULP data (we use 72 hours) Look at the location information of the logins (We use ASN data) If a user logs in from “x” locations or more (we use 6 ASNs) in the time period, there is a strong possibility that the password has been compromised We also look for logins from more than 2 countries Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Multiple ASN logins Copyright (c) 2018 The Trustees of Columbia University in the City of New York

ASN details Copyright (c) 2018 The Trustees of Columbia University in the City of New York

User login by country Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Investigation of possible compromise password From the multiple country login report, this users logins appear to be suspicious 3      mc4313           Taiwan : United States : China  (LENEL) 3 Countries in 72 hours, very far apart, including a campus swipe Copyright (c) 2018 The Trustees of Columbia University in the City of New York

GULP of users account Swipe at CU China & Taiwan Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Notes on GULP of user Use Start/End to limit size of report Colors make it easy to see jumps in CIDR ranges Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Look up first suspicious IP address Click on it in Hostname column - China 218.27.82.114 Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Check of second IP shows it is from Taiwan 103.98.74.117 Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Now look at the timestamps New York swipe at 10:32 Taiwan login at 07:30 China login at 05:21 That is China to NY in about 5 hours Either this person has their own Transporter, or more than 1 person is using the account  Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Audit requirements One of the things that Auditors often ask is how do you monitor the logins of employees to sensitive systems GULP is the perfect answer – you know who logged in from where and can even setup an “off hour” filter to look for unusual logins Copyright (c) 2018 The Trustees of Columbia University in the City of New York

When did a user become a Super User? Copyright (c) 2018 The Trustees of Columbia University in the City of New York

MAC spoofers GULP correlates User, IP and MAC Using some additional information, you can look for multiple MAC addresses being use by the same ID from the same Jack or location (We have written some additional tools, but that is a different presentation ) Copyright (c) 2018 The Trustees of Columbia University in the City of New York

DHCP: Prior lease found: 1866DA157F2A had 128. 59. 54 DHCP: Prior lease found: 1866DA157F2A had 128.59.54.141 at 03-apr-2018 09:41:20 GMT DHCP: No subsequent lease found, querying ARP cache for subsequent bound ARP: 1866DA157F2A had 128.59.54.141 at 04-Apr-2018 12:55:16 MAC Address confirmed as 1866DA157F2A

Switchmgr – who is plugged into a jack

128.59.54.141- Joel’s IP from GULP

Now let’s find the port

And the jack is …

Summary GULP is a powerful and useful tool for bringing together disparate pieces of information. GULP can be used in a “free love” or a managed environment. Once you have GULP, it will quickly become the “go to” tool for any question that involves WHO or WHERE Copyright (c) 2018 The Trustees of Columbia University in the City of New York

To get your Build your own GULP kit, send email to me Make sure to include GULP Kit in the subject line Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Questions? Copyright (c) 2018 The Trustees of Columbia University in the City of New York

Joel Rosenblatt Joel at columbia.edu 212 854 3033