1. GULP Grand Unified Logging Program Do you REALLY know who your users are
SPC
April 12, 2018
Joel Rosenblatt
Director, Computer & Network security
Columbia University, CISO

2. Columbia Network Environment
Large research university
Decentralized management structure
Over 250,000 network nodes
Over 100,000 MAC addresses active on average
Decentralized computer support
No sniffing traffic or scanning machines allowed
“Free Love” IP address assignments
No university wide, corporate like, firewalls
130,000 accounts
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

3. Initial problems to solve
We wanted to offer pain free use of our network to visiting people
We needed to reduce the overhead of registering machines
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

4. Free Love The solution is ….
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

5. What is “Free Love”
From
“Free Love” and Secured Services, by Vace Kundakci
“Free love” allowed all computers, whether public or private, wired or wireless, in residence halls, at the libraries, in faculty and staff offices, or anywhere else on campus to connect directly to the network, and thereby to the world, without further ado.
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

6. NEW Problems to solve How do you answer the question…
Who is using a certain IP address?
Who is using a certain MAC address?
When was a certain IP address being used by a certain user?
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

7. GULP The NEW solution is …
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

8. Grand Unified Logging Program - GULP
Problem – How do you know who is using an IP/MAC address without registration?
GULP processes the logs from 14+ different services that require authentication
GULP includes information from LENEL
It processes information from DHCP and the ARP cache to associate MAC address with IP address
GULP correlates all information
A user can be tracked by IP, MAC, or UNI – even if the IP is not on the Columbia network
The data is kept for 28 days and then purged
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

9. Basic GULP workflow
Pull all logs that associate an authenticated user, process, timestamp and IP address
Dump information into a database
Pull information from the network that associates IP address, MAC address and time (DHCP and ARP cache)
Add network information into appropriate records in the database
Includes ID Card transactions – “Door Swipes”
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

10. Some technical stuff … We are currently pulling logs from servers
We use cron to run scripts to pull different logs at different times depending on service
We use a Perl parser designed for each log to extract the relevant data
A script runs overnight to correlate the user>IP>MAC mapping
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

11. Network security vs Public Safety
What machine used that IP address at 3:00pm
Was the machine with MAC address XX connected to the network yesterday
How many MACs used that jack
Who used that IP address at 3:00pm
Did the person named John Doe log in to the network yesterday
How many people used that IP address – and when
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

12. Nifty Web interface
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

13. Sample GULP for UNI Joel
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

15. Gulp for IP
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

17. Sample GULP for audrey0
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

19. Question
No one has seen this student for 10 days, can you tell me anything?
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

20. Lost person procedure Look up ID of missing person using GULP
Analyze login records for location and times
Work with Public Safety to establish if this information matches up with missing person report
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

21. Question
A (faculty, staff, student) received this anonymous from Yahoo – can you tell me who sent it?
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

22. Procedure to track down some anonymous email senders
Get IP address of sender from headers (this does not work easily with Gmail)
Pop into GULP
See what comes up
We have found that, quite often, the offender will fire off the nasty , then login to our systems to check on their own , once they authenticate, GULP has them
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

23. Question
We got a call from LE that someone is applying for Credit Cards using the identities of employees, can you help?
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

24. Procedure to help Law Enforcement find Bad Guys
Get some data from LE – in this case, we got the IP address that the applications were being submitted from
Pop into GULP and see what you get
P.S. The person is currently in jail
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

25. Question
(Department that runs their own network – I know you have them )
We can’t find this machine anywhere. All I know is the IP address, can you help?
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

26. Procedure to find lost computers
Take the IP address and pop into GULP
The user or users of that computer will be displayed – then it is a simple matter of calling them and asking where they are
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

27. GULP data mining Use GULP data to discover compromised passwords
Use GULP data to satisfy Audit requirements
Use GULP data to expose MAC spoofers
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

28. Compromised Password Discovery
Create a daily process that looks at the last few days of GULP data (we use 72 hours)
Look at the location information of the logins (We use ASN data)
If a user logs in from “x” locations or more (we use 6 ASNs) in the time period, there is a strong possibility that the password has been compromised
We also look for logins from more than 2 countries
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

29. Multiple ASN logins
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

30. ASN details
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

31. User login by country
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

32. Investigation of possible compromise password
From the multiple country login report, this users logins appear to be suspicious
3      mc4313           Taiwan : United States : China  (LENEL)
3 Countries in 72 hours, very far apart, including a campus swipe
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

33. GULP of users account
Swipe at CU
China & Taiwan
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

34. Notes on GULP of user Use Start/End to limit size of report
Colors make it easy to see jumps in CIDR ranges
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

35. Look up first suspicious IP address Click on it in Hostname column - China
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

36. Copyright (c) 2018 The Trustees of Columbia University in the City of New York

37. Check of second IP shows it is from Taiwan
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

38. Copyright (c) 2018 The Trustees of Columbia University in the City of New York

39. Now look at the timestamps
New York swipe at 10:32
Taiwan login at 07:30
China login at 05:21
That is China to NY in about 5 hours
Either this person has their own Transporter, or more than 1 person is using the account 
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

40. Audit requirements
One of the things that Auditors often ask is how do you monitor the logins of employees to sensitive systems
GULP is the perfect answer – you know who logged in from where and can even setup an “off hour” filter to look for unusual logins
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

41. When did a user become a Super User?
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

42. MAC spoofers GULP correlates User, IP and MAC
Using some additional information, you can look for multiple MAC addresses being use by the same ID from the same Jack or location (We have written some additional tools, but that is a different presentation )
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

43. DHCP: Prior lease found: 1866DA157F2A had 128. 59. 54
DHCP: Prior lease found: 1866DA157F2A had at 03-apr :41:20 GMT DHCP: No subsequent lease found, querying ARP cache for subsequent bound ARP: 1866DA157F2A had at 04-Apr :55:16 MAC Address confirmed as 1866DA157F2A

44. Switchmgr – who is plugged into a jack

45. Joel’s IP from GULP

46. Now let’s find the port

47. And the jack is …

48. Summary
GULP is a powerful and useful tool for bringing together disparate pieces of information.
GULP can be used in a “free love” or a managed environment.
Once you have GULP, it will quickly become the “go to” tool for any question that involves WHO or WHERE
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

49. To get your Build your own GULP kit, send to me Make sure to include GULP Kit in the subject line
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

50. Questions?
Copyright (c) 2018 The Trustees of Columbia University in the City of New York

51. Joel Rosenblatt Joel at columbia.edu 212 854 3033