© Hortonworks Inc Hadoop and Kerberos: The madness beyond the gate Steve 2015

Page 2 Me: Before Kerberos

© Hortonworks Inc. Page 3 Me: After Kerberos

© Hortonworks Inc. Page 4

© Hortonworks Inc Leave now if you want to retain your life of naïve innocence Page 5

© Hortonworks Inc. Page 6

© Hortonworks Inc export HADOOP_USER="root" Page 7

© Hortonworks Inc Modern Hadoop clusters are locked down through Kerberos Page 8

© Hortonworks Inc Discover Kerberos before Kerberos discovers you Page 9

© Hortonworks Inc Kerberos: the gateway to hell Page 10

© Hortonworks Inc. This is not a metaphor Art: Andrés Álvarez Iglesias

© Hortonworks Inc KPKP Kerberos is the gateway Page 12 Authentication Service Ticket Granting Service Principal (P, TGS, n 1 ) {K P.TGS, n 1 } KP, {ticket(P,TGS)} KTGS Ticket(P, TGS) = (TGS, P, t start, t end, K PT) KPKP {K P.S, n 2 } KP, {ticket(P,S)} KS {auth(P)} KP.TGS,{ticket(P,TGS)} KTGS, S,n 2 K TGS Kerberos Domain Controller Client auth(P) KP.TGS = {P, time)} KP.TGS

© Hortonworks Inc Every service is a principal Page 13 short names: alice bob oozie namenode hdfs yarn HTTP

© Hortonworks Inc. Page 14 Entering the darkness

© Hortonworks Inc. Page 16

© Hortonworks Inc HDFS Bootstrap: Kerberos Login Page 17 shared keytab in /etc/hadoop log in to kerberos tickets for TGS

© Hortonworks Inc HDFS Bootstrap: DNs register with NN Page 18 shared keytab in /etc/hadoop DN registration Ticket for ExportedBlockKeys Request ticket for

© Hortonworks Inc. Hadoop Tokens Issued and tracked by individual services (HDFS, WebHDFS, Timeline Server, YARN RM, …) Grant some form of access: Block tokens, Delegation Tokens Can be passed on to other processes Renewable via service APIs (RPC, HTTP) Revocable in server via service APIs Page 19 read: O'Malley 2009, Hadoop Security Architecture

© Hortonworks Inc HDFS IO: Block Tokens Page 20 Obtain ticket for BlockToken BlockToken: userId, (BlockPoolId, BlockId), keyId, expiryDate, access-modes open("file")

© Hortonworks Inc Delegation Tokens delegate access Page 21 BlockToken HDFS Delegation Token BlockToken HDFS Delegation Token Token Obtain ticket for Request delegation token

© Hortonworks Inc Launch Context YARN app launch Page 22 HDFS Delegation Token HDFS Launch Context AM/RM HDFSAM/RM HDFS AM/RM Obtain ticket for Request delegation token AM/RM Token Obtain tickvet for AM/RM' Refresh AM/RM

© Hortonworks Inc That which must not be named: UGI if(!UserGroupInformation.isSecurityEnabled()) { stayInALifeOfNaiveInnocence(); } else { sufferTheEnternalPainOfKerberos(); } UserGroupInformation.checkTGTAndReloginFromKeytab(); UserGroupInformation.getLoginUser() // principal logged in as UserGroupInformation.getCurrentUser() // principal acting as Page 23

© Hortonworks Inc UGI.doAs() UserGroupInformation bob = UserGroupInformation.createProxyUser("bob", UserGroupInformation.getLoginUser()); FileSystem userFS = bob.doAs( new PrivilegedExceptionAction () { public FileSystem run() throws Exception { return FileSystem.get(FileSystem.getDefaultUri(), conf); } }); Page 24

© Hortonworks Inc Hadoop = "my.kerberos.principal") public interface MyRpc extends VersionedProtocol { … } public class MyRpcPolicyProvider extends PolicyProvider { public Service[] getServices() { return new Service[] { new Service("my.protocol.acl", MyRpc.class) }; } public class MyRpcSecurityInfo extends SecurityInfo { … } META-INF/services/org.apache.hadoop.security.SecurityInfo org.example.rpc.MyRpcSecurityInfo Page 25

© Hortonworks Inc IPC Server: get the current user identity Messages.KillResponse killContainer(Messages.KillRequest request) { UserGroupInformation callerUGI; try { callerUGI = UserGroupInformation.getCurrentUser(); } catch (IOException ie) { LOG.info("Error getting UGI ", ie); AuditLogger.logFailure("UNKNOWN", "Error getting UGI"); throw RPCUtil.getRemoteException(ie); } … Page 26

© Hortonworks Inc IPC Server: Authorize String user = callerUGI.getShortUserName(); if (!checkAccess(callerUGI, MODIFY)) { AuditLog.unauthorized(user, KILL_CONTAINER_REQUEST, "User doesn't have permissions to " + MODIFY); throw RPCUtil.getRemoteException( new AccessControlException( + user + " lacks access " + MODIFY_APP.name())); } AuditLog.authorized(user, KILL_CONTAINER_REQUEST) Page 27

© Hortonworks Inc SASL: RFC4422 Page 28

© Hortonworks Inc. REST: SPNEGO (+ Delegation tokens) Page 29 Jersey + java.net httpclient? “if lucky it'll work” HADOOP-11825: Move timeline client Jersey+Kerberos+UGI support into a public implementation

© Hortonworks Inc. Testing Page 30

© Hortonworks Inc. Error messages to fear Art: Andrés Álvarez Iglesias Failure unspecified at GSS-API level (Checksum failed) No valid credentials provided (Failed to find any Kerberos tgt) Server not found in Kerberos database Clock skew too great Principal not found No valid credentials provided (Illegal key size)

© Hortonworks Inc. Topics Avoided Not Covered Zookeeper JAAS Trying to use HTTPS in a YARN application Trying to use Full REST in a YARN application System properties to debug Kerberos & SPNEGO Group management HADOOP_PROXY_USER Page 32

© Hortonworks Inc. Questions? Art: Andrés Álvarez Iglesias

© Hortonworks Inc. Zookeeper SASL to negotiate security: System.setProperty("zookeeper.sasl.client", "true"); Permissions are not transitive down the tree Page 34 List perms = new ArrayList<>(); if (UserGroupInformation.isSecurityEnabled()) { perms(new ACL(ZooDefs.Perms.ALL, ZooDefs.Ids.AUTH_IDS)); perms.add(new ACL(ZooDefs.Perms.READ,ZooDefs.Ids.ANYONE_ID_UNSAFE)); } else { perms.add(new ACL(ZooDefs.Perms.ALL, ZooDefs.Ids.ANYONE_ID_UNSAFE)); } zk.createPath(path, null, perms, CreateMode.PERSISTENT);

© Hortonworks Inc System Properties for debugging -Dsun.security.krb5.debug=true -Dsun.security.spnego.debug=true export HADOOP_JAAS_DEBUG=true Page 35

© Hortonworks Inc. Services RPC authentication via annotations & metadata in JAR YARN Web UIs: rely on RM proxy for authentication Authentication != Authorization Add audit logs on service endpoints YARN services: come up with a token refresh strategy: keytab everywhere; keytab in AM; update from client Page 36

© Hortonworks Inc. JAAS Java Authentication and Authorization Service Core Kerberos classes and types ( Principal ) Text files to configure –Different for different JVMs –Need to double escape \ for windows paths UGI handles setting up a JAAS context & logging in Page 37

© Hortonworks Inc. Glossary Simple Authentication and Security Layer (SASL) GSSAPI Generic Security Service Application Program Interface (RFC others) JAAS: Java Authentication and Authorization Service Simple and Protected GSSAPI Negotiation Mechanism (SPNEGO) Page 38