The Scaling IQ Test: When Dev and Admin Collide Richard Campbell Strangeloop Networks

 Background  After thirty years, done every job in the computer industry you’ve ever heard of  Currently  Co-Founder and Product Evangelist for Strangeloop Networks  Co-Host of.NET Rocks!  Host of RunAs Radio Richard Campbell

 Every web application has this meeting eventually  Sooner is always better  The goal is to trade information  What IT needs to know about the app  What Dev needs to know about the operating environment The IT/Dev Meeting

 Who needs to be in the room?  The architect/senior dev  Seniors devs that know the features in detail  IT personnel that will operate the application  Senior personnel that know the entire network The IT/Dev Meeting

 When does the meeting need to happen?  When the application is being designed (collected as requirements)  While the application is being developed  After the application is deployed  After the application has crashed horribly  When the application is too slow The IT/Dev Meeting

 Starting the meeting  What are the priorities  Reliability  Performance  Scalability  Accuracy  Put them in order, every site has different priorities The IT/Dev Meeting

 What IT Needs to Know  What’s in the web.config file (a great starting point)  What load balancing strategies will work for the application  Any known performance bottlenecks The IT/Dev Meeting

  None (Anonymous)  Windows (Active Directory, Basic, etc)  Forms-Based Web.Config

  Global connection strings, paths etc  Make sure they’re being used!  Remove dead strings  These can be critical in failover/disaster recovery scenarios Web.Config

  Decide on how errors should be displayed to the customer (internal or external)  Defaults are really not enough  You can create separate pages for each error (handle 404 page not found differently from 500 internal server error) Web.Config

  In-process vs. out-of-process  More dependencies  Affects options around load balancing Web.Config

 Find out what load balancing will work with the application  In-process session requires “sticky” load balancing  You only get to load balance the first request  Talk through server failure effects Load Balancing

 Discuss known performance issues  Night time processing that conflicts with existing work  Administrators work that significant impacts performance of regular users  What parts of the application are more scalable than others? Performance Bottlenecks

 The Network Diagram (in detail!)  How to get at production log data  What redundacy/failover/disaster recovery options there are Things Dev Need to Know

The Network Diagram How developers see it

The Network Diagram Closer to reality

 Production logs are the truth of what happened with the application  Providing developers with production logs gives them a chance to help out  Provide access to the backups of the logs  Saying “I’ll give them to you when you ask” is not enough  You’re looking for proactive analysis Production Logs

 All DR strategies require at least some coding support  SQL Server failover still needs to have queries retried to be seamless  What happens between the time a server fails and the load balancing strategy detects it?  Is losing request acceptable in your scenario? Disaster Recovery

 Switching to a backup site  Are DNS changes needed?  What references within the application need to be changed?  What does a switch-back look like?  Practice practice practice!  Don’t let your first failover test with an application be a real failure! Disaster Recovery

 What follow ups are there for management?  You’ve probably made some business-related decisions, make sure you have buy-in  When do we need to meet again?  Preferably before the next disaster After the Meeting

 IT is invariably on the front lines of an application failure  But when should development be brought in?  Post-mortem is often not enough The Cooperative Firefight

 Make a strategy to involve development during the firefight  They often have deep insight into how the application works and so can understand why it might fail  Just make sure they’re educated to not make the problem worse  This is NOT a time for fixing code The Cooperative Firefight

 Have the meeting early  Repeat as necessary  Each group must learn from the other  Assist and seek assistance during a firefight Summary