How We Maintain a Healthy Bb Environment Sheila Chandler Director, Learning Systems Virginia Commonwealth University

Presentation Overview Background –Who am I? –Virginia Commonwealth University (VCU) –VCU Blackboard Environment ____________________________________ How We Maintain a Healthy Bb Environment Sharing Lessons Learned

Who am I? High School – Computer programming (IBM) University of Louisville –Sociology Degree –First programming job (PDP 11/45 - BASIC) Community Hospital of Central California (MUMPS, BASIC) Virginia Commonwealth University (VCU) -Accounting Degree Auditor of Public Accounts – CPA, CISA VCU –Programmed on Web Course in a Box (1998 to 2000) –Blackboard purchased WCB (2000)

About VCU Richmond, VA Students: 32,000+ Faculty: 1,927 Adjunct: 1,161 Staff: 2,586 Monroe Park, MCV, Qatar, NOVA

Bb App Environment Bb Application (2000 to present) –Bb Classic Enterprise (7.3 -> v9 – May 2010) –Bb Content Management System Building Blocks –Bb Faculty Tools (VCU developed) –CPS Connection (Clickers) –Echo360 Content Creator –Health Check (modified) –Help Desk Tools (VCU developed) –Wimba –SafeAssign –SoftChalk

Bb User Environment Fall Courses 2031 Instructors Students (85%) 723 Organizations Spring Courses 1983 Instructors Students (87%) 804 Organizations

Lesson Learned #1 Designate Bb Coordinator and Support Team VCU Support Team (Tribe) –Technology Services Learning Systems Help Desk University Computer Center Networking –Center for Teaching Excellence –Support staff in Schools (School of Nursing, Business, etc) –Vendor (Blackboard)

Lesson Learned # 2 Recognize Mission Critical Path to Mission Critical –Student demand (85%+) –Faculty use ( , gradebook, document storage) –Administration/Staff found uses –Keep statistics – show growth

Lesson Learned # 3 - Scale Out

Lesson Learned #4 Request System Audit

Lesson Learned # 5 Monitor and Take Action Hobbit (Internal) Pingdom (External) Custom Scripts –Check db connections –Search logs for Oracle errors –Scan Snapshot files (header & footer) –Monitor SAN space (set to expand at 90%)

#5 Monitor and Take Action System cleanup –Rotate logs –Run Bb Cleanbb once a week –Restart app servers twice a week Track usage and react to trends Ask Users –My VCU Portal Polls –Annual Tech Fair suggestion box –Talk to Students on Campus

#5 Monitor and Take Action External Listserv –ASU list ) –BIE list ) –Developers ) Internal VCU Listserv –Bb-core –Bb-mgmt –Bb faculty list

Lesson Learned #6 Document and Communicate Technical –Bb Diary –Internal wiki – notes –BbCMS – Documentation –Emergency Call list End-User –Help Desk - –Blogs (Tech Services, Bb system-wide, Bb faculty) –Learning Systems Newsletter –Technology Services Knowledge Base (

#6 Document/Communicate Regular Meetings –Weekly LS meeting –Monthly Bb Tech meeting (agenda & minutes) –Monthly Center for Teaching Excellence meeting –Periodic meetings with IT Directors of Schools

#6 Document/Communicate Special Meetings –Special Task Force (iptables fix) Discovered that net.ipv4.netfilter.ip_conntrack_tcp_be_liberal = 1 doesn't survive an iptables restart –Bb TSM bi-weekly meetings Periodically request to review Bb tickets

Lesson Learned #7 “No such thing as a SMALL change” ALL Changes follow Written Procedures –Development Environment (maybe multiple times) –Test Environment (always multiple times) –Production

#7 “No such thing as a SMALL change” System Maintenance – Regularly Scheduled –Second and Fourth Friday of the month –Perform Downtime Checklist One week before downtime During downtime After downtime –Group IM (Instant Messenger) –Follow emergency communication plan (if needed)

Lessons Learned Summary 1.Designate Bb Coordinator and Support Team 2.Recognize Mission Critical 3.Scale out 4.Request System Audit 5.Monitor and Take Action 6.Document and Communicate 7.“No Such Thing as a SMALL Change”

Additional Information –Sheila Chandler –Bb Connections site – Thank you to… –Learning Systems Staff –University Computer Center Staff –Networking Services –Center for Teaching Excellence