Copyright © 2014 Splunk Inc. Transform E-‐ Commerce the Domino’s Pizza Way Domino’s Pizza Russell Turner Disclaimer During the course of this presentaFon, we may make forward-‐looking statements regarding future events or the expected performance of the company. We cauFon you that such statements reflect our current expectaFons and esFmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-‐looking statements, please review our filings with the SEC. The forward-‐looking statements made in the this presentaFon are being made as of the Fme and date of its live presentaFon. If reviewed aQer its live presentaFon, this presentaFon may not contain current or accurate informaFon. We do not assume any obligaFon to update any forward-‐looking statements we may make. In addiFon, any informaFon about our roadmap outlines our general product direcFon and is subject to change at any Fme without noFce. It is for informaFonal purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligaFon either to develop the features or funcFonality described or to include any such feature or funcFonality in a future release. 2 Domino’s Pizza Overview ! Founded in 1960 ! Tom Monaghan sells Dominos in 1998 ! Went public 2004 ! World leader in pizza delivery ! More than 10,000 corporate and franchised stores in US and internaFonal markets ! 2013 sales: $8 billion 3 Domino’s Pizza Technology Overview ! Responsive design released this year ! In August Patrick Doyle our CEO announces that we are officially an e-‐commerce company ! NaFve applicaFons for iPhone, Android, Kindle and Windows Phone ! Introduced “Dom” this year, our voice ordering feature in the naFve apps ! Easy ordering with Pizza Profile ! MulFple data centers ! InternaFonal hosFng 4 My Background and Role ! ! Russell Turner, Manager of Site Reliability Engineering My Team is responsible for ensuring our online customers have the best experience possible – – – – Maintain ecommerce upFme Middleware, infrastructure, servers, global and local load balancing ArchitecFng and deployments of new business iniFaFves Closely Fed into development workflow 5 How We Started ! POC’ed Splunk for the first Fme in 2009 (within the Infrastructure team) ! Needed a soluFon to analyze and aggregate logging data from our OS (Linux and Solaris) and middleware in a Fmely manner ! InfoSec team used HP ArcSight for log aggregaFon, but Splunk offered the following advantages: – – – – – – Faster and easier searches in Splunk Real-‐Fme insights Beier reporFng with Apache access logs Much faster alerFng in Splunk Cost and scalability Ease of deployment “We now even help our security team.” 6 Splunk at Domino’s Today ! ! ! ! ! ! Two indexers with Distributed Search Splunk deployed across two data centers (live and failover) Four different producFon environments Teams using Splunk: Site reliability team, InfoSec and developers 100-‐120GB data indexed per day between both instances Dozen unique users per month Splunk Apps: Deployment Monitor, Google Maps, Search, Field Extractor, Google Maps, Splunk on Splunk, Splunk App for Unix, soon to be JMX MI Datacenter 360+ forwarders 7 VA Datacenter Using Splunk For Data CorrelaFon? Domino’s Splunk Environment: ApplicaFon logs (over 20 types) Logs from 900 Virtual and physical servers, Linux/Solaris Proprietary System Logs Middleware, Database logs Apache Web Server Logs Enter Splunk Before Splunk ! Gathering logs manually ! “Million Fmes easier with Splunk” ! SiQing through aggregated Java messages from middleware (Grep) ! ProacFve alarms alert us to dips in our sales ! VERY ReacFve ! Baselining and trending 8 Splunk at Domino’s: Ops + App Management ! Dashboards, Maps, Alerts and Reports for the following: – API events – Perfmon and other performance monitors – DBConnect queries of ancillary databases including Service Desk and CRM systems ! ! ! Significant reducFon in troubleshooFng Fme Streamlined developer insight into debugging development code ParFal Order to Cash system health monitoring – No store visibility for my team 9 Splunk at Domino’s: InfoSec and Stores ! ! ! ! ! ! Monitor Security logs in real-‐Fme Monitor ApplicaFon logs in real-‐Fme, was scheduled Monitor system performance in real-‐Fme, easier to build custom dashboards and reports than System Center Single locaFon for mulFple teams to query data, includes Security, Helpdesk and QA Universal Forwarder smaller footprint than Java based agent Universal Forwarder managed through exisFng soQware deployment infrastructure 10 How Do We Cross That Bridge? Splunk Professional Services to the Rescue ! Splunk became our most important tool but was stood up quickly on subpar hardware we had laying around ! As an organizaFon we had matured greatly yet our implementaFon was the sFll the same ! How can we migrate to new hardware, Fe our two instances together, start leveraging new features and disturb producFon monitoring as liile as possible? 11 But Wait… “Don’t wave your Johnny Manziel money sign at me, you said this was about Business AnalyFcs!” -‐Angry Patroit Lady 12 Eye Candy 13 14 15 16 Insights 17 Splunk For OperaFonal Analysis of Payment Processing ! Measuring response Fme for various order channels ! Instant analysis of cash vs. credit card ordering performance ! TroubleshooFng card processor issues 18 PromoFonal Support ! ! ! ! What we are selling, orders per minute, coupon usage, etc. Online ordering trends, efficiency of markeFng promoFons Splunk provides us answers 24-‐48h prior to analysis from our data warehousing tools 50% off promoFon 19 Splunk for Domino’s MarkeFng Before Splunk • Someone at midnight pulling data and crunching numbers daily Splunk dashboard to track 50% off online coupon promoFon Results • Automated informaFon • Report submiied to our leadership team, including the CIO and CEO • Monitoring promoFon success in real-‐Fme 20 Splunk for GEO Sales Tracking ! ! ! Splunk RESTful APIs integrate with Domino’s GEO sales tracking applicaFons (Java based) Sales monitoring by regions We have been able to idenFfy ISP outages in certain regions 21 Tracking Easy Orders ! ! Easy Ordering was released this year and we needed a way to quickly pull reports on how many orders are being placed using the feature Overall number of Easy Orders in a given period is a report that our Data Warehouse team can easily create but Splunk gives us the ability to see this data in real Fme 22 NaFve Voice vs NaFve App ! ! Another new feature released this year for iPhone/iPad and Android was voice ordering. Our Windows plaxorm always had it We could easily pull reports that showed how many users had used the voice feature to submit their order, again in real Fme 23 Security 24 Splunk at Domino’s: Security ! Dashboards, Maps, Alerts and Reports for the following: – Windows event logs – ApplicaFon WhitelisFng event logs – Remote access event logs – Proxy event logs – Threat Intelligence logs – Custom security event logs 25 AHA! Moments with Splunk Russell: IT team started taking Splunk home and working on their own Fme with Splunk. “Splunk is much bigger than system health monitoring tool. We are si8ng on a gold mine of data!” ADD NOTE FROM SECURITY TEAM 26 Best PracFce RecommendaFons ! ! ! ! ! ! Build a full blown POC to demonstrate Splunk’s value Create a cross funcFonal team to define real use cases to demonstrate Splunk’s effecFveness. Plan your Splunk deployment (distributed environment); understand where config files live Splunk documentaFon is helpful – use it! Leverage the huge online community Take scoping seriously, especially sizing… Once that sweet data starts rolling in you’ll want to turn on even more 27 Splunk at Domino’s: Future Create real Fme dashboards for any departments to view OLO health, not just reports mailed to the LT ! Use Splunk for more key performance analyses ! Expand Splunk Apps deployment: Linux and Unix monitoring, VMware App, F5 integraFon ü OpFmize middleware applicaFon logs for Splunk consumpFon ü Start to leverage Splunk to monitor corporate applicaFons built on our stack (Liferay) and Store health ! 28 Results with Splunk Reduced MTTR ProacFve AlerFng and Baselining Cost Savings Real-‐Fme alerFng = proacFvity Historical baselining and trending has been huge Issue resoluFon from 2-‐3 hours to less than 5 minutes. Saved $300,000 vs. alternate APM tools Engineering resources freed up for other needs. 29 OperaFonal Intelligence Tracking: business relevant informaFon, trends, promoFon success, customer behavior Lessons Learned ! ! ! ! ! ! ! Use AD/LDAP for role based access Always have a Search Head in front of your users, simplifies security Always have a Heavy Forwarder in front of your Indexers, simplifies maintenance and improves stability and performance (WHOOPS!) Always name your log source at the Universal Forwarder, manage using Puppet, System Center or exisFng deployment infrastructure Assume you will collect all data in the environment when creaFng iniFal indexes, don’t use default and don’t create indexes ad-‐hoc. i.e. plan index use Logs are Garbage In/Garbage Out, encourage developers to use key value or json format, 10x easier than trying to parse poorly wriien logs Have a defined owner who can control and fund backlog/Splunk work, without this you are setup to fail 30 Summary ! ! ! ! Splunk empowers us to beier uFlize our technology to gain a compeFFve edge, not just monitor system resources! Helps to ensure excepFonal customer saFsfacFon Enables us to be agile and could allow markeFng to decisions based on real-‐Fme promoFon data Splunk helps us not just save cost but boosts morale as well 31 My Summary (conFnued) ! ! ! By leveraging Splunk in creaFve ways we’ve been able to expand our relaFonships with team outside of IS It’s given us exposure to other parts of the business that we likely would never have been exposed to I know in the video I said I didn’t want to become the “Real-‐Fme Monitoring Team” but embrace that role 32 Thank You! 33
© Copyright 2024 ExpyDoc
