IT ran its Melbourne Cup well before race day
Tabcorp and Sportsbet share their Cup Day tips on how to prepare for workload surges
The first Tuesday in November is a special day for Australians, as 'The race that stops a nation', the Melbourne Cup, makes a once-a-year flutter all-but-compulsory.
The workload faced by betting agencies therefore soars on the day, as punters flock to betting shops and hit the web to back their preferred beasts.
For Tabcorp, a wagering, gaming and media outfit that has annual turnover in excess of $AUD3bn, Melbourne Cup day is the busiest of the year.
Tabcorp Chief Information Officer, Kim Wenn, told The Reg that “Last year Tabcorp processed approximately 56 million transactions on Melbourne Cup Day.”
Transaction loads hit 1,100 transactions per second and traffic to the company's website, tab.com.au, saw 58,000 concurrent customers clicking away.
“This year we expect in excess of 75,000 concurrent customers with transaction rates expected to peak beyond 1,000 transactions per second just before the Melbourne Cup jump,” Wenn said.
Getting ready for that load requires months of effort. Wenn said “We start preparing for the following year the day after the Cup is run,” but things get serious in May when “Operational readiness and scheduling of staff over key race days begins.”
Wenn said her IT team focuses on three things ahead of the cup.
Operational support efforts sees Tabcorp work to ensure “all of the support teams are available and onsite where required. This includes our internal team and a large number of key vendor partners.”
Performance testing is another focus, and sees what Wenn called “extensive performance testing across all of our distributions channels and backend host systems” before the big day.
The IT team also goes into lockdown, prohibiting changes to systems before the busy period of the Spring Carnival and its culmination on Melbounre Cup Day. “We freeze system changes ahead of the Carnival period to ensure operational stability,” Wenn explained. “Only those changes that are identified as absolutely essential will be applied and only after rigorous testing and change management processes have been followed.”
That freeze – and a preference for on-site IT – means Tabcorp can't take advantage of cloudbursting to give it extra capacity.
“Tabcorp infrastructure is scaled to meet the loads for Melbourne Cup day,” Wenn said. “While there are potential cost savings opportunities in external cloud computing, we need to ensure the absolute integrity of our customer data and 100% availability of our systems.”
“ At this time we believe this is best achieved through having these systems run and supported in-house. In addition, Tabcorp has developed an internal ‘cloud’ and we are progressively migrating our systems to this environment. Over time we expect (subject to regulatory approval) that our core wagering platforms will be able to take advantage of our internal capability.”
Come the big day, Wenn hopes her team has little to do.
“Our preparations leading up to Cup day are there to ensure that the systems run smoothly and human intervention is not required,” she said. Come Cup Day, however, all hands are on deck.
“Having said that, all systems are monitored closely for any deviation in expected performance. Monitoring is undertaken by data centre operations our applications support team and third level development teams. If actions need to be taken, our incident processes will kick in. This ensures that any actions that may be required are balanced against the risk of change and that senior IT and business management are kept informed and consulted where necessary.”
Darwin-based online gaming company Sportsbet also prepares in advance for Cup day.
Senior Systems Administrator Scott Rosicka told The Reg the company relies on a three-tiered storage array, plus a RAM SAN, to handle web traffic that will peak at 4.000 bets per minute on the day of the race.
Like Tabcorp, Sportsbet tries to clear the IT decks before the big race, shifting non-critical data out of production systems to free up space.
Lead-up races like the Caulfield Cup – the second-busiest day of the year but still with less traffic than Melbourne Cup - give the company a chance to test its preparations.
Rosicka said he pulled an all-nighter in the week before the Caulfield Cup, and has conducted load tests on production systems, to ensure all is in readiness. ®
El Reg's local connections (aka a brother who likes a punt and used to be a sports journo) suggest Zabeelionaire is the nag on which to do your dough. It has impeccable bloodlines thanks to Zabeel and Sir Tristram and is at tasty odds at the time of writing.