Reducing Friction with DataOps
- Details
- Written by Paul Lewis
Businesses are intolerant of inefficiency of any kind, especially if it takes the form of people, process, or technological bottlenecks. Businesses expect their day-to-day operations, their hierarchies and organizational structures, their supply chains, and their IT operations to run as smoothly and efficiently as possible… to be frictionless.
On the other hand, we can easily imagine scenarios in which friction functions as a kind of necessary, essential, and – in the proper context – advantageous constraint.
At a basic level, friction is what makes Usain Bolt go. It’s what enables Simone Biles to stick her spectacular landings. Most of us can easily imagine how friction operates as a positive factor in the design of mechanical parts too: tires, brakes, transmissions, flywheels, and belt-driven pulleys all depend on friction to work.
Like it or not, friction fulfills an important function in any system.
Friction in data management and data governance
This is a good segue to the crux of this post. The points I want to make are two-fold. First, friction is actually a useful concept in both disciplines. Governance, for example, is the inevitable product of friction between two or more opposing forces or entities – divergent values, priorities, purposes – that come into conflict with one another. In a sense, governance is friction. Albeit friction of a necessary and essential type.
Second, friction is a positive force because it makes certain things hard to change. If this point seems counter-intuitive, then think about your high-value investment in your management information systems (MIS) infrastructure. Companies pour millions of dollars into designing and maintaining their MIS infrastructures. As much as companies like to complain about the state of their data mart, data cube and ETL assets, people depend on operational reports, dashboards, KPIs, and scorecards to support day-to-day business decision making. As a result, IT doesn’t introduce new data sources without thoroughly testing and trouble-shooting them. This is one reason it takes so long to provision new data or analytics in a conventional data warehouse architecture. Another reason is that, until recently, IT didn’t have any other options. Now it does. I’ll say more about this below.
Self-service is a response to friction
If it seems that IT’s priorities are at odds with the self-service ethic, it’s because they are. This is less of a problem than most of us realize. If you think about it, we’re used to framing the relationship between data management/data governance and self-service as antagonistic: a zero-sum collision of top-down authority with bottom-up insurgency. Thanks to technological and economic disruption, it’s now possible to see it in terms of both, not a zero-sum relationship.
First, some obligatory background. Self-service tools were a response to users’ frustration with unnecessary forms of IT friction, such as overly restrictive data management and data governance controls. Self-service forced IT to reassess what was necessary in both data management and data governance. Consistent, conformed data is less necessary for self-service, analytic discovery, data science, data mining, and machine learning. On the contrary, access to the data is essential for these and similar disciplines.
Fast-forward to the present… DataOps
IT is under pressure to incorporate data from more and varied sources into the operational reports, dashboards, and scorecards that power critical day-to-day business decision making. There’s pressure to enhance these MIS assets with new KPIs and new analytics. There’s pressure to use these assets to accommodate new practices, such as self-service discovery, data science, and machine learning, as part of new development paradigms for which they’re fundamentally ill-suited.
That’s the crux of the problem. IT is under pressure to accommodate an entirely different data management paradigm: DataOps. Traditional data management presupposed a centralized site for data access and data processing. This was the data warehouse. In the DataOps paradigm, the data warehouse is one among many data sources and data processing engines. Data isn’t just created in a diversity of contexts, it is accessed, moved, and processed in a diversity of contexts too.
DataOps assumes that for data access and data processing:
- Data may originate at the enterprise edge and be processed before being vectored to its many and varied destinations.
- Data engineers and data scientists will build multi-stage pipelines which vector data to and from multiple data sources and/or processing engines.
- Data will stream continuously from IoT and telemetry signalers, on- and off-premises RESTful services, and the like.
Retrofitting an MIS infrastructure for DataOps involves introducing radical changes into well-understood and strictly governed data integration processes. It also expects data type diversity beyond just OLTP systems. Unlike MIS, DataOps presupposes a federated model. You can’t square this circle by retrofitting it.
Squaring the circle
So how do you square this circle? How do you accommodate a set of demands and priorities that seem to be at cross purposes with one another? The solution as I see it is actually pretty simple: you cut the Gordian Knot. You don’t even try to bend your existing MIS infrastructure to support fundamentally new DataOps-like practices and use cases.
The sensible thing to do is to draw a square around your circle – i.e., to create an environment just for DataOps. You can exploit disruptive technologies, like public and private cloud services; converged infrastructure; and distributed object storage, to build a parallel DataOps environment that supports self-service and similar use cases. Even better, your parallel environment would consume exactly the same data that flows into your governed MIS systems – extracted from the same OLTP sources and applying the same transformations. But it won’t impact, or, above all, change your MIS infrastructure.
Best of all, you can slipstream new sources of data and new types of analytics from the parallel DataOps environment into your bread-and-butter MIS infrastructure once they’re proven to be useful and mature. Call it whatever you like: a data lake, a data hub, or a data refinery; the important thing is that your DataOps environment is ideally suited for self-service discovery, data science, machine learning (ML), and artificial intelligence (AI). Like it or not, DataOps requires a fundamentally different kind of friction – i.e. different constraints and controls with respect to how data is sourced, how rapidly it is provisioned, how it is engineered and/or changed, how it is used – than core MIS.
Making friction your friend
To sum up, there’s essential friction in both environments. It’s just different kinds of friction. Some controls (i.e. controls that govern the use of personally identifiable information - PII) will be consistent across both environments. Even so, the parallel DataOps environment might make use of automated mechanisms, such as data masking/hashing or differential privacy algorithms, to enable data scientists and ML engineers to use data that contains PII without exposing the PII in a way that is inconsistent with governance goals or controls. In addition, some contractual and regulatory requirements will also be consistent across both environments. The parallel DataOps environment lets organizations diversify the insights they produce, as well as carefully and deliberately incorporate this diversity into their core MIS processes. In this way, the organization can introduce predictive and prescriptive analytics into its fragile (and tightly controlled) MIS infrastructure.
We’re transitioning to a model in which we manage, secure, and control data in a federated way. At Hitachi Vantara, we like to call this “Edge to Outcomes.” Let friction and the feedback it generates help you navigate this shift. Let friction help you determine the pace at which you shift your MIS systems, processes, and workloads to other contexts – be it the on-premises private cloud, the virtual private cloud, the public cloud or, conversely, get enriched with data or analytics that originate in the cloud. Let friction help you determine the pace at which you use IoT data from the edge to enrich operational reports, KPIs, dashboards, and other business-critical MIS assets. Let friction help you determine the pace at which you make data and analytics available to or from external sources. Let friction help you determine which data you need to keep in place where it can be processed in situ. Let friction be your friend, not your foe.
Paul Lewis is Global Vice President Industry and Enterprise Architecture of Hitachi Vantara and an advisory board member with The IT Media Group.
Past Attendees
ADP - VP Architecture & Infrastructure
AESO - VP, Information Technology
Agnico Eagle Mines - VP, IT
Agrium - Global Mgr., IT Security
Agrium - Senior Director IT Shared Services
Aimia - SVP & Global CIO
Ainsworth Engineered - Director IT
Air Canada Vacations - Director IT
Alberta Energy Regulator - Director, Office of the CIO
Anthem Properties - VP IS
AON Risk Solutions Canada - Head of IT
Avison Young - VP Global Enterprise Architecture & Integration
Aviva Canada - VP, Architecture & Strategy
Bank of America Merrill Lynch - CTO
BC Ferry Services - VP & CIO
Bell Business Markets - Director, Strategy & Planning
Bell Canada - National Director, Digital Transformation
Bellatrix Exploration - Director, Information Technology
Bentall Kennedy - VP IT
BFL CANADA - CIO
BFL CANADA - Director, Cybersecurity & IT Risk Management
Black Press - CTO
BlackBerry - VP Corporate IT
BMO Financial Group - Director, Technology & Operations Transformation
BMO Financial Group - Head of Services Delivery
Bombardier Aerospace - CISO
Bonavista Petroleum - Head of IT
Borden Ladner Gervais LLP - Global CIO
Bow Valley College - Director, IT Services
Bridgewater Bank - Head of IT
BuildDirect - VP IT
Bulk Barn - Head, IT
Burnco - CIO
Caisse de Depot et Placement du Quebec - VP, IT Planning, Architecture, Governance, Operations
Calfrac Well Services - Head of IT
Canada Goose - CTO
Canada Live - VP of Technology
Canada Mortgage and Housing - VP, Information & Technology
Canada Protection Plan - Head of IT
Canadian Depository for Securities - CIO
Canadian Direct Insurance - CTO
Canadian Payments Association - VP & CIO
Canucks Sports - Head of IT
Capgemini - Service Delivery Director
CAPREIT - CIO
Cardel Homes - VP MIS
Cargojet - CIO
CBI Health Group - CIO
CCS Corp. - VP IT
CDSPI - Board Director
Centerra Gold - Director IT & Comm
CI Global Asset Management - VP of Enterprise Infrastructure $amp; Operations
CIBC - Senior Director, Infrastructure Planning & Engineering
CIBC - SVP & CIO, Retail and Business Banking Technology
CIBC Mellon - AVP, Enterprise Architecture
CIBC Mellon - SVP & CIO
Cineplex Entertainment - CTO
City of Brampton - Senior Manager, IT Architecture & Planning
City of Richmond Hill - CIO
City of Toronto - Director of Strategic Planning & Architecture
CN Rail Service - Chief Information Security Officer
Coast Capital Savings - VP Technology
Colliers - Head of Technology & Data
Concordia University - AVP & CIO
Crescent Point Energy - Head of IT
Dairy Farmers of Ontario - Head of IT and Administration
Dale Parizeau Morris Mackenzie - VP, IT
Davies Ward Phillips & Vineberg LLP - Director, Information Technology
DealerTrack Canada - Director, Technology
Defence Construction Canada - Corporate Manager, IT
Deloitte - Director, Risk Advisory
Dentons - Canada CIO
Devon Energy - Director, Integrated Business Services
Direct Cash - VP IT & Security
Dynamic Tire Corp - CIO
D+H Partnership - VP, Head of Canadian Mortgage Technology
eHealth - EVP, Technology
eHealth Ontario - VP, IT Systems & Services
Encana - Director, InfoSec
Enbridge Inc. - VP, Technology and Information Services
Enerflex - CIO
Enerplus - VP. IS
ENMAX - VP, IT & PMO
Equitable Bank - CIO
Equitable Bank - CISO
Equity Financial Trust - VP, IT
Essential Energy Services - Director, IT
Expedia Cruise Ship Centers - VP IS
FGL Sports - VP, Information Technology
Finastra - SVP, Head Technology Managed Services
Fix Auto Canada - COO & SVP
Flightnetwork.com - CIO
Freedom Mobile - Head, Customer Applications, Experience, & Strategy
FT Services - CIO
FundServ - CIO
Genus Capital Management - CTO
Genworth Financial Inc. - VP IT
Geotab - Board Member
Golder Associates - CTO
Gran Tierra Energy - Director IT
Grant Thornton LLP - CIO
Grand River Hospital - Director, Data Governance & Analytics
Greenwin Inc - VP, Information Technology
Groupe Dynamite - Director, IT
GSK Canada - IT Director
GTAA - Acting CIO
H&R Block Canada - VP IT
Haventree Bank - VP, Technology
Hewitt Equipment Ltd. - VP & CIO
Hitachi Vantara - GVP & Global CTO
Home Trust Company - CIO
Home Trust Company - CTO
Home Trust Company - VP & CISO
Horizon North Logistics - CIO
HSB Canada - VP IT
IBM Canada - Associate Partner, Payments Industry
Indigo Books and Music - CIO
Interac Corp - Director, Platform Engineering
ivari - SVP & CIO
JP Morgan Chase Canada - Executive Director, Information Risk Management
Keyera Energy - Director, Information Technology
KFC Canada - CTO
KnowledgeOne - CIO
LaFarge Canada - Director, IT
Landmark Cinemas Canada - VP, IT
LAWPRO - CIO
LCBO - Director, Applications Systems
LCBO - SVP & CIO
Leisureworld Senior Care Corp - VP IS
Lifeguard Digital Health - Chief Security & Informatics Officer
Loblaw Companies Ltd - Senior Director, Customer Engagement Technology
London Drugs - GM IT
Loto-Quebec - Corporate Director, InfoSec
Magna International Inc - VP & Global Leader, IT (CIO)
Manulife - Global Head of Private Markets & Real Estate Technology
March Networks - VP Professional Services & CIO
MaRS Discovery District - Managing Director, Fintech and Commerce
McCain Foods Limited - Manager InfoSec
McInnis Cement - Director of Information Technology
Medical Pharmacies Group - VP, Information Technology
MEG Energy - Manager, Information Technology Solutions & Services
Metrolinx - EVP & CIO
Minto Group - VP IT
MMM Group - CIO
Montreal Police Service - CIO
Morguard Investments - CIO
Moulding & Millwork - CIO
MullenLowe Group - Global CIO
National Bank of Canada - Information Security Officer
National Capital Commission - Chief, IT infrastructure & Support Services
NHL Players' Association - Head, Security & Technology
Northbridge Financial Corp - CIO
OEC Group Canada - Vice President, Information Technology and Client solutions
ODAIA - CEO
Oildex - VP, Architecture & Infrastructure
OPTrust - AVP, Enterprise Data Services
Olympia Financial Group - CIO
OMERS - EVP, Data & Technology
OMERS - SVP IT
OMERS - SVP, Data & Advanced Analytics
Ontario Pension Board - CTO
Ontario Teachers' Pension Plan - SVP, Product & Delivery
Ontario Trillium Foundation - CIO
Osum Oil Sands Corp - Manager, IS
Ottawa Police Service - CIO
Pacific Western Transportation - CIO
Packers Plus - Global IT Director
Pason Systems - Manager, Digital Communications & Corporate IT
Patient News - CTO
Peel District School Board - CIO
Pengrowth Corp - Director IS
Penn West Exploration - Snr. Manager, IT Operations
Peterson Investment Group - Head of IT
PFB Corp. - CIO
Pizza Pizza - CIO & VP, IT
Precision Drilling - VP, IT
Precision Drilling - Director, IT Infrastructure & Security
PSP Investments - Snr. Director, Internal Audit & Business Infosec
Public Works and Government Services Canada - Director, IT Security Directorate
PwC - Managing Director, Real Estate Technology Advisory
Pythian - CTO
Qantas - Global CIO
Queen's University - Director, Information Technology
RBC Royal Bank - Head of Application Security, Data Protection & Security Consulting
RBC Royal Bank - VP, Technology Platforms & Risk Management
RBC Royal Bank - Global Cybersecurity VP
Regal Lifestyle Communities - CIO
Revera Inc. - CIO
Revera Inc. - Security Architect
Rheem Manufacturing - CISO & Enterprise Architect
Ricoh Canada - VP,IT
RioCan Property Services - VP IT
Roche - Head of IT Americas – Operations
Rogers Communications - SVP, Customer Experience IT
ROM - CIO
Russel Metals - VP,IS
Salvation Army Canada - Board Director
SCI Group - CIO
Scotiabank - Head, Systems Architecture & Platform Modernization
Scotiabank - VP - International Systems Technology
Scotiabank - Head, System Architecture & Platform Modernization
Scotiabank - Global Head, GBM Compliance & Transformation
Sears Canada - Divisional VP, Information Technology Services
Secure Energy Services - GM, IT
Shaw - Head, Customer Applications, Experience, & Strategy
Shaw Communications - VP, Technology Operations
Shaw Communications - Director, Risk Management
SMART Technologies - Director, IS Corporate Services
Smartcentres - Director IS, IT
SmartOne Solutions - President & CIO
Societe de Transport de Montreal - Division Head - Security and Compliance
Street Capital Financial - CIO
Sun Life Financial - AVP, Data & Business Intelligence Services
Sun Life Financial - VP Application Ops & Services
Sunco Communication - COO
Suncor Energy Inc. - Director, Application Portfolio Optimization, I&PM, Business Services
Symcor - CTO, VP Technology Services
Talisman Energy - SVP IT & Business Services
TD Bank - Enterprise Architect
Teknion - SVP, CIO
TELUS - Chief Security Architect
Tervita Corporation - VP, Information Technology
The Hudsons Bay Company - VP Technology
The Hudson's Bay Company - SVP & CIO
The Source - VP, Information Technology
TMX Group - CISO & Global Head of Infrastructure Services
Toromont Industries - VP & CIO
Toronto District School Board - Chief Technology Officer
Toronto Hospital for Sick Children - Director of Technology
Toronto Parking Authority - CIO
Toronto Police Services - CISO
Toronto Transit Commission (TTC) - Chief Enterprise Architect
Toronto Transit Commission (TTC) - CIO
Toyota Canada - National Manager, IS
Transamerica Life Canada - CIO
Trican Well Services Ltd. - Director, Business Information Systems
Tridel Corporation - CIO
Trillium Health Partners - IT Director, Applications & Clinical Informatics
UFA Cooperative - VP & CIO
University of Calgary - Executive Director, Development Services
University of Ottawa - CIO
University of Ottawa - Senior Director IT Services & Infrastructure
University of Toronto - Director, Centre for Management & Technology
University of Waterloo - Director, Technology Entrepreneurship
Valencia Risk - Managing Director
Vancity - VP Technology & Solutions
Viterra - Director Enterprise Technology
Wawanesa Mutual Insurance Company - Director of Innovation Outpost
World Health - Director IT
Wolseley Canada - CIO & COO
WSIB - Board Director
Yellow Pages Group - Director - Enterprise Data Management
York Region District School Board - CIO
York University - Board Director
Technology Archive
- ► 2024 (1)
- ► 2023 (1)
- ► 2022 (1)
- ► 2019 (3)
- ► 2018 (1)
- ► 2017 (2)
- ► 2016 (2)
- ► 2015 (1)
- ► 2014 (3)
- ► 2013 (5)
- ► 2012 (4)
Lessons from CrowdStrike on Safeguarding Your Data with Compliance, Continuity and Disaster Recovery Strategies.
Recently I went for my annual physical where my routine blood work gave me some interesting news.
Generative AI (GenAI) has captured the imagination of enterprises across the globe, with nearly all organizations recognizing it as a top-five priority. However, a recent report from 
AI factories and foundries moved into the popular lexicon when Nvidia CEO Jensen Huang used these terms 
