HomeSitemap This email address is being protected from spambots. You need JavaScript enabled to view it.

Technology

New technologies and adoption trends that influence IT strategy, and the issues and opportunities they represent.

Reducing Friction with DataOps

Businesses are intolerant of inefficiency of any kind, especially if it takes the form of people, process, or technological bottlenecks. Businesses expect their day-to-day operations, their hierarchies and organizational structures, their supply chains, and their IT operations to run as smoothly and efficiently as possible… to be frictionless.

On the other hand, we can easily imagine scenarios in which friction functions as a kind of necessary, essential, and – in the proper context – advantageous constraint.

At a basic level, friction is what makes Usain Bolt go. It’s what enables Simone Biles to stick her spectacular landings. Most of us can easily imagine how friction operates as a positive factor in the design of mechanical parts too: tires, brakes, transmissions, flywheels, and belt-driven pulleys all depend on friction to work.

Like it or not, friction fulfills an important function in any system.

Friction in data management and data governance

This is a good segue to the crux of this post. The points I want to make are two-fold. First, friction is actually a useful concept in both disciplines. Governance, for example, is the inevitable product of friction between two or more opposing forces or entities – divergent values, priorities, purposes – that come into conflict with one another. In a sense, governance is friction. Albeit friction of a necessary and essential type.

Second, friction is a positive force because it makes certain things hard to change. If this point seems counter-intuitive, then think about your high-value investment in your management information systems (MIS) infrastructure. Companies pour millions of dollars into designing and maintaining their MIS infrastructures. As much as companies like to complain about the state of their data mart, data cube and ETL assets, people depend on operational reports, dashboards, KPIs, and scorecards to support day-to-day business decision making. As a result, IT doesn’t introduce new data sources without thoroughly testing and trouble-shooting them. This is one reason it takes so long to provision new data or analytics in a conventional data warehouse architecture. Another reason is that, until recently, IT didn’t have any other options. Now it does. I’ll say more about this below.

Self-service is a response to friction

If it seems that IT’s priorities are at odds with the self-service ethic, it’s because they are. This is less of a problem than most of us realize. If you think about it, we’re used to framing the relationship between data management/data governance and self-service as antagonistic: a zero-sum collision of top-down authority with bottom-up insurgency. Thanks to technological and economic disruption, it’s now possible to see it in terms of both, not a zero-sum relationship.

First, some obligatory background. Self-service tools were a response to users’ frustration with unnecessary forms of IT friction, such as overly restrictive data management and data governance controls. Self-service forced IT to reassess what was necessary in both data management and data governance. Consistent, conformed data is less necessary for self-service, analytic discovery, data science, data mining, and machine learning. On the contrary, access to the data is essential for these and similar disciplines.

Fast-forward to the present… DataOps

IT is under pressure to incorporate data from more and varied sources into the operational reports, dashboards, and scorecards that power critical day-to-day business decision making. There’s pressure to enhance these MIS assets with new KPIs and new analytics. There’s pressure to use these assets to accommodate new practices, such as self-service discovery, data science, and machine learning, as part of new development paradigms for which they’re fundamentally ill-suited.

That’s the crux of the problem. IT is under pressure to accommodate an entirely different data management paradigm: DataOps. Traditional data management presupposed a centralized site for data access and data processing. This was the data warehouse. In the DataOps paradigm, the data warehouse is one among many data sources and data processing engines. Data isn’t just created in a diversity of contexts, it is accessed, moved, and processed in a diversity of contexts too.

DataOps assumes that for data access and data processing:

  • Data may originate at the enterprise edge and be processed before being vectored to its many and varied destinations.
  • Data engineers and data scientists will build multi-stage pipelines which vector data to and from multiple data sources and/or processing engines.
  • Data will stream continuously from IoT and telemetry signalers, on- and off-premises RESTful services, and the like.

Retrofitting an MIS infrastructure for DataOps involves introducing radical changes into well-understood and strictly governed data integration processes. It also expects data type diversity beyond just OLTP systems. Unlike MIS, DataOps presupposes a federated model. You can’t square this circle by retrofitting it.

Squaring the circle

So how do you square this circle? How do you accommodate a set of demands and priorities that seem to be at cross purposes with one another? The solution as I see it is actually pretty simple: you cut the Gordian Knot. You don’t even try to bend your existing MIS infrastructure to support fundamentally new DataOps-like practices and use cases.

The sensible thing to do is to draw a square around your circle – i.e., to create an environment just for DataOps. You can exploit disruptive technologies, like public and private cloud services; converged infrastructure; and distributed object storage, to build a parallel DataOps environment that supports self-service and similar use cases. Even better, your parallel environment would consume exactly the same data that flows into your governed MIS systems – extracted from the same OLTP sources and applying the same transformations. But it won’t impact, or, above all, change your MIS infrastructure.

Best of all, you can slipstream new sources of data and new types of analytics from the parallel DataOps environment into your bread-and-butter MIS infrastructure once they’re proven to be useful and mature. Call it whatever you like: a data lake, a data hub, or a data refinery; the important thing is that your DataOps environment is ideally suited for self-service discovery, data science, machine learning (ML), and artificial intelligence (AI). Like it or not, DataOps requires a fundamentally different kind of friction – i.e. different constraints and controls with respect to how data is sourced, how rapidly it is provisioned, how it is engineered and/or changed, how it is used – than core MIS.

Making friction your friend

To sum up, there’s essential friction in both environments. It’s just different kinds of friction. Some controls (i.e. controls that govern the use of personally identifiable information - PII) will be consistent across both environments. Even so, the parallel DataOps environment might make use of automated mechanisms, such as data masking/hashing or differential privacy algorithms, to enable data scientists and ML engineers to use data that contains PII without exposing the PII in a way that is inconsistent with governance goals or controls. In addition, some contractual and regulatory requirements will also be consistent across both environments. The parallel DataOps environment lets organizations diversify the insights they produce, as well as carefully and deliberately incorporate this diversity into their core MIS processes. In this way, the organization can introduce predictive and prescriptive analytics into its fragile (and tightly controlled) MIS infrastructure.

We’re transitioning to a model in which we manage, secure, and control data in a federated way. At Hitachi Vantara, we like to call this “Edge to Outcomes.” Let friction and the feedback it generates help you navigate this shift. Let friction help you determine the pace at which you shift your MIS systems, processes, and workloads to other contexts – be it the on-premises private cloud, the virtual private cloud, the public cloud or, conversely, get enriched with data or analytics that originate in the cloud. Let friction help you determine the pace at which you use IoT data from the edge to enrich operational reports, KPIs, dashboards, and other business-critical MIS assets. Let friction help you determine the pace at which you make data and analytics available to or from external sources. Let friction help you determine which data you need to keep in place where it can be processed in situ. Let friction be your friend, not your foe.


Paul Lewis is Global Vice President Industry and Enterprise Architecture of Hitachi Vantara and an advisory board member with The IT Media Group.

Past Attendees


ADP - VP Architecture & Infrastructure

AESO - VP, Information Technology

Agnico Eagle Mines - VP, IT

Agrium - Global Mgr., IT Security

Agrium - Senior Director IT Shared Services

Aimia - SVP & Global CIO

Ainsworth Engineered - Director IT

Air Canada Vacations - Director IT

Alberta Energy Regulator - Director, Office of the CIO

Anthem Properties - VP IS

AON Risk Solutions Canada - Head of IT

Aviva Canada - VP, Architecture & Strategy

Bank of America Merrill Lynch - CTO

BC Ferry Services - VP & CIO

Bell Business Markets - Director, Strategy & Planning

Bell Canada - National Director, Digital Transformation

Bellatrix Exploration - Director, Information Technology

Bentall Kennedy - VP IT

Black Press - CTO

BlackBerry - VP Corporate IT

BMO Financial Group - Head of Services Delivery

Bombardier Aerospace - CISO

Bonavista Petroleum - Head of IT

Borden Ladner Gervais LLP - Global CIO

Bow Valley College - Director, IT Services

Bridgewater Bank - Head of IT

BuildDirect - VP IT

Bulk Barn - Head, IT

Burnco - CIO

Caisse de Depot et Placement du Quebec - VP, IT Planning, Architecture, Governance, Operations

Calfrac Well Services - Head of IT

Canada Mortgage and Housing - VP, Information & Technology

Canadian Depository for Securities - CIO

Canadian Direct Insurance - CTO

Canadian Payments Association - VP & CIO

Canucks Sports - Head of IT

Capgemini - Service Delivery Director

CAPREIT - CIO

Cardel Homes - VP MIS

Cargojet - CIO

CCS Corp. - VP IT

CDSPI - Board Director

Centerra Gold - Director IT & Comm

CIBC - Senior Director, Infrastructure Planning & Engineering

CIBC - SVP & CIO, Retail and Business Banking Technology

CIBC Mellon - AVP, Enterprise Architecture

CIBC Mellon - SVP & CIO

Cineplex Entertainment - CTO

City of Brampton - Senior Manager, IT Architecture & Planning

City of Richmond Hill - CIO

City of Toronto - Director of Strategic Planning & Architecture

CN Rail Service - Chief Information Security Officer

Coast Capital Savings - VP Technology

Concordia University - AVP & CIO

Crescent Point Energy - Head of IT

Dairy Farmers of Ontario - Head of IT and Administration

Dale Parizeau Morris Mackenzie - VP, IT

Davies Ward Phillips & Vineberg LLP - Director, Information Technology

DealerTrack Canada - Director, Technology

Defence Construction Canada - Corporate Manager, IT

Deloitte - Director, Risk Advisory

Dentons - Canada CIO

Devon Energy - Director, Integrated Business Services

Direct Cash - VP IT & Security

Dynamic Tire Corp - CIO

D+H Partnership - VP, Head of Canadian Mortgage Technology

eHealth - EVP, Technology

eHealth Ontario - VP, IT Systems & Services

Encana - Director, InfoSec

Enbridge Inc. - VP, Technology and Information Services

Enerflex - CIO

Enerplus - VP. IS

ENMAX - VP, IT & PMO

Equity Financial Trust - VP, IT

Essential Energy Services - Director, IT

Expedia Cruise Ship Centers - VP IS

FGL Sports - VP, Information Technology

Finastra - SVP, Head Technology Managed Services

Fix Auto Canada - COO & SVP

Flightnetwork.com - CIO

Freedom Mobile - Head, Customer Applications, Experience, & Strategy

FT Services - CIO

FundServ - CIO

Genus Capital Management - CTO

Genworth Financial Inc. - VP IT

Golder Associates - CTO

Gran Tierra Energy - Director IT

Grant Thornton LLP - CIO

Greenwin Inc - VP, Information Technology

Groupe Dynamite - Director, IT

GSK Canada - IT Director

GTAA - Acting CIO

H&R Block Canada - VP IT

Haventree Bank - VP, Technology

Hewitt Equipment Ltd. - VP & CIO

Hitachi Vantara - GVP & Global CTO

Home Trust Company - CIO

Home Trust Company - CTO

Home Trust Company - VP & CISO

Horizon North Logistics - CIO

IBM Canada - Associate Partner, Payments Industry

Indigo Books and Music - CIO

ivari - SVP & CIO

JP Morgan Chase Canada - Executive Director, Information Risk Management

Keyera Energy - Director, Information Technology

KnowledgeOne - CIO

LaFarge Canada - Director, IT

Landmark Cinemas Canada - VP, IT

LCBO - Director, Applications Systems

LCBO - SVP & CIO

Leisureworld Senior Care Corp - VP IS

Lightstream Resources - Head, Information Services

London Drugs - GM IT

Loto-Quebec - Corporate Director, InfoSec

Magna International Inc - VP & Global Leader, IT (CIO)

Manulife - Global Head of Private Markets & Real Estate Technology

March Networks - VP Professional Services & CIO

MaRS Discovery District - Managing Director, Fintech and Commerce

McCain Foods Limited - Manager InfoSec

McInnis Cement - Director of Information Technology

Medical Pharmacies Group - VP, Information Technology

MEG Energy - Manager, Information Technology Solutions & Services

MMM Group - CIO

Montreal Police Service - CIO

Morguard Investments - CIO

Moulding & Millwork - CIO

National Bank of Canada - Information Security Officer

National Capital Commission - Chief, IT infrastructure & Support Services

NHL Players' Association - Head, Security & Technology

Northbridge Financial Corp - CIO

OEC Group Canada - Vice President, Information Technology and Client solutions

Oildex - VP, Architecture & Infrastructure

Olympia Financial Group - CIO

OMERS - EVP, Data & Technology

OMERS - SVP IT

Ontario Pension Board - CTO

Ontario Trillium Foundation - CIO

Osum Oil Sands Corp - Manager, IS

Ottawa Police Service - CIO

Pacific Western Transportation - CIO

Packers Plus - Global IT Director

Pason Systems - Manager, Digital Communications & Corporate IT

Patient News - CTO

Peel District School Board - CIO

Pengrowth Corp - Director IS

Penn West Exploration - Snr. Manager, IT Operations

Peterson Investment Group - Head of IT

PFB Corp. - CIO

Pizza Pizza - CIO & VP, IT

Precision Drilling - VP, IT

Precision Drilling - Director, IT Infrastructure & Security

PSP Investments - Snr. Director, Internal Audit & Business Infosec

Public Works and Government Services Canada - Director, IT Security Directorate

PwC - Managing Director, Real Estate Technology Advisory

Qantas - Global CIO

Queen's University - Director, Information Technology

RBC Royal Bank - Head of Application Security, Data Protection & Security Consulting

Regal Lifestyle Communities - CIO

Revera Inc. - CIO

Revera Inc. - Security Architect

Ricoh Canada - VP,IT

RioCan Property Services - VP IT

Roche - Head of IT Americas – Operations

Rogers Communications - SVP, Customer Experience IT

ROM - CIO

Russel Metals - VP,IS

Scotiabank - Head, Systems Architecture & Platform Modernization

Scotiabank - VP - International Systems Technology

Scotiabank - Head, System Architecture & Platform Modernization

Scotiabank - Global Head, GBM Compliance & Transformation

Sears Canada - Divisional VP, Information Technology Services

Secure Energy Services - GM, IT

Shaw - Head, Customer Applications, Experience, & Strategy

Shaw Communications - VP, Technology Operations

Shaw Communications - Director, Risk Management

SMART Technologies - Director, IS Corporate Services

Smartcentres - Director IS, IT

SmartOne Solutions - President & CIO

Societe de Transport de Montreal - Division Head - Security and Compliance

Street Capital Financial - CIO

Sun Life Financial - AVP, Data & Business Intelligence Services

Sun Life Financial - VP Application Ops & Services

Suncor Energy Inc. - Director, Application Portfolio Optimization, I&PM, Business Services

Symcor - CTO, VP Technology Services

Talisman Energy - SVP IT & Business Services

TD Bank - Enterprise Architect

Teknion - SVP, CIO

TELUS - Chief Security Architect

Tervita Corporation - VP, Information Technology

The Hudsons Bay Company - VP Technology

The Hudson's Bay Company - SVP & CIO

The Source - VP, Information Technology

TMX Group - CISO & Global Head of Infrastructure Services

Toromont Industries - VP & CIO

Toronto District School Board - Chief Technology Officer

Toronto Hospital for Sick Children - Director of Technology

Toronto Transit Commission (TTC) - Chief Enterprise Architect

Toronto Transit Commission (TTC) - CIO

Toyota Canada - National Manager, IS

Transamerica Life Canada - CIO

Trican Well Services Ltd. - Director, Business Information Systems

Tridel Corporation - CIO

Trillium Health Partners - IT Director, Applications & Clinical Informatics

UFA Cooperative - VP & CIO

University of Calgary - Executive Director, Development Services

University of Ottawa - CIO

University of Ottawa - Senior Director IT Services & Infrastructure

University of Toronto - Director, Centre for Management & Technology

University of Waterloo - Director, Technology Entrepreneurship

Vancity - VP Technology & Solutions

Viterra - Director Enterprise Technology

Wawanesa Mutual Insurance Company - Director of Innovation Outpost

World Health - Director IT

Wolseley Canada - CIO & COO

Yellow Pages Group - Director - Enterprise Data Management

York Region District School Board - CIO

Technology Archive