The energy industry is a data-rich environment. Across the sector, companies collect a wide variety of data in huge volumes, often at high velocity. Geologists gather seismic data to pinpoint potential new sources of gas and oil. Renewable companies continually watch the sun and the wind to both identify changing patterns and optimally locate energy generation sites. National Oil Companies (NOCs) are using drones in remote locations to gather visual information on, for example, pipeline leaks. And refineries and processing plants increasingly depend on big data to predict production failure and activate remedial maintenance, thereby reducing downtime.
The industry as we know it today simply wouldn’t exist without its data. But despite its importance, most energy companies do something strange with the vast majority of the data they generate: They throw it away. There are practical reasons for that. In aggregate, the equipment installed on an offshore oil platform generates between one and two terabytes of data every day. Right now, those facilities just don’t have the data bandwidth to export most of that data for analysis. As a result, they monitor the data stream for things they know are important, like indications that there is a safety or production issue. If there’s nothing there to raise concern, the remaining data may be stored locally for a while, but it is ultimately deleted.