WellBet – wisdom of the crowd in oil exploration

By Jonny Hesthammer and Hans Gebauer

Over the past four decades, commercial exploration success rate has remained surprisingly constant at around 25% worldwide. The same success rate is also found for more than 300 wildcat wells offshore Norway during the past decade. The introduction of 3D seismic and other technologies combined with an ever increasing knowledge base have not changed this. Although it can be argued that while technology improves prospects become harder to find, it is certainly worthwhile exploring the experience aspect in more detail. Is it at all possible to change the established 25% commercial success rate? This question is one of the reasons for establishing the WellBet concept, which is all about making an educated guess on the outcome of NCS prospects to be drilled. This playful application of the prediction market theory (of which the stock exchange is probably the best known example) in the oil & gas industry has been run with geoscience students at the university of Bergen as well as with the team of Atlantic Petroleum Norge since 2012, including not only exploration experts but other team members such as IT and accounting.

Prediction markets produce accurate results

Prediction markets utilize inputs from various participants in game-like environments and have been shown in several scientific studies as well as within companies and different markets to produce accurate results by combining the knowledge of many via market-based aggregation mechanisms. In other words, it builds on the idea that the average opinion of a crowd of semi-experts is better than the view of a single expert. Now, go ahead and tell that to your exploration manager if you dare.

The WellBet experiment started in 2012 with three wells targeting the Garantiana, Albert and Salina prospects. Interestingly, the students did better than the oil industry employees in these first examples – an intriguing result which allowed at least the hypothesis that it was not a given that the real experts were better than semi-experts (geoscience students in this case). And it could potentially indicate that the prediction market theory works for oil exploration as the predictions were quite accurate. However, the number of wells was certainly not high enough to draw sound conclusions.

The experiment was therefore continued and expanded with many more wells. So far, students have participated in 18 wells while the Atlantic employees have participated in 19 wells (not all wells are the same). This provides enough data for an analysis and a comparison of the students and the oil industry experts. In order to set the results in context and to test against a single expert, the predictions of Atlantic’s exploration manager Susanne Sperrevik are presented separately.

Fascinating numbers

Overall, the numbers are fascinating. When considering the ability to predict a discovery (both technical and commercial), Atlantic employees are right in 83% and students in 72% of their predictions. Apparently, and perhaps luckily, oil industry employees perform better than students, although not by a lot. Regardless, Atlantic’s exploration manager Susanne outperforms both groups with 89% correct prediction success rate.

To add complexity, phase was also included in the prediction game as a third parameter and participants had to choose between a completely dry well, the occurrence of oil, gas or both oil and gas, thus increasing the complexity significantly. The result for the three parameters is a 68% correct prediction success rate for the employees of Atlantic, 54% for students and – again superior – 79% for Susanne. Not surprisingly perhaps, oil industry employees seem to be better at predicting phase than students.

However, when it comes to real business, it is not enough to predict the correct outcome of a well. Since most wells do not encounter a commercial discovery (keep in mind the, on average, 25% commercial success rate), you could argue that it is possible to bet on a dry well each time, and, overall, you would do great. As such, key commercial aspects to consider are: (a) should an oil company farm-in to an opportunity and (b) are you actually able to capture the commercial discoveries (missing these would be terrible for an oil company over time). With this in mind, you can state that if a prediction market group believes in a commercial discovery, you should farm in. If, on the other hand, the group believes that the well will not be a commercial discovery, you should not farm in. We define this as the “farm-in success rate” (i.e. evaluating if the oil company made the right commercial decision). The farm-in success rate in the WellBet groups so far is 79% for the Atlantic employees, 72% for the students and, once again, a remarkable 89% for Atlantic’s exploration manager.

Anyway: It does not help if you make a right decision, but miss the actual commercial discoveries by always stating that the oil company should not farm in to an opportunity (the risk averse approach). WellBet handles this as well: So far, the student well population contains 6 commercial discoveries, which is the same as the Atlantic well population although not all discoveries are the same. So did students and employees actually capture these opportunities?

As a matter of fact, they did. In 4 out of the 6 (67%!) cases the two groups correctly predicted the exploration wells to make commercial discoveries. However, Susanne once again exceeded this excellent quota with an impressive 80% prediction accuracy. Why Atlantic’s exploration manager is deviating from the general prediction market theory can only be speculations at this stage. However, there are numerous studies indicating that intuition, based on knowledge acquired over many years, applied correctly (!) can be an extremely powerful tool as documented by Daniel Kahneman who received the Nobel Prize in economics for his work which was later published in the bestselling book “Thinking, fast and slow”.

Taking it a step further

In order to further test and expand on the potential of prediction markets for hydrocarbon exploration, WellBet was launched in May 2014 on Geonova – a Norwegian-based online community for geoscientists (http://www.geonova.no/well-bet/) where it is accessible for a broader audience. On Geonova, prospects being drilled are presented with publicly available information such as the exact location of the well and a description of the prospect including a map with geological data, information about the operator and partners, and weblinks to relevant articles, presentations and the official drilling permit by the Norwegian Petroleum Directorate (NPD).

Participants need to indicate whether they predict a technical (yes, no) as well as a commercial discovery (yes, no), what kind of fluids will be found (dry, oil, gas or oil & gas) and their level of knowledge about the prospect (nothing, little, quite a lot). Finally, participants are asked to provide information about their age (less than 25, 25-40, 40+) and their background (oil industry, finance, academia incl. students, other).

Bets are placed anonymous. The bet on a prospect ends when the prospect has been drilled and the result is made public with a press release from the NPD, the operator or one of the partners. At this time, we can all see how the community prediction compares to the actual outcome.

So far, 10 WellBet have been completed on Geonova on prospects located on the Norwegian Continental Shelf (NCS), with a total number of 236 bets placed by members of the community. Most participants in the Geonova WellBet were in the age group of 40+ with a background in the oil & gas industry, although they indicated that they possessed little detailed knowledge about the actual prospects. The success rate for predicting technical and commercial discoveries as well as phase, i.e. 3 different parameters, is 53,33% which must be considered very good – and at the same level as the university students who’s score is currently at 54%. The overall discovery success rate – predicting technical and commercial discoveries but not the, perhaps most difficult parameter, phase – is somewhat higher at 60%. Interestingly, the very important farm-in success rate – the rate at which the community predicts correctly a commercial success or commercial failure – lies at 70% and thereby remarkably at quite the same excellent level as the groups of geoscience students (72%) and oil industry employees of Atlantic Petroleum Norge (79%).

With the above results, it may well be worth reflecting on the drivers for decision making within oil companies as this study indicates that it may be possible to avoid dry wells – and still capture the majority of discoveries: While geoscience students and Atlantic Petroleum employees so far can demonstrate an impressive 67% commercial discovery prediction success rate (from 6 discoveries made), WellBet participants have so far correctly predicted both commercial discoveries currently made within the population of 10 wells. From the wells included in WellBet to date, the oil companies seem to remain around the historic 25% commercial success rate (in WellBet, the oil companies have succeeded with 2 out of 10 wells which equals 20% commercial success rate). Meanwhile, the WellBet participants online outperform the oil companies, which is exactly what happened in the examples including university students and Atlantic Petroleum employees. Is it time to think differently on how drilling decisions are made and is it valid to ask if the oil companies, and perhaps particularly smaller ones, in general take on too much risk?

Community seems a bit too positive

So where do WellBet participants go wrong? So far, the community members seem to be a bit too positive on making commercial discoveries – an interesting observation considering that the majority of the group is represented by oil industry employees (is there a link between this optimism and the 75% commercial failure rate we observe for exploration wells?). Can these initial results be improved? Studies on prediction markets show that if participants are put in a more binding setup, e.g. when money is involved or names are exposed, results improve (for the interested reader, we recommend the book “Oracles: How Prediction Markets Turn Employees into Visionaries” by Donald Thompson). In the case of the employees of Atlantic, this is solved with transparency by showing the name of the participants. Maybe the WellBet participants would lose some of their optimism if their reputation – or their own money – were at stake.

To conclude, the prediction market mechanism seems to be providing highly valuable data, although more data are needed to generate resilient results. But we must also keep in mind that most people would probably not guess on Johan Sverdrup being a commercial discovery. As such, if we used WellBet to decide, this giant would probably not have been discovered. But a valid question could be: Should small companies drill such high-risk wells or should that be left to the giants such as Statoil? While WellBet may not be suitable to identify outliers, the concept may prove helpful for small oil companies with limited capital available for drilling. It seems that taking the input from a knowledgeable crowd into consideration can lead to better decisions, thus preventing companies with limited amount of money available from participating in poor opportunities.

Another key puzzling question to ask is why the students with their limited knowledge could do so well? What parameters do they consider when making their decision? Is it the track record of the license partners, the success rate of the operator, distance to nearest discovery etc.? It seems clear that they can manage to do quite well with much less information and experience generally available to them than the industry experts.

The outlier in this brief study is Atlantic’s exploration manager, Susanne Sperrevik, who for some reason – maybe by correct application of experience based intuition – outperforms all and thereby contradicts a fundamental paradigm of prediction markets: The wisdom of the crowd. No rule without exception!

It remains to be seen how the community will perform over time. Based on the results of the first bets in WellBet run in the three different groups, we can state so far that non-experts can have a relevant saying in finding oil and we should watch closely how the WellBet accuracy develops with increasing number of participants.