The annual Olive Oil Award Zurich ceremony, held in April each year, is a well-established part of Switzerland's olive oil agenda. However, the Zurich Olive Oil Competition organised by the ZHAW under the direction of Annette Bongartz this year looks back on an unusual edition. COVID-19 not only prevented an official award ceremony in Schloss Grüental high above Lake Zurich, but also made it impossible to adequately examine the submitted olive oil samples prior to publication of the results.
A review by Silvan Brun
According to the conclusion of the Olive Oil Award Zurich this year, 100 percent of all olive oil samples sent in should correspond to the highest quality category Extra Virgin. 109 out of 109 olive oil samples were found to be free from defects in sensory analysis. This is not a pleasant result, but first and foremost surprising. It is clear that as a rule producers, retailers and bottlers send products to competitions of which they think they would score well. Similar trends are also seen in other food competitions. Thus, for example, it is not surprising that many food products awarded at competitions are not even available for sale later on. It is true some products from the same company carry the identical label as the winning product, even praising themselves with a sticker pointing out to consumers the success of the contest, but nevertheless there is usually something completely different inside the packaging. A completely different product. Lower quality, higher availability. Particularly in the case of olive oil this practice is taking on ugly proportions. A systematic fraud, and hardly anyone is looking at it!
Some olive oil producers I know even manage to deliberately produce two different products for one and the same label. One in a very small quantity, of which they send samples to the various competitions and are often awarded prizes, and another in much larger quantities, which they put on sale nicely decorated with symbols of victory. If ever a "prize-winning" olive oil you have bought was faulty, you now know why. Consequently, the results of an olive oil competition should generally be read with special care.
The coronavirus forces panels to adopt non-standard test methods
With the lockdown, the guillotine fell for the olive oil competitions. Group gatherings were no longer permitted in many places. In Italy and Spain, moreover, only those businesses that were indispensable for ensuring the provision of basic services were allowed to operate. Olive oil competitions were understandably not amongst them. Also, the required safety distance between the panel members could not be maintained within the panels. Consequently, the competition organisers were suddenly faced with a question of economic significance: Cancel the competition and reimburse the applicants for the high fees they had paid, or continue the competition under unilaterally changed contractual conditions by ordering the oils to be tasted at home by the individual assessors?
«Panel testing can be an extremely effective tool when it comes to protecting consumers.»
- Silvan Brun, evoo.expert
More than a few organisers have chosen the second option. They allowed the olive oils sent in to be tasted in the private habitats of the respective panel members. Under uncontrolled and uncontrollable conditions. Far away from an accredited standardised sensory laboratory. This turned the panel into a simple group of assessors, and under these conditions it should not have called itself a panel anymore. Provided that the terms of the contract between the organiser and the sending party so provide, olive oils can of course be analysed by sensory analysis in a test procedure that differs from the standardised test method in force in the European Union. After all, it does not have to be the panel test provided by the legislator in olive oil competitions to determine the supposedly best entered olive oils. However, and this is the main point of my criticism, is it then questionable whether the test results determined in "freestyle manner" can actually be reliable? After all, many olive oils use the prizes won in the competitions to adorn themselves - regardless of whether the olive oils were analysed by a panel that complies with the standards set out in the regulations or by an ordinary group of assessors working without standardised technical aids.
The panel test, if correctly applied, which requires that both the panel leaders and the panel members understand their roles, is a reliable tool in order to determine the quality of virgin olive oils. In fact, the quality of an olive oil can be much better identified by means of a correct panel test than by a chemical analysis. Panel tests can therefore be an extremely effective tool in protecting consumers from poor quality olive oil. And this is something that is urgently needed. This is impressively demonstrated by a study on the quality of olive oil in the Swiss retail trade in 2019 carried out by the IOF - International Olive Foundation. While 93 % of 178 chemically analysed olive oils complied with the chemical-physiological requirements set for the quality designation "Extra Vergine", on the other hand only 22 % out of 183 olive oils analysed by means of a panel test (whereby this test group also included the 178 olive oils that were chemically analysed) were able to meet the requirements for the sensory performance of an Extra Virgin olive oil. The IOF Olive Oil Panel found defects in a large number of oils which had not been detected by conventional chemical analysis.
High revenues at the Olive Oil Award Zurich
Olive oil competitions usually restrict themselves to the sensory evaluation of the olive oils sent in, which is explained by the relatively high cost of official chemical analysis and the logistical effort involved. The tasters or panel members work on a voluntary basis and are at best compensated for the costs of travel and meals. Quite often, however, not even that. On the other hand, there are sometimes exorbitant revenues that the organisers record thanks to the oil entries by the producers, bottlers and traders of olive oil. In addition, there are possibly further subsidies from trade associations, consortia and agricultural-political organisations. In order to participate in the latest edition of the Olive Oil Award Zurich, applicants had to pay 600 Swiss francs for an olive oil. For two oils 900 francs. For three, 1,200, and for each additional oil another 300 francs. 109 olive oils participated in the Olive Oil Award Zurich this year. 101 olive oils can be found on the "List of all oils OOA2020" published by the ZHAW. Split up by applicant. The Migros Cooperative Association entered four oils, as did Almazaras de la Subbética from Córdoba. Monini from Umbria even five. Finca la Torre from Málaga or Di Bennardo from Zurich one each. Adding up the participation fees according to the regulations of the Olive Oil Award Zurich 2020, the total amount is 48,000 Swiss francs. This is an average of around 475 francs per oil. Not too bad!
«The Olive Oil Award aims to increase consumer confidence in the product "Extra Virgin Olive Oil" and to promote transparency in the olive oil market. The project offers a comprehensive overview of the quality of the current olive oil offerings on the Swiss and EU markets.»
- ZHAW, Olive Oil Award Zurich
50'000 francs participation fees - expect what you get in return
The regulations of the Olive Oil Award Zurich provide detailed information on this. According to article 8 of the competition regulations, participants in the competition are granted the right to have the results determined by the panel published in the daily and specialist press and on the internet under www.oliveoilaward.ch. However, participants can be sure that before the results are published, the Swiss Olive Oil Panel will carry out a three-stage expert assessment of the olive oils they have submitted. This three-step evaluation procedure is divided firstly into a pre-tasting of the oils carried out by three tasters of the Swiss Olive Oil Panel, in which the order of tasting for the panel tests I and II is determined; secondly into the first panel test, in which at least eight tasters of the Swiss Olive Oil Panel evaluate the olive oils sent in in accordance with Regulation (EEC) No 2568/91, a description of the aroma and an assessment of harmony and, in addition, the selection of olive oils that qualify for awards for the Panel Test II; thirdly, in the second panel test, in which at least eight examiners of the Swiss Olive Oil Panel taste the olive oils from Panel Test I worthy for awards for a second time and identify the winning oils for the Gold, Silver, Bronze and Special Prizes categories.
By means of a dubious NIR method, which is carried out by Dr. Christian Gertz and which has so far failed to prove it can provide reliable results (from anticipated data, results are interpreted which the system cannot actually provide), the olive oils are additionally tested on a few chemical parameters. Along with the otherwise usual award ceremony and the individual information to the participants, which includes a detailed test report, the participants are also granted the right to a so-called consumer test, in which 60 to 80 opinions on each olive oil can be represented and after which the most popular and attractive olive oil is awarded. All in all, this represents a quite solid package of measures, which seems to justify the high participation fees. The participant gets a good deal for his money!
What the organisers of the Olive Oil Award Zurich 2020 could not know, just as little as anyone else, was that in spring 2020 a 160 nm small virus would almost lock up one third of the world's population in its own houses and homes as a result of large-scale lockdowns, leaving the economy on the verge of a historic slowdown. Hardly anything is possible any more. Governments are imposing drastic virus containment measures. Hygiene, distance, quarantine and isolation are the key words of this spring. The Olive Oil Award Zurich is no exception to the measures decreed by the Federal Council. Human gatherings of more than five people have been prohibited in Switzerland since 16 March 2020. There is a minimum distance of two metres to people not living in the same household. Consequently, panel work is no longer possible in Wädenswil. Eight to twelve tasters plus the head of testing for hours in the same room, slurping and coughing actively, which is common in olive oil panels, does not work anymore. The service contractually guaranteed to the participants in the competition was no longer able to be provided and the Olive Oil Award Zurich was at risk of a massive loss of revenue due to the possibility of having to repay the participation fees.
The question of scientific excellence: When the kitchen becomes an ISO and SAS accredited sensory laboratory
Damage was mitigated by the Zurich Olive Oil Award in the measure of continuing olive oil testing in a decentralised manner at the various locations of the individual testers. In response to my investigation, the panel leader and organiser of the Olive Oil Award Zurich, Annette Bongartz, wrote that a suspension and thus a refund of the testing fees to the participants was "in fact at no time" an option. It is not clear from the official communication how far the participants were informed about this substantial change in the testing procedure. The organisers merely write on their website: "[...] The first evaluation, which was already carried out at the end of February / beginning of March 2020, could still take place within the usual framework, i.e. "in situ" on site in the sensory lab of the ZHAW. The second evaluation - after the "lockdown" of the ZHAW laboratories - was then carried out "virtually". For this purpose, the olive oils to be assessed were sent by parcel post to the members of the panel. In order to provide the tasters with the olive oil samples, the panel management had to open the original samples accordingly, split them and fill them into a third-party container, which they could then send to the tasters by parcel post. Arriving at the various destinations spread across the country, the tasters had to analyse the samples by sensory analysis. Alone. Possibly, however, they also did it in twos or threes. But probably all of them in different environments. Someone may have tasted the samples in the living room. Someone else might have done in the office. And another probably at the kitchen table or perhaps in the cellar room. It's possible that the tasters tasted the samples at different times of the day. Perhaps in different physical and mental conditions. And yet, according to the ZHAW on its website, the "home workplaces" are integrated into the accreditation 17025 of the ZHAW sensory testing laboratory. My kitchen an ISO-accredited sensory laboratory? That cannot and must not be the aim of the Zurich University of Applied Sciences.
An unconventional test methodology
The head of the Swiss Olive Oil Panel, Annette Bongartz, relies on ISO and SAS accreditations and believes that this justifies that her Olive Oil Award olive oil tests are scientific. When I asked her whether the practice of home testing is compatible with her quality claim of scientificity, she wrote to me: "We always work as a panel and as described above we have been successfully demonstrating the validity of the results from sample shipments to our accreditation body for years. This procedure is not a short-term reorientation of the way we work in our panel, but a long-established procedure within our accredited testing laboratory. These home testing stations, Bongartz continues, are by the way audited by the Swiss Accreditation Service SAS.
My question how the very unconventional test procedure is compatible with the scientific claims of the ZHAW remains unanswered. Or, to further clarify the problem: How does the Olive Oil Award Zurich guarantee a constant ambient temperature of the samples during their dispatch by regular mail? According to the International Olive Council (COI), records of the controlled temperature of the samples must always be able to be kept without interruption. Are the organisers of the Olive Oil Award Zurich able to do this? Or, can they physically supervise their testers while they perform the sensory analysis at their respective home testing stations, far from the sensory laboratory in Wädenswil? What about the technical facilities provided for in the COI regulations? Are all home testing stations equipped with tasting glasses and heating devices in accordance with COI/T.20/Doc. No 5/Rev. 1/2007? After all, such heating devices are not cheap, given the fact that they cost around 400 euros in the 4-version. And those who are used to working with official tasting glasses also know how difficult it is to clean them. A rinse with distilled water is actually prescribed by the COI. And it is unlikely if all home inspectors had such standardised tasting glasses, that every household would have the same dishwasher and the same detergent. Drying off with a fresh and nicely scented kitchen towel is not in it. It is at least inconceivable that the used oily glasses would be taken by the home inspectors to the actual location of the sensory lab for cleaning after each performance - particularly not now, in the days of Corona.
Many other questions arise, however. For example, what about the environmental conditions at the home-based test stations? Are there color-neutral walls, odor-free surfaces, adequate lighting, room ventilation? Is one free from disturbances there? Can I, as an organisation of a competition that promotes its credibility with independence, neutrality and professionalism, guarantee all these essential conditions and requirements?
«One must not know everything.»
- Horaz, Roman satirist
Well, if I wanted to be a member of the jury of the Olive Oil Award Zurich, I would have to move. I, for my part, have no such room in my home which could meet these admittedly very high standards. However, the panel members of the Swiss Olive Oil Panel might possibly do. I am not happy with Ms. Bongartz' first answer and therefore confront her with the above questions, which for me are quite elementary, and in view of the facts that have come to my attention so far, I also ask her to admit that the testing procedure of the Olive Oil Award Zurich 2020 is not in accordance with the COI guideline COI/T.28/Doc. No 1/Rev. 5/2019. In response, Bongartz writes: «We do not have to admit anything here. The home testing sites of our assessors are defined within our accredited testing laboratory and are regularly checked. The guidelines are clear and implemented accordingly. You don't have to "worry" at all ... - and the service we provide is absolutely reasonable in relation to the financial expenses of our participants.» The manner in which they acted was integer and serious. The data quality has been validated, said Bongartz in her reply to my further insistence.
She misses to put her cards on the table, admitting that the testing procedure at this year's Olive Oil Award Zurich was anything but compliant with the rules. «One must not know everything», the Roman satirist Horace was wont to say. Of course, specifications can be clear and implemented accordingly by the home testers. But they are not the specifications made by the COI for conducting sensory analyses of olive oils within a standardised scientific framework, but rather arbitrary instructions from the Swiss Olive Oil Panel of the ZHAW. If all home testers were equipped with the appropriate technical facilities, the panel leader would be very happy to confirm this in order to dispel any remaining doubts.
The results of the Olive Oil Award 2020 are supposed to be a reflection of the Swiss market
«The Olive Oil Award aims to increase consumer confidence in the product "Extra Virgin Olive Oil" and to promote transparency in the olive oil market. The project offers a comprehensive overview of the quality of the current olive oil supply on the Swiss and EU markets», the Olive Oil Award Zurich writes on its website. According to this, all discussions about the quality of olive oil in the Swiss retail trade would be invalid. To conclude, out of 109 oils submitted to the Olive Oil Award Zurich, 109 received the rating "Extra Virgin". A quota of 100 %. However, this figure contrasts with another one. One from a self-financed independent study. The number 21. Of 183 olive oils tested completely in accordance with the latest version of Regulation (EEC) No. 2568/91, in the comprehensive study by IOF - International Olive Foundation, only 21 % meet the requirements for "Extra Virgin". Just about a quarter of the samples were classified as lampante olive oil, slightly more than half could be assigned to the second quality class "Virgin". But, there is another essential difference to the work of the Zurich University of Applied Sciences. While the ZHAW received olive oils from olive oil producers, bottlers and traders for the Olive Oil Award 2020 and was even paid for it, IOF had bought the samples for its study with its own funds from the retailers' respective shops.
«Basically, the experience of recent years is that applicants do not necessarily submit products for competition which they consider to be already "questionable".»
- Annette Bongartz, panel leader Olive Oil Award Zurich
Migros and Coop should be relieved about the good news from the Grüental Castle in Wädenswil. Their oils all scored "Extra Virgin" at the Olive Oil Award Zurich. Seven products were even awarded gold, silver or bronze. The image has been rehabilitated after the olive oils marketed by the two major distributors did not perform very well in the IOF study. When the results of the IOF study were published, the Migros Cooperative Association told CH Media that the test results were not comprehensible to Migros. Migros makes very strict demands on its extra-virgin olive oils according to Migros.
New crop, new luck. Olive oil brands that did not perform well in the recent IOF study are recognised as Extra Virgin by the Olive Oil Award Zurich. It is clear that it is difficult to compare the individual results with each other, since on the one hand other production batches and possibly even olive oils from different harvesting campaigns were most likely tested, and on the other hand only "fresh" bottled oils were assessed in the Olive Oil Award Zurich 2020. However, on balance, this should not change IOF's study conclusion that extra virgin olive oils are hardly to be found in the Swiss retail trade. Annette Bongartz confirms this indirectly by answering the question of the possible risk that there could be entrants who, although they are successful with one batch in the Olive Oil Award Zurich, send another batch - possibly of inferior quality - for sale to food retailers: «Basically, the experience of recent years is that applicants do not necessarily submit products for competition which they consider to be already"questionable". That is why the quota of "defective oils" was and is generally very low, not only in our competition. Incidentally, the prize won in each case is only valid for the submitted LOT. This is explicitly pointed out in our communication to our participants. It goes without saying that there is a risk that products with the same label but a different LOT will subsequently be marketed using one of the prizes won. This would be a violation against our regulations. Effective control, however, is hardly possible for us.»
Bongartz does not dispute my conclusion that the Olive Oil Award Zurich project is not, as claimed by the organisers, a comprehensive overview of the quality of the current olive oil supply on the Swiss and EU markets, despite claims to the contrary on the ZHAW website. «The Olive Oil Award Zurich makes no claim to representativeness, as the participants send in their oils of their own free will.» And further: «The OOA project has now been running for almost 18 years, and our findings regarding the quality of olive oil on the Swiss market - and beyond - date from this period. The process of quality development is not one that can be published "just like that" and certainly cannot be broken down to a single year - it is a process that takes years. To this end, we are in contact with our participants (producers, traders, etc.) and provide them with the correspondingly well-founded information about their oils. By the way, this is not only done within the framework of the competition, but also regularly as part of our services, independent from the OOA.» In addition, the competition, according to Bongartz, offers consumers a transparent presentation of the diversity of the sensory qualities of olive oil.
But how can consumers be sure that what they buy on the basis of the publication of the results, for example from Migros, Coop or an online retailer, actually corresponds to what was advertised? Checking whether producers comply with the regulations of the Olive Oil Award Zurich when marketing their oils is hardly possible for the Wädenswil-based organisation, and it would be absurd to demand this from it. Here one may address an appeal to the participants' personal responsibility. However, in the interests of sincere and transparent communication, it would be mandatory, when publishing the competition results, to note the corresponding lot and best before date under the participant and winner olive oils shown. Furthermore, it must also be noted that the regulations of the Olive Oil Award Zurich 2020 do not specify the scope to which the applicants may advertise and market their olive oils with regard to the use of the word/picture trademarks of the Olive Oil Award Zurich after the publication of the results or the awarding of the prize. The panel leader does not respond to my renewed question as to where I can find the communication mentioned by Ms. Bongartz, which was intended to regulate this marketing issue. Is it wrong to conclude from this that this propagated document does not even exist and that, as a result, the winning companies can stick the awards on any bottling without committing any wrongdoing?
«As Hölderlin freely put it "with relentlessness, necessity accomplishes on a great day what hardly centuries succeed in doing", the head of the Swiss Olive Oil Panel defies the laws in force and introduces just one sub-category for oils of "questionable qualities" within the first quality class of virgin olive oil.»
- Silvan Brun, evoo.expert
109 oils in competition. 109 buying recommendations?
By legal definition, an extra virgin olive oil is of the highest quality and can therefore be described as a very good olive oil, even if the legislator does not write the latter in this wording. An extra virgin olive oil fulfils the chemical-physiological requirements relevant for this quality class and, in addition, does not have any sensory defects, is at least somewhat fruity and ideally even pungent and, depending on the type of olive, also bitter. Within this first quality class there are of course qualitative differences. Some products are of outstanding quality. But there are no products of inferior quality within this quality class. They are at least always very good. Good products may still be labelled "virgin olive oil" (without extra) and therefore belong to the second quality class. Such olive oils are marketable. In some countries this second quality class is even well developed, for example in Spain. As is well known, not everything can always be of the best quality, we are perfectly aware of that. Otherwise the Coop Cooperative would have had to switch the entire range from Asparagus over olive oil to Yuzu to "fine food" long ago. But so far it has not done so.
The Olive Oil Award Zurich judged all the olive oils submitted as "Extra Virgin", i.e. quasi as "Fine Food" products. However, Annette Bongartz does not want to recommend buying all 109 oils entered in the competition. «A purchase recommendation does not depend exclusively on our evaluation, but also and in particular on the expectations and needs of the consumers», she writes. Bongartz explains that although all the oils have been confirmed as "Extra Virgin", a very precise and meaningful differentiation of the various sensory characteristics and thus also of the quality is achieved through the harmony evaluation. «Hence, there are also oils with a harmony of less than 5.0 which show a very low, not to say "questionable" quality», panel leader Bongartz continues.
Extra virgin olive oils with a harmony score of 5.0 or less can have a questionable quality? I would like to have this explained to me in greater detail. «Of course there are qualitative differences within the extra virgin category. We all know that. The "declassification" is a process based on the "panel test" (according to EU directive). If the panel test shows that an oil is "extra virgin", then this must first of all be accepted. However, the quality index of the "harmony evaluation", as defined in our methodology, is a way to differentiate the different sensory qualities of olive oils within the "extra virgin" category», explains Annette Bongartz. Let's keep it that way, according to the Olive Oil Award Zurich, there are extra virgin olive oils of questionable quality which can be identified by the lower harmony rating. As Hölderlin freely put it «with relentlessness, necessity accomplishes on a great day what hardly centuries succeed in doing», the head of the Swiss Olive Oil Panel defies the laws in force and introduces just one sub-category for oils of "questionable quality" within the first quality category of extra virgin olive oil. If four or five out of eight tasters find the oil to be tested to have a slight or even a severe defect, the panel leader must understand that something may be wrong with this sample. Consequently, it should be tested again. It then seems much more likely that some of the tasters have an insufficient test performance rather than that the test object is without defects, i.e. extra virgin. Hence, in my opinion, it is a bad habit to take the path of least resistance and classify the oil with the questionable quality as "Extra Virgin" despite clear indications from the panel. But of course, this path of least resistance is the preferred one, especially for market participants who are not so concerned with quality, but rather with quantity, which is ultimately always at the expense of the consumer. They pay too much money for oil which is not good enough. A so-called extra virgin olive oil with "questionable" quality and a harmony value according to Bonartz et. al of maximum 5.0 points.
«If we give up this diversity just because olive oil competitions demand harmonious oils and therefore sometimes even large food retail chains explicitly ask for high harmony values, we risk losing great cultural and culinary values.»
- Silvan Brun, evoo.expert
According to the definition of the Swiss Olive Oil Panel, extra virgin olive oil with a harmony value of 4.5 to 5.0 points is "just about ok" (see graphic interpretation / definition of the "harmony value (Figure: ZHAW)). What does "just ok" mean? If it is an olive oil with fruitiness combined with the absence of defects, the oil is to be classified as "Extra Virgin", which by legal definition corresponds to the first quality class and must therefore be considered a very good olive oil. To be or not to be; this is the question here (Hamlet of Shakespeare). Extra virgin or not?
Harmony parameter is subject to arbitrary interpretation
The harmony parameter has a great potential for distortion. Under this term, Annette Bongartz not only cares for the balance between the three positive attributes provided for by the legislator, "fruitiness", "bitterness" and "pungency", but also and especially for the purity and complexity of the aroma (all of which are attributes not mentioned by the legislator). Is it possible to combine three sensory parameters, which are in principle completely different, in a single descriptor (the harmony parameter) and to make a statement that ultimately qualifies the olive oil? No, this is simply not possible (read the monograph on the harmony parameter here). This is an impossible task for the tasters anyway. Or, to put it again with the description of Dr. Gertz' dubious NIR procedure read earlier in this blog: From data to be assumed, results are interpreted which the system (here panel) actually cannot provide.
If we look at the above two illustrations, it is clear that the descriptor "harmony" does not make it possible to represent the sum of positive attributes quantitatively and qualitatively and to be able to conclude whether olive oil A or olive oil B is better. Both are probably very good. Due to their different characteristics, however, they will have different scopes of use in the kitchen. Nevertheless, the Olive Oil Award Zurich is likely to evaluate Olive Oil B with a higher harmony, which is why it would consequently perform better and have a better chance of winning an award. This will lead to an unfavourable influence on the olive oil market. Not only harmonious oils should be in demand, but first and foremost authentic oils, those that reflect the characteristics of the respective varieties, stand for terroir and their making. If we give up this diversity just because olive oil competitions demand harmonious oils and therefore sometimes even large food retail chains explicitly ask for high harmony values for their olive oils, we are in danger of losing great cultural and culinary values. Olive oil competitions forget the most important thing: they evaluate a condiment. And this should never be tested only outside of its field of application if you want to make a statement about whether this condiment is a good one or not. Yet this is how olive oil competitions and especially the Swiss and the German Olive Oil Panel have been breeding uniformity instead of diversity for the inexperienced consumer. It is therefore not wrong if the Swiss Olive Oil Panel is also regarded as a kind of culinary guardian. We must not allow this to continue. We must give the consumer the responsibility of being able to decide for himself which olive oil he likes to choose for which kind of dish.
The main purpose of olive oil panels is to ensure that the oils offered for sale are of the quality advertised on the label. That is the most important thing. That is the core task of olive oil panels. Nothing more. The rest is left to the market and consequently to the consumers.