# Rafał Rybnik

---

# Kleszcz, miliarder i drabina Rube Goldberga
_2026-05-21_

# Kleszcz, miliarder i drabina Rube Goldberga

### Dlaczego prawdziwe afery są krótsze, brzydsze i nudniejsze niż konspiracyjne filmiki

![[ChatGPT Image 21 maj 2026, 01_19_24.png|*Esej o rozpoznawaniu fałszywej złożoności — narracji, które imitują analizę systemową, a są literacką maszyną przyczynową. Kleszcz, Gates i alfa-gal to wygodny przykład: każde ogniwo da się sprawdzić w bazie danych, a mimo to ich suma jest fałszywa.*|450]] 

## Punkt wyjścia: kleszcze, Gates i drabina, której nie ma

W mediach społecznościowych krąży narracja, którą można streścić jednym tchem:

> *Farmerzy znajdują kleszcze Lone Star. Te kleszcze poruszają się trzy razy szybciej od innych gatunków i przenoszą alfa-gal — substancję wywołującą  długotrwałą alergię na czerwone mięso. W 2009 roku było kilkadziesiąt przypadków, dziś jest ich 450 000. Bill Gates finansował Oxitec, firmę modyfikującą genetycznie kleszcze. Gates inwestuje w Impossible Foods. CIA odtajniła dokumenty o uzbrajaniu kleszczy przeciwko Kubie. Plum Island było tajnym laboratorium broni biologicznej, z którego wyciekła borelioza. Wniosek: alfa-gal to sposób Gatesa na wymuszenie popytu na zamienniki mięsa.*

Każdy element tej narracji zawiera ziarno prawdy. Ale to nie jest łańcuch przyczynowo-skutkowy — to kolaż faktów spojonych domysłem.

Co **rzeczywiście** jest prawdą:

- Kleszcz Lone Star (*Amblyomma americanum*) **rzeczywiście** porusza się ok. 3 razy szybciej niż kleszcz czarnonogi, co potwierdzają AVMA i URI TickEncounter.
- **Rzeczywiście** zwiększa się liczba przypadków AGS. CDC raportuje ponad 110 000 udokumentowanych przypadków w latach 2010–2022 i szacuje, że po uwzględnieniu osób niezdiagnozowanych liczba ta może sięgać 450 000 [@cdcmmwr2023a; @cdcmmwr2023b].
- Fundacja Gatesa **rzeczywiście** sfinansowała Oxitec: 1,28 mln USD w 2021 roku i 4,8 mln USD w 2023 roku. Finansowanie dotyczyło jednak *Rhipicephalus microplus* (kleszcza bydlęcego), a nie *Amblyomma americanum*. To zupełnie inny gatunek, a same badania laboratoryjne prowadzono w Wielkiej Brytanii. W USA kleszcz bydlęcy to gatunek praktycznie wyeliminowany od 1943 roku.
- Artykuł *Beneficial Bloodsucking* [@crutchfield2025] **rzeczywiście** istnieje. To filozoficzny eksperyment myślowy, nie program polityczny.
- S. Matthew Liao (NYU) **rzeczywiście** opublikował tekst *Human Engineering and Climate Change* [-@liao2012], w którym pojawia się akapit o kleszczu Lone Star. To tekst akademicki, a nie wezwanie do działania.
- Plum Island **rzeczywiście** było obiektem biologicznej broni ofensywnej od 1952 roku (początkowo pod U.S. Army Chemical Corps, od 1954 pod USDA), aż do zakończenia programu w 1969 roku przez prezydenta Nixona [@carroll2004]. Hipoteza, że wyspa jest źródłem boreliozy, została jednak **odrzucona**: DNA *Borrelia burgdorferi* znaleziono u Ötziego, człowieka sprzed 5300 lat [@keller2012], oraz w okazach myszy z Massachusetts z 1894 roku.
- Operacja Mongoose **rzeczywiście** obejmowała rozważania i działania sabotażowe wobec Kuby== [@churchcommittee1975mongoose]. Raport Komisji Churcha potwierdza między innymi chemiczne obezwładnianie pracowników cukrowni; konkretne twierdzenia o skażeniu transportów cukru do ZSRR wymagają ostrożności. W żadnym dokumencie nie ma natomiast mowy o ofensywnym wykorzystaniu kleszczy.

Każdy z tych faktów da się zweryfikować. Łączącego je łańcucha przyczynowego — nie.

I to jest ważne rozróżnienie, bo intuicja podpowiada coś przeciwnego: jeśli każde z dziewięciu twierdzeń da się sprawdzić w bazie danych, dlaczego ich suma miałaby być fałszywa? Odpowiedź leży nie w treści, lecz w konstrukcji. Konspiracja nie jest tu listą rzeczy, które się wydarzyły. Jest **maszyną narracyjną o określonym układzie**, w której każde ogniwo musi zadziałać w ściśle określonym czasie i kierunku, żeby końcowy wniosek się domknął. A o takich maszynach wiemy z innych dziedzin — inżynierii, teorii niezawodności, ekonomii informacji — że są kruche w sposób zwykle niewidoczny na pierwszy rzut oka, ale wychodzący na jaw podczas próby implementacji. Zanim jednak przejdę do omówienia przyczyn i skutków tej kruchości, warto zobaczyć, jak wyglądają afery, które **rzeczywiście** miały miejsce.

## Jak wyglądają prawdziwe spiski

Sceptyczny obserwator może odpowiedzieć: *„Skoro mówisz, że konspiracje to konstrukcje fabularne, prawdziwe spiski nie powinny się zdarzać. A przecież zdarzały się poważne, ukryte i skoordynowane afery”.*

To prawda. I właśnie dlatego warto przyjrzeć się im z bliska, bo strukturalnie różnią się od scenariusza z filmiku o kleszczach.

### 1. Sugar Research Foundation (1965–1967)

Sugar Research Foundation zapłaciła trzem badaczom z Harvardu — Markowi Hegstedowi, Robertowi McGandy’emu i Fredowi Stare’owi — za napisanie przeglądu literatury, który minimalizował rolę sacharozy w rozwoju choroby wieńcowej i przesuwał odpowiedzialność na tłuszcze nasycone. Tekst ukazał się w 1967 roku w *NEJM*, bez ujawnienia źródła finansowania. Pięćdziesiąt lat później Cristin Kearns wraz z zespołem z UCSF odnalazła w archiwach Harvardu pełną dokumentację zleceń, faktur i wewnętrznej korespondencji SRF [-@kearns2016].

**Mechanizm:** zapłacić konkretnym autorom za konkretny tekst. Jeden krok między pieniędzmi a publikacją.

### 2. Exxon Knew (1977–1989)

Wewnętrzni naukowcy Exxonu — zwłaszcza James Black w prezentacji dla zarządu z 1977 roku oraz autorzy raportu „Greenhouse Effect” z 1982 roku — modelowali tempo ocieplenia z dokładnością, którą trzy dekady później potwierdziły szacunki IPCC. Mimo to korporacja przez dwadzieścia lat finansowała Global Climate Coalition, American Petroleum Institute i sieć ośrodków analitycznych publicznie podważających te same wnioski. Wewnętrzne dokumenty trafiły do prasy w 2015 roku (*InsideClimate News*, *LA Times*). Ilościową analizę dokładności modeli Exxonu wobec późniejszej rzeczywistości przedstawili Supran, Rahmstorf i Oreskes [-@supran2023].

**Mechanizm:** wewnątrz mówimy A, na zewnątrz finansujemy mówienie B. Dwa kroki, ale oba w obrębie jednej organizacji i jej kontrahentów.

### 3. VW Dieselgate (2009–2015)

W oprogramowaniu sterującym silnikami Diesla w 11 milionach samochodów Volkswagen Group umieszczono mechanizm oszukujący testy emisji („defeat device”) — fragment kodu wykrywający warunki badania (statyczne obciążenie, brak ruchu kierownicy) i przełączający silnik w tryb zgodny z normą tylko podczas kontroli. W normalnej jeździe emisja NO_x przekraczała normę nawet 40-krotnie. Oszustwo wyszło na jaw, kiedy International Council on Clean Transportation, chcąc użyć VW jako pozytywnego przykładu z Europy, zleciła West Virginia University niezależne testy drogowe. Wyniki nie zgadzały się z laboratoryjnymi i tę sprzeczność trzeba było wyjaśnić.

**Mechanizm:** jedna funkcja w jednym fragmencie oprogramowania w jednej firmie. Brak efektów wtórnych poza prostym wykryciem.

### 4. Boeing 737 MAX / MCAS (2017–2019)

Podczas certyfikacji 737 MAX, typu wprowadzonego do służby w 2017 roku, Boeing nie ujawnił pilotom ani FAA istnienia systemu MCAS, który mógł samodzielnie pchać dziób samolotu w dół na podstawie odczytu z jednego, niezdublowanego czujnika kąta natarcia. Po dwóch katastrofach — Lion Air JT610 w 2018 roku i Ethiopian ET302 w 2019 roku, łącznie 346 ofiar — wyszło na jaw, że FAA delegowała znaczną część certyfikacji samemu Boeingowi w ramach programu Organization Designation Authorization. Boeing umniejszał znaczenie MCAS i nie przekazał kluczowych informacji o systemie ani FAA Aircraft Evaluation Group, ani pilotom — częściowo po to, by uniknąć kosztownych szkoleń symulatorowych dla pilotów przeszkolonych już na starszej wersji 737 [@boeing2020house]. W 2021 roku DOJ zawarł z firmą ugodę o odroczeniu ścigania (DPA); status prawny sprawy zmieniał się w kolejnych latach.

**Mechanizm:** ukryć funkcję oprogramowania w certyfikacji i przejąć część własnego nadzoru regulacyjnego. Dwa kroki, oba w obrębie relacji Boeing–FAA. Klasyczne przechwycenie regulatora, a nie kaskada efektów wtórnych.

### 5. Theranos (2003–2018)

Elizabeth Holmes zbudowała wycenę 9 mld USD wokół technologii oznaczania parametrów krwi z kropli z palca — technologii, która nigdy nie zadziałała. W praktyce Theranos używał komercyjnych analizatorów Siemensa, fałszując źródło wyników. Demaskacja — artykuły Johna Carreyrou w *Wall Street Journal* w 2015 [@carreyrou2018], wyrok skazujący Holmes w 2022.

**Mechanizm:** powiedzieć, że ma się technologię, której się nie ma. Jeden krok. Skala zaufania była ogromna, mechanika oszustwa — banalna.

### 6. COINTELPRO (1956–1971)

Program kontrwywiadowczy FBI Hoovera — infiltracja, dezinformacja, prowokacje policyjne, kampanie zniesławiające wymierzone w ruch praw obywatelskich, Black Panthers, organizacje antywojenne. Najsłynniejsze elementy to anonimowy list wysłany Martinowi Lutherowi Kingowi z sugestią samobójstwa (1964) i zabójstwo Freda Hamptona w policyjnym nalocie (1969). Ujawnione przez fizyczne włamanie aktywistów do biura FBI w Media w marcu 1971; pełną skalę pokazała Komisja Churcha [@churchcommittee1976bookIII].

**Mechanizm:** jedna instytucja, jasny cel polityczny, brutalne narzędzia operacyjne. Nie ma drabiny — jest cel i są ofiary.

### 7. Tuskegee (1932–1972)

US Public Health Service przez 40 lat obserwowała nieleczony syfilis u 399 czarnych mężczyzn z Alabamy [@jones1993]. Po wprowadzeniu penicyliny jako standardu leczenia w 1947 roku badanych nie poinformowano i nie leczono; w niektórych przypadkach aktywnie odciągano ich od terapii. Ujawnione w 1972 roku przez sygnalistę Petera Buxtuna [zob. też @brandt1978].

**Mechanizm:** *nie podawać leku, którego wymaga standard.* Bezczynność jako działanie. Zero ogniw poza pojedynczą decyzją kontynuowaną przez cztery dekady.

### Co je łączy strukturalnie

Wszystkie siedem przypadków ma kilka cech, które są **dokładnym przeciwieństwem** scenariusza z filmiku o kleszczach:

|Cecha                          |Realne afery                                                    |Konspiracja z filmiku                          |
|-------------------------------|----------------------------------------------------------------|-----------------------------------------------|
|Liczba ogniw przyczynowych     |1–2                                                             |6+                                             |
|Liczba zaangażowanych podmiotów|1 organizacja (+ jej wykonawcy)                                 |wiele niezależnych podmiotów                   |
|Mechanizm                      |bezpośredni (kłamstwo, sabotaż, bezczynność)                    |kaskadowy (rynki, ekologia, biologia, polityka)|
|Cel                            |banalny: zysk lub kontrola                                      |wieloetapowy plan globalny                     |
|Ujawnienie                     |postępowania sądowe / sygnalista / fizyczna kradzież dokumentów |brak — system poznawczo zamknięty             |
|Czas od działania do ujawnienia|5–50 lat                                                        |bez terminu                                    |

**Wniosek:** realne afery są zwykle prostsze, krótsze i głupsze, niż sugerują narracje konspiracyjne. Mają jednego wyraźnego sprawcę, jeden mechanizm i jeden cel. Ujawniają je nudne, zinstytucjonalizowane procedury — procesy cywilne, FOIA, inspektorzy generalni — a nie genialni śledczy rozszyfrowujący kaskady zdarzeń.

## Drabina Rube Goldberga — eksperyment myślowy

*Rube Goldberg (1883–1970) był amerykańskim rysownikiem i inżynierem, którego cykl rysunków przedstawiał absurdalnie zawiłe maszyny wykonujące trywialne czynności, na przykład dwudziestoogniwowy łańcuch fizycznych zdarzeń kończący się wytarciem ust serwetką. Termin „maszyna Rube Goldberga" wszedł do języka angielskiego w 1931 roku jako synonim konstrukcji, która osiąga prosty cel maksymalnie zawiłą drogą. Brytyjski odpowiednik to „Heath Robinson contraption". Używam go tutaj dlatego, że konspiracyjne narracje typu „pestycyd → miedź → satelita" mają dokładnie tę strukturę: trywialny cel i maksymalnie zawiły łańcuch deterministycznych ogniw. Dosłownie należałoby mówić o „maszynie Goldberga" — i tak ją będę nazywał, kiedy chodzi o sam mechanizm. Mówię „drabina" wtedy, gdy chcę podkreślić jednowymiarowy, sekwencyjny układ zależności: każde ogniwo musi domknąć następne, jak kolejny szczebel, bo inaczej cała konstrukcja się zawala.*

Wyobraź sobie taki plan:

> *Sprzedaję rolnikom w regionie X wadliwą partię pestycydu. Lokalny niedobór pestycydu A zmusza ich do wykupywania pestycydu B, który firma normalnie dostarcza do regionu Y. W Y rosną ceny żywności, więc górnicy w lokalnej kopalni miedzi są niedożywieni. Wydajność spada o 5%. Globalne ceny miedzi zaczynają się wahać. Producent komponentów satelitarnych przesuwa harmonogram. Wrogie państwo opóźnia start swojego satelity komunikacyjnego o pół roku. W tym czasie mój system globalnej komunikacji wchodzi na rynek jako pierwszy i staje się standardem.*

To brzmi jak fabuła książki Dana Browna albo kolejnego sezonu *Mr. Robot*. **I właśnie dlatego nie zadziała.**

Można zarzucić, że ten konkretny przykład jest karykaturalny — pestycyd → miedź → satelita brzmi jak fabuła klasy B. Słuszne. Można sobie wyobrazić wersję bardziej życiową: przejęcie startupu konkurenta, presja regulacyjna w jednym kraju, kampania medialna, lobby branżowe, zmiana standardu certyfikacyjnego, wymuszony exit konkurencji. Mniej egzotyki, wciąż sześć ogniw. Sedno nie leży w stopniu absurdu pojedynczego ogniwa, tylko w tym, ile ich trzeba ułożyć w odpowiedniej kolejności — i każdy taki łańcuch, niezależnie od tego, jak realistycznie brzmi, podlega tej samej matematyce kompozycji, którą za chwilę przedstawię.

### Dlaczego nie działa

#### Matematyka kompozycji

Załóżmy, że każde z 7 ogniw ma 60% szans powodzenia. To naprawdę optymistyczne założenie, bo efekty drugiego rzędu na rynkach surowców rzadko działają w przewidywanym kierunku. Całkowite prawdopodobieństwo wynosi wtedy:

$$0{,}6^{7} \approx 2{,}8\%$$

To oczywiście model zabawkowy, który zakłada, że szczeble drabiny są niezależne, każdy ma to samo prawdopodobieństwo, a planista nie koryguje kursu w trakcie. Realne systemy są bardziej elastyczne. Ale intuicja zostaje. Każdy dodatkowy warunek, który *musi* zajść, istotnie obniża wiarygodność całego planu.

Przy 50% na ogniwo $0{,}5^{7} \approx 0{,}78\%$ — czyli ponad 3,5-krotnie mniej niż dla ciągu zdarzeń "sześćdziesięcioprocentowych". Nawet przy budżecie rzędu majątku Gatesa zwrot z takiego planu jest katastrofalny w porównaniu z narzędziami bezpośrednimi: przejęciem konkurenta, zatrudnieniem lobbysty albo sfinansowaniem korzystnego badania naukowego.

#### Zapasowe ścieżki w systemach

Ten scenariusz zakłada, że niedobór pestycydu A prowadzi do wykupywania pestycydu B, a to do wzrostu cen w innym kraju. Globalne łańcuchy dostaw mają jednak **więcej zapasowych ścieżek**, niż intuicyjnie się wydaje — właśnie po to, żeby nie załamywać się pod wpływem punktowych szoków. Lokalny niedobór zostanie uzupełniony importem z kilku innych źródeł, zanim ktokolwiek zauważy oczekiwany efekt. Arbitraż rynkowy zjada zaplanowane skutki uboczne.

#### Problem rozproszonej wiedzy (Hayek)

Probabilistyka i zapasowe ścieżki w systemach to dopiero dwie warstwy odpowiedzi. Trzecia, mocniejsza, dotyczy tego, czy planista-spiskowiec w ogóle może mieć dane, których potrzebuje. Żeby drabina z naszego przykładu zadziałała, ktoś na górze musiałby znać równocześnie:

- elastyczność popytu na pestycyd B w regionie X w bieżącym sezonie,
- dietę i krzywą wydajności górników w tej konkretnej kopalni,
- udział tej kopalni w globalnej produkcji miedzi,
- wpływ ceny miedzi na koszt komponentów elektronicznych przy danej mieszance technologii,
- wykaz materiałowy satelity wrogiego państwa,
- jego harmonogram, rezerwy strategiczne i polityczny próg tolerancji opóźnień.

Każda z tych danych jest albo niedostępna, albo obciążona szumem, albo zmienia się szybciej, niż trwa realizacja planu. I nie jest to deficyt, który da się załatać większym budżetem na wywiad albo lepszym modelem AI. To ograniczenie strukturalne, które Friedrich Hayek opisał w *The Use of Knowledge in Society* [-@hayek1945] — diagnoza w głównym nurcie ekonomii informacji pozostaje przyjmowana jako trafna[^Polanyi] [zob. m.in. @stiglitz1994; @boettkecoyne2015], a krytyki dotyczą zwykle implikacji politycznych, nie samej diagnozy.

[^Polanyi]: W literaturze ekonomicznej i menedżerskiej nazywa się to dziś *information dispersion* albo *local knowledge problem*; w polskim piśmiennictwie: *problem rozproszonej wiedzy*, *wiedza lokalna* lub *wiedza milcząca* — to ostatnie pojęcie częściowo nakłada się z koncepcją Michaela Polanyiego, brytyjskiego filozofa nauki o węgierskich korzeniach (brata socjologa Karla Polanyiego), który w *The Tacit Dimension* z 1966 roku wprowadził pojęcie *tacit knowledge*.

Argument Hayeka, w wersji operacyjnej, brzmi mniej więcej tak. Wiedza istotna dla decyzji ekonomicznych nie jest sumą faktów zgromadzonych centralnie. To *milcząca, lokalna, kontekstowa wiedza milionów uczestników* — który dostawca jest w stanie zwiększyć produkcję o 5%, czyj kierownik zmiany przejmuje obowiązki w czasie urlopu, kiedy konkretna fabryka przechodzi remont kapitalny. Ta wiedza nie istnieje w żadnym repozytorium, bo w dużej części nie istnieje *w formie jawnej*. Żyje w nawykach, decyzjach z ostatniego tygodnia i telefonach między dwiema osobami, które się znają. Rynek wykorzystuje ją pośrednio — uczestnicy podejmują decyzje na podstawie własnej wiedzy lokalnej, a wyniki tych decyzji odczytujemy ze zmian cen. Centralny planista — nawet z nieskończonym budżetem i siecią agentów wywiadu — tej wiedzy nie zbierze, bo nie istnieje ona w żadnej formie obserwowalnej z zewnątrz. Istnieje tylko w działaniu osób, które się nią posługują.

Hayek pisał to o socjalistycznej kalkulacji ekonomicznej. Ten sam argument działa także przeciwko prywatnemu planiście-spiskowcu, który próbuje zrealizować siedmioetapową kaskadę przez gospodarkę globalną. Drabina Rube Goldberga w realnym świecie podlega prawu Murphy'ego w wersji probabilistycznej — jeśli każdy ze szczebli *może* pójść źle z prawdopodobieństwem 1-p, to przy n szczeblach szansa, że *wszystkie* pójdą dobrze, spada wykładniczo. A takie planowanie zawodzi nie dlatego, że brak na nie pieniędzy albo mocy analitycznej. Zawodzi dlatego, że istotne dane nie istnieją w postaci możliwej do pozyskania.

### Nawet jeśli kiedyś zadziałała pozostaje niewidoczna

Załóżmy, że gdzieś w XX wieku ktoś naprawdę zrealizował siedmiostopniową drabinę i osiągnął zamierzony efekt. Czy bylibyśmy w stanie to wykryć?

Nie. Z dwóch powodów:

1. **Po fakcie taki łańcuch jest nieodróżnialny od koincydencji.** Każde z ogniw ma niezależne, bardziej prawdopodobne wyjaśnienie: cykle rynkowe, politykę, pogodę, zmianę technologiczną.
2. **Architektura poznawcza takich hipotez jest niefalsyfikowalna.** Każde obalone ogniwo można schować pod formułą: „ale inne plany mogły działać”. To czyni je nie tylko niedowodliwymi, ale też nieobalalnymi — czyli poznawczo zamkniętymi.

Z punktu widzenia historyka i naukowca **drabina Rube Goldberga nie istnieje, dopóki nie zostawi twardego śladu**: notatki, przelewu, kodu, świadka. A im więcej ogniw, tym mniejsza szansa, że choć jedno z nich zostawi taki ślad we właściwym miejscu.

## Co zrobiłby gracz z nieograniczonym kapitałem

Tu argument zwykle używany do *uzasadnienia* drabin obraca się przeciwko nim.

> *„Mając nieograniczony kapitał, mogę uruchomić sto planów i wystarczy, że jeden zadziała”.*

To matematycznie poprawne, ale prowadzi do **innej** strategii niż drabina: do **dywersyfikacji krótkich, prostych ataków**, a nie do budowania długich łańcuchów.

Powodów jest kilka:

- **Wariancja wyniku.** Pojedyncza drabina to gra „wszystko albo nic” przy bardzo niskim prawdopodobieństwie. Sto krótkich ataków daje rozkład z dodatnią wartością oczekiwaną.
- **Czas.** Drabina wymaga sekwencji rozciągniętej w czasie. Krótkie ataki działają równolegle.
- **Atrybucja.** Sto krótkich ataków rozprasza się w szumie. Jedna drabina zostawia rozpoznawalny wzór, jeśli ktokolwiek zacznie go szukać.
- **Korekta kursu.** Niezależne ataki pozwalają wycofać się z tych, które nie działają. Drabina, która załamuje się w połowie, traci wszystko, co już w nią zainwestowano.

I to właśnie obserwujemy historycznie. Microsoft lat 90. nie konstruował kaskady prowadzącej do dominacji przeglądarki — równolegle wiązał produkty, zawierał umowy z producentami sprzętu i prowadził wojnę z Sunem. Sacklerowie nie projektowali łańcucha „OxyContin → uzależnienie → fentanyl” — agresywnie marketingowali jeden lek, kalkulując, że efekt uzależnienia również zwiększy sprzedaż. Exxon nie planował łańcucha — przez dekady finansował dziesiątki niezależnych ośrodków analitycznych.

Realny skoncentrowany kapitał kupuje **prostotę i skalę**, nie wielowarstwową elegancję.

Tu warto wprowadzić jedno rozróżnienie, bo bez niego cały ten argument może wyglądać na obronę naiwnego optymizmu wobec dużych aktorów. Czym innym jest *wieloetapowy centralny plan* — drabina Goldberga, którą tu analizuję — a czym innym *systemowy efekt bez architekta*, czyli złożone skutki wytwarzane przez rynki, platformy i sieci interesów *bez* jednego planisty. Realne szkody na poziomie systemu — koncentracja kapitału, kruchość globalnych łańcuchów dostaw, dynamika algorytmów rekomendacyjnych spychających dyskurs w skrajności — zwykle wyłaniają się spontanicznie, a nie z czyjegoś projektu. Nikt ich nie zaplanował krok po kroku. Po prostu wyłaniają się z milionów drobnych decyzji podejmowanych przez aktorów, którzy starają się optymalizować pod lokalne warunki. Konspiracje z YouTube próbują udowodnić coś znacznie mocniejszego — istnienie ukrytego planisty, który *przewidział* i *zsynchronizował* biologię, epidemiologię, inwestycje, zachowania konsumentów i rynek zamienników mięsa. I to właśnie ich strukturalna słabość. Zakładają intencjonalnego projektanta tam, gdzie sama dynamika rynku, technologii i polityki wystarcza, by wyjaśnić obserwowane zjawiska.

## Dlaczego skomplikowane plany czyta się z zapartym tchem — i dlaczego to nie jest komplement

Antonowi Czechowowi przypisuje się zasadę, którą do dziś wykłada się na pierwszym semestrze kursów scenopisarskich: jeśli w pierwszym akcie na ścianie wisi strzelba, to w trzecim musi wystrzelić; w przeciwnym razie nie powinno jej tam być. Zasada Czechowa nie jest tylko poradą estetyczną. To definicja gatunku. Fikcja narracyjna działa, bo każdy element wprowadzony na początku ma swoje miejsce w łańcuchu przyczyn prowadzącym do finału. Autor pisze od końca — od wniosku do przesłanek — i wycina wszystko, co nie służy temu łańcuchowi.

Rzeczywistość ma odwrotną logikę. W rzeczywistości większość strzelb nigdy nie wystrzeli, większość predykcji nigdy się nie spełni, a większość ogniw łańcucha się nie domknie. Dlatego, że nie ma żadnego autora, który pisze historię od jej finału. A to, co w fikcji zwykle jest defektem — luźny detal, zbędna postać, scena bez konsekwencji dla głównej osi fabularnej — w świecie fizycznym i społecznym jest domyślnym stanem rzeczy.

Konspiracyjna narracja należy gatunkowo do fikcji, mimo że nosi kostium reportażu — niczym paradokumenty typu *Trudne sprawy*. Każda jej cegiełka spełnia funkcję uwiarygadniającą. Kleszcz Lone Star szybciej biega. Gates dotuje Oxitec. Na Plum Island było coś biologicznego. Ktoś napisał coś o konieczności jedzenia owadów. Wszystkie te elementy *muszą* prowadzić do tego samego finału, bo inaczej historia się nie domknie. Autor takiej narracji pracuje jak scenarzysta. Zaczyna od wybranej przez siebie konkluzji (np. Gates wymusza zamienniki mięsa) i dobiera fakty wstecz. Dlatego brzmi spójnie. I dlatego jest fałszywa — bo świat, w którym żyjemy, ma znacznie więcej luźnych detali niż domykających się historii.

Diagnoza wykracza poza literaturoznawstwo. Steven Pinker w *How the Mind Works* [-@pinker1997], a wcześniej Tooby i Cosmides [-@toobycosmides2001] argumentowali, że fikcja narracyjna jest u ludzi wyspecjalizowanym poznawczo narzędziem symulacji, które pozwala bez kosztów modelować sytuacje społeczne i przewidywać konsekwencje działań. Działa, bo ewolucyjnie opłacało się mieć aparat, który pozwala czytać intencje, przewidywać motywy i widzieć sprawców w ciągach zdarzeń. Teorie spiskowe wykorzystują dokładnie ten aparat. Oferują sprawcę (Gatesa), motyw (zysk z Impossible Foods) i łańcuch zdarzeń, który ma sens. Mózg wyewoluowany do wykrywania ukrytych zamiarów u członków plemienia nagradza spójną narrację poczuciem zrozumienia niezależnie od tego, czy spójność odpowiada faktom.

Eliezer Yudkowsky w *Harrym Potterze i metodach racjonalności* [-@yudkowsky2015] ujmuje to dosadnie: skomplikowane, wieloetapowe plany działają tylko w fikcji, bo tam wszystkie postacie pisze ten sam autor. W realnym świecie postacie mają własne intencje, własną niewiedzę, własne błędy oraz dni, w których budzik nie zadzwoni na czas.

Konspiracyjny filmik z żółtymi napisami na YouTube jest w tym ujęciu dobrze napisanym opowiadaniem w dokumentalnym kostiumie. Stąd jego siła emocjonalna i jednocześnie jego słabość poznawcza: działa, bo jest dobrze opowiedziany, nie dlatego, że jest prawdziwy. Być może najprostsza heurystyka pozwalająca na odsianie fikcji od faktów to zadać sobie pytanie: czy dałoby się z tego nakręcić sezon serialu? Jeśli tak — to prawdopodobnie mamy do czynienia z fabułą w dokumentalnym kostiumie, nie z aferą, którą ktoś rzeczywiście odkrył.

## Wnioski operacyjne

Hipoteza „ludzie z dużymi pieniędzmi prowadzą skomplikowane, wieloetapowe gry, których nie widzimy” jest sensowna jako *założenie wstępne*. Problem zaczyna się wtedy, gdy stosujemy ją do hipotez wymagających **drabiny przyczynowej**, a nie do hipotez wymagających **skoncentrowanej skali**. Z całej tej analizy wynikają trzy reguły robocze i jedna konstatacja na marginesie.

**1. Policz ogniwa.** Liczba ogniw przyczynowych to jeden z najlepszych sygnałów wiarygodności hipotezy spiskowej. Im więcej ogniw, tym mniejsze prawdopodobieństwo, że jest prawdziwa. Wszystkie siedem omówionych wyżej afer ma 1–2 ogniwa. Konspiracja "kleszczowa" ma ich sześć lub więcej. Reguła robocza powyżej trzech ogniw — podnosimy próg dowodowy. Nie wystarczy, że każdy fakt z osobna da się sprawdzić w bazie danych. Trzeba pokazać twardy ślad *przyczynowego połączenia* między ogniwami (np. notatkę, przelew, korespondencję, świadka), który łączy ogniwo A z ogniwem B w jeden zaplanowany ruch. Bez tego mamy listę faktów, nie łańcuch.

**2. Sprawdź najtańszy fakt.** Najtwardsze, najtańsze do zdobycia fakty weryfikujemy *zanim* zaczniemy oceniać strukturę. *Amblyomma americanum* (Lone Star) i *Rhipicephalus microplus* (kleszcz bydlęcy) to inny rodzaj, inny gospodarz i inny kontynent. Chronologia: pierwsze przypadki AGS opisane przez Comminsa i Plattsa-Millsa [-@commins2009] są wcześniejsze niż jakikolwiek grant Fundacji Gatesa dla Oxitec. Lokalizacja laboratorium Oxitec to Oxford w Wielkiej Brytanii, nie USA. Każdy z tych faktów obala konspirację bez potrzeby dyskutowania o intencjach Gatesa. Dyskusja o intencjach jest *najdroższą* drogą sprawdzania hipotezy i powinna być ostatnia, nie pierwsza.

**3. Oddziel mechanizm od archiwum.** Emocjonalna gratyfikacja narracji jest sygnałem alarmowym, nie potwierdzeniem. Jeśli historia „wszystko wyjaśnia” — od kleszczy przez Plum Island po Impossible Foods — to znaczy, że *została napisana tak, aby wszystko wyjaśniała*, a nie odkryta przez śledczych próbujących wyjaśnić konkretną rzecz. Tu warto rozróżnić dwie rzeczy — *mechanizm* afery i *archiwum* dookoła niej. Mechanizm realnej afery zwykle jest prosty — jedna decyzja, jeden fragment kodu, jedna umowa. Tuskegee to *nie podawać leku*. Boeing to *nie ujawnić systemu*. Exxon to *wewnątrz mówić A, na zewnątrz finansować B*. Ale archiwum dookoła — śledztwa, FOIA, ławy przysięgłych, raporty kongresowe — narasta przez dekady, bo proces odkrywania prostego mechanizmu jest długi i kosztowny. Konspiracyjny filmik daje się zrozumieć w pięć minut nie dlatego, że ma prosty mechanizm, tylko dlatego, że *nie ma nic do odkrycia* — wszystko jest już ułożone pod tezę. Jeśli zebranie dowodów nie boli, prawdopodobnie jeszcze nie dotknęliśmy faktów.

**Bonus. Krytyka kapitału nie potrzebuje drabin.** To nie jest reguła robocza, tylko obserwacja, którą warto mieć z tyłu głowy stosując trzy poprzednie. Skoncentrowany kapitał filantropijny realnie kształtuje programy badawcze. To poważny problem strukturalny, niezależny od kleszczy. Krytyka autorstwa Linsey McGoey [-@mcgoey2015], Ananda Giridharadasa [-@giridharadas2018] i innych badaczy filantrokapitału nie wymaga hipotezy wieloetapowej. Pokazuje coś prostszego i mocniejszego. Kiedy jeden aktor finansuje znaczną część globalnego budżetu zdrowia publicznego, *program badań się dostosowuje*. Bez żadnej kaskady. Mocniejszy argument przeciwko modelowi finansowania Fundacji Gatesa to ten, który nie potrzebuje żadnych drabin. Konspiracja kradnie energię argumentowi, który byłby zasadny.

Drabiny Rube Goldberga są atrakcyjne narracyjnie, bo wyjaśniają *wszystko*. Realne afery wyjaśniają *jedną rzecz*, ale za to solidnie popartą materiałem dowodowym. Pierwsze są fikcją literacką. Drugie są udokumentowanymi faktami.

## Źródła

::: {#refs}
:::

---

# How to market to smart people
_2024-08-14_

# How to Market to Smart People

### Strategies for Building Trust and Loyalty with Savvy Consumers

Marketing to smart people requires a different approach than traditional marketing strategies. Smart consumers are discerning, sceptical, and highly informed. Here's how you can tailor your approach to appeal to this audience effectively.

![[hero.jpg|*Image generated by DALL-E*|450]]

## 1. Start with a Product That Stands Out

It should go without saying, but it's crucial: your product must be exceptional. If your product is mediocre, marketing efforts will be wasted. Smart consumers expect products that are not only functional but also innovative, flexible, and deep. When designing features, don't just cover basic use-cases; anticipate the advanced scenarios that a smart person might envision. Flexibility is key. However, don't fall into the trap of trying to be everything to everyone. A well-focused, polished product will resonate more than one that tries to do too much.

![[fig-tesla.jpg|*TESLA Model S Folio + 2007 Concept Art Design Renderings von Holzhausen ([source](https://www.ebay.com/itm/296178970903))*]]

Tesla Model S exemplifies this approach. Tesla didn't just create an electric car; it created a high-performance, luxurious, and innovative vehicle that appealed to tech-savvy, environmentally conscious consumers. Its deep integration with software and constant over-the-air updates keep the product evolving, ensuring it continues to meet the advanced expectations of its users.

## 2. Emphasise Openness and Transparency

Smart people value transparency and autonomy. Make your product as open as possible — the ideal scenario being open-source. When you open your protocols and provide Software Development Kits (SDKs), you allow these users to engage with your product on a deeper level, tailoring it to their unique needs. Additionally, make your support system transparent. By openly addressing issues and showing how you resolve them, you build trust and credibility, which are critical for this audience.

![[fig-wordpress.png|*WordPress contributor page invites users to actively participate in the platform's development, exemplifying its commitment to openness and transparency through clear and accessible community involvement options. ([source](https://make.wordpress.org/))*]]

WordPress is a prime example of a product that emphasises openness and transparency. As an open-source platform, WordPress allows developers to access, modify, and contribute to its codebase, fostering a large and active community. Additionally, WordPress's transparent development process, including open discussions about upcoming features and changes, enables users to feel involved and informed. This openness has helped WordPress dominate the content management system market, particularly among smart, tech-savvy users who value the ability to customise and control their own websites.

## 3. Simplify the Test-Drive Process

Smart consumers don't have the patience for cumbersome trials. Facilitate easy, no-strings-attached access to your product. Avoid crippled evaluation versions or long-winded registration processes. Ideally, provide a direct download link, allowing them to install and test your product with minimal friction. The assumption is that most products are subpar, so your goal is to quickly demonstrate otherwise.

> 📺 [Video: "Slack is free to try for as long as you'd like"](https://www.youtube.com/watch?v=IIqPSWFFSXg) ([source](https://slack.com/))

## 4. Encourage Discovery, Don't Push

Smart people dislike being bombarded with information. Instead of aggressive marketing, position your product where your audience is likely to find it. Engage in content marketing, sharing knowledge in forums, blogs, or communities where your target audience spends time. This approach not only feels more authentic but also aligns with how smart people prefer to discover new products — through research, not ads.

![[fig-dropbox.png|*In Dropbox's referral program users can earn up to 16 GB of free storage by inviting friends to join, effectively leveraging word-of-mouth and organic growth through user-driven discovery. ([source](https://www.referralcandy.com/blog/dropbox-referral-program))*]]

Dropbox initially grew by relying heavily on word-of-mouth and viral marketing rather than traditional advertising. They placed their product where tech-savvy users were discussing file-sharing solutions, such as on Hacker News and Reddit. Their strategy of offering extra storage space for referrals also encouraged organic growth through discovery.

## 5. Avoid Hyperbolic Claims

Smart consumers are turned off by exaggerated claims and marketing buzzwords. Be straightforward about what your product does and how it stands apart from the competition. Instead of using vague terms like "fast," "scalable," or "reliable," provide context or evidence that makes these claims meaningful. For example, if your product is open-source and cross-platform, explain why that matters, especially if your competitors are proprietary or limited to certain platforms.

![[fig-37signals.png|*The 37signals corporate website stands out with its simplicity. ([source](https://37signals.com/))*]]

37signals (Basecamp) has always taken a straightforward approach to marketing its project management software. Their website and promotional materials focus on specific features and how they solve real-world problems, avoiding fluff and emphasising usability, simplicity, and efficiency. They also make it clear why they're different from more complex, feature-heavy tools like Jira.

## 6. Rethink Online Advertising

Online ads have largely lost their effectiveness with savvy consumers, who often block or ignore them. Instead of investing heavily in traditional online advertising, focus on content that provides real value. Well-written articles, case studies, and insightful blog posts can do more to build your brand with smart people than any banner ad ever could.

![[fig-patagonia.png|*Patagonia: "We are in business to save our home planet." ([source](https://eu.patagonia.com/gb/en/activism/))*]]

Patagonia is an excellent example of a company that has rethought traditional online advertising. Rather than relying on conventional digital ads, Patagonia focuses on impactful content that aligns with its brand values. For instance, they produce and promote high-quality documentaries and storytelling campaigns about environmental activism, which resonate deeply with their audience. These initiatives not only engage smart, socially conscious consumers but also build a strong, authentic brand presence without the need for intrusive online ads.

## 7. Respect the Competition

Bashing the competition is a quick way to lose credibility, especially if you do it on their turf. Criticizing competitors, particularly in spaces where their supporters congregate, can backfire, making you look unprofessional or insecure. Instead, focus on articulating your product's strengths without directly attacking others. Let your product's superiority speak for itself.

> 📺 [Video: "All the people with the power to create use an Apple."](https://www.youtube.com/watch?v=njos57IJf-0)

Apple has often taken a subtle approach when discussing its competition. Rather than directly attacking competitors, Apple focuses on its own product's design, innovation, and user experience. For example, in their "Get a Mac" campaign, Apple highlighted the strengths of their product in a lighthearted manner without being overly aggressive toward Windows PCs.

## 8. Showcase Real-World Success

Testimonials and case studies are powerful tools, but they need to be genuine and specific. Instead of merely listing client names, detail how these customers are using your product and the tangible benefits they've experienced. The best endorsements are unsolicited and come from forums or discussions where users voluntarily share their positive experiences. These carry far more weight than curated testimonials.

![[fig-salesforce.png|*Salesforce's customer stories page contains detailed case studies highlight how various companies, such as Bombardier and Wyndham, have successfully used Salesforce to solve specific challenges, demonstrating real-world impact and building credibility through genuine customer experiences. ([source](https://www.salesforce.com/customer-stories/))*]]

Salesforce effectively uses case studies to demonstrate how different companies have successfully implemented their CRM solutions. These case studies often include detailed narratives about the challenges the companies faced before using Salesforce and how the software helped them overcome these obstacles. By presenting detailed stories rather than just names, Salesforce builds credibility and demonstrates real-world impact.

## 9. Establish Thought Leadership

Beyond having a great product, it's essential to be recognized as an expert in your field. Publish quality content that educates your audience on broader industry trends and challenges, rather than overtly selling your product. When you offer insights that help smart people succeed — regardless of whether they choose your product or a competitor's — you build trust and position yourself as a leader in the space.

![[fig-hubspot.png|*HubSpot Academy's offers free, online courses and certifications, demonstrating HubSpot's commitment to thought leadership by providing valuable, practical education on marketing, sales, and business skills to empower professionals and establish expertise in the industry. ([source](https://academy.hubspot.com/))*]]

HubSpot has mastered the art of content marketing by providing a wealth of free resources on inbound marketing, sales strategies, and customer service. Through their blogs, webinars, and HubSpot Academy, they educate their audience on industry best practices without pushing their software overtly. This strategy has positioned HubSpot as a thought leader in marketing and sales, making their software a natural choice for those who value their expertise.

## Conclusion

Marketing to smart people isn't about flashy ads or aggressive sales tactics. It's about creating something of value, being honest and transparent, and building trust through genuine expertise and openness. When you respect your audience's intelligence and provide them with a product that truly solves their problems, you not only earn their business but also their loyalty and respect.

---

# Introduction to Fourier analysis of time series
_2021-01-28_

# Introduction to Fourier analysis of time series

### How to detect seasonality, forecast and fill gaps in time series using Fast Fourier Transform

![[hero-fourier-animation.gif|*Mehmet E. Yavuz (2021). Fourier Series Animation using Harmonic Circles (link), MATLAB Central File Exchange. Retrieved January 24, 2021.*|450]]

In this article, I will show you the uses of the Fourier transform in time series analysis. We will use the Fast Fourier Transform algorithm, which is available in most statistical packages and libraries. Visualisations and code examples in Python supplements this article.

All are available in [this notebook (Google Colab)](https://colab.research.google.com/drive/10VADEg8F5t_FuryEf_ObFfeIFwX-CxII?usp=sharing).

Although this topic often seems complicated, I will convince you that even basic use of Fourier analysis can give good results.

## How to analyse weather data using Fourier analysis

Let's assume that we work on some weather data.

| ds | temperature |
|---|---|
| 2008-01-01 | 0.49680824612499985 |
| 2008-01-02 | -2.8211083458333337 |
| 2008-01-03 | -5.7556919375 |
| 2008-01-04 | -7.374025225 |
| 2008-01-05 | -6.109025095833332 |

*Full data: [weather_sample.csv](https://gist.github.com/fischerbach/90236e9d96dcbba16d77311f017a50c4)*

In our dataset is the average daily temperature for a certain location.

![[fig-temperature-raw.png]]

![[fig-temperature-trend.png|*Although there are outlier measurements in the dataset, the general trend is consistent over time. (fig. by author)*]]

As we can see, a sinusoidal trend prevails throughout the year at a certain fixed frequency. If we can know what frequency this is, we can decompose the seasonality of this time series. This could be useful for improving forecasts or dealing with missing measurements.

## Fourier analysis

Our goal is to take this single-variable periodic time series and decompose it into simpler periodic functions.

According to the theorem formulated by Joseph Fourier, any periodic function, no matter how trivial or complex, can be expressed as a composition (combination) of periodic components, known as the Fourier series.

![[fig-fourier-series.png]]

The method for expressing a function as a sum of sines and/or cosines, and for recovering the function from those components is called Fourier analysis. It does not matter if the function is non-sinusoidal. Any periodic time series is an infinite sum of sinusoidal components with coefficients.

Fourier analysis is the process of obtaining the spectrum of frequencies H(f) comprising a time-series h(t) and it is realized by the Fourier Transform (FT). Fourier analysis converts a time series from its original domain to a representation in the frequency domain and vice versa.

![[fig-frequency-spectra.png|*Typical examples of frequency spectra of some periodic time series composed of sinusoidal components. (fig. by author)*]]

In simpler words, Fourier Transform measures every possible cycle in time-series and returns the overall "cycle recipe" (the amplitude, offset and rotation speed for every cycle that was found). Classical Fourier transform is for continuous functions. As our data is discrete, we will use a discrete counterpart of the Fourier transform.

We will apply the Fast Fourier Transform (FFT), an algorithm that computes the discrete Fourier transform (DFT) of a time series, or its inverse (IDFT). DFT has a great number of applications in physics, digital signal processing and compression, e.g. in MP3 and JPEG formats.

![[fig-jpeg-blocks.png|*The JPEG algorithm is brutally simple: it divides an image into blocks and applies Fourier transform on each. (fig. source)*]]

A good introduction to the FFT can be found in the NumPy library documentation:

> [Discrete Fourier Transform (numpy.fft) — NumPy Manual](https://numpy.org/doc/stable/reference/routines.fft.html)

## How to detect time-series seasonality using Fast Fourier Transform

In the time-series data, seasonality is the presence of some certain regular intervals that predictably cycle on the specific time frame (i.e. weekly basis, monthly basis). Decomposing seasonal components from time-series data can improve forecasting accuracy.

There are many approaches to detect seasonality in time series, such as classical decomposition and seasonal extraction in ARIMA. We will use Fast Fourier Transform (FFT) to find the period of dominating seasonal components of time series. As an example, data that has one strong seasonal effect and residuals. Intuition suggests that temperature should have at least one year period. Let's check it.

The procedure to extract seasonality from time-series is straightforward:

- Apply Fourier transform on the dataset to get frequency domain.
- Sort descending frequency domain by coefficients.
- Take the highest of these and get the periods by dividing 1 by the frequency.

```python
nobs = len(data['temperature'])
temperature_ft = np.abs(rfft(data['temperature']))
temperature_freq = rfftfreq(nobs)

plt.figure(figsize=(10, 7))
plt.plot(temperature_freq[2:], temperature_ft[2: ])
annot_max(temperature_freq[2:], temperature_ft[2: ])
    
plt.xlabel('frequency (1/day)')
plt.show()
```

![[fig-fft-spectrum.png|*Distinct spikes at certain frequencies represent harmonics (component sine function). Annual seasonality is dominant here. (fig. by author)*]]

| freqency [1/day] | y | period [days] |
|---|---|---|
| 0.0027 | 23642.48 | 364.15 |
| 0.0023 | 1837.26 | 430.36 |
| 0.0025 | 1528.34 | 394.50 |
| 0.0004 | 1396.18 | 2367.00 |
| 0.002 | 1296.69 | 338.14 |

*Full data: [seasonality.csv](https://gist.github.com/fischerbach/dc3d2b9f7fc1b7b94b0dd380a9077791)*

The result is quite predictable. Because of the seasons, the temperature fluctuates with the annual period. The result is consistent with this intuition.

## Sidenote: Spectral leakage

Because we have data from multiple full periods, we are unlikely to have to worry about spectral leakage. When we conduct FFT on a finite length time series, we assume the data repeats itself to infinity and connects the endpoint with the start point. But when there is a discontinuity, it spreads as a peak to the surrounding frequencies.

![[fig-spectral-leakage.png|*Example of spectral leakage (fig. by Author)*]]

Here is an interesting article on how to deal with this problem:

> [Signal Processing: Why do we need taper in FFT](http://qingkaikong.blogspot.com/2017/01/signal-processing-finding-periodic.html)

## How to smooth time series using Inverse Fast Fourier Transform

Another use of FT is smoothing time series data. You can think about it as low-pass filtering that can be easily performed to remove components with a certain frequency and up, while information containing low-frequency components are retained. This can lead to the removal of unnecessary random fluctuations from the time series.

One might ask, why not use a moving average method or exponential smoothing. These methods do not exploit the observed periodicity in the data, So, for example, if there are gaps in the dataset, you can fill them with its estimations by adjusting the IFT parameters. This does not mean that we have to limit ourselves only to functions with noticeable seasonality.

Fluctuations of any function can be cut out to the desired frequency range. A similar approach can be used to get rid of seasonality in the data to increase the stationarity of the time series. In that case, a high-pass filter should be used.

The procedure is simple:

- Move to the frequency domain. (Fourier transform)
- Remove undesired frequencies.
- Move back to the time domain. (Inverse Fourier transform)

![[fig-smoothing-animation.gif|*Animation showing how successive filtering thresholds affect Fourier smoothing. The red area represents the range of removed frequencies. (fig. by author)*]]

Let's use this procedure.

![[fig-smoothing-input.png]]

![[fig-smoothing-result.png|*Fourier smoothing (fig. by author)*]]

## Takeaways

As we can see Fourier analysis can help us capture the seasonality and can be used to decompose the time series data. Besides its usefulness, Fourier analysis is not a universal tool for all problems. FT is great for quickly creating predictive models for data with strong seasonality. Special events in your data, such as unexpected weather conditions, would not be predicted.

So, you can use Fourier Analysis whenever:

- You notice periodicity in data
- You need a quick forecasting model
- You want to fill gaps in the data

## References

- Measuring Forecast Accuracy
- https://www.kaggle.com/residentmario/signal-decomposition-with-fast-fourier-transforms
- http://195.134.76.37/applets/AppletFourAnal/Appl_FourAnal2.html
- https://medium.com/towards-artificial-intelligence/seasonality-detection-with-fast-fourier-transform-fft-and-python-1021986d1e4f
- https://numpy.org/doc/stable/reference/routines.fft.html
- http://qingkaikong.blogspot.com/2017/01/signal-processing-finding-periodic.html

---

# How to predict solar energy production
_2021-01-10_

# How to predict solar energy production

### Efficient use of renewable energy sources with machine learning

**Cite as:** Rybnik, R. (2021). How to predict solar energy production: Efficient use of renewable energy sources with machine learning. Zenodo. <https://doi.org/10.5281/zenodo.20562980>

**Code & data:** [solar-energy-forecasting-code-and-data.zip](https://zenodo.org/records/20562980/files/solar-energy-forecasting-code-and-data.zip)

## Abstract

Solar photovoltaic (PV) energy production is highly dependent on weather, which makes its output difficult to predict. This article presents a practical, data-driven approach to forecasting the daily energy output of a rooftop PV installation in north-eastern Poland, using production data collected through the SolarEdge API together with historical weather data from Meteoblue and computed daylight information. Several forecasting methods are compared — naive and moving-average baselines and Facebook's Prophet, with and without external weather regressors — and evaluated with standard forecast-accuracy statistics. The results show that adding a small number of readily available weather variables lets Prophet predict PV output with satisfactory accuracy and anticipate sudden changes in production.

**Keywords:** solar energy, PV forecasting, Prophet, time series, machine learning, weather data

Solar power systems could be a key tool for energy production for the present and future generations. Solar energy is environmentally friendly and provides electricity to places where it is difficult to build conventional infrastructure. Photovoltaic (PV) cells become cheaper each year. Solar energy is cheaper than ever.

![[fig-solar-cost.png|*The average cost of solar panels has fallen 65% from $7.34 per watt in 2010, to $2.53 per watt in 2019. (source: HomeGuide, https://homeguide.com/costs/solar-panel-cost#kwh)*]]

However, it has two huge obstacles: energy is produced only during the daytime and the amount of energy produced is highly dependent on the weather. Machine learning algorithms combined with weather data have the potential to overcome these barriers, which results with more efficient use of renewable energy sources.

In this article, you will learn about predicting time series depending on external (weather) conditions. I will show you how to improve your predictions using the domain knowledge of the target variable. We will go quickly through collecting data about energy production from the solar farm through its producers API. Next, we will analyze gathered data and select features for time-series forecasting. Finally, we will train and test several predictive models and identify the best, most suitable for the described case.

## Why?

The problem with setting up solar power sources is that it is difficult to predict how much power it will generate once it's built. Of course, we can quite accurately [calculate the amount of solar energy reaching the Earth's surface under ideal weather conditions](https://www.pveducation.org/pvcdrom/properties-of-sunlight/calculation-of-solar-insolation). However, weather fluctuations and the unique microclimate of each location affect current energy production by renewable energy installations. This forces scheduling of tasks that require a lot of power.

To maximize the environmental and financial benefits of using renewable energy, we need to alter our habits. Making the electricity system intelligent and flexible will help optimize our behaviour and manage household energy consumption. Energy-consuming tasks can be scheduled for days or even hours of best weather conditions.

Lastly, knowing your system's production tells you how quickly your solar system will return the investment.

![[fig-roi.png]]

## Device

My friends have PV cells for more than a year.

![[fig-guesthouse.jpg|*Czaplisko Siedlisko is a guesthouse located at the shore of Jeziorak lake, among forest and meadows. Bird's songs will accompany you every day. Modern stud wall house with 1.5 hectares of land at your disposal — our proposal of an unforgettable vacation! (https://czapliskosiedlisko.pl/)*]]

The whole installation has 31 panels that occupy 53 square metres of the roof. Inverter manages electricity generation. It is a device that converts direct current (produced by PV cells) into alternating current (used by household appliances), and manages the entire installation. Unused surplus is fed into the public electricity grid, which reduces the owner's electricity bill.

![[fig-panels.png]]

The inverter is manufactured by Solar Edge, which has equipped it with internet connectivity. The device sends statistics about energy production to the company, who makes it available to the owner of the installation via a mobile application, a website and API.

## Acknowledgements

> Special thanks are going to [Czaplisko Siedlisko](https://czapliskosiedlisko.pl/) for sharing data from their solar farm.
>
> Also, thanks [Meteoblue](https://www.meteoblue.com/) for providing historical weather data for the location of the panels.

## Data collecting

All code and data used in this article are available here: [Solar Energy Forecasting in north-eastern Poland](https://doi.org/10.5281/zenodo.20562980).

First, we need to collect and store data from SolarEdge API.

```json
{
    "energy": {
        "timeUnit": "DAY",
        "unit": "Wh",
        "measuredBy": "INVERTER",
        "values": [
            {
                "date": "2020-12-09 00:00:00",
                "value": 2911.0
            }
        ]
    }
}
```

*A typical response to an API request*

To avoid repetition in the code, I sketched a simple API client.

```python
class SolarEdgeClient:
    def __init__(self, key):
        self.key = key
    
    def show_key(self):
        print(self.key)
    
    def getDataPeriod(self, site_id):  
        url = f"https://monitoringapi.solaredge.com/site/{site_id}/dataPeriod?api_key={self.key}"
        response = requests.request("GET", url, headers={}, data = {})
        data = response.json()
        return (data['dataPeriod']['startDate'], data['dataPeriod']['endDate'])

    def getSiteEnergy(self, site_id, start_date, end_date, time_unit='DAY'):
        url = f"https://monitoringapi.solaredge.com/site/{site_id}/energy?timeUnit={time_unit}&endDate={end_date}&startDate={start_date}&api_key={self.key}"
        response = requests.request("GET", url, headers={}, data = {}) 
        data = response.json()
        
        return data['energy']['values']
    
    def getSiteDetails(self, site_id):
        url = f"https://monitoringapi.solaredge.com/site/{site_id}/details?api_key={self.key}"
        response = requests.request("GET", url, headers={}, data = {}) 
        data = response.json()
        
        return data['details']
    
    
    def read_site_energy(self, site_id, start_date, end_date, time_unit='DAY'):
        site_details = self.getSiteDetails(site_id)
        datelist = pd.date_range(start_date, end=end_date).tolist()
        energyData = pd.DataFrame()
        for date in datelist:
            dailyData = self.getSiteEnergy(site_id=site_id, start_date=date.strftime("%Y-%m-%d"), end_date=date.strftime("%Y-%m-%d"), time_unit=time_unit)
            temporary_df = pd.DataFrame.from_records(dailyData)
            if(time_unit != 'DAY'):
                temporary_df['date'] = temporary_df['date'].apply(lambda date: Helpers.localtime_to_utc(date, site_details['location']['timeZone']))
            energyData = pd.concat([energyData, temporary_df])
        return energyData
```

It is worth noting that if you are dealing with time-series data, especially in hour or greater frequencies, you must take care of the time zones of all data sources. I recommend you convert datetimes to UTC timezone or to UNIX timestamps as soon as you import a timezone-aware dataset. For this purpose, I wrote some helper functions, which I have collected in one class as static functions. This makes the code more readable, without the risk of functions' name conflict.

```python
from astral.sun import sun
from astral.sun import daylight
from astral import LocationInfo

class Helpers:
    @staticmethod
    def localtime_to_utc(date, local_timezone="Europe/Warsaw"):
        local = pytz.timezone(local_timezone)
        naive = datetime.datetime.strptime (date, "%Y-%m-%d %H:%M:%S")
        local_dt = local.localize(naive, is_dst=True)
        utc_dt = local_dt.astimezone(pytz.utc)
        return utc_dt.strftime ("%Y-%m-%d %H:%M:%S")
    
    @staticmethod
    def generate_daylight(start_date, end_date):
        datelist = pd.date_range(start_date, end=end_date).tolist()
        daylightData = pd.DataFrame(columns=['date','daylight','sunrise','sunset'])
        for date in datelist:
            daylight = Helpers.getDaylight(date)
            daylightData = daylightData.append({
                "date": date, 
                "daylight": daylight['duration'],
                "sunrise": daylight['sunrise'],
                "sunset": daylight['sunset']
            }, ignore_index=True)
        return daylightData
    
    @staticmethod
    def getDaylight(start_date, timezone="Europe/Warsaw"):
        city = LocationInfo("Skitlawki", "Poland", timezone, LATITUDE, LONGITUDE)
        s = sun(city.observer, date=start_date)
        return {
            "duration": (s['sunset'] - s['sunrise']).seconds,
            "sunrise": s['sunrise'].strftime ("%Y-%m-%d %H:%M:%S"),
            "sunset": s['sunset'].strftime ("%Y-%m-%d %H:%M:%S")
        }
```

## We look at the data and establish success indicators

The first step to develop a predictive model for specific time series is to identify and understand the underlying pattern of the data over time. Data collected in the previous section contains the electricity output of the solar PV system, which is measured in kilowatt-hours (kWh). Just like in a smartphone's battery, the kilowatt-hours unit describes how much energy is produced and can be used to power household appliances. 1 kilowatt-hour corresponds to the amount of energy consumed per hour by a 1,000-watt device.

| date | value |
|---|---|
| 2020-03-05 | 13895.0 |
| 2020-03-06 | 3245.0 |
| 2020-03-07 | 32336.0 |
| 2020-03-08 | 4228.0 |
| 2020-03-09 | 26802.0 |
| 2020-03-10 | 28464.0 |
| 2020-03-11 | 8922.0 |
| 2020-03-12 | 29895.0 |

*Full data: [energy_daily_sample.csv](https://zenodo.org/records/20562980/files/solar-energy-forecasting-code-and-data.zip)*

Another metric is the current power production (measured in kilowatts), which describes how much power at the specific moment a solar panel can provide. For example, if the PV cells produce 1000 W at a specific moment, it is theoretically possible to supply two devices with the requirement of 450 W each and 100W remains. But to power, a 2500 W milling machine may require additional power to be obtained from the public grid.

After consulting the receivers of my forecasts, we have established that the information about the daily expected number of kilowatt-hours will be most valuable to them. The actual home equipment does not require sophisticated energy flow management techniques, so the moment the power of the system will not apply in the foreseeable future. In the following, we will therefore focus on the total energy produced (or to be produced) on a given day. The forecast horizon will be 5 days.

![[fig-daily-energy.png|*Not surprisingly, it shows an annual fluctuation that is tied to the seasons.*]]

## Simple forecasts

Before we will go deeper into forecasting based on external variables, let's try to make a prediction having historical values of the target variable — daily electricity amounts produced by PV cells.

**Naive forecast.** This is a very basic method in which the predicted value is simply the value of the most recent observation. In our case, it will be energy produced the day before the prediction date.

**Moving average.** Another basic forecasting method is moving average. Moving average is rather a technical tool to analyse the time series, however, is extremely useful for forecasting long-term trends. In this case, we will use a simple moving average, which is a calculation that takes an arithmetic mean of a specific number of days in the past.

![[fig-moving-average.png]]

Code below tests those methods.

```python
data = energy.copy()
data['cap'] = data['y'].max()
data['floor'] = 0.0
simple_methods = []
### Naive
MODEL = 'Naive'
data[MODEL] = data['y'].shift(periods=1)
arg = (data[MODEL], data['y'])
simple_methods.append([MODEL, *FES.all(*arg)])

### Moving Average
MODEL = 'Moving Average'
data[MODEL] = data['y'].rolling(4).mean().shift(periods=1)
arg = (data[MODEL], data['y'])
simple_methods.append([MODEL, *FES.all(*arg)])

simple_methods_df = pd.DataFrame(simple_methods, columns= ['Method', 'ME', 'MSE', 'RMSE', 'MAE', 'MPE', 'MAPE', 'U1', 'U2'])
display(simple_methods_df.set_index('Method'))
simple_methods_df.to_html('naive_and_moving_average.html', index=False)
                                                 
plot_data = data['2020-06':'2020-07']
fig = plt.figure(figsize=(40,14))
plt.xlabel('date',fontsize=40, labelpad=25)
plt.ylabel('energy [Wh]',fontsize=40, labelpad=25)
plt.plot(plot_data['y'], linewidth=2.5)
plt.plot(plot_data['Naive'], linewidth=0.75)
plt.plot(plot_data['Moving Average'], linewidth=0.75)
plt.legend(['Real value','Naive','Moving Average'], prop={'size': 20})
plt.title('Naive and moving average',fontsize=40, pad=30)
                                                 
# plt.show()
plt.savefig('naive_and_moving_average.png')
```

> Note: We haven't split the dataset for training and test sets yet. By definition of those methods, there is no risk of using data from the future.

## How to evaluate forecasts (1)

Now, for both methods, we should calculate several forecast evaluation statistics.

If you want to know more about forecast evaluation statistics, please read my article:

> [Forecast evaluation statistics with examples in Python](https://fischerbach.medium.com/forecast-evaluation-statistics-with-examples-in-python-6b540ef751c2)

The table below shows the results of the evaluation:

![[fig-eval1-table.png]]

As expected, these methods are not very accurate. We will use those as a benchmark to evaluate more sophisticated ones. Because, if a more complicated model is less accurate than those forecasts, then there is something wrong. The main indicator of success is the ability of the model to inform about sudden changes in values of time series. No matter how well the trend fits the data, it can't inform in time of sudden changes in production volume (because those are induced by external conditions).

In this context, to achieve our goal, we need a more advanced solution.

## We need the Prophet

Our second shot will be [Prophet](https://facebook.github.io/prophet/) library from Facebook. The Prophet is a procedure for forecasting time series data based on an additive model where non-linear trends are fit with yearly, weekly, and daily seasonality, plus holiday effects.

Creators of the Prophet advertise it as giving a reasonable forecast on messy data with no manual effort.

So, let's test this declaration.

```python
from fbprophet import Prophet
from fbprophet.plot import add_changepoints_to_plot
from fbprophet.diagnostics import cross_validation

# (...)

def prophet_without_regressors(train, test):
    m = Prophet(daily_seasonality=False, weekly_seasonality=False, yearly_seasonality=True, growth='logistic')
    m.fit(train)
    forecast = m.predict(test)
    forecast['yhat'] = forecast['yhat'].apply(lambda x: 0 if x<0 else x)
    return forecast.set_index('ds')['yhat']
```

## How to evaluate forecast (2)

For the reliability of forecasts, the model can't be trained using future data.

![[fig-prophet-train.png]]

Therefore, for further evaluation and comparison of forecasts, we will use training and test sets. The size of the training set is typically about 20% of the total sample, depending on how large is the sample and the forecasting horizon. Since we have more than a year's data and, also, the forecast is only to cover the next few days, we can use a more sophisticated method.

Instead of splitting the whole sample once, there are a series of training and test sets, separated from the sample as shown below.

![[fig-crossval-split.png]]

Each training set is used for the creation of a new model and then we calculate evaluation statistics using the corresponding test set.

```python
def cross_validation(data, model=prophet_without_regressors):
    date_range = (data['ds'].min()+datetime.timedelta(days=365), data['ds'].max())
    datelist = pd.date_range(*date_range, freq='5D').tolist()
    evaluation_statistics = []
    for i in range(len(datelist)):
        if (i == len(datelist) - 1):
            continue
        train_end = datelist[i]
        test_start = datelist[i] + datetime.timedelta(days=1)
        test_end = test_start + datetime.timedelta(days=5)
        if(test_end > data['ds'].max()):
            test_end = data['ds'].max()
        else:
            test_end = datelist[i] + datetime.timedelta(days=5)
        train_end = datetime.datetime.strftime(train_end, '%Y-%m-%d')
        test_start = datetime.datetime.strftime(test_start, '%Y-%m-%d')
        test_end = datetime.datetime.strftime(test_end, '%Y-%m-%d')
        
        train = data[:train_end].copy()
        test = data[test_start:test_end].copy()
        train['ds'] = train.index
        test['ds'] = test.index

        evaluation_statistics.append([f":{train_end} | {test_start} - {test_end}",*FES.all(model(train, test), test['y'])])
    evaluation_statistics = pd.DataFrame(evaluation_statistics, columns= ['period','ME', 'MSE', 'RMSE', 'MAE', 'MPE', 'MAPE', 'U1', 'U2'])
    return evaluation_statistics
```

It reproduces the real way of evaluating the forecast, as forecasts will be converted once every 5 days.

![[fig-prophet-forecast.png]]

The Prophet is not doing better than the moving average. By the nature of the problem, it cannot take full advantage of the possibility of matching seasonality (after all, sunlight does not show weekly or monthly seasonality). MAPE for Prophet is slightly lower than for moving average. This suggests that it reacts to trend changes a little faster. Another statistic — Theil's U₂ is less than 1, so both models are doing better than naive.

![[fig-prophet-table.png]]

However, after a visual inspection of the chart of Prophet's forecasts, it is encouraging to add external variables.

## What are the factors that affect solar panels' efficiency?

### Theory

The kilowatt-hour production of a solar system depends on how much sunlight hits the panels and for how long. ([source](http://web.archive.org/web/20240619041327/https://www.solarunitedneighbors.org/news/predict-solar-output-production-2/))

The amount of available sunlight depends on the local solar resource, weather conditions, panel orientation, amount of shading, system design and type of mounting (ground or roof mount).

Every installation is different. Things like the angle of PV, wiring and location-specific weather conditions could lead to short-term variability in results.

### Location

Different parts of Earth's surface receive different amounts of sunlight.

![[fig-world-sunlight.png|*Most of the data were obtained for the exact location of the solar farm.*]]

### Weather conditions

Variables that may have a physical impact on the production of electricity by PV cells:

- air temperature,
- precipitation,
- snowfall,
- relative humidity,
- cloud cover,
- sunshine duration,
- solar radiation.

![[fig-cloud-cover.gif|*It is easy to see that a larger cloud cover reduces the efficiency of solar panels.*]]

Cloud coverage does not mean that no sunlight will reach solar panels, but the amount will be reduced. Moreover, PV cells are often installed on the roofs of high buildings. Because of the lack of availability of reliable forecasts for every variable, not all of them will be used in the final model.

The approach proposed in this article is customizable for all installations and is independent of any device-specific characteristics (such as declared efficiency of PV cells). Also, the side goal is to use regressors easily accessible in most weather forecast services.

### Temperature

Interestingly, air temperature is important not because it correlates with sun activity only. The efficiency of the PV cell depends on its temperature, which increases when the temperature drops and decreases when the temperature rises.

## Weather dataset

As you know from previous sections, reliable weather data is crucial to the short- and long-term prediction of energy production. We should distinguish between two types of weather data: historical (from real-world measurements) and forecasted.

Just like in our future solar energy model, the weather forecast is also modelled. When creating models you should always focus on the domain of the problem and decide whether the model should be trained using historical weather forecasts or rather using past real measurements.

![[fig-weather-forecast.png|*Typical weather forecast (source: Meteoblue)*]]

For example, if you have to predict the purchase of tickets to an amusement park, you better take into account the weather forecast, because consumers often consider it when they are planning such activities. The advantage of this approach is also that (assuming the stability of the weather forecast error itself) the weather forecast error will have an easier predictable impact on the final predictions.

On the other hand, if you are to predict the results of physical quantity measurements, it will be better to take into account real measurements of past predictors. Then the resultant forecast error will have a more difficult to analyse random distribution (resulting from interferences between the resultant model and weather forecast model). But, at the same time, the model will be resistant to changes in weather modelling.

In the solution described in this article, I used historical data for the weather, because available sunlight is variable depending on real physical conditions. Unfortunately, there are no weather stations near the location of the solar panels. But simulation data could be a good compromise.

## Meteoblue

Courtesy of [Meteoblue](https://www.meteoblue.com/), I was given access to historical weather data for the location of the solar panels. Their product [history+](https://www.meteoblue.com/en/historyplus) offers consistent weather simulation data in hourly resolution since 1985. Higher-resolution data are available since 2008 for nearly every place on Earth. Meteoblue history+ is a paid service, but you can test it for free with full historical data for Basel, Switzerland. For other locations, data is limited to the last 2 weeks.

![[fig-meteoblue-vars.png|*Available variables (screenshot from meteoblue.com)*]]

The data has been downloaded into a CSV file and requires minor processing.

## Daylight

There is one other data source that I have used. Day length for every location on Earth changes during the year, with an average of 12 hours of light.

![[fig-daylight.png|*It is easy to see that the solar energy produced is dependent on the length of the day.*]]

We should generate the length of each day of the year and the times of sunrises and sunsets. They will be needed later to resample the weather data.

```python
from astral.sun import sun
from astral.sun import daylight
from astral import LocationInfo

class Helpers:
    @staticmethod
    def localtime_to_utc(date, local_timezone="Europe/Warsaw"):
        local = pytz.timezone(local_timezone)
        naive = datetime.datetime.strptime (date, "%Y-%m-%d %H:%M:%S")
        local_dt = local.localize(naive, is_dst=True)
        utc_dt = local_dt.astimezone(pytz.utc)
        return utc_dt.strftime ("%Y-%m-%d %H:%M:%S")
    
    @staticmethod
    def generate_daylight(start_date, end_date):
        datelist = pd.date_range(start_date, end=end_date).tolist()
        daylightData = pd.DataFrame(columns=['date','daylight','sunrise','sunset'])
        for date in datelist:
            daylight = Helpers.getDaylight(date)
            daylightData = daylightData.append({
                "date": date, 
                "daylight": daylight['duration'],
                "sunrise": daylight['sunrise'],
                "sunset": daylight['sunset']
            }, ignore_index=True)
        return daylightData
    
    @staticmethod
    def getDaylight(start_date, timezone="Europe/Warsaw"):
        city = LocationInfo("Skitlawki", "Poland", timezone, LATITUDE, LONGITUDE)
        s = sun(city.observer, date=start_date)
        return {
            "duration": (s['sunset'] - s['sunrise']).seconds,
            "sunrise": s['sunrise'].strftime ("%Y-%m-%d %H:%M:%S"),
            "sunset": s['sunset'].strftime ("%Y-%m-%d %H:%M:%S")
        }
```

## Collecting all data for the final set

Let's gather all data (energy, weather and daylight) in one DataFrame.

```python
dataset = pd.merge(left=meteoblue, right=daylight)
dataset['sunrise'] = pd.to_datetime(dataset['sunrise'])
dataset['sunset'] = pd.to_datetime(dataset['sunset'])
dataset['is_day'] = dataset[['ds','sunrise','sunset']].apply(lambda x: x['sunrise'] <= x['ds'] <= x['sunset'],  axis=1)
display(dataset.head())

df = dataset.loc[dataset['is_day']].drop('is_day', axis=1)

aggregations_dict = {
    'Temperature [2 m elevation corrected]': 'mean',
    'Temperature [1000 mb]': 'mean',
    'Temperature [850 mb]': 'mean',
    'Temperature [700 mb]': 'mean',
    'Precipitation Total': 'sum',
    'Snowfall Amount': 'sum',
    'Relative Humidity [2 m]': 'mean',
# (...)
    'Temperature': 'mean',
    'Soil Temperature [0-10 cm down]': 'mean',
    'Soil Moisture [0-10 cm down]': 'mean',
    'Vapor Pressure Deficit [2 m]': 'mean',
    'daylight': 'mean',
}


for key in aggregations_dict.keys():
    df[key] = pd.to_numeric(df[key])

df = df.resample('D', on='ds').agg(aggregations_dict).reset_index()
df = pd.merge(left=data.drop('ds', axis=1).drop(['cap','floor'], axis=1), right=df, left_on='ds', right_on='ds')
```

The resulting set has 46 columns, so a sample is shown here as raw CSV (full file: [final_dataset_sample.csv](https://zenodo.org/records/20562980/files/solar-energy-forecasting-code-and-data.zip)):

```text
ds,y,Temperature [2 m elevation corrected],Temperature [1000 mb],Temperature [850 mb],Temperature [700 mb],Precipitation Total,Snowfall Amount,Relative Humidity [2 m],Wind Speed [10 m],Wind Direction [10 m],Wind Speed [80 m],Wind Direction [80 m],Wind Gust,Wind Speed [900 mb],Wind Direction [900 mb],Wind Speed [850 mb],Wind Direction [850 mb],Wind Speed [700 mb],Wind Direction [700 mb],Wind Speed [500 mb],Wind Direction [500 mb],Cloud Cover Total,Cloud Cover High [high cld lay],Cloud Cover Medium [mid cld lay],Cloud Cover Low [low cld lay],CAPE [180-0 mb above gnd],Sunshine Duration,Shortwave Radiation,Direct Shortwave Radiation,Diffuse Shortwave Radiation,Mean Sea Level Pressure [MSL],Geopotential Height [1000 mb],Geopotential Height [850 mb],Geopotential Height [700 mb],Geopotential Height [500 mb],Evapotranspiration,FAO Reference Evapotranspiration [2 m],Temperature,Soil Temperature [0-10 cm down],Soil Moisture [0-10 cm down],Vapor Pressure Deficit [2 m],daylight,ds,cap,floor
2019-05-09,32425.0,12.64,12.65,2.47,-5.80,0.30,0.0,54.31,...,56060,2019-05-09,71061.0,0.0
2019-05-10,32269.0,13.85,13.60,2.99,-5.70,0.20,0.0,72.25,...,56276,2019-05-10,71061.0,0.0
```

*Final dataset*

## Forecasting

To select external variables from a previously prepared dataset, we focus on two characteristics. First is the physical influence of variables, such as temperature and solar power reaching PV cells' surface. Second is the correlation with the target variable. As we cannot be sure that some weather-related variable isn't correlated somehow with another, which is directly tied with the effectiveness of solar farm, but unavailable to measure directly.

```python
correlation_table = df.corr()
target_correlation_table = correlation_table.loc[correlation_table.index == 'y'].transpose()

regressors = target_correlation_table.loc[np.abs(target_correlation_table['y']) >= 0.1].index.values
display(regressors)
```

The method of selecting variables was straightforward. First, we should reject variables with a close to zero correlation with the target variable. Let's see how the model is doing with the remaining variables.

```python
def prophet_with_regressors(train, test):
    m = Prophet(daily_seasonality=False, weekly_seasonality=False, yearly_seasonality=True, growth='logistic')
    for col in train.columns:
        if col in regressors and col not in ['y', 'cap', 'floor','Naive','Moving Average']:
            m.add_regressor(col)

    m.fit(train)

    forecast = m.predict(test)
    forecast['yhat'] = forecast['yhat'].apply(lambda x: 0 if x<0 else x)
    
    return forecast.rename(columns={'yhat':'y'}).set_index('ds')['y']
```

![[fig-model-full-1.png]]

![[fig-model-full-2.png]]

Quite a good one. Not only does the model have the best statistics so far, but it is also able to inform in advance about sudden changes in production values. The MSE is an order of magnitude smaller than previous attempts!

## Limiting the number of necessary regressors

Variables such as Shortwave Radiation are obviously linked to solar electricity production, but at the same time are not easily accessible in the weather forecast. Therefore, we will now try to limit the number of variables to the necessary minimum and to those that can be found in any weather forecast.

![[fig-model-limited-1.png]]

![[fig-model-limited-2.png]]

Quite close to the 'full' model.

## Conclusions

With the use of a relatively small number of weather variables, electricity production by PV cells can be predicted with satisfactory accuracy. Importantly, the predictions of explanatory variables are easily accessible in most weather forecasts. However, if we use all measurable variables, the model achieves correspondingly better results.

## Future directions

Predictive models, aware of how prices, supply and demand are shaped and knowing future forecasts of energy production, can prepare recommendations for owners as to their behaviour on the energy market. This can create additional revenue opportunities for users of renewable energy sources.

My friends have a smart home solution. They already integrate their lights in it and schedule dishwasher or washing machine cycles with potential high PV activity. In the future, all of their power plugs could be scheduled by solar prediction. Another direction is to provide the forecasts in the form of a kitchen dashboard.

## References

- https://www.solaredge.com/sites/default/files/se_monitoring_api.pdf
- http://www.geo.mtu.edu/KeweenawGeoheritage/MiTEP_ESI-2/Solar_energy_and_latitude.html
- https://otexts.com/fpp2/accuracy.html
- https://www.thegreenage.co.uk/article/the-impact-of-temperature-on-solar-panels/

---

# Measuring forecast accuracy
_2021-01-03_

# Measuring Forecast Accuracy

### Forecast evaluation statistics with examples in Python

If I had to choose one basic skill in data science that is the most useful, it would be time series forecasting. Predicting the future value of something contributes to making better decisions. Therefore, it is crucial to be sure that you can rely on forecasting. The choosing, construction and interpretation of forecast accuracy metrics are just as important as making forecasts.

What actually makes up the accuracy of the forecast?

![[hero.png|*Unless stated otherwise, all pictures in the article are by the author.*|450]]

## How to choose evaluation statistics

Choosing a accuracy estimation method often depends on the domain of the problem. In my career, I have encountered a situation where a hastily chosen metric has caused the client's dissatisfaction with the forecasting results optimized for KPI inconsistent with the specific business case. For example, the model may have a low mean square error, but at the same time doesn't predict sudden deviations from "everyday normal" values or trend changes.

This article will show you the fundamental forecast evaluation statistics that you can use to build and test your predictive models. We will calculate and interpret them using concrete examples: the mean error (ME), the mean squared error (MSE), the mean absolute error (MAE), the mean percentage error (MPE), the mean absolute percentage error (MAPE), and Theil's U-statistics. We will interpret and discuss examples in Python in the context of time-series forecasting data.

![[fig-metrics-overview.png|*The statistic metrics are shown in this article.*]]

> Note: As I am focusing on specific metrics in this article, I do not address the subject of cross-validation and the splitting of the data set into training and testing. As this is an important concept, I plan to address this topic in future texts.

## Why?

Imagine that you work for an electricity producer. Your task is to create a model that predicts the daily production of electricity in a small wind farm. This will make it possible to plan and reduce the production of electricity in a conventional power station at times of high efficiency from renewable sources.

![[fig-windfarm.png]]

You have created a couple of prediction models. How to choose the best one to do the task?

## Dataset

To allow the calculation of all presented statistics, a CSV file has been created.

| ds | y | Naive | Moving Average | Advanced #1 | Advanced #2 |
|---|---|---|---|---|---|
| 2020-04-01 | 11534.0 |  |  | 10579.0 | 13435.0 |
| 2020-04-02 | 16063.0 | 11534.0 |  | 23928.0 | 18007.0 |
| 2020-04-03 | 34138.0 | 16063.0 |  | 34263.0 | 27723.0 |
| 2020-04-04 | 34091.0 | 34138.0 |  | 41914.0 | 35713.0 |
| 2020-04-05 | 58414.0 | 34091.0 | 23956.0 | 47230.0 | 53892.0 |
| 2020-04-06 | 61080.0 | 58414.0 | 35676.0 | 50566.0 | 59580.0 |
| 2020-04-07 | 56992.0 | 61080.0 | 46931.0 | 52275.0 | 59721.0 |
| 2020-04-08 | 61045.0 | 56992.0 | 52644.0 | 52697.0 | 61959.0 |
| 2020-04-09 | 36327.0 | 61045.0 | 59383.0 | 52151.0 | 47474.0 |
| 2020-04-10 | 54514.0 | 36327.0 | 53861.0 | 50929.0 | 53416.0 |

*First 10 rows of 91. Full data: [fes.csv](https://gist.github.com/fischerbach/f13975e943c855c11769a07ee2f44da0)*

This spreadsheet contains an example involving two months of observations and four alternative sets of forecasts (made with naive, moving average and two more advanced methods) of a variable of interest (denoted as y).

## Technical details

The notation used below is fairly standard, with yᵢ and fᵢ as a variable of interest and forecast respectively, and n as a number of observations. Full source code used for preparing this article:

> [Full source code (Jupyter notebook)](https://gist.github.com/fischerbach/14e90b32e55b635591934a890ae20f60)

To avoid repetition in the code, I sketched a simple class FES, with static methods that calculate each statistic. This makes the code more readable, without the risk of functions' name conflict.

## Forecast evaluation statistics

By an "error" we mean uncertainty in forecasting, or, in other words, the difference between the predicted value and real value. It is a yᵢ — fᵢ component in most of the following formulas.

## Mean error

The mean error is the average of all the errors in a set.

![[formula-me.png]]

This is a very simple statistic. Unfortunately is biased due to the offsetting effect of positive and negative forecast errors, which may conceal forecasting inaccuracy. Because of that, ME isn't very helpful to model evaluation. However, it is very easy to understand, even for a layman (this is not always an advantage, because of the limitations described above). ME can quickly present the symmetry of the error distribution, which can be useful in assessing a specific model.

```python
def me(f,y):
    f = f.reset_index(drop=True).values.flatten()
    y = y.reset_index(drop=True).values.flatten()
    df = pd.DataFrame({'f_i':f, 'y_i': y})
    df['e'] = df['y_i'] - df['f_i']
    return np.mean(df['e'])
```

## Mean absolute error

A remedy for the inaccuracy of the mean error is the use of the mean absolute error.

![[formula-mae.png]]

The MAE uses absolute values of errors in the calculations, which overcomes the cancellation of errors with opposite signs. It gives us an average of all values of errors, no matter whether they were positive or negative.

```python
def mae(f, y):
    f = f.reset_index(drop=True).values.flatten()
    y = y.reset_index(drop=True).values.flatten()
    df = pd.DataFrame({'f_i':f, 'y_i': y})
    df['e'] = np.abs(df['y_i'] - df['f_i'])
    return np.mean(df['e'])
```

## Mean square error

Just like the MAE, mean square error overcomes the cancellation of positive and negative errors.

![[formula-mse.png]]

Also, the MSE places a greater penalty on large forecast errors than the MAE.

![[fig-error-vs-square.png|*Error vs square error. Note that both plots have different scales.*]]

```python
def mse(f, y):
    f = f.reset_index(drop=True).values.flatten()
    y = y.reset_index(drop=True).values.flatten()
    df = pd.DataFrame({'f_i':f, 'y_i': y})
    df['e'] = np.square(df['y_i'] - df['f_i'])
    return np.mean(df['e'])
```

## Root mean square error

Root mean square error is the standard deviation of the errors.

![[formula-rmse.png]]

RMSE shares advantages of MSE and is commonly used in forecasting and regression analysis to verify experimental results. Furthermore, it has the advantage of having the same units as the forecasted variable, so it is easier to directly interpret.

```python
def rmse(f, y):
    return np.sqrt(mse(f,y))
```

## Mean percentage error

Mean percentage error is the average of percentage errors by which each forecast differs from corresponding real observed values.

![[formula-mpe.png]]

This statistic is easy to understand because it provides the error in terms of percentages. As in the ME, positive and negative forecast errors can offset each other, so it can be used to measure bias in the forecasts. The disadvantage of this statistic is that it is not suitable for datasets containing observed values, which are equal to zero.

```python
def mpe(f, y):
    f = f.reset_index(drop=True).values.flatten()
    y = y.reset_index(drop=True).values.flatten()
    df = pd.DataFrame({'f_i':f, 'y_i': y})
    df['e'] = df['y_i'] - df['f_i']
    df['pe'] = 100*(df['e']/df['y_i'])
    return np.mean(df['pe'])
```

## Mean absolute percentage error

The mean absolute percentage error overcomes the problem with error offsetting and works best if there are no extremes to the data (and no zeros).

![[formula-mape.png]]

```python
def mape(f, y):
    f = f.reset_index(drop=True).values.flatten()
    y = y.reset_index(drop=True).values.flatten()
    df = pd.DataFrame({'f_i':f, 'y_i': y})
    df['e'] = df['y_i'] - df['f_i']
    df['ape'] = 100*np.abs(df['e']/df['y_i'])
    return np.mean(df['ape'])
```

## Theil's U statistics

There is some confusion about Theil's forecast accuracy coefficient, caused probably by Theil himself. He proposed two different formulas at different times under the same name, both labelled with U. More about this topic you find in Forecast Evaluation using Theil's Inequality Coefficients.

### U₁

The values of U₁ are in the range (0, 1).

![[formula-u1.png]]

The greater accuracy of the forecast, the lower will be the value of the U₁.

```python
def u1(f,y):
    y = y.reset_index(drop=True).values.flatten()
    f = f.reset_index(drop=True).values.flatten()
    df = pd.DataFrame({'f_i':f, 'y_i': y})
    df['(f_i - y_i)^2'] = np.square(df['f_i'] - df['y_i'])
    df['y_i^2'] = np.square(df['y_i'])
    df['f_i^2'] = np.square(df['f_i'])
    return (np.sqrt(np.mean(df['(f_i - y_i)^2'])))/(np.sqrt(np.mean(df['y_i^2']))+np.sqrt(np.mean(df['f_i^2'])))
```

### U₂

Theil's U₂ tells how much more (or less) accurate a model is relative to a naïve forecast.

![[formula-u2.png]]

U₂ has a lower bound of 0 (which indicates perfect forecast), hasn't an upper limit. When the value of U₂ thing exceeds 1, it means that the forecast method becomes doing worse than naive forecasting.

![[fig-interpreting-u2.png|*Interpreting Theil's U₂*]]

```python
def u2(f,y):
    y = y.reset_index(drop=True).values.flatten()
    f = f.reset_index(drop=True).values.flatten()
    df = pd.DataFrame({'f_i+1':f, 'y_i+1': y})
    df['y_i'] = df['y_i+1'].shift(periods=1)
    df['numerator'] = np.square((df['f_i+1'] - df['y_i+1']) / df['y_i'])
    df['denominator'] = np.square((df['y_i+1'] - df['y_i']) / df['y_i'])
    df.dropna(inplace=True)
    return np.sqrt(np.sum(df['numerator'])/np.sum(df['denominator']))
```

## Model evaluation

Let's go back to our dataset. It contains the results of 4 different predictive models.

![[fig-model-results-1.png]]

![[fig-model-results-2.png]]

The table below contains all the statistics described in this article, calculated for each model.

![[fig-stats-table.png|*The specific information provided by each statistics is clear. For example, Advanced model #1 is not much better than a simpler moving average model in terms of U₂. However, the moving average's mean error is further from zero, which suggests a bias in the forecast.*]]

It is worth noting that the advanced method has a square error of an order of magnitude smaller than any other. It is also a top performer in other statistics.

## Conclusions

If the main indicator of success was the ability of the model to inform about sudden changes in the predicted variable, then "Advanced #2" will be the preferred method. This is consistent with the intuition and visual evaluation of the plot. However, this method is the most complex in terms of calculation and requires the use of external variables. But in such an important domain of the problem as electricity generation, this is an acceptable cost.

The "Advanced #1" and moving averages methods, on the other hand, are suitable for rough estimates. The above conclusions would be impossible to draw without looking at alternative statistics, which shows that no forecast evaluation statistic is redundant as each has information to impart.

I used the described statistics in practice when creating models:

> **[How to predict solar energy production](https://fischerbach.medium.com/how-to-predict-solar-energy-production-887ce31ec9d1)** — Efficient use of renewable energy sources with machine learning
>
> **How to Forecast Website Traffic** — Local Pageview Projection Tool using Weather Data

## References

- http://www.treasury.act.gov.au/documents/Forecasting%20Accuracy%20-%20ACT%20Budget.pdf
- https://www.economicsnetwork.ac.uk/showcase/cook_forecast
- https://www.jstor.org/stable/2352722?seq=1
- https://stackoverflow.com/questions/54931514/theils-u-1-theils-u-2-forecast-coefficient-formula-in-python
- https://stats.stackexchange.com/questions/345178/interpretation-of-theils-u2-statistic-forecasting-methods-and-applications
- https://docs.oracle.com/cd/E40248_01/epm.1112/cb_statistical/frameset.htm?ch07s02s03s04.html
- https://www.statisticshowto.com/mean-error/
- https://en.wikipedia.org/wiki/Mean_percentage_error
- https://arxiv.org/abs/1905.11744

---

# Conjoint analysis: Discrete choice modelling in market research
_2020-11-20_

# Conjoint analysis: Discrete choice modelling in market research

### Choice-Based Conjoint Analysis

![[hero.png|*In choice-based conjoint analysis, a set of products is presented to consumers in a similar manner to the real marketplace situation. They decide which one is the most attractive for them. (fig. by author)*|450]]

In the [previous article](https://towardsdatascience.com/how-to-develop-perfect-product-using-conjoint-analysis-1c2d9e4beb5d), I introduced a conjoint analysis and provided some examples of how useful the market research method is. I recommend you to read it first.

Choice-based conjoint analysis (CBC, or: discrete choice modelling, discrete choice experiment, experimental choice analysis, quantal choice models) uses discrete choice models to collect consumer preferences. The main difference distinguishing choice-based conjoint analysis from [the traditional full-profile approach](https://netlabe.com/how-to-develop-perfect-product-using-conjoint-analysis-1c2d9e4beb5d) is that the respondent expresses preferences by choosing a profile from a set of profiles, rather than by just rating or ranking them. The basic idea of choice-based conjoint analysis is to simulate a situation of real market choice.

After reading this article, you will know:

- what are the uses of choice-based conjoint analysis,
- how to design experiment,
- how to analyse collected data.

## Design of experiment

![[fig-choice-between-products.png|*The choice between different products. Products are described as sets of different combinations of attribute levels — profiles. (fig. by author)*]]

In this method, a set of profiles is presented to respondents and they decide which one is, for various reasons, the most attractive for them. You simply ask respondents to choose the most attractive (preferred) profile from a set of alternatives.

A nice example of a well-designed choice-based conjoint survey you find [here](https://websitedemos.sawtoothsoftware.com/cbc-baseball/). Authors, Sawtooth Software, provide professional software tools for conjoint analysis.

## Applications

So, when you want to develop a new or modify an already existing product, choice-based approach flexibility of configuration is preferred over other conjoint methods. In general, choice-based conjoint analysis is used to measure preferences (e.g. attribute importance), and the willingness to pay for products and services. By asking respondents to choose the most preferred profile, CBC forces them to make trade-off decisions between different products in a competitive, similar to the real market, environment.

This approach enables you to find out how to purchase likelihood is influenced by various product attributes and their levels (values).

### Pricing

For example, you can find what is the optimal price for a new product.

![[fig-amp-price.png|*In this scenario, the optimal price for the energy drink AMP is $1.39, where the purchase probability is highest at 15%. ([source](https://www.quantilope.com/en/method-choice-based-conjoint-analysis))*]]

Or what attributes have the greatest influence on consumers willingness to pay a premium price?

![[fig-glass-bottle.png|*Between a can, glass bottle, and plastic bottle, consumers are willing to pay more for a glass bottle when purchasing an energy drink. ([source](https://www.quantilope.com/en/method-choice-based-conjoint-analysis))*]]

### Product optimization

You can also, as in most conjoints, find out which product features have the greatest impact on consumers' purchase decisions.

![[fig-attribute-importance.png|*Product price (42%) and brand (33%) have the greatest impact, accounting for 75% of a consumer's purchase decision of energy drinks. ([source](https://www.quantilope.com/en/method-choice-based-conjoint-analysis))*]]

![[fig-sample-task.png|*Sample Choice-Based Conjoint Analysis Task ([source](https://www.quantilope.com/en/method-choice-based-conjoint-analysis))*]]

Other problems that can be studied using CBC:

- How to combine features to create the best product?
- Which products alternatives could be sold for the best price?
- How important is each attribute in the matter of purchasing decision?
- How do different features compare to others?
- How sensitive is the price to changes in levels of attributes?

As you can see, you can use CBC in multi-attribute studies or in complex scenarios of purchasing paths for a better representation of real situations.

## Advantages

So, choice-based conjoint analysis is a great tool for market simulation. By selecting one of the proposed variants of the product, respondents simultaneously and unknowingly evaluate the attributes that characterize the profiles.

Indeed, respondents make a simultaneous assessment of all attributes, as in the case with actual market decisions. In traditional conjoint analysis methods respondent assesses the attributes in pairs in isolation from other parameters. This leads to the under- or overestimation of the importance of certain attributes, especially such specific attributes as the price or brand. In contrast, the choice-based conjoint analysis gives you the ability to obtain more realistic estimates of the value (significance) of individual attributes that respondents are associated with their chosen attribute levels.

Another advantage of a choice-based approach over traditional conjoints is the ability to learn which attribute values or their combinations may discourage the consumer from buying any of the products available on the market.

Depending on the problem studied, respondents have or not a possibility to refrain from choosing, e.g. by selecting "none" when no profile meets their expectations. When you will have to decide whether to give that possibility to the respondent or not, you should take into consideration the best resemblance to the situation on the real marketplace.

![[fig-laptops-cars.png|*Things like laptops or cars are more serious and thoughtful purchases than groceries. (fig. by author)*]]

Consumers in case of lack of perfect alternative more likely would refrain from purchasing smartphone (e.g. because they have still working old device) than wine (e.g. because they invited friends for dinner). The utility of a combination of attributes that is not chosen is a threshold value that should be taken into account when defining a new profile that is acceptable to the potential buyer.

CBC can also measure the main effects and interactions between them. CBC is more effective than full-profile in profile assessment because it requires less effort from respondents. This requires a smaller number of decisions from respondents than the traditional conjoint analysis method.

## Disadvantages

But like any method, the CBC has limitations. The choice procedure results in less informative data than the ranking or rating assessment procedures. A choice-based experiment requires the collection of a large number of observations in order to obtain reliable parameter estimators. Therefore, the costs of such an experiment may be higher than the costs of an experiment carried out for traditional conjoint analysis.

![[fig-large-sample.png|*CBC requires the collection of a large number of observations in order to obtain reliable parameter estimators. (fig. by author)*]]

Algorithms required to analyse collected data are also more sophisticated. Depending on the design of a particular experiment, it may be difficult to achieve a reliable utility function in the continuous field of attribute levels.

Another disadvantage of this type of conjoint analysis is that standard estimation methods only allow for modelling at the aggregate level. The data collected as a result of a choice-based experiment does not allow the estimation of separate utility models (part-worth utilities) for each of the respondents on an individual level. However, as we will show later in the case study, you can segment the market and estimate part-worth utilities for each segment of consumers at least.

Discrete choice procedure in comparison with a ranking or positional assessment procedure leads to the collection of data of lesser informative value. Especially, if you include too many parameters displayed at the same time, the respondent will have to mentally process a large amount of information. This leads to an effort that is disproportionate to the added value and higher costs of conducting the study. So remember, you should only include a limited number of attributes and their levels to avoid respondents' information overload.

Choice-based conjoint analysis is not adaptive by design. In the case of a large number of attributes or their values, a correspondingly larger sample must be collected. Then you should consider using adaptive methods such as adaptive choice-based conjoint analysis or hybrid methods.

Note: CBC tests products that are fixed. If the consumer can customize the product, consider creating a menu-based study.

## Data analysis

The process of choosing between profiles is probabilistic, as consumers do not always act in a predictable and consistent manner. This means that the consumer, under the same conditions and from the same set of profiles, can make different choices at different times. That's why choice-based conjoint analysis shares assumptions with random utility theory. In random utility theory, we assume that people generally choose what they prefer, and when they do not, this can be explained by random factors.

So, let's propose a random utility function with deterministic and random components.

![[formula-utility.png]]

![[formula-utility-2.png]]

The random component has a very precise meaning. It is a source of inconsistencies in the choices of the consumer over time and must not be explainable by other factors. It could be the result of the actual emotional state of the consumer, his or her special needs at this particular time. However, if you could propose a model for these needs, this won't be a random phenomenon.

Next, we can propose a linear model for random utility:

![[formula-linear.png]]

![[formula-linear-2.png]]

An assumption in aggregate-level models is the homogeneity of parameters. The parameters representing the average value for the population. For the estimation of model parameters, a specific distribution of the random component is assumed, which leads to different probabilistic models. Most often it is assumed that the random component has a normal or Gumbel distribution. Therefore a binary probit model or a polynomial logit model is obtained accordingly.

Although the possibility of heterogeneous preferences among the population is ignored in aggregate-level models, there are methods for using choice-based conjoint analysis to segment consumers using additional data. For example, sympathy for anchovy is not normally bell-shaped distributed. Rather than that, distribution has two "humps", reflecting the overlapping of two very different populations: people who like anchovy and who don't.

As you will see in the example study, you can split consumers into segments that have different part-worth utilities.

### Hierarchical Bayes estimation

Although aggregate-level estimation of preferences is sufficient in forecasting the market share of a new product, in many situations, it is still desirable to obtain estimates of every individual consumer's preference structure. From data collected by choice-based conjoint experiment part-worths at the individual level cannot directly be estimated. It's because the dataset is too sparse. But you can Hierarchical Bayes methods in post-processing to recover individual preference heterogeneity even with insufficient degrees of freedom.

[More about HB.](https://sawtoothsoftware.com/resources/technical-papers/hierarchical-bayes-why-all-the-attention)

## Case study

Let's analyze the example study from "[Using cluster analysis and choice-based conjoint in research on consumers preferences towards animal origin food products. Theoretical review, results and recommendations](http://www.ighz.edu.pl/uploaded/FSiBundleContentBlockBundleEntityTranslatableBlockTranslatableFilesElement/filePath/1129/str171-184.pdf)".

### The business problem

Consumers are becoming more aware of food of animal origin. They shift their interests towards products that are safe, nutritious, produced through ethical and environment-friendly methods. The aim of the study is to determine which characteristics of the product (eggs) are of most importance to the consumer.

Note: in the original study, there is also an important analysis of methods of market segmentation. Market segmentation is beyond the scope of this article, but I recommend that you familiarize yourself with the methods described in the source study.

### Attributes

Attributes selected to further research are a farming method, hen breed, nutrition claims, egg size, package size and price. Their levels (values) are described in the table below.

![[fig-egg-attributes.png|*Attributes and levels used in the conjoint survey design. ([source](http://www.ighz.edu.pl/uploaded/FSiBundleContentBlockBundleEntityTranslatableBlockTranslatableFilesElement/filePath/1129/str171-184.pdf))*]]

Attributes and levels were selected after reviewing previous studies on consumer preferences and by direct assessment of their importance by the research team.

### Experiment design

Each respondent saw similar screens (with 3 different products at a time) with all the attributes defined in accordance with the established levels (presented in Tab. 1) and had to choose one of them. Each respondent saw a dozen screens with the question "Which product would you choose?".

![[fig-egg-experiment.png|*Respondents choose from 3 different alternatives of product. (fig. by author)*]]

Every screen contains 3 different profiles and respondents had to choose one of them. Importantly, there was no "none of those" option. As the authors of the study argue, this is similar to the real situation, when a person goes shopping and wants to buy eggs. Usually, he or she is forced to choose from what is available on the shelf and rather buy anything, than refrain from buying eggs.

The questionnaire contained choice-based questions, socio-demographic questions and questions about food selection habits, nutritional beliefs and preferences.

Examples of additional questions:

- How often do you buy organic eggs?
- Information on the packaging is very important to me.
- I need to know what the product contains.
- Organic eggs are better than non-organic eggs.

The scale was 1–7, where 1 means "I strongly disagree…" and 7 means "I strongly agree…".

### Data collecting

Answers from nearly 1000 respondents aged 21+ were collected using Computer Assisted Personal Interviewing (CAPI). The sample was selected to be representative of the polish population for region, age and gender.

### Analysis

After collecting data, [Hierarchical Bayesian networks](http://www.aurelielemmens.com/wp-content/uploads/2019/04/Conjoint_HB_2018_shortened.pdf) are used to analyze it.

> K-means clustering algorithm. The aim of the K-means algorithm is to divide M-points in N-dimensions into K-clusters in order to minimize the within-cluster sum of squares. We seek "local" optima solutions so that no movement of a point from one cluster to another will reduce the within-cluster sum of squares.

A detailed statistical algorithm is described e.g. [here](https://en.wikipedia.org/wiki/K-means_clustering) and [here](https://towardsdatascience.com/understanding-k-means-clustering-in-machine-learning-6a6e67336aa1).

![[fig-kmeans-clusters.png|*K-means clustering algorithm: the part-worth utilities with the relative importance of attributes for 4 identified clusters. ([source](http://www.ighz.edu.pl/uploaded/FSiBundleContentBlockBundleEntityTranslatableBlockTranslatableFilesElement/filePath/1129/str171-184.pdf))*]]

### Results

The most important attributes were "price" and "farming method". Other ("breed", "nutrition claims", "size", and "package") were defined as less important but were taken into consideration later on. Regarding mean relative importance, there are two clusters focused on price (Cluster 1 — RI — 59% and Cluster 3 — RI — 53%) whereas Cluster 4 does not perceive the price as the only important egg attribute (RI — 39%). It can be seen that segments that consider "price" as extremely important pay less attention to attributes related to animal welfare.

## Takeaways

As you can see, choice-based conjoint analysis is a useful tool. Furthermore, in combination with other methods, like clustering algorithms, it can circumvent some of its limits.

## References

- https://www.slideshare.net/surveyanalytics/webinar-a-beginners-guide-to-choicebased-conjoint-analysis
- https://digitalcommons.lsu.edu/cgi/viewcontent.cgi?article=2685&context=gradschool_dissertations
- https://help.xlstat.com/s/article/choice-based-conjoint-cbc-in-excel-tutorial?language=en_US
- https://www.quantilope.com/en/method-choice-based-conjoint-analysis
- https://www.researchgate.net/publication/23505678_A_HIERARCHICAL_BAYES_APPROACH_TO_MODELING_CHOICE_DATA_A_STUDY_OF_WETLAND_RESTORATION_PROGRAMS
- https://docs.displayr.com/wiki/Random_Utility_Theory

---

# Conjoint analysis: How to develop the perfect product
_2020-11-09_

# Conjoint analysis: How to develop the perfect product

### Make data your unfair competitive advantage

![[hero.png|*Conjoint analysis is based on the idea the relative attributes and their levels considered jointly can be measured better than when considered in isolation. (fig. by author)*|450]]

Conjoint analysis is a market research method used to measure customer preferences and the importance of various attributes of products or services. In this method, products or services (real or hypothetical) are presented to respondents (e.g. potential consumers) as a set of profiles. Each profile is described by attributes and their levels. Those attributes and their levels are explanatory variables. On the basis of the collected assessments (preferences) from consumers, a breakdown (decomposition) of total preferences is made, using statistical methods, by calculating the share of each attribute in the estimated total utility of the profile.

This way, marketers can identify which combination of attributes is going to perform on the market or which will be the most sought after combination in terms of the consumer's point of view before actually introducing a new product.

Fundamental for conjoint analysis is the idea that attributes (and their levels) of the product can increase or decrease consumer's willingness to buy it. Conjoint analysis measures how people value each attribute of a product.

Conjoint analysis is useful in all areas that involve human decision making, like:

- Marketing
- Product development
- Human resources
- Politics

This article focuses on an introduction to conjoint analysis. After reading this, you will know what are the applications, types and advantages of conjoint analysis. In future posts, we will discover conjoint analysis using some methods from data science stack.

![[fig-multitude-products.png|*You probably faced the situation where you have to decide which product to buy among a multitude of others. Do you remember how important each attribute of the product was for your final decision? (pic. by [Victoriano Izquierdo](https://unsplash.com/@victoriano))*]]

Products or services have multiple features and attributes these days.

Some product or services, like laptops, cars, financial service, smartphones, professional equipment have an extensive list of features and functionalities. Things like fast-moving consumer goods (FMCG) are more homogeneous and they are mainly differentiated by brand, packaging, and prices.

It is essential for the business to know what factors are causing customers to choose your product.

Imagine you are working on the product line, which will be added to your company's portfolio. How would you determine product attributes and features that are important for your customers? Or what functionalities are customers focused on when making their purchase decision?

In other words, before you introduce a new product or service, you have to determine what is your consumer's ideal product, and how big will be demand for it. If you understand how people make choices, you can create better alternatives and try to predict their choices.

Conjoint analysis can help you obtain answers for those and more questions.

## Applications of conjoint analysis

Conjoint analysis can provide insights into vast business problems, such as:

- Understanding consumers
- Concept testing
- Product evaluation
- Market simulation
- Segmentation of the market

![[fig-applications.png]]

Using the conjoint analysis, you can find the answer to the following questions:

- What feature or attribute of a product is most influential in terms of market success?
- What are consumers focused on when making their purchase decisions?
- How attributes influence price?
- What is the elasticity of demand by attributes?
- How sensitive are consumers to price shifting?
- What trade-offs do consumers make?
- How does the modification of the product affect demand for it?
- What is the optimal combination of features?

As you can see, conjoint analysis is a powerful and realistic tool for market research, which can provide insights into a wide array of business objectives. And this everything is possible with a few product-related questions.

## Let's ask customers

Ok, but why can't we simply ask consumers what attributes they value most?

In a classical approach, we would first ask consumers to rate each attribute and then conclude which ones have the most impact on individual preferences. The disadvantage of this method is that simply consumer answering everything is important.

You can see this in results, where average scores of each attribute are not variable enough to implicate meaningful conclusions. For example, most people would like more features for a lower price. Isn't enlightening, huh?

Furthermore, some attributes are mutually exclusive. For example, fuel consumption is an important characteristic, if you are buying a car. It is obvious that the same car can't have the highest fuel efficiency and engine power simultaneously.

However, we want to know how much customers want each feature and how much they would pay for it. Although in the real world we can't afford to have absolutely everything, as consumers, we have to make trade-offs between various attributes of a product.

The trade-off is a compromise achieved between having two desirable but incompatible features.

Conjoint analysis tries to replicate this behaviour and doesn't allow people to simply say "everything is important". We ask for ratings or decisions between profiles of products — bundles containing some of the key attributes of a product, and then predict the most likely behaviour of the consumer.

In other words — calculate the most likely utility function for each consumer and consumers as a whole. (More about utility functions in the next posts.)

This is the main factor that sets the conjoint analysis apart from classical decision methods. Conjoint analysis asks the question "which product would you pick?", while classical methods ask "how good is this product?" or "is that feature important for you?".

![[fig-heuristics.png|*Customers employ a variety of heuristics when evaluating product alternatives and choosing in the marketplace. (fig. by author)*]]

Using conjoint analysis, you can understand how customers make their choices, what trade-offs people would make when given different product features and different levels of its attributes and how attributes interact with each other.

## Example of conjoint analysis

To illustrate how simple and robust is basic conjoint analysis, let's do some as an exercise. We will conduct one of the traditional types of conjoints — Full-Profile Conjoint Analysis. It is relatively simple to demonstrate.

There are other types, like Adaptive Conjoint Analysis (ACA), which is generally more suitable for larger problems.

See also my article about Choice-based Conjoint Analysis:

> [Conjoint Analysis: Using discrete choice modelling in market research](https://fischerbach.medium.com/choice-based-conjoint-analysis-dafcff135c2)

### The business problem

Watermelon LLC — newly funded producer of smartphones asks for your help in understanding their market environment.

![[fig-watermelon.png|*Watermelon LLC wants to make a phone that will conquer the market. (fig. by author)*]]

They want you to know what features and functionalities they should focus on in the first place when they are planning their portfolio of the devices.

They want to know:

- What functionalities of a smartphone are most influential in terms of market success?
- What do consumers focus on when making their purchase decisions?
- How does each attribute influence price?

### Attributes

![[fig-attributes-independent.png|*Attributes should be independent, mutually exclusive. Brand, quality and product life expectancy may all measure the same thing. (fig. by author)*]]

After a conversation with their product manager, you choose these attributes and levels:

Available memory:

- 32 GB
- 64 GB

Screen size:

- 5''
- 6''

5G-capability:

- YES
- NO

Price:

- €999
- €1199

Rather than educated guesses, it's usually a good idea to put some extra effort into qualitative research. One of the possibilities is focus groups.

As we have only 4 attributes to explore, Full-Profile is an excellent approach. It is especially useful when the number of attributes is no more than 6 and thus it is relatively simple to demonstrate.

Attributes are independent aspects of a product or a service (brand, price, size, colour, etc.). Each attribute has varying degrees, or "levels". Another advantage of having fewer attributes is that too many features may cause respondents to simplify looking only at 2–3 most important.

### Profiles generation

The Full-Profile factorial experiment takes into account all combinations of levels of individual factors, so-called profiles. It is mean that you should generate profiles as every possible combination of levels of attributes.

Most major statistical software has the functionality of designing such experiments.

![[fig-profiles-list.png|*List of profiles (tab. by author)*]]

### Data collecting

To collect market research data by asking potential consumers to rank generated previously profiles, you can:

- survey your clients through an online questionnaire,
- show them printed cards of each profile.

Sidenote: modern approaches of conjoint analysis also include the possibility to collect actual choice data, e.g. data where the person actually makes a decision between several alternatives.

![[fig-profiles-ranking.png|*Ranking of profiles (tab. by author)*]]

### Building a regression model

Once we have a set of values of the dependent variable, we can try to fit the model for data. We will use multiple regressions for this.

Assuming that the consumer uses some internal additive point system to evaluate the overall attractiveness of each profile, we can introduce a simple linear model.

![[formula-utility.png|*Utility function as linear combinations of part-worth utilities of each attribute (eq. by author)*]]

Parameters of the model, which we call part-worths, are numerical scores that represent how much each attribute influences the consumer's decision to choose a particular profile. We use the Ordinary Least Squares method to calculate each b-parameter's numerical value.

Available input:

- Attributes
- Levels
- Respondents

Output of our model:

- Part-worth utilities (b-parameters) for each level
- Importance scores for each attribute
- Ability to perform simulations

![[fig-explanatory-vars.png|*Set of explanatory variables values. (fig. by author)*]]

![[fig-dependent-var.png|*Set of dependent variable values. (fig. by author)*]]

![[fig-linest.png|*Regression can be made using Google Sheets function LINEST. (fig. by Author)*]]

![[fig-regression-results.png|*Results of regression — price has a major influence on purchase decisions. (fig. by author)*]]

Sidenote: In the real world scenario you should obtain answers from many respondents and then use means of a ranking of each profile.

## Conclusions

As you can see in the results, one attribute has a major influence on purchase decisions. It is a price and it is not surprising. (Of course, answers are intentionally biased for this example.)

Let's repeat regression without considering price as an attribute.

![[fig-without-price.png|*Without taking into account the price, the available memory is most important for this respondent. (fig. by author)*]]

Now, memory is the number one attribute of this respondent.

We know what to recommend to Watermelon LLC already:

- They should focus on making a cheap smartphone with large available memory.
- The second most important feature of a smartphone is screen size.
- 5G-capability is not crucial.

I prepared an interactive website with Full-Profile Conjoint Analysis: <https://fischerbach.github.io/full_profile_example/>

You can try it yourself and check how the profiles' ranking influences the calculated importance of attributes.

## Takeaways

I hope you understand now what powerful tools conjoint analysis is. Rather than ask directly your consumer about their preferred attributes of the product, you survey them using more realistic questions about their product-focused preferences. When consumers are forced to make difficult trade-offs, we learn what they truly value.

## References

- https://sawtoothsoftware.com/academics/teaching-resources
- http://www.dobney.com/Conjoint/conjoint_simple.htm