Polar bear mother and cup in the Arctic ocean

Automatic Marine Mammal Mitigation by Infrared Imaging

Use of loud seismic airguns and naval sonars in marine surveys is feared to potentially injure whales and seals when nearby these activities.  It is therefore commonly requested to shut-down such operations when marine mammals are sighted within a predefined exclusion zone.  Monitoring such exclusion zone for marine mammals is usually done by visual marine mammal observers.  However, night-time, reduced visibility (rough seas, fog, rain, etc.) and observer fatigue or distraction all limit the reliability of visual monitoring. To facilitate monitoring 24/7 with unwavering vigilance, our work aims at developing an automatic whale detection system utilizing a commercial, cooled 360° thermal imaging sensor.

After more than 10 years of research ashore and at sea, we captured thousands of whale cues (mostly blows and bodily displays) on video and analysed the data for dependency on environmental conditions and specificity. We established that IR imaging will work worldwide in any climate zone (rather than, as initially suspected, in polar and subpolar regions only), 24/7 and for months in a row, except for some rare and transient environmental conditions (fog being the major impediment).

Based on this comprehensive  operational at-sea experience, we reframed our Tashtego software and are now in the final stages of finishing the most performant automatic detection system ever, Tashtego PRO, with all the bells and whistles you can imagine.  It allows for instant replay of any detection, localization of detected events, automatic adjustment to the ships superstructure, auto focus, auto contrast, and on and on…

Projects


2013 -

Bundesministerium für Bildung Forschung und Technologie funded project (BMBF FKZ 03F0479I) funded project:  MAPS-IR: Development, Validation und First Use of an IR-Imaging based System for automatic Detection of whales in the environs of ships and platforms. 

2015-2016

Bundesministerium für Bildung Forschung und Technologie funded project (BMBF FKZ 03F0725A):  THETA: Thermography for the benefit of animal and species protection: automatic detections of whales, seals and walruses in warm and cold seas.

2014-2016

Office of Naval Research (ONR Award Number: N000141310856 ) funded project: ETWW: Exploring the thermal limits of IR-based automatic whale detection

 

 

2015-2018

Canadian Environmental Studies Research Fund (ESRF) funded project: Assessing the quality of marine mammal detections using three complementary methods.

2019-2020

AWI funded project: Tashtego PRO

Milestones

 

Project Idea

ANT XX

Proof of concept during Polarstern expedition PS …. using hand-held IR camera (uncooled Bolometer)

ANT XX

Testing detection ranges with passively gimballed directional FLIR cameras A40 (uncooled Bolometer)

 

Acquisition of FIRST Navy cooled and actively gimballed IR sensor

 

beta-version of Whale detection software “Tashtego”

ANT xx

 

First trial of FIRST-Navy

Gohl reis emit Daniel

 

The publication demonstrates the efficiency of thermal imaging based automatic whale detection in waters cooler than 10°C.

 

 

Peer reviewed publications / patents

2013

Peer reviewed publication:  Automatic Round-the-Clock Detection of Whales for Mitigation from Underwater Noise Impacts. 

2015

United States Patent No. US 8,941,728 B2, January 27, 2025: Method for automated real-time acquisition of marine mammals

2020

Peer reviewed publication: Scaling the Laws of Thermal Imaging–Based Whale Detection, J-Tech

2020

Peer reviewed publication: HR. Smith, DP. Zitterbart, TF. Norris, MFlau, EL. Ferguson, CG. Jones, O Boebel, VD. Moulton, A field comparison of marine mammal detections via visual, acoustic, and infrared (IR) imaging methods offshore Atlantic Canada, Marine Pollution Bulletin,Volume 154,2020, 111026, ISSN 0025-326X, https://doi.org/10.1016/j.marpolbul.2020.111026.

(http://www.sciencedirect.com/science/article/pii/S0025326X20301442)

Field Work

ANT 21.2

Dec 2003 - Jan 2004

~1h operational

6 blows captured

9MB data saved

Use of a hand-held IR camera (uncooled bolometer, 304x224 pixels, 30 fps) to explore if whale blows can be recognized in IR video.

(+) Whale blows are clearly discernable against polar   background within a few hundred meters.

(-) Handheld system is highly unstable and unsuitable for long-term marine mammal monitoring.

ANT 22.3

Jan 2005 - Apr 2005

Deployment of two directional FLIR ThermoVision A40 cameras (uncooled bolometers, 320x240 pixels, horizontal Field of View 24° each, 25 fps) mounted in fixed position on monkey deck of RV Polarstern.  First try of custom developed recording and detection software (provided by external company).

(+) 41 blows recorded, 8 automatically. 

(-) Unheated housing with "windshield wipers" which regularly feeze over.  Ship's pitch and roll causes instable vide due to fixed mounting.  Software revision too expensive, development discontinued.

ANT 23.2 Polarstern

Nov 2005 - Dec 2005

Use of same cameras facing strictly forward, but mounted under roof of Polarstern's crow's nest and passively gimballed against pitch.  Custom software develped by U of Chemnitz, forming the start to later software developments.

(+) Slightly heated housing inhibits freeze-up of lense.

(-) Limited FoV causes missing of blows occurring athwardship and aft.  Passive gimbal subject to high winds in crows nest, misdirecting camera downwards.  Passive gimbal subject to swinging unless damped, which, however, would require complex mechanic subject to freezing. Passive gimballing diemscontinued.

ANT 23.6

Jun 2006 - Aug 2006

Setup as above, but taking the system to the austral winter. 

(+) Camera setup endured several storms up to Bft 12.

(-) Ice build up on protective Germanium lense, to be removed manually ethanol (98

ANT 24.3 Polarstern

Feb - Apr 2008

360h operational

80 GB data collected

37 encounters capured

Operation of 2 FLIR ThermoVision A40 cameras with 7° and 12° lense mounted in crows nest.  Camears can be rotated in directions of interest.

(+) Minke whales, which tend to keep greater distance to ship and exhibit weaker blows captured with high resolution lenses at distance up to 1km.

(-) Directional system requires manual pointing of cameras or a ring of 30 cameras to provide full circumferential monitoring.

ARK 24.1

2009

400h operational

2 TB data collected

Acceptance test of newly acquired FIRST-Navy inrared imager: 7200 x 586 pixels, horizontal FOV 360° at 5 Hz, cooled line sensor. Gimbal and sensor head mounted in crow's nest of RV Polarstern.

(+) system operational for entire expedition.  Whale blows perceptible in IR images up to 3 km distance.

(-) occasional stutter video stream

ANT 26.3

Jan 2010 - Apr 2010

436h operational

2.7 TB data

Rewrite of detectiong kernel to facilitate real-time processing and modification of GUI to facilitate use by MMOs. Implementation of hot-pluggable storage media. Ship- and geo-referenced mapping of blow positions established.

(+) 19 of 20 encounters (each comprising several blows) detected by new kernel

(-) Failure of sensor head due to damaged mechanical slip-ring

ANT 27.1

Oct 2010 - Nov 2010

690 h operational

Operation of FIRST-Navy during Polarstern's transit from Bremerhaven to Cape Town.  Implementation of a picture-in-picture visual camera system.

(+) Two ship-whale encounters recorders.

(-) Problems with synchronization of rotation of sensor head and image acquisition.  Failure of sterling cooler after 15 days.

ANT 27.2

2010 2011

1410 h operational

145 autodetections

Operation of FIRST-Navy in Southern Ocean (Weddell Sea) together with high-resolution visual camera controlled by IR system

Tashtego software developments: Implementaton of advanced horizon tracking, replacement of detection algorithm with more efficient system, thermal imager status information, Improved image acquisiton, interface to visual detections software Walog.

(+) Sterling cooler of sensor head achieved nearly 3 months lifetime before failure.

(-) false alerts triggered by swarms of birds and rapid motion of horizon in image

ANT 28.2

Dez 2011 - Jan 2012

751 h operational

123 TB data collected

Operation of FIRST-Navy infrared imager in Southern Ocean (Weddell Sea), also during darkness with autodetector.

Tashtego software developments: Introduction of dedicated bird classification layer to reduce false alerts.  F

(+) Continous operation for entire expedition. First night time autodetections of whale blows.

(-) (-) focus need occasional (4 times during expediton) adjustment. 

ANT 28.4

Mar - Apr 2012

586h operational

93 TB data collected

Operation of FIRST-Navy infrared imager in Southern Ocean (Weddell Sea)

Tashtego software developments: Introduction of dedicated bird classification layer to reduce false alerts.  F

(+) Continous operation for entire expedition.

(-) focus need occasional (4 times during expediton) adjustment. 

ARK 27.3

Aug - Oct 2012

1563h operational

Operation of FIRST-Navy infrared imager in Arctic Ocean.

(+) Successful operaton of sensor for entire expedtion without any member of the IR team on board.

(-)

ANT 29.2

Nov 2012 - Jan 2013

726h operational

10 TB data collected

>887 IR cues logged

Operation of FIRST-Navy infrared imager in Southern Ocean (Weddell Sea).

(+) Sensorhead #1 was operated for 5 months before failure of mechanical slip ring failure causes drift in image orientation. Tashtego runs quasi-continously for > 1months.

(-) Sensorhead #2 fails after 11 days of opertion

Stradbroke Island, AUS

Jun - Jul 2014

854h operational

 

 

 

Operation of FIRST-Navy thermal imager at coast of Stradborke Island, Australia.  Watere temperature 22°-24°C.

 

(+)

(-)

Nova Scotia, CAN

Kauaii, USA

Jan - Mar 2016

926h operational

>6000 cetacean cues autodetected

 

Operation of FIRST-Navy thermal imager on both, south and north coasts of Kauaii, USA.  Water temperature 26°C. Parallel operaton of directional IRCAM dual (MWIR and LWIR) band camera. 

(+) Porpoising spinner and common dolphins are clearly perceptible in IR footage.

(-) Stand-up paddlers along with recreational boating introduce new signals causing false positives, a hindrance which howerver is unlikely to occurr in off-shore seismic and renewable energy construction settings.

RV Leeway Odyssey

Jul - Aug 2017

 

 

Operation of FIRST-Navy infrared imager off south coast of Newfundland. 

(+) Continous operation of IR system during entire survey.

(-) The small (38m length) ship exhibited significant roll and pitch, exceeding the gimbals compensation limits and leading to poor (instable) video data impeding automatic whale detectons