Polar bear mother and cup in the Arctic ocean

Automatic Marine Mammal Mitigation by Infrared Imaging

Use of loud seismic airguns and naval sonars in marine surveys is feared to potentially injure whales and seals when nearby these activities.  It is therefore commonly requested to shut-down such operations when marine mammals are sighted within predefined exclusion zones.  Monitoring such exclusion zone for marine mammals is usually done visually by marine mammal observers.  However, night-time, reduced visibility (rough seas, fog, rain, etc.) or observer fatigue and distraction all limit the reliability of visual monitoring. To facilitate monitoring 24/7 with unwavering vigilance, our work aims at developing an automatic whale detection system utilizing a commercial, cooled 360° thermal imaging sensor.

After more than 10 years of research ashore and at sea, we captured thousands of whale cues (mostly blows and bodily displays) on video and analysed the data for dependency on environmental conditions and specificity. We established that IR imaging will work worldwide in any climate zone (rather than, as initially suspected, in polar and subpolar regions only), 24/7 and for months in a row, except for some rare and transient environmental conditions (fog being the major impediment).

Based on this comprehensive operational at-sea experience, we reframed our Tashtego software and are now in the final stages of finishing the most performant automatic detection system ever, Tashtego PRO, with all the bells and whistles you can imagine. It allows for instant replay of any detection, localization of detected events, automatic adjustment to the respective ship's superstructure, auto focus, auto contrast, and on and on…

False color video of two dolphins leading a humpback whale, swimming along the south coast of Kauai, HI, USA.  Recorded by the IRCAM GEMINIS 327k ML dual band (LWIR + MWIR) infrared camera at night on 29 Jan 2016 at 22:16:18 Local Time.

Projects

2013 -

Bundesministerium für Bildung Forschung und Technologie funded project (BMBF FKZ 03F0479I):  MAPS-IR: Development, Validation and First Use of an IR-Imaging based System for Automatic Detection of whales in the environs of ships and platforms

2015-2016

Bundesministerium für Bildung Forschung und Technologie funded project (BMBF FKZ 03F0725A):  THETA: Thermography for the benefit of animal and species protection: automatic detections of whales, seals and walruses in warm and cold seas.

2014-2016

Office of Naval Research (ONR Award Number: N000141310856 ) funded project: ETAW: Exploring the thermal limits of IR-based automatic whale detection.

2015-2018

Canadian Environmental Studies Research Fund (ESRF) funded project: Assessing the quality of marine mammal detections using three complementary methods.

2019-2020

AWI funded project: Tashtego PRO: Rerendering Tashtego.

Milestones

2003

Project idea

2004

Proof of concept using a rented hand-held IR camera (uncooled bolometer).

2005

Systematic acquisition of IR data with two passively gimballed directional FLIR Thermovision A40 cameras (uncooled Bolometer).  Trial version of detection software tests.

2009

Acquisition of FIRST Navy cooled and actively gimballed IR sensor.

2010

Beta-version of automatic whale detection software “Tashtego” tested.

2011

Software "Fedallah" for off-line analysis of IR data released for internal use.

2013

1st peer-reviewed publication demonstrates the efficiency of thermal imaging based automatic whale detection in waters cooler than 10°C.

2015

Patent on detection process details granted in US, EU and RUS.

2016 Software licensed to RDE for use with FIRST Navy sensor sales.
2020 2nd peer-reviewed publication compares visual, acoustic and infrared imaging based whale detection methods.
2020 3rd peer-reviewed publication explores autodetection in warm climates.

Peer reviewed publications / patents

2013

Peer reviewed publication:  Zitterbart, D. P. , Kindermann, L. , Burkhardt, E. and Boebel, O. (2013) Automatic Round-the-Clock Detection of Whales for Mitigation from Underwater Noise Impacts , PLoS ONE . doi:https://doi.org/10.1371/journal.pone.0071217 , hdl:10013/epic.41931

2015

United States Patent No. US 8,941,728 B2, January 27, 2015: Method for automated real-time acquisition of marine mammals.

2020

Peer reviewed publication:  Zitterbart, D. P. , Smith, H. R. , Flau, M. , Richter, S. , Burkhardt, E. , Beland, J. , Bennett, L. , Cammareri, A. , Davis, A. ,  Holst, M. , Lanfredi, C. ,  Michel, H. , Noad, M. , Owen, K. ,  Pacini, A. , Boebel, O. (2020)
Scaling the Laws of Thermal Imaging–Based Whale Detection, Journal of Atmospheric and Oceanic Technology, 37 (5), pp. 807-824 . doi:https://doi.org/10.1175/JTECH-D-19-0054.1 , hdl:10013/epic.6f13adf9-f6e1-4516-b5a8-bb0e9afbe2bf

2020

Peer reviewed publication:  Smith, H. R. , Zitterbart, D. P. , Norris, T. F. , Flau, M. , Ferguson, E. L. , Jones, C. G. , Boebel, O. and Moulton, V. D. (2020)
A field comparison of marine mammal detections via visual, acoustic, and infrared (IR) imaging methods offshore Atlantic Canada, Marine Pollution Bulletin, 154 , p. 111026. doi:https://doi.org/10.1016/j.marpolbul.2020.111026 , hdl:10013/epic.d95bad2c-ec8b-4eb3-bc43-83f6f43dc02e

Field Work

ANT 21.2

Dec 2003 - Jan 2004

~1h operational

6 blows captured

9MB data saved

Ship based use of a hand-held IR camera (uncooled bolometer, 304x224 pixels, 30 fps) to explore if whale blows can be recognized in IR video.

(+) Whale blows are clearly discernable against polar background within a few hundred meters.

(-) Handheld system is highly unstable and unsuitable for long-term marine mammal monitoring.

ANT 22.3

Jan 2005 - Apr 2005

Deployment of two directional FLIR ThermoVision A40 cameras (uncooled bolometers, 320x240 pixels, horizontal Field of View (FOV) 24° each, 25 fps) mounted in fixed positions on monkey deck of RV Polarstern.  First try of custom developed recording and detection software (provided by external company).

(+) 41 blows recorded, 8 automatically. 

(-) Unheated housing with "windshield wipers" which regularly feeze over.  Ship's pitch and roll causes unstable videos due to fixed mounting.  Software revision too expensive, development discontinued.

ANT 23.2 Polarstern

Nov 2005 - Dec 2005

Use of same cameras facing strictly forward, but mounted under roof of Polarstern's crow's nest and passively gimballed against pitch.  Custom software develped by U of Chemnitz, forming the start to later software developments.

(+) Slightly heated housing inhibits freeze-up of lense.

(-) Limited FOV causes missing of blows occurring athwartship and aft.  Passive gimbal subject to high winds in crows nest, misdirecting camera downwards.  Passive gimbal subject to swinging unless damped, which, however, would require complex mechanics subject to freezing. Passive gimballing discontinued.

ANT 23.6

Jun 2006 - Aug 2006

Setup as above, but taking the system to the austral winter. 

(+) Camera setup endured several storms up to Bft 12.

(-) Ice build up on protective Germanium lense, to be removed manually ethanol (89%).

ANT 24.3 Polarstern

Feb - Apr 2008

360h operational

80 GB data collected

37 encounters capured

Operation of 2 FLIR ThermoVision A40 cameras with 7° and 12° lense mounted in crows nest.  Cameras can be rotated in directions of interest.

(+) Minke whales, which tend to keep greater distance to ship and exhibit weaker blows, were captured with high resolution lenses at distance up to 1km.

(-) Directional system requires manual pointing of cameras or a ring of 30 cameras to provide full circumferential monitoring.

ARK 24.1

2009

400h operational

2 TB data collected

Acceptance test of newly acquired FIRST-Navy infrared imager: 7200 x 586 pixels, horizontal FOV 360° at 5 Hz, cooled line sensor. Gimbal and sensor head mounted in crow's nest of RV Polarstern.

(+) system operational for entire expedition.  Whale blows perceptible in IR images up to 3 km distance.

(-) occasional stutter video stream.

ANT 26.3

Jan 2010 - Apr 2010

436h operational

2.7 TB data

Rewrite of detection kernel software to facilitate real-time processing and modification of GUI to facilitate use by MMOs. Implementation of hot-pluggable storage media. Ship- and geo-referenced mapping of blow positions established.

(+) 19 of 20 encounters (each comprising several blows) detected by new kernel.

(-) Failure of sensor head due to damaged mechanical slip-ring.

ANT 27.1

Oct 2010 - Nov 2010

690h operational

Operation of FIRST-Navy during Polarstern's transit from Bremerhaven to Cape Town.  Implementation of a picture-in-picture visual camera system.

(+) Two ship-whale encounters recorders.

(-) Problems with synchronization of rotation of sensor head and image acquisition.  Failure of Sterling cooler after 15 days.

ANT 27.2

2010 2011

1410h operational

145 autodetections

Operation of FIRST-Navy infrared imager in the Southern Ocean (Weddell Sea) together with high-resolution visual camera controlled by IR system.

Tashtego software developments: Implementation of advanced horizon tracking, replacement of detection algorithm with more efficient system, infrared imager status information, Improved image acquisition, interface to visual detection software "Walog".

(+) Sterling cooler of sensor head achieved nearly 3 months lifetime before failure.

(-) false alerts triggered by flocks of birds and rapid motion of horizon in image.

ANT 28.2

Dez 2011 - Jan 2012

751h operational

123 TB data collected

Operation of FIRST-Navy infrared imager in the Southern Ocean (Weddell Sea), also during darkness with autodetector.

Tashtego software developments: Introduction of dedicated bird classification layer to reduce false alerts.

(+) Continuous operation for entire expedition. First night time autodetections of whale blows.

(-) Focus needs occasional (4 times during expedition) adjustment. 

ANT 28.4

Mar - Apr 2012

586h operational

93 TB data collected

Operation of FIRST-Navy infrared imager in the Southern Ocean (Weddell Sea).

Tashtego software developments: Introduction of dedicated bird classification layer to reduce false alerts.

(+) Continuous operation for entire expedition.

(-) Focus needs occasional (4 times during expedition) adjustment. 

ARK 27.3

Aug - Oct 2012

1563h operational

Operation of FIRST-Navy infrared imager in the Arctic Ocean.

(+) Successful operation of sensor for entire expedtion without any member of the IR team on board.

(-)

ANT 29.2

Nov 2012 - Jan 2013

726h operational

10 TB data collected

>887 IR cues logged

Operation of FIRST-Navy infrared imager in the Southern Ocean (Weddell Sea).

(+) Sensorhead #1 was operated for 5 months before failure of mechanical slip ring causing drift in image orientation. Tashtego runs quasi-continuously for > 1 month.

(-) Sensorhead #2 fails after 11 days of operation.

Stradbroke Island, AUS

Jun-Jul 2014

854h operational

 

 

 

Land based operation of FIRST-Navy infrared imager at coast of Stradbroke Island, Australia.  Water temperature 22°-24°C.

(+)

(-) Coastal setting introduces surf as part of image. For proper processing, this requires clipping of image. 

Cape Race, CAN

Jul-Aug 2015

146h operational

 

Land based FIRST-Navy infrared imager operated from a cliff at Cape Race, Newfoundland. Water temperatures around 16°C.

(+) 700 cetacean cues logged.

(-) Coastal birds introduce large numbers of false positives, a hindrance which is less prevalent in off-shore seismic and renewable energy construction settings.

Kauai, USA

Jan-Mar 2016

926h operational

> 6000 cetacean cues autodetected

 

Land based operation of FIRST-Navy infrared imager on both, south and north coasts of Kauai, HI, USA.  Water temperature 26°C. Parallel operation of directional IRCAM GEMINI 372k dual (MWIR and LWIR) band camera. 

(+) Porpoising spinner and common dolphins are clearly perceptible in IR footage.

(-) Stand-up paddlers and small boats bopping up and down on waves introduce new, long lasting thermal signatures causing false positives, a hindrance which however is unlikely to occurr in off-shore seismic and renewable energy construction settings.

Cape Race, CAN

Jul-Aug 2016

ca. 400h operational

 

Land based operation of FIRST-Navy infrared imager from a cliff at Cape Race, Newfoundland. Water temperatures around 16°C.

(+) 2nd stage classifier for birds implemented.

(-) Coastal birds introduce large numbers of false positives, a hindrance which is less prevalent in off-shore seismic and renewable energy construction settings.

Arctic Ocean

PS101

Sep-Oct 2016

Ship based operation of FIRST-Navy infrared imager on RV Polarstern.

(+) Operational use to support bridge-based polar bear watch.

(-) none.

RV Leeway Odyssey

Jun-Aug 2017

 

 

Ship based operation of FIRST-Navy infrared imager off south coast of Newfoundland.  New generation featuring an athermal lense and optical slip ring for data transfer.

(+) New sensor generation exhibits even better image sharpness and insensitivty of focus to temperature changes. Continuous operation of IR system during entire survey.

(-) The small (38m length) ship exhibited significant roll and pitch, exceeding the gimbals compensation limits and leading to poor (unstable) video data impeding automatic whale detections.

Polarstern

PS115/1

Aug - Sep 2018

~60 TByte data collected

Ship based operation of FIRST-Navy infrared imager on RV Polarstern for marine mammal mitigation during seismic survey of BGR.

(+) Operated by external personell after 1-day training.

(-) On occasion, ship's antennas cause frequent false positives.

 

 

Polarstern

MOSAiC

Ship based operation of FIRST-Navy infrared imager on RV Polarstern for polar bear surveillance.

PS128

765 h operational

FIRST-Navy infrared imager operated productively on board Polarstern from 12 Jan 2022 until 13 Feb 2022, sensor head A040 SN0002

PS134

773 h operational

FIRST-Navy infrared imager operated productively on board Polarstern from 18 Jan 2023 until 19 Feb 2023, sensor head A040 SN0002