graphicsmagick:coder_HEIC_fuzzer: Use-of-uninitialized-value in residual_coding

[bobfriesenhahn]

Due to GraphicsMagick oss-fuzz testing, OSS-Fuzz issue 494799445 was created. I am reporting it here (rather than libheif) given that the involved code appears to be in libde265.

This is the reported stack trace to where uninitialized memory is allocated and accessed:

WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x57cbd5d8d3d8 in decode_coeff_abs_level_remaining(thread_context*, int) libde265/libde265/slice.cc:2497:40
#1 0x57cbd5d8d3d8 in residual_coding(thread_context*, int, int, int, int) libde265/libde265/slice.cc:3365:15
#2 0x57cbd5d8e2e8 in read_transform_unit(thread_context*, int, int, int, int, int, int, int, int, int, int, int, int) libde265/libde265/slice.cc:3686:16
#3 0x57cbd5d9276d in read_transform_tree(thread_context*, int, int, int, int, int, int, int, int, int, int, int, PredMode, unsigned char, unsigned char) libde265/libde265/slice.cc:4002:5
#4 0x57cbd5d923d9 in read_transform_tree(thread_context*, int, int, int, int, int, int, int, int, int, int, int, PredMode, unsigned char, unsigned char) libde265/libde265/slice.cc:3977:5
#5 0x57cbd5d99833 in read_coding_unit(thread_context*, int, int, int, int) libde265/libde265/slice.cc:4617:9
#6 0x57cbd5d86a08 in read_coding_quadtree(thread_context*, int, int, int, int) libde265/libde265/slice.cc:4676:5
#7 0x57cbd5d86a08 in read_coding_quadtree(thread_context*, int, int, int, int) libde265/libde265/slice.cc:4676:5
#8 0x57cbd5d86a08 in read_coding_quadtree(thread_context*, int, int, int, int) libde265/libde265/slice.cc:4676:5
#9 0x57cbd5d9eab9 in decode_substream(thread_context*, bool, bool) libde265/libde265/slice.cc:4780:5
#10 0x57cbd5da1487 in thread_task_ctb_row::work() libde265/libde265/slice.cc:5029:3
#11 0x57cbd5db02c3 in worker_thread(thread_pool*) libde265/libde265/threads.cc:164:11
#12 0x57cbd5db104c in std::__1::__invoke_result_impl<void, void ()(thread_pool), thread_pool*>::type std::__1::__invoke[abi:ne220000]<void ()(thread_pool), thread_pool*>(void (&&)(thread_pool), thread_pool*&&) /usr/local/include/c++/v1/__type_traits/invoke.h:87:27
#13 0x57cbd5db104c in void std::__1::__thread_execute[abi:ne220000]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct>, void ()(thread_pool), thread_pool*, 0ul, 1ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct>, void ()(thread_pool), thread_pool*>&, std::__1::__integer_sequence<unsigned long, 0ul, 1ul>) /usr/local/include/c++/v1/__thread/thread.h:159:3
#14 0x57cbd5db104c in void* std::__1::__thread_proxy[abi:ne220000]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct>, void ()(thread_pool), thread_pool*>>(void*) /usr/local/include/c++/v1/__thread/thread.h:167:3
#15 0x7d0113c44608 in start_thread /build/glibc-B3wQXB/glibc-2.31/nptl/pthread_create.c:477:8
#16 0x7d0113b67352 in __clone /build/glibc-B3wQXB/glibc-2.31/sysdeps/unix/sysv/linux/x86_64/clone.S:95

Uninitialized value was created by a heap allocation
  #0 0x57cbd4ec7ccc in operator new[](unsigned long) /src/llvm-project/compiler-rt/lib/msan/msan_new_delete.cpp:53:37
  #1 0x57cbd5d0ddbf in slice_unit::allocate_thread_contexts(int) [libde265/libde265/decctx.cc:107](https://github.com/strukturag/libde265/blob/36ad04841c209cb8b3577ec2723d431b6bf7efa0/libde265/decctx.cc#L107):21
  #2 0x57cbd5d1d121 in decoder_context::decode_slice_unit_WPP(image_unit*, slice_unit*) [libde265/libde265/decctx.cc:827](https://github.com/strukturag/libde265/blob/36ad04841c209cb8b3577ec2723d431b6bf7efa0/libde265/decctx.cc#L827):14
  #3 0x57cbd5d19e62 in decoder_context::decode_slice_unit_parallel(image_unit*, slice_unit*) [libde265/libde265/decctx.cc:785](https://github.com/strukturag/libde265/blob/36ad04841c209cb8b3577ec2723d431b6bf7efa0/libde265/decctx.cc#L785):11
  #4 0x57cbd5d186f5 in decoder_context::decode_some(bool*) [libde265/libde265/decctx.cc:555](https://github.com/strukturag/libde265/blob/36ad04841c209cb8b3577ec2723d431b6bf7efa0/libde265/decctx.cc#L555):13
  #5 0x57cbd5d154e8 in decoder_context::read_slice_NAL(bitreader&, NAL_unit*, nal_header&) [libde265/libde265/decctx.cc:513](https://github.com/strukturag/libde265/blob/36ad04841c209cb8b3577ec2723d431b6bf7efa0/libde265/decctx.cc#L513):9
  #6 0x57cbd5d1fa38 in decoder_context::decode_NAL(NAL_unit*) [libde265/libde265/decctx.cc:1055](https://github.com/strukturag/libde265/blob/36ad04841c209cb8b3577ec2723d431b6bf7efa0/libde265/decctx.cc#L1055):11
  #7 0x57cbd5d2012e in decoder_context::decode(int*) [libde265/libde265/decctx.cc:1143](https://github.com/strukturag/libde265/blob/36ad04841c209cb8b3577ec2723d431b6bf7efa0/libde265/decctx.cc#L1143):16
  #8 0x57cbd5c99013 in libde265_v1_decode_next_image2(void*, heif_image**, unsigned long*, heif_security_limits const*) [libheif/libheif/plugins/decoder_libde265.cc:382](https://github.com/strukturag/libheif/blob/5ec43ffdb2940ca0df28504c4174f9cc19d4d2ec/libheif/plugins/decoder_libde265.cc#L382):18
  #9 0x57cbd5bc0463 in Decoder::get_decoded_frame(heif_decoding_options const&, unsigned long*, heif_security_limits const*) [libheif/libheif/codecs/decoder.cc:417](https://github.com/strukturag/libheif/blob/5ec43ffdb2940ca0df28504c4174f9cc19d4d2ec/libheif/codecs/decoder.cc#L417):11
  #10 0x57cbd5bc1e84 in Decoder::decode_single_frame_from_compressed_data(heif_decoding_options const&, heif_security_limits const*) [libheif/libheif/codecs/decoder.cc:468](https://github.com/strukturag/libheif/blob/5ec43ffdb2940ca0df28504c4174f9cc19d4d2ec/libheif/codecs/decoder.cc#L468):17
  #11 0x57cbd56cc49c in ImageItem::decode_compressed_image(heif_decoding_options const&, bool, unsigned int, unsigned int, std::__1::set<unsigned int, std::__1::less<unsigned int>, std::__1::allocator<unsigned int>>) const [libheif/libheif/image-items/image_item.cc:987](https://github.com/strukturag/libheif/blob/5ec43ffdb2940ca0df28504c4174f9cc19d4d2ec/libheif/image-items/image_item.cc#L987):19
  #12 0x57cbd56c17a7 in ImageItem::decode_image(heif_decoding_options const&, bool, unsigned int, unsigned int, std::__1::set<unsigned int, std::__1::less<unsigned int>, std::__1::allocator<unsigned int>>) const [libheif/libheif/image-items/image_item.cc:731](https://github.com/strukturag/libheif/blob/5ec43ffdb2940ca0df28504c4174f9cc19d4d2ec/libheif/image-items/image_item.cc#L731):60
  #13 0x57cbd5a9ac76 in HeifContext::decode_image(unsigned int, heif_colorspace, heif_chroma, heif_decoding_options const&, bool, unsigned int, unsigned int, std::__1::set<unsigned int, std::__1::less<unsigned int>, std::__1::allocator<unsigned int>>) const [libheif/libheif/context.cc:1339](https://github.com/strukturag/libheif/blob/5ec43ffdb2940ca0df28504c4174f9cc19d4d2ec/libheif/context.cc#L1339):34
  #14 0x57cbd5683e21 in heif_decode_image [libheif/libheif/api/libheif/heif_decoding.cc:235](https://github.com/strukturag/libheif/blob/5ec43ffdb2940ca0df28504c4174f9cc19d4d2ec/libheif/api/libheif/heif_decoding.cc#L235):81
  #15 0x57cbd525a6e3 in ReadHEIFImageFrame /src/graphicsmagick/coders/heif.c:1158:17
  #16 0x57cbd5255ec6 in ReadHEIFImage /src/graphicsmagick/coders/heif.c:2000:10
  #17 0x57cbd4fbd8ea in ReadImage /src/graphicsmagick/magick/constitute.c:1682:13
  #18 0x57cbd4f5fdc2 in BlobToImage /src/graphicsmagick/magick/blob.c:785:13
  #19 0x57cbd4ed8ee2 in Magick::Image::read(Magick::Blob const&) /src/graphicsmagick/Magick++/lib/Image.cpp:1601:5

These two test case files were provided:

clusterfuzz-testcase-minimized-coder_HEIC_fuzzer-5897318355501056.gz

clusterfuzz-testcase-coder_HEIC_fuzzer-5897318355501056.gz