Week Of: September 09, 2024

🛠 New Features

🚀 Improved Duplicate URL Detection

• Implemented a feature to better detect and manage duplicate URLs, resolving an issue we saw with Al Jazeera.

🔧 Remove "Beta" Label from Narratives

• We've removed the "Beta" label to clear any confusion and reflect its maturity.

📑 A bunch of Narrative improvements

• Updated our data pipeline processes in Dagster and narrative-api to support multiple FCQs per workspace.
• Updated coredata inference and classifier training to ignore inactive FCQs, ensuring relevant and accurate data processing.
• Released an optimised version of our narrative writer and inference code, significantly improving processing speeds.
• Resolved an issue where V11 and V12 topic detection jobs were overwriting each other’s assets in S3 by implementing a more granular partitioning strategy.

Full Changelog

Feature

• Duplicate URL detection
• Remove "Beta" label from Narratives
• support multiple FCQ per workspace - datapipeline process dagster and narrative-api
• Classifier training and coredata inference to ignore inactive FCQs
• Fix & release faster narrative writer and inference code ,

Bug

• Topic detection jobs should partition on both fcq_id and version