Week Of: October 20, 2024 - Nov 18, 2024

🚀 This Week's Highlights

🛠 New Features

• Pass User's Timezone to Vizzly: Narrative details dashboard now respects user timezone. Date filters apply correctly based on the user's local time.
• Narrative Aggs with Timezone Support: All queries automatically incorporate timezone offsets for more accurate data representation.
• Similarity Lookup in Inference: New functionality allows for similarity checks within inference processes. 🎉
• Centroid Calculation for Topic Detection: Enhanced topic detection with new centroid calculation capabilities.
• Exports UI Update: Users can now select fields using checkboxes for exports concerning favorability, Newsguard, and reach. Also, the UI is updated for sorting narratives and mentions via sidebar sorting.

🪠 Improvements

• Copy Mention URL Button: Easily copy mention URLs directly, simplifying user workflows when accessing articles.
• GitHub Scripts for Microstreams: New scripts enhance CI and support Porter for streamlined microstreams processes.
• Narrative API Enhancements: Introducing hierarchical narrative fetching with layer parameters, improving narrative structuring.

🐞 Bug Fixes

• Dagster Job Table Alteration: Resolved issues caused by unqualified SQL when altering tables.
• BGOV User Password Fix: Fixed an issue where a user was created with an invalid password, correcting authentication errors.
• Favorability Enrichment Writing: Addressed cases where favorability enrichments weren't always reaching the operational system.

That's a wrap! Keep exploring new features and stay tuned for more updates! 🚀

Full Changelog

Feature

• Pass browser's timezone to vizzly narrative details dashboard
• Narrative aggs need to support timezone offsets
• Add similarity lookup to inference
• Add centroid calculation functionality to topic detection
• Data science analysis - narrative detection failing with michal's example workspace
• Narratives: scheduled runs to respect frequency and cron time
• Exports UI for favorability, news guard, reach
• Export Mentions - Should include new enrichments (favorability, newsguard, reach)
• Sort narratives by favorability + newsguard and implement sidebar sorting UI for narratives (+mentions)
• Enrichment export -use friendly names for exports
• Topic Hierarchies flow to write LAYER_NARRATIVES to postgres
• Narrative Layer Postgres DDL Updates
• Vizzly - friendly name mappings for favorability & newsguard ratings
• Copy mention URL button
• Serve the right parent id dashboard based on the team or group
• Create github scripts for microstreams
• Create boilerplate for microstreams reddit ingestion
• Create proto file supporting Microstreams reddit mention.
• Narrative API get hierarchical narratives
• Migrate existing filters and implement enrichment filters into side panel UI
• Write reach in consolidated location
• button to select all narratives in narrative list ,

Bug

• dagster job failing due to table alter
• bgov user - somehow has invalid password
• enrichments - favorability not written sometimes

Week Of: October 14, 2024

🐞 Bug Fixes

• Pending Threat Scores: We resolved an issue where narratives displayed "pending" threat scores.

• Data Discrepancy: Resolved inconsistencies between Snowflake and OpenSearch data.

⚙️ Chores & Maintenance

• Reddit Embed Update: Enhanced the Reddit mention drawer to allow for previewing Reddit mentions in-app.

• Vizzly Dashboard Setup: Customized Vizzly dashboards for BGOV users to facilitate tailored, easy-to-manage user experiences.

Full Changelog

Feature

• Script to assign delorean group to a billion bgov users
• Metric computation implementation - 2024-02-23
• Help Alex get Dagster running in local
• Investigate the performance of HDBScan clustering operating on "streaming" data
• Similar and duplicate narrative consolidation
• TACFI dedicated favorability classifiers
• Threat score - trend component methodology
• Write final report notebook & documentation
• Update classifier training dataset construction
• Agglomerative Clustering Class
• Favorability: UI & API updated to allow exporting of the new field
• Map mention_type to channel in narrative aggs table
• Snowflake aggregations on narrative_id
• Add an info icon with boolean tips
• Unclustered mentions being run through inference
• Get vizzly narrative details shipped behind a feature flag

Chore

• Reddit mention drawer should use Reddit embed
• Explore Topics: Hide Recommendations section
• Vizzly for BGOV
• Test out the bookmarks migration script again

Bug

• Seeing threat scores as "pending"
• Investigate data discrepancy between snowflake and opensearch

Week Of: October 06, 2024

Most of this week was backend work to support enriching data and delivering better Dashboard features in-app.

🛠 New Features

• Hide Nonsensical Filters: When viewing narrative details, we now hide irrelevant filters like keyword, narrative, and threat score for cleaner results. Dive deeper into our Favorability MVP here.

🔧 Improvements

• Focused Cluster Query Configurations: Added configurations for the narratives API, including backfill dates and update frequencies for more tailored outputs.
• OS Writer for Generic Lists: Supports writing a generic enrichment under specific prefixes based on configuration. Check it out here.

🐞 Bug Fixes

• Tweet Truncation: Addressed an issue with ingested tweets having truncated text due to changes in the Twitter API or our pipeline.

🔍 Monitoring Chores

• FCQ Alerts: Implemented an alert system via Grafana to notify us if the FCQs don't run narrative detection in over 36 hours, ensuring our narrative insights stay fresh.

Full Changelog

Feature

• Write to new snowflake tables for smc_v8 & wmc_v6 that have enrichments field
• Focused Cluster Query additional configurations
• Hide AI summaries from BGOV version
• Create Vizzly Query Engine Layer
• Favorability - hide nonsensical filters when narrative details is shown
• Favorability - write the enrichments to OS
• favorability: coredata updated to write inference to common schema
• Favorability: os mapper updated to write fields to OS
• Vizzly PoC - Narratives details dash
• Update OS writer to support generic lists

Chore

• Alert on narrative detection when fcqs don't run for over 36 hours

Bug

• Some change in twitter API or data pipeline is leading to ingested tweets with truncated text

Week Of: September 30, 2024

Low feature release week as the team's been working through major user-facing product updates for end of sprint.

🐞 Bug Fixes

• Resolved a query bug affecting mention counts.
• Updated workspace preview functionality to reflect changes immediately.
• Fixed an issue with domain searches that was causing mention previews to return zero.
• Addressed an issue with the Alerts UI where mentions export was broken.
• Corrected a bug that disabled narratives when editing workspace titles.
• Clarified narrative enabling boundaries for large workspaces.

Full Changelog

Feature

• UI: top-level status module
• Favorability: dagster job to backfill inference that match the workspaces
• Favorability: dagster job to apply inference on newly ingested mentions that match the workspaces
• create snowflake tables for smc_v8 & wmc_v6 that have enrichments field ,

Bug

• Mention Counts broken for complex queries
• edit workspace: preview not updating when sources or query changes
• domain search causes mention preview count to return zero
• Alerts UI - export mentions broken
• fix narratives being disabled on workspace title edit
• Able to enable narratives even for large workspace

Week Of: September 23, 2024

🛠 New Features

• Narratives Status View: Implemented a view that displays the progress of narrative generation at the top of the narratives view! No more asking engineering about the status of a Narrative.
• Two-Column Narrative View: Narratives can now be displayed side-by-side, allowing you to see more Narratives with the same screen real estate.

🐞 Bug Fixes

• BigInt ID Conversion: Too much data! We resolved an issue where we had too many records in our database for the int datatype. The issue is on-going but should not have any customer facing impact.
• Explore Topics Filter: Corrected a bug preventing topic filters from applying correctly in the Explore Topics feature. Report.

🚀 Upcoming!

Our team is hard at work on some exciting user-facing features for the next sprint! Here’s what’s on the way:

• Custom Favorability Models: Integrating custom ML favorability models for Paul’s workflows, allowing sorting, visualization in Narrative details, result export, and representation in the Universal Dashboard.
• NewsGuard Data Integration: Offering similar features—sorting, visualizing, and exporting—just like with custom favorability models.
• Reach Metrics: Adding reach data, including estimated domain traffic and user follower stats.
• Narrative VoT by Channel: Initiating planning and development for Narrative Value of Truth metrics by channel.
• York Release: Launching foundational updates to enable these features, including automatic Twitter ingestion rule creation when new Workspaces are created.
• Narrative Quality Enhancements: Improving narrative quality, with progress on layered data generation and to reduce narrative drift.

Additionally, we're trying to move the Vizzy dashboard to Snowflake, which will enable much more flexible visualizations.

We really want this to happen! Please encourage the Eng team, share your feedback along the way, and help provide the space needed to get this work done!

Full Changelog

Feature

• Expose FCQ data to client
• Implement narratives status view
• Revive narrative list UX improvements and ship it ,

Bug

• convert big IDs to strings in Flux API input/output
• mention id is now a bigint
• Alerts UI - Feed not loading due to a bigint issue.
• Explore topics - cannot filter topic
• New narratives layout - narrative card - overflow issue flashes when transitioning between 2 column and 1 column views
• New narratives layout - narratives tab - lots of empty space at the bottom in two column layout
• New narratives layout - narrative card - tag picker has stacking issue with graph
• New narratives layout - narrative card - list of tags is vertical and scrolls vertically and only one shows at at time