Data Engineering

In the first of the C# Quick Start series, I'll show how to set up connections between C# and MongoDB. In future parts we'll move on and work through: Database Create, Read, Update, and Delete (CRUD) operations Map MongoDB to C# objects Look at MongoDB aggregation How to use LINQ with MongoDB As you already know, C# is a general-purpose language and MongoDB is a general-purpose data platform. Together, C# and…

MongoDB Blog · 2026-11-03T20:09:41Z

Learn how to migrate Google BigQuery workloads to an open lakehouse on AWS built on Apache Iceberg and Amazon S3. This post explains why an open lakehouse future-proofs your data, presents the target architecture on AWS, and walks through a step-by-step migration: assess, design, convert, migrate, cut over, and optimize.

AWS Big Data Blog · 2026-11-02T17:21:17Z

For builders using AI coding tools, speed is everything. Integrations that require generating a credential, filing an access request, or waiting on a custom build can stop momentum (and productivity) in its tracks. Until now, connecting an external application like an AI coding assistant to a MongoDB Atlas account introduced some of that friction. The only way to do so was by sharing long-lived service account…

MongoDB Blog · 2026-11-03T20:08:21Z

Amazon SageMaker Unified Studio now offers designations: named permission sets that give administrators granular, least-privilege control over what each project member can do. Learn how designations work and how to create and assign your first one from the console and the AWS CLI.

AWS Big Data Blog · 2026-10-30T17:22:32Z

Generative AI is rapidly evolving from experimenting with models to relying on intelligent, autonomous multi-agent workflows that can reason, act, and adapt in real time. Together, Microsoft and MongoDB are defining the future of AI by providing companies everywhere a robust, secure, scalable foundation for building innovative, next-generation AI agents. Microsoft Azure Foundry provides a unified, enterprise-grade…

MongoDB Blog · 2026-11-03T20:02:58Z

A repeatable process for keeping Apache HBase on Amazon EMR highly available. Part 1 covers triaging an HBase coordinator that is slow to start, a RegionServer that leaves the cluster, and regions that stay in transition, with the web UI pages, commands, and log signatures that point to each cause.

AWS Big Data Blog · 2026-10-29T18:25:48Z

Today’s cloud-driven world is characterized by the requirement to scale—and to do so massively and faster than ever. But standing up secure and compliant environments on the cloud is complex and requires huge efforts, as teams have to figure out networking, policies, identity, and governance requirements across numerous projects. At MongoDB, we’re focused on empowering enterprise customers to build faster, smarter…

MongoDB Blog · 2026-11-03T20:02:16Z

The next generation of Amazon OpenSearch Service serverless now supports time series collections. Define the boundaries for cost, performance, security, access, and data retention using simple controls for data tiering, time field selection, and data retention, then turn raw events into insights with the next-generation OpenSearch UI.

AWS Big Data Blog · 2026-10-27T21:52:04Z

As enterprise AI agent adoption scales, the absence of centralized, organization-level tool infrastructure is producing compounding costs. When adoption is built around optimizing for deployment speed, enterprises expose themselves to a combination of risks: duplicated engineering effort, security exposure, and operational opacity. Every enterprise needs its own shared tool registry, one that reflects its specific…

MongoDB Blog · 2026-11-03T20:01:13Z

The Amazon OpenSearch Service Search Relevance Agent turns a low NDCG score into a better search query in minutes. Describe the problem through the Ask AI chat, and the agent analyzes your User Behavior Insights data, generates an improved query, and validates it with an offline Search Relevance Workbench experiment, all without leaving the UI.

AWS Big Data Blog · 2026-10-27T21:31:20Z

Welcome back to the last part of our series on Building a Financial Crime Mitigation Platform. We have shown how MongoDB ideally supports the technical capabilities of modern digital financial operations as a unified data platform. In case you missed it, be sure to check out the series overview . In today’s world, fraud management systems need to react in the moment. Immediacy has become a must-have characteristic…

MongoDB Blog · 2026-11-03T20:00:16Z

Amazon OpenSearch intelligent ingestion is a fully managed, serverless capability that turns documents and structured data in Amazon S3 into RAG-ready, searchable indexes with no pipelines to manage. Learn how it works and when to use Ingestion Jobs or Ingestion Streams.

AWS Big Data Blog · 2026-10-23T21:50:38Z

Editor’s note: This post is the second in a five-part series exploring how MongoDB Atlas Resource Policies help you strengthen database security and enforce consistent guardrails across your organization. Read the first post on IP access list management here . Future posts will cover topics like blocking wildcard IP access and restricting access by cloud provider or region. Imagine protecting your home with only a…

MongoDB Blog · 2026-11-03T19:56:14Z

Trusted Identity Propagation with AWS IAM Identity Center now works across Regions. Learn how to share AWS Lake Formation-governed Amazon S3 and Amazon Redshift data in one Region and query it from another as the actual corporate user, with fine-grained access control and per-user auditability in AWS CloudTrail.

AWS Big Data Blog · 2026-10-23T19:50:12Z

Editor’s note: This post is the first in a five-part series exploring how MongoDB Atlas Resource Policies help you strengthen database security and enforce consistent guardrails across your organization. Future posts will cover topics like defense-in-depth, blocking wildcard IP access, and restricting access by cloud provider or region. Picture this: Your development team spins up a new MongoDB cluster for a…

MongoDB Blog · 2026-11-03T19:55:16Z

The Amazon OpenSearch Service Search Relevance Agent automates search relevance tuning. You describe a problem in plain language, and the agent investigates your behavioral data, proposes an improved query, and validates it with an offline experiment. This post walks through the agent's architecture and a complete example investigation.

AWS Big Data Blog · 2026-10-22T17:43:30Z

At MongoDB Research, we’re exploring how to make cutting‑edge AI models smaller, faster, and more flexible. In this blog post, we present LEAF ( Lightweight Embedding Alignment Framework ), a new knowledge distillation framework for text embedding models. We’re also releasing two best‑in‑class embedding models trained with LEAF: mdbr-leaf-ir —23M parameter model optimized for retrieval‑focused tasks like semantic…

MongoDB Blog · 2026-11-03T19:54:38Z

Learn how to trace two invisible threats in a multi-agent AI pipeline, prompt injection propagation and cross-tenant data exfiltration, using Amazon OpenSearch Service and Amazon Bedrock. Instrument every agent hop with embeddings, provenance, and PII detection, reconstruct the attack path, and add guardrails that stop these attacks before they reach production.

AWS Big Data Blog · 2026-10-19T16:10:31Z

Where money moves, crime evolves. As banking becomes fully digital—with instant payments, open finance APIs, and mobile-first onboarding—it has opened new roads for cyberattacks, forcing financial institutions to respond in real-time. With just seconds to detect, prevent, and react to suspicious activity—without disrupting the customer experience—banks face an immense challenge that only modern, AI-powered solutions…

MongoDB Blog · 2026-11-03T19:53:52Z

Learn how to set up Amazon Redshift Data Sharing between a producer and consumer cluster and use dbt Cloud to transform the shared data into analytics-ready models, with live, zero-copy access and no ETL between clusters.

AWS Big Data Blog · 2026-10-15T18:01:07Z

We are excited to announce the public preview of Automated Embedding in MongoDB Vector Search (available in Public Preview in MongoDB Community Edition ), a groundbreaking feature designed to make building sophisticated, AI-powered applications easier than ever. Since 2023, MongoDB has offered integrated vector search alongside our operational database, and last year we announced the acquisition of Voyage AI…

MongoDB Blog · 2026-11-03T17:17:18Z

Amazon Redshift now automatically resolves many concurrent writes to Apache Iceberg tables that don't actually overlap, committing changes that previously failed and had to retry. Learn why these writes conflict, how the engine rebases non-overlapping ones across operation, partition, and file checks, and the throughput gains it delivers.

AWS Big Data Blog · 2026-10-13T20:16:48Z

The federated Copilot connector for MongoDB Atlas gives organizations a governed way to bring live operational data into the flow of work across Copilot experiences. To operate efficiently, businesses require timely information about the systems that underpin their organizations. For example, they need to be able to answer questions like: Which incidents are open? Which orders are delayed? What fields and data types…

MongoDB Blog · 2026-11-03T14:00:00Z

AWS Glue Data Quality is now available natively on Amazon EMR, bundled in AWS runtime for Apache Spark (emr-spark-8.1.0). Write the same DQDL rulesets you use in AWS Glue directly in your Amazon EMR Spark jobs. This post walks through two patterns: lightweight inline validation and an Apache Iceberg Write-Audit-Publish workflow with managed rulesets.

AWS Big Data Blog · 2026-10-12T23:40:26Z

As applications grow, database teams need to make increasingly complex decisions about how to distribute data, manage workloads, and scale their infrastructure. Sharding enables MongoDB to scale horizontally, but managing a sharded cluster can introduce its own challenges. Understanding cluster health, diagnosing uneven load, and choosing an appropriate shard key can require piecing together metrics across multiple…

MongoDB Blog · 2026-11-02T15:13:14Z

Learn how to build an enterprise-scale cross-account streaming pipeline with Amazon Kinesis Data Streams using the Kinesis Producer Library (KPL) and Kinesis Client Library (KCL) across three separate AWS accounts, with resource-based policies, cross-account IAM, and a shared customer managed AWS KMS key for maximum security isolation.

AWS Big Data Blog · 2026-10-12T15:48:07Z

If you arrived here without reading the first part, I recommend starting with Agentic WMS — Part 1: Where AI Agents Add Value . There, we introduced the WMS scenario, explored the replenishment problem, and discussed where an AI agent can add value without replacing deterministic application logic. In this second part, we will move from the design to the first steps of the implementation. We will look at how the…

MongoDB Blog · 2026-10-29T17:33:15Z

Troubleshooting a multi-service, MWAA-orchestrated ETL pipeline often means hunting across scattered Amazon CloudWatch log groups. This post shows how to centralize your ETL logs in Amazon OpenSearch Service and query them in plain language through an MCP server on Amazon Bedrock AgentCore, so you find root causes faster.

AWS Big Data Blog · 2026-10-06T16:17:46Z

Transaction routing is a delicate balancing act between authorization rates, execution cost, and latency. A declined transaction can mean a lost sale, making payment optimization one of the most direct levers for top-line revenue growth. Yet payment operations teams remain trapped in manual operational cycles: Pulling batch CSV reports, analyzing spreadsheets, and manually updating rule tables in legacy gateways…

MongoDB Blog · 2026-10-29T16:38:20Z

Learn how to pair DuckDB with AWS Glue 6.0 to run SQL-centric ETL on a single worker, reading Parquet from Amazon S3 and writing Apache Iceberg tables to Amazon S3 Tables. This post walks through a complete, runnable example and compares measured cost and runtime against an equivalent Apache Spark job on the same Glue runtime.

AWS Big Data Blog · 2026-10-06T16:13:02Z

This guest post by independent security researcher Ben Zimmermann was developed in collaboration with the Kingfisher team at MongoDB. Ben specializes in large-scale credential exposure research across public source-hosting platforms. TL;DR: I scanned 7,256,531 public repositories on Gitee (China's largest indigenous Git hosting platform), using both TruffleHog and Kingfisher , and found 26,400 unique verified-live…

MongoDB Blog · 2026-10-28T16:17:24Z

Amazon EMR 8.1.0 adds multi-catalog support through the RedirectingSessionCatalog, so you can query Iceberg, Delta Lake, Hudi, and Hive tables through a single catalog and join data across AWS accounts without copying it. This post shows how to put these capabilities into practice on Amazon EMR Serverless.

AWS Big Data Blog · 2026-10-06T16:05:27Z

JetBrains released a new version of DataGrip that includes support for MongoDB and ships with the MongoDB Shell out of the box. A few months ago, JetBrains released the first version of DataGrip that supports MongoDB. DataGrip is a professional DataBase IDE and is a great tool for advanced data exploration and analytical queries for your data in MongoDB and Atlas. If you use any of the other JetBrains commercial…

MongoDB Blog · 2026-10-28T01:07:06Z

Amazon Redshift now supports Iceberg materialized views. Compute an aggregation once in Amazon Redshift and store the result as a standard Apache Iceberg table in Amazon S3, queryable by Amazon Athena, Apache Spark, Amazon SageMaker, and AWS Glue. This post covers use cases, incremental refresh, and a step-by-step getting-started guide.

AWS Big Data Blog · 2026-10-05T21:41:49Z

Since we kicked off MongoDB’s series of 2024 events in April, we’ve connected with thousands of customers, partners, and community members in cities around the world—from Mexico City to Mumbai. Yesterday marked the nineteenth stop of the 2024 MongoDB.local tour, and we had a blast welcoming folks across industries to MongoDB.local London, where we discussed the latest technology trends, celebrated customer…

MongoDB Blog · 2026-10-28T01:04:18Z

Amazon Redshift automates much of its own tuning, but periodic operational reviews still pay off. Learn how the review_cluster tool in the open-source Amazon Redshift MCP server runs a full cluster diagnostic from one natural-language request, evaluating 12 diagnostic areas and returning prioritized, documentation-linked recommendations in minutes.

AWS Big Data Blog · 2026-10-05T15:44:45Z

Amazon CodeWhisperer is trained on billions of lines of code and can generate code suggestions — ranging from snippets to full functions — in real-time, based on your comments and existing code. AI code assistants have revolutionized developers’ coding experience, but what sets Amazon CodeWhisperer apart is that MongoDB has collaborated with the AWS Data Science team, enhancing its capabilities! At MongoDB, we are…

MongoDB Blog · 2026-10-28T00:58:19Z

The KIP-848 consumer protocol in Apache Kafka 4.0 redesigns consumer group rebalancing to eliminate stop-the-world pauses. This post explains how the consumer protocol works on Amazon MSK, how to enable it on MSK Standard and Express brokers, and how to diagnose and resolve slow rebalancing issues.

AWS Big Data Blog · 2026-10-01T16:43:54Z

Nearly a decade ago, I joined MongoDB as a Senior Product Manager to help build the company’s new cloud product, MongoDB Atlas. Our customers had been telling us they wanted to bring MongoDB’s familiar developer experience to the cloud, with the reliability and confidence teams needed to run in production. Atlas was our answer. Today, we’re celebrating 10 years of MongoDB Atlas, the generational data platform for AI…

MongoDB Blog · 2026-06-25T17:28:40Z

Debugging Apache Airflow DAG failures across services like AWS Glue, Amazon EMR, and Amazon Athena is slow and manual. In this post, we show you how to build a custom Airflow plugin that integrates with Amazon Bedrock to automatically analyze DAG task failures and deliver on-demand root cause analysis on Amazon MWAA.

AWS Big Data Blog · 2026-09-29T16:17:09Z